Venue: MIDL 2023 Conference

Reviews:

final_score: rating: 5: Strong accept
review_content: summary: The common metrics such as IoU, mAP, etc., that are used for evaluation of object detection and localization in natural images are inadequate for use in a clinical setting. The authors propose a more insightful metric, RoDeO, for evaluation of pathology detection in medical imaging. RoDeO accounts for the shape, location, and the class of pathology detected in the image and provides a summarized score(authors recommend using the sub-metrics along with the summary score to grasp the full picture). In this paper, RoDeO metrics have been obtained along with the baseline metrics on a large dataset to establish it's usefulness.
strengths: The paper is written very well. The motivation is clearly established.

The writing is such that the shortcomings of the metrics currently used in pathology detection and the method to address them with the new proposed metric are easy to comprehend.

The experimental setup and the evaluation of the machine learning model are convincing.
weaknesses: I really like the way this paper has been written and I do not see any major weakness to mention. It is quite convincing that the proposed metric can provide a more insightful evaluation of pathology detection models. 
recommendation: Oral
review_title: null

final_score: rating: 2: Weak reject
review_content: summary: After addressing the weaknesses of often used evaluation metrics for pathology detection, the authors propose RoDeO, a metric designed for this exact problem. Each component of RoDeO is introduced to improve over different issues with alternative metrics, such as over-, underprediction, shape similarities, and a meaningful aggregation of the sub-metrics to yield a single value. The metric is shown to be robust and outperform the other methods.
strengths: The authors propose a robust and explainable evaluation metric for object detection using a combination of three sub-metrics that provide a better insight into the behavior of trained machine learning models (or other detection algorithms) than single evaluation metrics. Both the problem of object detection and machine learning methods can benefit from more issue-specific explainability, therefore the manuscript certainly shows scientific merit.

The authors also perform a thorough assessment of the alternative metrics and exploit their weaknesses to build RoDeO which, by design, becomes more useful in extreme scenarios than the metrics to compare to.
weaknesses: Although I believe that a mindfully constructed evaluation metric is certainly useful, the selected evaluation metrics for comparison are quite arbitrary. As an example, a quick glance at the confusion matrix makes it very clear that neither the average precision metric nor the accuracy will be able to handle over- and underprediction, however metrics such as precision and recall could paint a more detailed picture. These metrics are briefly mentioned, however they are not included in the evaluations. I believe a more concise collection of baseline metrics would lead to a more interesting evaluation of RoDeO, further strengthening the paper.
detailed_comments: - Page 2 row 6 introduces the IoU abbreviation, however it has been used before.
- The labels for the colormaps on Figure 3 are cropped. In the appendix, similar figures look OK.
- "Since these are typically hard to obtain in pathology detection, and earlier works have used significantly lower thresholds". This sentence from page 2 is difficult to follow, consider rephrasing it. If I understood the meaning correctly, perhaps you should remove the "since".
- I believe the impact of the work and of RoDeO would benefit largely from providing the source code to its implementation.
recommendation: []
review_title: null

final_score: rating: 4: Weak accept
review_content: summary: The authors introduced Robust Detection Outcome (RoDeO), a new metric for pathology detection based on three types of errors that cause the detection quality to deteriorate: classification errors, localization errors, and shape mismatch. Its quite interesting. However, the experiments were limited to be generalized in view point of developing metrics.  

strengths: Robust Detection Outcome (RoDeO), a new metric for pathology detection based on three types of errors that cause the detection quality to deteriorate: classification errors, localization errors, and shape mismatch. Its quite interesting Clinically unmet needs based new detection evaluation metrics were suggested.
weaknesses: The experiments were limited to be generalized in view point of developing metrics. It should be evaluated in various kinds of different datasets. In general viewpoint, this evaluation metrics also can be applicable to natural image detection problem
detailed_comments: well writen.
recommendation: Oral
review_title: null

Meta-review: metareview: The paper proposes a metric to better reflect pathology detection in medical images than the IoU metric used in computer vision, with chest x-rays as a specific application.

The reviewers agree on the motivation and clarity of the paper. One of the reviewers has several questions about inner workings of the metric, and there is a lively discussion, which we thank the authors and the reviewers for. 

I am somewhat surprised that the paper is motivated by specific clinical requirements of pathology detection, but is mostly based in computer vision work, with only few sources from medical imaging or clinical venues. There have been a lot of efforts in the community to address appropriate metric choice, see https://arxiv.org/pdf/2206.01653.pdf . However, despite this omission I think there is value in how the validation is done, which could be also relevant for other metrics in used, so I think it would be worth discussing the paper at the conference. 
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)