Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yang Yang
authors: Wenhai Wang
authors: Zhe Chen
authors: Jifeng Dai
authors: Liang Zheng
reviews: 
Comments: This paper propose a box stability score (BoS score) for unsupervised detector evaluation on the real-world test sets. It has been well proved that BoS has a strong, positive correlation with detection accuracy measured by mAP.  BoS is the first metric proposed for unsupervised evaluation of object detection. Experiments demonstrate that it performs well with different network backbones and outperforms confidences based measurements.
Strengths: 1. This paper is well written and organized. The idea is clear and easy to follow.
2. Novelty: BoS is the first metric proposed for unsupervised evaluation of object detection.
3. Correlation between BoS and mAP is well demonstrated.
4. Good and consistent performance of BoS compared with exsiting confidence based metrics across meta-datasets, backbones, and iou thresholds.
Weaknesses: 1. My main concern on the proposed method is how it can be applied in the real world object detection and what it benefits. The proposed method has requirements on the detector's performance (mAP>15%) on target data. However, in real-world scenarios, we cannot be certain if the detector performs well enough to meet this requirement. We don't know how reliable is the BoS score. Can BoS be used to  improve the detection results?

2. Another issue is that the evaluation of BoS is primarily focused on single-class scenarios, and it does not perform as well in multi-class settings. In real-world applications, we often encounter multiple bounding boxes detected across different classes. Consequently, the accuracy of matching between the original prediction and the prediction after feature perturbation may be compromised. It remains uncertain whether the performance of BoS will degrade when dealing with multi-class data.

3. Additionally, the length of this paper exceeds the 9-page limit.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 4 excellent
reviews: 
Comments: This paper find that under feature map dropout, good detectors tend to output bounding boxes whose locations do not change much, while bounding boxes of poor detectors will undergo noticeable position changes. The proposed box stability score has a strong, positive correlation with detection accuracy.
Strengths: + This paper is clear and easy to read.
Weaknesses: - I find it hard to understand the experiments. In Table 1, the authors say "we report its ground truth mAP (%) on different unseen test sets in the header, the estimated mAP (%) in the main table". However, the numbers for estimated mAP reported in the main table are very low. So what do the numbers in Table 1 represent for? Same for Table 2, Table 3 and Table 4. What are the best numbers in each column?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: This paper provides a novel way to evaluate object detection systems without ground truth data, based on the stability of bounding boxes, and shows promising results in estimating detection accuracy. The authors propose a method for evaluating the quality of bounding boxes generated by object detection models and their correlation with detection accuracy. The paper introduces a metric named"box stability score" (BoS) and demonstrates a strong positive correlation between BoS scores and detection accuracy (mAP). The method is the first for AutoEval in object detection. Experiments on vehicle and pedestrian detection tasks demonstrate the effectiveness of the BoS score in estimating mAP and outperforming confidence-based measurements.
Strengths: 1. This paper proposes a metric, the box stability score (BoS), for object detection. The metric is validated that it has a strong positive correlation with the detection accuracy (mAP). The correlation provides an agent for evaluating detectors on real-world datasets lacking ground truth bounding boxes. It can better test the generalization performance of various detectors. It first applies the AutoEval algorithms for object detection.

2. The paper is well organized. The proposed method BoS is simple but effective for label-free detection evaluation. The discussions are sufficient to explain the insights of the paper. 

3. The paper is well written. It's easy to follow the main contributions of the paper.

4. The experiments compare BoS with extended classification AutoEval methods. The comparison demonstrates that the BoS can better measure the detection tasks under label-free scenarios.
Weaknesses: 1. The BoS is correlated with the accuracy on the dataset level but failed on the individual level. It does not make sense. This part needs more explanation and provides more evidence.

2. A limitation section is needed for a comprehensive analysis.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The authors discover and examine the phenomenon that - in an object detection setting - the stability of bounding box detection (in the presence of MC dropout variability) is strongly correlated to the ability of the detector to generalize to unseen datasets. Aside from a comprehensive set of experiments comparing the approach to various alternatives, the authors also spend time discussing what their insights are into the metric, as well as what it *cannot* do (a valuable insight into the metric),
Strengths: - An important discovery since dropout is performed on the neural network and so is dataset-independent. The fact that it works makes perfect sense as the approach is effectively testing the detector for robustness to unexpected/untrained differences in internal weights, which  can occur with distinctively-different datasets (especially real data), and is something that dropout is essentially simulating.

- Results are comprehensive, covering multiple detection problems and comparing against several alternative algorithms.

- The authors also spend a couple of sections on summarising what their scoring metric really is, and what it could be used for (and importantly what it could NOT be used for ie: as a loss function). This kind of clear elucidation is invaluable.
Weaknesses: - I really don't have any weaknesses to mention - I can fully see this becoming part of the standard toolkit for evaluating detectors
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent