Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works as a benchmark of out-of-distribution object detection (OOD-OD), which evaluates the detector's generalization ability to a different data (open-world) domain. While many OOD methods have been proposed to amortize the domain gap, DetectBench, consisting of four datasets, shows the current OOD generalization algorithms no longer work on complex object detection tasks. This paper leads to an in-depth discussion on the progress of OOD work, and provides a foundation for OOD-OD research.
strength_and_weaknesses: [Strength]
1. This paper works on an interesting and practical problem of OOD generalization of object detection models. The train-test discrepancy exists in many CV problems, and object detection has been well-studied and applied in research and industry. This DetectBench can work as a fundamental benchmark to study the OOD problem in object detection.

2. The benchmark covers multiple conditions in train-test discrepancy: weather, scene, time, and simulation. They can fairly work together to evaluate the OOD progress

3. Extensive experiments of 16 detectors and 12 OOD algorithms are measured on DetectBench. Those results are promising for the next generation of OOD-OD research.

4. The author has several new assumptions from the empirical studies. For example, they claim that improvement in object detection does not always transfer to OOD object detection, and the success in OOD is also not always consistent between classification and object detection, 

[Weakness]
1. The technical contribution is relatively limited, for example, a better OOD-OD method can be proposed/suggested given the benchmark result. 

2. Although it is sufficient to have more than ten OOD algorithms to draw the conclusion, but the re-implementation of a different problem may affect the observation. For example, using only 3 possible values for the hyper-parameter in all the methods may not be enough. 

3. The synthetic CtrlShift dataset is very fine-grained on a single object class with a single instance per image, therefore, it is lack of generalization. A better motivation is also supposed to explain the importance of the 'car' class in object detection, and why car color and snow level are selected for the only domain shift options. 



clarity,_quality,_novelty_and_reproducibility: clarity/quality:
- The paper is well-written and easy to follow. The tables are clear to compare different set-ups and results

novelty:
- Good. Most of the OOD studies are on classification problems. And this work shows the OOD method is not always working on object detection tasks.

reproducibility:
- The author provides their code with a short readme file. Each experiment requires 8 V100 GPUs, which should be reproducible in the laboratory. 
summary_of_the_review: This work is a benchmark of out-of-distribution object detection. Although it has limited technique contribution, it is marginally above the threshold because the problem is novel and less studied.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is a benchmark for out of distribution generalization in object detection. The authors create a benchmark to evaluate out of distribution generalization algorithms specifically for the task of object detection (object classification and bounding box regression), using multiple well known datasets in specific settings, while also introducing a new synthetic dataset that can be used to study correlation and diversity shift.

The contributions are:
- A new benchmark DetectBench to evaluate OOD generalization in object detectors (authors call this OOD-OD).
- A new dataset CtrlShift to study diversity and attribute shift in OOD-OD.
- A good evaluation and important insights on the effect of diversity and attribute shift on OOD generalization algorithms in object detectors, signaling the object detectors are more affected by diversity shift than attribute shift. This is an important conclusion that can guide future object detector and OOD-OD method design to improve OOD generalization performance.
- The evaluation also indicates that standard OOD generalization algorithms (designed for classification), do not transfer successfully to object detection, indicating the need for more specialized OOD generalization algorithms for this task.
strength_and_weaknesses: 
Strengths
- The paper is very well written and it is clear, ideas flow clearly, I have no further remarks about writing.
- An OOD generalization benchmark for object detection makes sense, as there is no dataset/benchmark specifically for this task, and the paper evaluation is of good quality, by combining standard object detectors with OOD generalization algorithms, to see if they work.
- It is a good idea to have test OOD generalization for object detection, as the authors argue, OOD generalization algorithms are not usually evaluated for more complex tasks like object detection, they usually are only evaluated for classification datasets.
- The proposed benchmark uses a selection of public datasets, covering multiple kinds of shifts, and additionally a synthetic dataset (CtrlShift) is proposed which has diversity and correlation shifts made in a controlled way, allowing the evaluation of the effect of both shifts combined. 
- The design of the DetectBench benchmark seems sensible, it contains multiple sub-benchmarks where train and test sets contain non-intersecting sets of conditions, like weather, scene types, time, and simulation/reality, which make a simple test set that is out of distribution with respect of the training set.
- I believe that the evaluation and initial benchmark results presented in the paper are correct. There is a good selection of OOD generalization algorithms to test, and of object detectors (Faster R-CNN, DETR, RetinaNet) for the main results, which makes the overall conclusions robust.
- Two major conclusions of this benchmark are that improvements to OOD generalization in classification settings do not transfer to improvements to OOD generalization in object detection, as tested on Faster R-CNN like on Table 4, and on sim2real for Faster R-CNN, DETR, and RetinaNet, and measured by mAP in the OOD setting. In some cases the performance actually decreases instead of increase, and that in the OOD setting, diversity shift seems to affect OOD generalization algorithms much more than attribute shift. These two conclusions are good contributions to the state of the art.
- While the main results (Table 4) are presented for Faster R-CNN on the whole DetectBench benchmark, there are additional results on the sim2real sub-benchmark using Faster R-CNN, DETR, and RetinaNet about OOD generalization algorithms, producing a more robust conclusion that OOD generalization algorithms fail to improve performance in object detectors.
- The design of CtrlShift, while simple, seems to do the job, as Figure 3 shows, there is correlation shift produced by changing the color of a car model, and for diversity shift there is simulated snow added to the environment, and both shifts can be combined to study their joint effect.

Weaknesses
- I think the only weakness in this paper is the name of the Benchmark, DetectBench is very generic, sounds like any detection benchmark, it should have OOD and generalization (or similar abbreviations) in the name, to distinguish it from OOD detection benchmarks and standard OOD generalization benchmarks. Please find a good name that includes all of this.

Minor Issues
- Figure 5 and related heatmaps would be better presented with the axis labeled in the figure itself, instead of relying on the caption. Just as Figure 3, Figure 5 should have axis labels for diversity and correlation shift. Additionally to this, Figure 5 should have the origin at the bottom left instead the top left, which would make the figure easier to interpret.
- In Table 3, I think it would be good to also provide the original mAP of these detectors, in order to put the benchmarked numbers into context, so the reader can easily see if mAP is lower than expected or not.
- I believe that in Table 2, simulation and reality are flipped in the sim2real description (last row, third column).
- Could you provide a bit more information on CtrlShift? I see that there are 2000 images, but how are these distributed according to the two kinds of shifts?
clarity,_quality,_novelty_and_reproducibility: To me this paper is clear, the writing and presentation of results is good, results and conclusions are clear and supported by the results.

About quality, I think this is a high quality paper, it has clear contributions, a clear problem and solution, and major insights that can influence the state of the art in OOD-OD and particular for object detector design. A lot of the performance improvements in object detection are only in in-distribution datasets, and this benchmark would help researchers also evaluate improvements in OOD generalization performance.

I believe this paper to be novel, there is no benchmark for the OOD-OD generalization task, and as the authors mention, OOD generalization algorithms are not usually tested in object detectors and that makes them fail. This is very clear to me.

summary_of_the_review: I believe that this paper should be accepted. There are only minor issues, there is a clear gap in the state of the art that this paper fills, a benchmark/dataset for out of distribution generalization for object detection tasks, and the paper does an excellent initial evaluation. This paper has good insights for future research, mainly that OOD generalization algorithms designed for classification fail to improve performance in object detection, and that object detectors are more sensitive to diversity than correlation shifts.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Building on top of BDD100k, Cityscapes and Sim10k, the proposed DetectBench in this work is an object detection benchmark for evaluating the OOD generalization performance of object detectors. The benchmark consists of 4 types of domain shifts, i.e., sim2real, weather, scene and time. The authors compared a number of object detectors on DetechBench, showing their yet poor OOD generalization performance. Using OOD generalization techniques developed for classification tasks, the object detection performance remains unsatisfactory, indicating OOD generalization of object detection is an underexplored task. 
strength_and_weaknesses: Strength: 
OOD generalization is an important task and is underexplored for object detection algorithms. This paper targets at this important problem. The authors proposed a OOD benchmark for object detection and compared a number of object detectors, empirically showing that existing solutions still struggle at dealing with domain shifts.

Weakness:
1. DetectBench does not differ much from previous benchmarks in this area. For example, [1] uses the eight shared categories in Cityscapes, Foggy Cityscape and BDD100k, and the Car category on these three datasets and additionally SIM 10k and KITTI. [2] also uses BDD100K, Cityscapes, Sim10K and KITTI as four domains, and train on three of them and test on the last one. The proposed CtrlShift dataset is different to these benchmarks, but is also limited to only two-dimension shift, and is not a very comprehensive new dataset with a large syn2real gap.

2. While considering a good number of object detectors, the authors mainly evaluate their performance on the constructed benchmarks. Beyond the general observation that these methods are not robust under domain shifts, there is no further insightful discussion, e.g., correlation between ID and OOD performance, different generalization behaviors under different shifts, how the model architecture would affect ID and OOD performance. More importantly, there generally miss a good justification that DetectBench is a qualified benchmark that allows cross-compare different methods. Especially, it is already shown by many previous works that current object detectors fail at generalizing to OOD scenes [1] [2] [3]. 

3. The compared OOD generalization baselines are not adapted to the task of object detection. All the OOD generalization methods are only used for the classification branch of the object detectors. The localization branch is not considered. It is important that this new benchmark should also investigate these OOD generalization techniques developed for object detection.

[1] Domain-Invariant Disentangled Network for Generalizable Object Detection
[2] Towards Domain Generalization in Object Detection
[3] OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images
clarity,_quality,_novelty_and_reproducibility: The paper requires major revision on writing and structuring. Especially, I find it is very hard to have a clear picture of the experiment setup of the work that greatly compromises the quality of the work.

For instance, in Table 3 and 4, there should include ID performance as the baseline. However, it is either very difficult to find or generally unclear what is the ID training set for each domain shift.

In Table 5, if the first line is the vanilla version of object detection training, it is unclear why applying OOD techniques would result in performance degradation.

In 5.2, the experiment and visualization are not clear and sufficient to support the conclusion that correlation shift is easier than diversity shift to tackle.


summary_of_the_review: This work targets an important problem, but the usefulness, novelty and effectiveness of the proposed benchmark are not convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes DetectBench - a benchmark for Out-of-Distribution (OoD) Object Detection, as most works mainly focus on OoD image classification. After introducing a new train-test split on existing datasets and newly proposed dataset CtrlShift, the paper benchmarks a wide range of detectors (Faster R-CNN, DETR, etc.) with different backbones, self-implemented OoD image classification algorithms (ERM, RSC, GroupDRO, etc.), and different detectors with same backbone (Faster R-CNN, RetinaNet, DETR) on these datasets. The main findings are that IID object detection algorithms are marginal on OoD object detection, and the authors propose a new synthetic dataset, which can be used to simulate the two types of distribution shifts based on user’s inputs.

Object detection is an important task for many applications, including safety-critical ones such as self-driving cars and health care. Accordingly, improving the ability of object detectors to deal with unseen distribution shifts is highly relevant. This benchmark is thus well-motivated.
strength_and_weaknesses: Strong points:

1. The paper is well written.
2. Details of the datasets are provided.
3. The authors run the benchmark on both two kinds of distribution shifts, proposed by [1].
4. The authors introduce a new synthetic dataset CtrlShift, of which API can be modified to simulate the two kinds of shifts mentioned above. 
5. This would be a good contribution to the community.
6. The authors extensively benchmark and run experiments on a wide range of detection algorithms.
7. Some algorithms from OoD image classification are also adapted to DetectBench.

Nevertheless, there are some concerns about the paper that make the findings not convincing. I am happy to raise the score if the authors can address the concerns.

Weak points:

1. The train-test split is still a bit weak. For example, in DomainBed [2], for PACS dataset they divide it into four domains. Using the training-domain splitting strategy, they train the models on three selected domains and test on the other. This is to determine if the distribution shift is actually present, or it is because a model or architecture simply performs better on a specific domain. In this paper, the authors split the Weather in BDD100K into six domains, but only select two domains to train and the rest four domains to test. What would happen if we simply flip the train-test split? Will the ranking still hold?
2. Missing the potential impact of the small-scale dataset on big models. Due to the dataset size, Sim2real has fewer data compared to BDD100K. This could make DETR models underperform compared to Faster R-CNN, as transformer models are generally data-hungry due to their size [5].
3. Missing justification for choosing a strong backbone for Faster R-CNN and other detectors, but a regular backbone for DETR. In the paper, the authors choose ResNeXt-101-64x4d for Faster R-CNN and other models but use ResNet50 for DETR and Deformable DETR. Backbone network plays a vital role in the performance, which has been observed in both IID Object Detection [3] and OoD classification [4]. I think it is better to use the same backbone in this table so that we can measure the effect of different detectors.
4. Missing results in (strong) Diversity Shift datasets. Diversity shift datasets such as Watercolor2k, Comic1k, Clipart1k [7] are missing from the paper. In the classification benchmark, the diversity shift datasets (PACS, VLCS, etc.) are much harder to imporve than correlation shift datasets (NICO, CelebA). From my experience, all OoD classification algorithms achieve very high accuracy by simply using ResNet50 as backbone on Correlation shift datasets, making gains of algorithms on this shift extremely minor.
5. The findings are limited in novelty and not very convincing. Section 5.1 highlights some findings in the paper, but I do not find two points very “surprising” nor convincing:
5.1) Finding 1 (The enormous achievements of object detection on IID datasets are marginal on the OOD condition): It is widely known that IID is a strong assumption in real-world use cases, and most models are sensitive to OoD tasks. This is not limited to only image classification, but also object detection, which has already been observed in BDD100K paper [6].
5.2) Finding 3 (The generalization inconsistency between classification and object detection of domain generalization algorithms happens among different detectors): As pointed out above in Weak points 2 and 3, Sim2real is a relatively small dataset for DETR (8.5k images), which possibly means results in Table 5 are not reflecting an accurate performance of DETR. Based on Table 3, the reviewer thinks Deformable DETR is a more suitable choice, and the conclusion “the degeneration of domain generalization algorithms has little relevance to the detectors” is probably a bit overclaimed.

Minor fixes:
1. Figure 4 X-axis is a bit hard to read. The authors may consider using vertical labels.
2. Finding 3 in Section 5.1, “Table 4.2” should be “Table 5”

References

[1] Ye, Nanyang, et al. “Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.” In CVPR (2021).

[2] Gulrajani, Ishaan, and Lopez-Paz, David. “In search of lost domain generalization.” In ICLR (2020).

[3] Zhu, Xizhou, et al. “Deformable DETR: Deformable transformers for end-to-end object detection.” In ICLR (2021).

[4] Angarano, Simone, et al. “Back-to-bones: Rediscovering the role of backbones in domain generalization.” arXiv preprint arXiv:2209.01121 (2022).

[5] Wang, Wen, et al. “Towards Data-Efficient Detection Transformers.” In ECCV (2022).

[6] Yu, Fisher, et al. “Bdd100k: A diverse driving video database with scalable annotation tooling.” arXiv preprint arXiv:1805.04687 2.5 (2018): 6.

[7] Inoue, Naoto, et al. “Cross-domain weakly-supervised object detection through progressive domain adaptation.” In CVPR (2018).
clarity,_quality,_novelty_and_reproducibility: See details above, and the authors provide the code but I haven't run it for reproducibility.
summary_of_the_review: I agree the proposed benchmark is well-motivated and the task is important for the research community. However, with the findings above, I currently give the paper a borderline reject score.

Update: My concerns are not entirely addressed after the rebuttal. I will keep the rejection score for this submission, due to unconvincing experimental results (without cross-validation), weak diversity shift benchmark, the limited novelty of the findings, missing related works, and the proposed benchmarks overlapping with previous works.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Paper proposes a new benchmark for out-of-distribution object detection (OOD-OD), which evaluates the detector's ability to generalize to new domains with changing data distribution. The paper was reviewed by four reviewers, ultimately receiving split scores:

- 1 x  marginally above the acceptance threshold 
- 1 x  accept, good paper
- 2 x reject, not good enough

The main concerns with the paper came down to a couple of core issues:

1. Limited technical contributions and novelty (JACh and aJ6J)
2. Similarity and lack of discussion regarding differences with existing benchmarks in this space (Sn2d and aJ6J)
3. Limited findings and lack of more through analysis; paper only focuses on classification and not localization branch (Sn2d and aJ6J)

The rebuttal provided by the authors did not do a lot in alleviating the aforementioned concerns. While lack of technical contributions [1] is reasonable for the dataset and benchmarking paper, significant concerns with [2] and [3] remain. AC has carefully considered these issues. Indeed the paper is not the first to propose OOD detection benchmark, as such the onus is on the authors to appropriately compare and differentiate their benchmark with others. This, however, was not effectively done either in the original submission, nor in the submitted revision. The papers suggest by reviewers remain un-cited and there appears to be no discussion comparing settings, splits, and findings in this paper vs. predecessors; for example,

* Domain-Invariant Disentangled Network for Generalizable Object Detection
* Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

While some of this discussion of this is given in rebuttal, this is insufficient and the paper itself needs to be adjusted and refocused to take these (and other provided) references into account. Further, AC agrees on the point made by Sn2d that analysis of object detection has to examine both classification and localization branch. For these reasons the decision is to Reject the paper at this time. Authors are encouraged to consider comments provided to them, revise the paper accordingly and, potentially, target another venue. 
justification_for_why_not_higher_score: The issues with similarity of the proposed benchmark to others and lack of analysis of localization are significant and need to be throughly addressed to warrant publication.
Decision: Reject