DetectBench: An Object Detection Benchmark for OOD Generalization Algorithms
Authors: Fan Wu
Nanyang Ye
Lanqing HONG
Chensheng Peng
Bikang Pan
Huaihai Lyu
Heyuan Shi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Out-of-Distribution
object detection
benchmark

Abstract: The consensus about practical machine learning tasks, such as object detection, is still the test data are drawn from the same distribution as the training data, which is known as IID (Independent and Identically Distributed). However, it can not avoid being confronted with OOD (Out-of-Distribution) scenarios in real practice. It is risky to apply an object detection algorithm without figuring out its OOD generalization performance. On the other hand, a plethora of OOD generalization algorithms has been proposed to amortize the gap between the in-house and open-world performances of machine learning systems. However, their effectiveness was only demonstrated in the image classification tasks. It is still an opening question of how these algorithms perform on complex and practical tasks. In this paper, we first specify the setting of OOD-OD (OOD generalization object detection). Then, we propose DetectBench consisting of four OOD-OD benchmark datasets to evaluate various object detection and OOD generalization algorithms. From extensive experiments on DetectBench, we find that existing OOD generalization algorithms fail dramatically when applied to the more practical object detection tasks. This raises questions over the current progress on a large number of these algorithms and whether they can be effective in practice beyond simple toy examples. For future work, we sincerely hope that DetectBench can serve as a foothold for OOD-OD research.

Under review as a conference paper at ICLR 2023
OOD-ODBENCH: AN OBJECT DETECTION BENCH- MARK FOR OOD GENERALIZATION ALGORITHMM
Anonymous authors Paper under double-blind review
ABSTRACT
The consensus about machine learning tasks, such object detection, is still the test data are drawn from the same distribution as the training data, which known as IID (Independent and Identically Distributed). However, it can not avoid being confronted with OOD -Out-of-Distribuus apply object detection algorithm without figuring out its OOD generalization performance. On the other hand, a plethora of OOD generalization algorithms has been proposed to amortize the gap between the in-house and open-world per- formances of machine learning systems. However, their effectiveness was only demonstrated in the image classification tasks. It is still an opening question of how these algorithms perform on more complex tasks. In this paper, we first specify the setting of OOD-OD (OOD generalization object detection). Then, we propose
scenarios real practice. It is risky to
consisting of four OOD-OD benchmark datasets to eval- uate various object detection and OOD generalization algorithms. From extensive experiments on KOD-ODBench we find that existing OOD generalization algo- rithms fail dramatically when applied to the more complex object detection tasks. This raises questions over the current progress rithms and whether they can be effective in practice beyond simple toy examples. For future work, sincerely hope that OOD-ODBench can serve foothold for OOD generalization object detection research.
large number of these algo-
INTRODUCTION
Modern object detection methods (Liu Liu ct al. 2021 Huang et al. 2019 Pang et al. 2019 Wu et al., 2019 Zhang al.) 2020a Sun et al. 2020 Zhu
2021 Ge 2021 have achieved many pro- gresses on various applications, such as autonomous driving and industrial defect detection. Tremen- dous efforts have been devoted improving an object detector's performance on standard datasets, such as MS-COCO (Lin et al.] (014). While these efforts have seen impacts industry (Redmon) Farhadi 2017 2018 [Bockkoskky al.] 2020 Ge et al., 2021], the im- provements are becoming marginal recently and most achievements are accompanied by an inherent assumption, i.e., the training data and the test data are IID (Independent and Identically Distributed). However, this assumption is unlikely to hold in real-world scenarios. For example, autonomous system suffers from different environmental conditions (Dai & Gool, 2018 Volk et al., 2019); a medical system fails work consistently among hospitals when data collected from different equipment (de Castro et al. 2019 Albadawy et al.] 2018 Perone et al. 2019). As consequence, models trained on IID dataset are susceptible subtle disturbance in test data distribution (Out- of of-Distribution) and fail to generalize to real scenarios (Torralba & Efros 201T). Previous research
jet al. 2016 Redmon
devoted encountering this train-test discrepancy can be summarized as either "less complex" or
"complex but not general". From the first perspective. plethora of Domain Generalization (DG)
algorithms (Arjovsky al. 2019 Ahuja et al. 2021 Li et al.] 2018h Sun Saenko 2016 Xu
et al. [2000c] Yan et al. (2020] Krueger et al. 2021 1 Pezeshki et al. 2020 Parascandolo et al. 2021
Koyama Yamaguchi 2021 1: Huang et al. 2020
Sagawa et al. 2019) concentrate on improving OOD generalization ability. But they are simply evaluated
the image classification. The effec- tiveness is unknown when applied to the complex task (object detection). On the other perspective, numerous Domain Adaption (DA) algorithms (Chen al. 2018 He & Zhang 2020 Rodriguez & Mikolajezyk 2019; Xu et al. 2020a [Su et al. 22000; Xu et al.) 2020b) (Soviany et al. 20199; Deng Chen al. 2021 aim to build optimal object detector that be generalized into
al.] 2020

---

Under review as a conference paper at ICLR 2023
Weather
Scene
Time
Simulation
Overcast
Residential
daytime
world
Truin
Rainy
Highway
night
real world
Figure 1: The setting illustration of out-of-distrriuttio generalization object detection (OOD-OD).
pre-specified target domain. However, it is hard to ensure performance consistency when dealing with unseen and infinite real-world domains. In this paper, we focus on OOD generalization object detection (OOD-OD) which aims at training detectors generalize to the testing data drawn from an unseen distribution distinct from the training distribution. See Table for more details and we provide the theoretical definition for OOD-OD Appendix [.
In this work, we propose DOD-ODDench, which four OOD-OD benchmarks are constructed with existing datasets, including BDD100K (Yu et al., 2018), Cityscapes (Cordts al. (016) and Roberson et al. 2016). As revealed by (Ye et al. 2021). data distribution shifts on classification datasets are dominated by correlation shift and diversity shift. We whether similar phenomenon also exists on detection datasets and we construct synthetic dataset named CtrlSift to quantitatively analyze generalization ability over the two kinds of distribution shifts of OOD-OD, respectively. With the above benchmark datasets, numerous experiments are conducted with detectors ranging from one(two)-stage to transformer-based and diverse OOD generalization algorithms carefully implemented popular detectors (Ren et al. 2015 Lin et al. 2017b).
Sim 1OK (Johnson-Roberson
Section revieew the related work dispersed in different research areas. Section provides clarifica- tion of different techniques and tasks with similar names. Section 4 introduces the implementation details of OOOD-ODBench, including the datasets, algorithms, and model selection methods. Finally, Section 5 discusses the experiment results OOD- DOD-ODeench and offers insightful recommenda- tions for future work. Our main contributions can be summarized as followed:
1. We propose OOD-ODBench, the first OOD generalization benchmark for object detection algorithms. Based the extensive experiment results, arrive at surprising conclusion: The enormous achievements in IID) object detection are marginal on OOD generalization object detection, and the OOD generalization improvements classification are hard to generalize to more complex tasks (i.e., object detection).
2. In OOD-ODBench, we propose a Sim2real benchmark for OOD generalization object de- tection analysis which measures the possibility of training models with low-cost simulated data to generalize well on real scenarios.
3. To further analyze the generalization ability under the different types of shifts, we con- struct synthetic dataset with designed shifts, namely CtriShift. The synthetic dataset can systematically measure the OOD generalization algorithms' performances under different types of distribution shifts.
4. From Benchmark results and analysis, we recommend future research to clearly investigate the diversity shift and the correlation shift on OOD generalization object detection before designing algorithms and then evaluate them comprehensively on the two-dimensional shift using CtrlShift.
RELATED WORK
2.1 OBJЕCT DETECTION
The task of object detection aims at classifying and localizing the objects in an image based on the assumption that test data drawn from the same distribution training data. Modern deep
2

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison between OOD generalization object detection and other tasks. X Y indicate the data drawn from distribution and Yi, Y2 represent category labels and bounding box labels respectively.
Setup
Task
Training inputs
Test inputs
Outputs
Unsupervised learning
classify / detect
X1
X1
y YI
Supervised learning
classify detect
X1 y
X2
Semi-supervised learning
classify / detect
X (rly
X
Y!
Transfer learning
classify detect
X Xhond d Xd+1 ydt1
Xd+1
yd+1
Domain generalization
classify
Xland ylond
Xd+1
yd+i
Domain adaption
detect
X Vl..n Xd+1
X
yd+1
1.2
OOD-OD
detect
X Xlaad Y ylund
xd+1.
Y1,2 Y
learning-based object detection models can be divided into three categories: two-stage detectors
201)))kkk2011])))) Dai et al., 2016; : He et al., 2017; Qiao et al. [001, Cai & Vasconcelos) 2019 Huang et al.
(Girshick al. 2014 GGraman & Darrell 2005 Girshick, 2015 Ren et al. [215 Lin et al. 2017a
2019
Pang et al. 2019 Wu et al. 2019
Sun al. 2220), one-stage detectors (Redmon al. 2016
Redmon & Farhadi 2017 2018
Bochkovskiy et al. [020 Ge et al. [202 Liu et al.] 2016: Lin
20176]
Zhou al. 2019
Tan et [2020] Law & Deng 2018, Tian et al. 2019 Zhang
et al.
2020a
Zhu et al. |
Liu Liu et al. 2021) and lightweight detectors with small components
(Howard et al. 2017; Sandler al. 2018; Howard et al., 2019 Zhang et al., 2018 Ma et al. 2018
Wang et al. 2018 landola et al.) 2016). Compared to one-stage detectors, two-stage detectors are
equipped with separate differentiable module to generate region proposals which are possible to contain objects. Lightweight detectors are usually proposed to improve real-time performance with small and efficient network. Recently, with the enormous success of applying transformer /Vaswani
et al. 2017) on computer vision, branch of transformer-basee detector (Zhu al. 2021 2021
Liu al.
has shaped up.
2.2
OOD GENERALIZATION
The task of OOD generalization is training on multiple datasets sampled from distinct domains and then generalizing to an unseen test domain. Models with OOD generalization ability typically have access multiple training datasets for the same task obtained from various environments. The purpose of OOD generalization algorithms is to learn from these diverse but relevant training settings before being applied to unknown testing environments. Driven by this motivation, many algorithms have been proposed throughout these years. These algorithms can be divided into: em- pirical risk learning (Vapnik) 1998: Sagawa al.
2019), invariant risk optimization (Arjovsky al.,
2019). domain adversarial learning (Ajakan et al.
2014 Li et al., 2018c Ruan et al.) (021), meta-
learning (Zhang al., 2020b Li et al. [20 Sa).
kernel function Li al. [2018b] Sun & Saenko,
2016). gradient-basse approach (Shi et al., 2021] Pezeshki et al. 2020 Bai et al. 22200; Parascan
dolo al.] 2021; (Shahtalebi et al. 2021 Koyama Yamaguchi] 2021 Rame et al.) [202])
1), risk
extrapolation (Krueger et al. [021] ) data processing (Xu et al. 2020c Yan et al. 2020)), transfer
learning (Blanchard et al. 2017 Xu & Jaakkola 2021), information bottleneck (Ahuja et al. 2021)
and self-supervised learning (Wang et al. 2020 Zhou et al. 2020).
OOD generalization for object detection is currently underexplored. Region Aware Proposal reweighTing (RAPT) (Zhang et al.) 2022) aims to eliminate dependence within Rol features for domain generalization. Declic-Dieettangeed Self-Distillatio (Wu Deng 2022) aims at disentan- gling domain- invariant representations. 3D-VField (Lehner et al. 2022 improves generalization 3D object detection.
2.3 OOD BENCHIARRK
Different domains data (Zhou et al. 2021 Wang et al. 2022) can be viewed as data drawn from different distributions and the distinct train- test domains are oft--of-Dstrrbbun
DomainBed Guul- rajani & Lopez-Pe Paz 2020) is large-scale benchmark suite for reproducing domain generalization research and facilitating the implementation of new algorithms. With the experiment results of four- teen algorithms seven datasets, the authors found that empirical risk minimization (Vapnik 1998)
3

---

Under review as a conference paper at ICLR 2023
Table Details datasets used in benchmarks. Quantity indicates the number of images in each domain. Total counts the total number of training and testing domains respectively.
Benchmark
Dataset
Domain
Train
Test
Quantity
Total
clear
42690
52699
overcast
10009
foggy
143
Weather
BDD100K
cloudy
5619
17888
rainy
5808
snowy
6318
city street
49628
69506
highway
19878
gas station parking lot residential
34
Scene
BDD100K
426
9943
9327
tunnel
156
daytime dawn dusk night
41986
47791
Time
BDD100K
5805
31900
31900
Sim10K
simulation
8500
8500
Sim2real
Cityscapes
reality
3975
3975
achieves state-of-the-ar performance across all datasets. OoD-Bench (Ye et al. 2021) identifies and measures two distinct kinds of distribution shifts in various OOD datasets. With tremendous empir- ical learning results, the authors recommend that algorithms should be comprchensively evaluated on two types of datasets dominated by correlation shift and diversity shift respectively.
CLARIFICATION OF TASKS
Domain Randomization techniques (Tobin et al. 2017 Tremblay et al. 2018 Zakharov et al., 2019 Yue al. 2019 Huang al. 2021) aim at providing enough simulated domains at training data so that models are possible to generalize to real-word world scenarios based on hypothesis that with enough variability in the data simulator, the real world may appear be specific variation of the simulation data which exists in the training set.
OOD Detection for Object Detection (Joseph al. 202 Du et al. 2022a Harakeh & Waslander,
2021 Riedlinger et al.) 2021: Dhamija et 2020 Miller al. 2019 Hall al. 2020; Deepshikha
ct al. 2021) can be formulated as a binary classification problem which distinguishes whether the
distribution of the incoming data is out of the distribution of the training data.
Open- World Object Detection (Joseph et al.] 2021 Zhao et al. 2022) initially learns a model which detect all the previously encountered categories, and incrementally updates the model when unseen classes come.
Open- Veen-Vocabulary Object Detection (Gu al. 2021 Zareian al. 2021 Du al. 2022b Bravo
et al. 2022) aims to train an detector which can detect various objects in any novel categories
described by arbitrary texts.
4 OOD-ODBENCE: IMPLEMENTATION DETAILS
4.1 BENCHMARKING DATASETS
In OOD-ODBench, we choose datasets to cover as many types of variations between training and test datasets as possible. Figure 2 lists some samples of the four benchmark datasets.
BDD100K (Yu et al. 2018) contains 80,000 labeled images (70,000 for training and 10,000 for val- idation) with ten annotated object categories, including bike, bus, motor, person, rider, traffic light, traffic sign, train and truck. Each image has three attribute labels which indicate the condition, including the weather, scene and time for data collection and remove the images with unde- fined attribute label. Specifically, we construct three OOD environments using the attribute labels, including Weather. Scene, and Time.
4

---

Under review as a conference paper at ICLR 2023
BOD1
Sim 10K
Citysoupes
Figure 2: Some samples of datasets included in OOD-ODBench. Note that BDD100K has ten categories while Sim and Cityscapes only use the annotated cars.
Correlation shift
Diversity shift
Origin
Figure 3: Some samples of CtrlShift and the illustration the two-dimensional shift.
Sim10K (Johnson- Roberrson et al. 2016) is a synthetic dataset containing 10,000 images (8,000 for training. 1,000 for validating and 1,000 for testing) with bounding box annotations for cars, which is rendered with the Grand Theft Auto V (GTA5) game engine.
Cityscapes (Cordts al. 2016) a large-scale database which focuses on urban street scenes. The dataset consists of around 5000 fine annotated images (2975 for training. 500 for validating and the rest for testing) with eight annotated instance categories. On OOD-ODBench, we consider the car recognition task to construct the Sim2real benchmark for simplicity and without loss of generality. We construct the Sim2Real benchmark which covers the "sim2real" scenario. The simulated images of Siml0K are used for training and the real images of Cityscapes are used for testing.
CtriShift is a synthetic dataset to analyze the two-dimension shift OOD generalization object detection. The Airsim simulator (Shah et al.) 2017) based the high-fidelity rendering software Unreal Engine 4 is used to generate samples in CtrlShift. We totally sample over 2000 simulated images from both the rural and urban environments which contain common objects, including build- ings, traffic lights, and vehicles. Moreover, every image comprises two attribute labels to construct controllable distribution shift. One is car color which indicates the color of the in the image,
5

---

Under review as a conference paper at ICLR 2023
Table 3: Experimental results of detectors performance measured by AP(%) MS COCO and the four OOD benchmarks of DOD-ODBench. All models are implemented by mmdetection (Chen
al. 2019 and loaded pretrained weights provided by open-mmlab (Contributors) 2018). Note that @ Faster in Libra R-CNN row represents applying Faster R-CNN as architecture. 1-101-64x4
represents modified ResNeXt-100 101 network architecture from (Xie et al. 2017). R-50 and R-101
represents ResNet backbones with 50 or 101 layers (He et al. 2016). OOD Avg calculate the average
accuracy on the four OOD benchmark.
Detector
Backbone
MS COCO
Sim2real
Weather
Scene
Time
OOD Avg
Faster R-Ch R-CNN
1-110-6-444
42.1
52.1
35.3
36.0
26.4
37.5
RetinaNet
X-101-64x44
41.0
49.3
34.7
35.8
24.6
36.1
Mask R-CNN
X-101-64x4d
42.8
52.9
34.8
35.8
26.8
37.6
CornetNet
Hourglass
41.2
27.8
29.7
30.8
22.5
27.7
YOLOv3
DarkNet-53
33.7
38.2
27.0
28.2
19.2
28.2
FCOS
X- X-101-64x40
42.6
50.8
35.6
36.5
24.8
36.9
Cascade R-CNN
1-10--64x4
44.7
53.1
36.3
36.7
24.1
37.6
MS R-CNN
X101-64x4d
43.0
52.5
35.5
35.7
25.5
37.3
Libra R-CNNG @ Faster
X-101-64x44
42.7
51.4
34.9
35.5
25.5
36.8
Double-Hod Head R-CNN
R-50
40.0
52.2
35.3
35.2
25.5
37.1
VarifocalNet
X-101-64x4d
50.4
56.1
38.7
39.0
27.4
40.3
Sparse R-CNN
R-101
46.2
54.1
36.3
36.4
28.2
38.8
DETR
R-50
37.8
23.5
24.4
25.4
Deformable DETR
R-50
46.8
53.3
35.1
35.3
26.1
37.5
Swin Transformer
Swin-B
51.9
58.4
36.6
32.0
28.9
39.0
YOLOX
YOLOX-X
50.9
51.1
33.1
34.8
29.4
37.1
the other is snow intensity which indicates the intensity of the weather snow added using Airsim's plugin. Cars are annotated in the dataset since the car is common object in existing datasets. To construct CtrlShift dominated by correlation shift, we restrict the white cars only existing in the training set and control the quantity ratio Pcorrelation of white data in the training set. In way that spuriously correlates strongly with the class label since the color white will be more relevant to the car label in the training when the Prorrelation increases, which increases the correlation shifts in the dataset. And for CtrlShift dominated by diversity shift, we render snow weather effects on each training datum with certain intensity Pdiversity and the increase of Pdiversity corresponds to a larger diversity shift. See Figure for some examples. Moreover, we provide an API to generate the training set and the testing set with specific choices Pdiversity and Pcorrelation-
4.2 DETECTION METHODS FOR COMPARISONS
We choose widely-used detectors trained with the empirical risk minimization (ERM) or OOD gen- eralization algorithms MOD-ODBenchh The comparisons of different detectors trained with ERM on OOD-ODBench can help answer the question that whether the progress made by recently pro- posed detectors generalizable to OOD data. The henchmarks of detectors trained with proposed OOD) generalization algorithms can indicate whether the OOD generalization algorithms proposed recently are still effective for object detection beyond toy image classification tasks.
Detectors. Object detection models (detectors) generally can be categorized into two genres: one- stage methods and two-stage methods. One-stage detectors predict the bounding boxes well as the categories of the objects. Two-stage detectors predict the bounding boxes first to indicate the possible locations objects. Then. two-stage detectors conduct classifications on the images within the bounding boxes to predict the categories of these images. Recently, with the tremendous success of transformer (Vaswani et al. (2017), transformer-bas based detectors become popular. We have selected one/two-stee and transformer-based algorithms ranging from 2015 to 2021 for our Object Detection OOD generalization benchmark. One-stage detectors: RetinaNet (Lin et al. 2017b), CornerNet Law & Deng 20188), YOLOv3 (Redmon & Farhadi 2018), FCOS (Tian et al. 2019). NerifocalNet
Zhang et al. 2020a). YOLOX (Ge et | (021)) Two-stage detectors: Faster R-CNN (Ren et al. (He et al. 2017). Cascade R-CNN (Cai & Vasconcelos 2019). Mask Scoring R-CNN (MS R-CNN) (Huang et al., 2019), Libra R-CNN (Pang et al. 2019), Double-Head R-
2015), Mask R-CNN
CNN (Wu al. 2019) and Sparse R- R-CNN (Sun et al.
2020). Transformer-hasee detectors: DETR

---

Under review as a conference paper at ICLR 2023
Table 4: Experimental results of domain generalization algorithms OOD generalization object detection. All algorithms are implemented by ourselves based on Faster R-CNN (Girshick) 2015) with ResNet-50 (He et al. 2016) backbone. Hyper-parameter of each algorithm chosen among 0.1, and 10 according to the average AP over the four benchmarks.
Algorithm Hyper-paramee parameters
Sim2real
Weather
Scene
Time
Average
ERM
44.1
24.4
24.1
17.7
27.6
IB-ERM
A 0.1
42.5
24.7
23.4
18.2
27.2
IRM
Airm 0.1
42.3
24.9
23.2
17.5
27.0
MMD
7mmd
42.9
24.1
23.9
18.4
27.3
CORAL
7mmd 10
42.6
24.4
22.8
17.7
26.9
VREx
Aerez
43.1
24.7
24.3
18.0
27.5
GS
Areg 10
40.3
19.9
18.4
15.2
23.5
IGA
Apenalty =
42.9
24.6
24.3
18.6
27.6
GroupDRO
Ngroupdro 1
42.8
24.5
23.6
18.1
27.3
RSC
Arsc
41.0
8.7
8.8
7.0
16.4
CAD
Aead
42.8
24.2
23.1
18.4
27.1
CausIRL
Acirl
42.4
23.8
23.4
17.1
26.7
Carion et al. 2020 Deformable DETR Zhu et al. 2021 and Swin Transformer (Liu et al.
2021).
OOD generalization algorithms. have adapted eleven algorithms from different OOD research areas to the classification branch in object detection, including Empirical Risk Minimization (ERM) [Vapnik] 1998) which aims to minimize the loss function overall the training domains. (IB-ERM)
Ahhuj al. 202T which applies information bottleneck constraint to address OOD generaliza- tion, Invariant Risk Minimization (IRM) (Arjovsky et al.] 2019) which aims at estimating invariant different domains, adversarial feature learning (MMD) (Li et al.) (018h) which imposes Maximum Mean Discrepancy (Gretton et al. (012)) to align the distributions among dif-
correlations
ferent domains, correlation alignment (CORAL) (Sun Saenko 2016) which aims at matching the mean and covariance of feature distributions, Variance Risk Extrapolation (VREx) (Krueger al., 2021) which performs form of robust optimization over extrapolated domains, Gradient Starvation (GS) (Pezeshki et al. (0200) which derives a regularization to overcome the gradient descent phenomenon across different domains, (IGA) (Koyama & Yamaguchi (02T) which uses parametrization trick to conduct feature searching and predictor training, Group Distributionally Ro- bust Optimization (GroupDRO) (Sagaaa et al. (2019) which increases the importance of each do- main with penalty loss, and Representation Self Solf-Challengin (RSC) (Huang al. (020) iteratively challenges the dominant features to force the model to activate the remaining features. Optimal Rep- resentations (CAD) (Ruan al. 2021) designs self- supervised objectives to obtain representations on which risk is minimal to any distribution. Invariant Causal Mechanisms (CausIRL) (Chevalley al. 2022) learns the invariant features by viewing the learning process as causal process and introduces unifying framework.
4.3 MODEL SELECTION METHOSS
Model selection methods can influence the final rankings of methods to a large extent, especially in O0D) generalization tasks (Gulrajani & Lopez-Paz 2020). However, there is no consensus on what parameters selection strategy should be used in OOD generalization research for object detection. In OOD-ODBench, choose the models trained at the last epoch
model selection method. This because the testing data is inaccessible and selecting models based on the trainset's performances may lead to excessive over-fitting for current methods since there is huge distribution gap between the training set and testing set. For future research, we strongly recommend that researchers should detail and include the model selection methods in OOD generalization object detection research.
EXPERIMENT RESULTS
5.1 BENCHMARK RESULTS

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation study of OOD algorithms with different detectors on Sim2real. Faster R-CNN
(Ren al., 2015), RetinaNet (Lin al. 201 7b) and DETR (Carion et al. 2020 all with ResNet- 50 He et 2016) backbone.
Algorithm
Hyper-parameters
Faster R-CNN
RetinaNet
DETR
Average
ERM
44.1
38.5
36.0
39.5
IB-ERM
Aa 0.1
42.5
38.8
35.1
38.8
IRM
Airm = 0.1
42.3
38.4
34.7
38.5
MMD
7mmd
42.9
14.0
36.2
31.0
CORAL
Tmmmd = 10
42.6
35.8
36.9
38.4
VREx
Avrez
43.1
37.6
36.9
39.2
GS
Xvng = 10
40.3
33.0
33.0
35.4
IGA
Apenalty
42,9
38.2
29.5
36.9
GroupDRO
Igroupdro =
42.8
37.3
36.2
38.8
RSC
Arsc 10
41.0
38,4
36.9
38.8
CAD
Acad =
42.8
37.9
35.5
38,7
CausIRL
Acirt
42.4
38.3
36.3
39.0
In this section, conduct numerical experi- ments on our benchmark to reveal the OOD generalization ability for existing algorithms and we provide further discussion in Ap-
Relative AP(%) comparing baseline
OOD Object Detection IID Object Detection
pendix A.5 All experiments are conducted Pytorch platform with eight Tesla V100 GPUs. We evaluate each algorithm using the
10
Average Precision (AP) from MS COCO (Lin et al. 2014]. For object detection algorithms, codes based mmdetection (Chen eet al. 2019) and for domain generalization al- gorithms, our codes stemmed from Do- mainBed (Gulrajani & Lopez-Paz (020). We draw several conclusions from the results.
MaskRCNN
ComerNet
FCOS
CascadeRCNN
LibraRCNN VanifocllNer
DefermableDETa Sparse SparseRCNHv
The enormous achievements of object de- tection on IID datasets are marginal on the OOD condition. Table summarizes the OOD results of various object detection algorithms. If we use the classic Faster R-CNN (Ren et al.) 2015) as our baseline, all successful algorithms on object detection only marginally improve OOD generalization ability by 2.8 AP (Varifo- calNet (Zhang et al. 2020at) overall the four OOD benchmarks. While the IID object detection benchmark (MS COCO test-dev (Lin et al. (2014), this performance improvement can be up to 14.8 AP comparing VarifocalNet (Zhang al.) 2020a). Figure 4 intuitively displaces the significant discrepancy between IID and OOD. What is responsible for these results? We suspect two factors: One that current researches simply stem from the ideal assumption of IID regardless of whether can be satisfied in real scenarios. The other is that the improvement on IID datasets may be a phe- nomenon of over-fitting since few works provide sufficient evidence that the causal features have been learned during the training process without evaluating on OOD benchmarks.
Figure 4: The improvements of recent object de- tection methods over the baseline IID and OOD respectively. While the improvements on IID datasets (MS COCO) generalizable to OOD scenarios. The compared baseline method is Faster R-CNN.
prominent, it is not
The tremendous success domaín generalization algorithms confronting OOD inconsistent between classification and object detection. We draw this conclusion from Table the exper- imental results reported on OOD-bench (Ye et al. 2021). The OOD results on the four benchmarks in Table suggest that the domain generalization algorithms degenerate slightly outperform the ERM /Vapnik 1998) which can be attributed to the hyper-parameters bias. Moreover, as for VREx Krueger et al. 2021) which is the best models on Correlation- Benthn (Ye al. 2021). AP drops by 0.1 comparing to ERM (Vapnik 1998) while VREx (Krueger et al. 2021 outperforms ERM (Vap-)
nik 1998) by 8.6 AP Correlation-Bence Ye al. 2021)
RSC (Huang et al. 2020) which the
8

---

Under review as a conference paper at ICLR 2023
0-
32
32
0.2
21
24
0.4
0.4
0.6
4
0.8
0.8
1.0
0
0.2
0.4
0.6
0.6
0.2 0.6
1.0
0.4 0.5
1.0
(a) ERM (Vapmik 1998
(b) IRM (Arjovsky et al. 2019)
(c) VREx (Krueger et al. 2021
Figure 5: Controlled distribution shifts experiments results of ERM, IRM and VREx. X-axis
Diversity shift and Y-axis is Correlation shift. Each block indicates the AP(%).
best models on Diversity-Bende (Ye et al. 2021 degenerates 11.2 AP while improves 0.6 accuracy comparing to ERM (Vapnik) 1998).
The generalization inconsistency between classification and object detection of domain gen- eralization algorithms happens among different detectors. As shown in Table 4.2 we choose popular two-stage detector, Faster R-CNN (Ren et al., 2015), one-stage detector, RetinaNet (Lin al., 2017b), and transformer- for implementing domain generalization algorithms. Obviously, ERM achieves the best average generalization ability the three detectors and generalization algorithms has little relevance to the detectors.
detector, DETR (Carion al. 2020), base models
conclude that the degeneration of domain
5.2 CONTROLLED DISTRIBUTION SHIFTS EXPERIMENTS
Previous experiments provide performance analysis the real scenarios for OOD generalization object detection. But it is hard to see which kind of distribution shift leads to performance de- generation. To systematically analyze the generalization performance under the influence of the two-dimensional distribution shift, we test the performance of Faster-RCNN trained by ERM (Vap-) nik 1998). and the top performers on previous datasets (IRM (Arjovsky al. 2019 and VREx (Krueger et al. (021)) on CtrlShift dataset with different settings of the correlation shift and di- versity shift. The results are shown in Figure [5] methods consistently (Vapnik) 1998 Arjovsky et al., 2019; Krueger et al. (021) achieve the best AP when both correlation shift and diversity shift are low. For FRM (Vapnik 1998), the performance evenly degenerates on two dimensions. In Figure [bb] with the increase of the two-dimension shift, the performance of IRM (Arjovsky al.) 2019) in the horizontal direction tends to degenerate faster than in the vertical direction. This indi- cates that IRM (Arjovsky et al. 2019) confronts correlation shift better than diversity shift. From Figure we can observe the sumilar phenomenon happens for VREx (Krueger et al. (202T) on the two-dimensional shift. This phenomenon demonstrates that existing OOD generalization algorithms may help mitigate performance degradation when confronted with correlation shifts. Whereas for diversity shift, key components complex mixture of both shifts in real datasets. For future research, we recommend that both shifts should be included in new benchmark datasets and algorithms should be evaluated on both types of distribution shifts simultaneously.
missing improve the generalization abilities, let alone the
CONCLUSION AND DISCUSSION
In this paper, we propose the first benchmark for OOD-OD tasks, named OOD-ODBench. The benchmark suite includes four benchmark datasets along with synthetic dataset generate trolled distribution shifts. The experimental results conducted on OOD-ODBench suggest that the enormous achievements in classical ITD object detection are marginal OOD generalization object detection. And the OOD generalization methods mainly tested on classification cannot generalize object detection tasks. This raises questions about existing progress object detection and OOD generalization algorithms. We appeal for more attention from the community for this problem to propose an OOD-OD method that is undoubtedly effective.

---

Yolov4: Optimal speed and Under review as a conference paper at ICLR 2023
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Shan-Chrittophe Gagnon-Auder, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. NeurIPS, 34, 2021.
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. neural networks. arXiv: 141.:444. 2014.
Ehab Albadawy, Ashirbani Saha, and Maciej A. Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Medical Physics, 2018 8.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv:1907.0289; 2019.
Haoyue Bai, Rui Sun, Langing Hong, Fengwei Zhou, Nanyang Ye, Han-Jia Ye, S-H Gary Chan, and Zhenguo Li. Decaug: Out-of-distributiion generalization via decomposed feature representation and semantic augmentation. arXiv:20/2.09382, 2020.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv: 1711 2017.
Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. accuracy of object detection. arXiv:2004. 10934, 2020.
Hong-Yuan
Maria A Bravo, Sudhanshu Mittal, and Thomas Brox. Localized vision-language matching for open-vocabulary object detection. arXiv preprint arXiv:2205.006 06660 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: High quality object detection and instance doi: 10.1 101.:///mm22211.2966666 URL
segmentation. TPAMI, pp. 1-1, 2019. ISSN 1939-3539.
httt:://xx. doi. org wttpo/////.../.ee.eeeeereeeeeeee 1109/tpami. 201 9. 2956516
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy. and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv: 1e0::0906..55 155, 2019.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster r-enn for object detection in the wild. In CVPR, 2018.
Yuhua Chen, Haoran Wang, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Scale- aware domain adaptive faster r-cnn. IJCV, 2021.
Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. Invariant causal mecha- nisms through distribution matching. arXiv preprint arXiv:2206. 2022.
MMCV Contributors. MMCV: OpenMMLab computer vision foundation. https://qithur
com/open-mmt ab/ mmcv 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CVPR, pp. 3213-3223, 2016.
Dengxin Dai and Lue Van Gool. Dark model adaptation: Semantic image segmentation from day- time to nighttime. international conference intelligent transportation systems, 2018.
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region-based fully convo- lutional networks. NeurlPS, 29, 2016.
Daniel Coelho de Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communications, 2019.
10

---

Under review as a conference paper at ICLR 2023
Kumari Deepshikha, Sai Harsha Yelleni, PK Srijith, and C Krishna Mohan. Monte carlo dropblock for modelling uncertainty in object detection. arXiv preprint arXiv:2108.03614 2021.
Jinhong Deng. Wen Li, Yuhua Chen, and Lixin Duan. Unbiased mean teacher for cross-doman
object detection. In CVPR, 2020.
Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pp. 1021-1030, 2020.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01197 2022a.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulayy object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022b.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: Exceeding yolo series in 2021. arXiv:2107.088: 084400, 2021.
Ross Girshick, Fast r-enn. ICCV, pp. 1440-1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 580-587, 2014.
Kristen Grauman and Trevor Darrell. The pyramid match kernel: Discriminative classification with sets of image features. In ICCV, volume 2, pp. 1458-1465. IEEE, 2005.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel two-sample test. JMLR, 2012.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:210411392/ 2021.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv:2007.01434, 2020.
David Hall, Feras Dayoub, John Skinner, Haoyang Zhang, Dimity Miller, Peter Corke, Gustavo Carneiro, Anelia Angelova, and Niko Sünderhauf. Probabilistic object detection: Definition and evaluation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1031-1040, 2020.
Ali Harakeh and Steven Waslander. Estimating and evaluating regression predictive uncertainty in deep object detectors. arXiv preprint
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari. Piotr Dollar, and Ross Girshick. Mask r-cnn. ICCV, Oct 2017.
Zhenwei He and Lei Zhang. Domain adaptive object detection via asymmetric tri triwway faster-renn. In ECCV, 2020.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et Searching for mobilenetv3. In ICCV, pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:170..04861 2017.
Jaxxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain random- ization for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6891-6902, 2021.

---

Under review as a conference paper at ICLR 2023
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross domain generalization. In ECCV, pp. 140-140. Springer, 2020.
Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cmn. In CVPR, 2019.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv:1602.07300 2016.
Matthew Johnson-Roberson Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annota- tions for real world tasks? In ICRA, 2016.
KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5830-5840, 2021.
Masanori Koyama and Shoichiro Yamaguchi. Out-of-distribution generalization with maximal in- variant predictor. 2021.
David Krueger, Ethan Caballero, Joern-Herrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Outoof-distribution generalization via risk extrapo- lation (rex). In ICML, pp. 5815-5826. PMLR, 2021.
Hei Law and Jia Deng. Comernet: Detecting objects as paired keypoints. In ECCV, pp. 765-781. Springer Verlag, 2018.
Alexander Lehner, Stefano Gasperini, Alvaro Marcos-Ramiro. Michael Schmidt, Mohammad- Ali Nikouei Mahani, Nassir Navab, Benjamin Busam, and Federico Tombari. 3d-vfield: Adversar- ial augmentation of point clouds for domain generalization in 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17295-17304, 2022.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning generalize: Meta- learning for domain generalization. In AAAI, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pp. 1400-5549, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, pp. 624 639. 2018c.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, pp. 2117-2125, 2017a
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV. pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2103.14030 2021.
Ningming Ma, Xiangyu Zhang, Hai- Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, pp. 116-131, 2018.
12

---

Under review as a conference paper at ICLR 2023
Dimity Miller, Feras Dayoub. Michael Milford, and Niko Sünderhauf. Evaluating merging strategies for sampling-bar basen uncertainty techniques in object detection. In 2019 International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2019.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra r-cnn: Towards balanced learning for object detection. In CVPR, 2019.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard vary. In ICLR, 2021.
Christian Perone, Pedro Ballester, Rodrigo C. Barros, and Julien Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with self-ensembling Neurolmage, 2019.
Mohammad Pezeshki, Sékou-Ouma Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. arXiv: Learning, 2020.
Siyuan Qiao, Liang- Chich Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In CVPR, pp. 10213-10224, 2021.
Liang-Chieh
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. urXiv:2109.02934 2021.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. CVPR. pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv:1804.02777 2018.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 28, 2015.
Tobias Riedlinger, Matthias Rottmann, Marius Schubert, and Hanno Gottschalk. Gradient- based quantification of epistemic uncertainty for deep object detectors. arXiv preprint 107:4:2100.04517, 2021.
Adrian Lopez Rodriguez and Krystian Mikolajczyk. Domain adaptation for object detection via style consistency. BMVC, 2019.
Yangjun Ruan, Yann Dubois, and Chris Maddison. Optimal representations for covariate shift. arXiv:2201.00057, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv: Learning, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chi Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520, 2018.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor, 2017. URL httt:://arxxt brg//bb/17053 05065
Soroosh Shahtalebi, Jean- Chan-Christophe Gagnon-Audet, Touraj Laleh, Mojtaba Faramarzi. Kartik Ahuja, and Irina Rish. Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. arXiv:2106.025 0222666006 2021.
Yuge Shi, Jeffrey Seely, Philip HS Torr, Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv:2104.09937, 2021.
Petru Soviany, Radu Tudor lonescu, Paolo Rota, and Nicu Sebe. Curriculum self-paced learning for cross-domain object detection. CVIU, 2019.
Peng Su, Kun Wang. Zeng Xingyu, Shixiang Tang, Dapeng Chen, Di Qiu, and Xiaogang Wang. Adapting object detectors with conditional domain normalization. In ECCV, 2020.
13

---

Under review as a conference paper at ICLR 2023
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pp. 443-450. Springer, 2016.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo. SparseR-CNN: End-to-end object detection with learnable proposals. arXiv:2011.12450 2020.
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In CVPR, pp. 10781-10790, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. drXiv:1904.01335. 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEFE, 2017.
Antonio Torralba and Alexei Efros. Unbiased look dataset bias. In CVPR, 2011.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 969-977, 2018.
Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, volume 30, 2017.
Georg Volk, Stefan Müller, Alexander von Bernuth, Dennis Hospach, and Oliver Bringmann. To- wards robust enn-based object detection through augmentation with synthetic rain variations. in- ternational conference on intelligent transportation systems, 2019.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
J Wang, Xiang Li, and Charles X Ling. Pelee: real-time object detection system on mobile devices. NewrIPS, 31, 2018.
Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. In ECCV, 2020.
Aming Wu and Cheng Deng. Single-domain generalized object detection in urban scene via cyclic- disentangled disf-distillatinn In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 847-856, 2022.
Yue Wu, Yinpeng Chen, Lu Yuan, Zicheng Liu, Lijuan Wang, Hongzhi Li, and Yun Fu. Rethinking classification and localization for object detection. arXiv:1904.06449 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR. pp. 1492-1500, 2017.
Chang-Dong Xu, Xing-Ran Zhao, Xin Jin, and Xiu-Shen Wei. Exploring categorical regularization for domain adaptive object detection. In CVPR, 2020a.
Minghao Xu, Hang Wang, Bingbing Ni, Qi Tian, and Wenjun Zhang, Cross-domain detection via graph-induced prototype alignment. In CVPR, 2020b.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjic Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In AAAI, volume 34, pp. 6502-6509, 2020c.
Yilun Xu and Tommi Jaakkola. Learning representations that support robust transfer of predictors. arXiv:21 10. 090094 2021.
14

---

Under review as a conference paper at ICLR 2023
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv:2001.00677, 2020.
Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms. MxXiv:200.00722 2021.
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving video database with scalable annotation tooling. N0l.:8805.4687, 2(5);6, 2018.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Srngiiovann--innenteen Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2100-21 10, 2019.
Sergey Zakharov, Wadim Kehl, and Slobodan Ili.. Deceptionnet: Network-driven domain ran- domization. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 2019.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14393-14490 14400, 2021.
Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sünderhauf. Varifocalnet: An iou-aware dense object detector. arXiv:2008. 13367, 2020a.
Marvin Mengxin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group shift. 2020b.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, pp.
2018.
Xingxuan Zhang, Zekai Xu, Renzhe Xu, Jiashuo Liu, Peng Cui, Weitao Wan, Chong Sun, and Chen Li. Towards domain generalization in object detection. arXiv preprint arXiv:2203. 14387, 2022.
Xiaowei Zhao, Xianglong Liu, Yifan Shen, Yuqing Ma, Yixuan Qiao, and Duorui Wang. Revisiting open world object detection. arXiv preprint arXiv:2201.0047), 2022.
Kaiyang Zhou, Yongxin Yang. Timothy M. Hospedales, and Tao Xiang. Deep image generation for domain generalisation. national conference on artificial intelligence, 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: A survey. arXiv preprint arXiv: 103.::20330025000 2021.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv:1904.07850, 2019.
Xizhou Zhu, Weijic Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021. URL httpp::/ wpwnreview.net/Seeee. http://www...ettttooomo/o/g/gggo
15

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 THEORETICAL ANALYSIS OF OOD GENERALIZATION OBJECT DETECTION
The OOD generalization object detection has been fragmentally re- searched in previous research, however, no rigorous definition of OOD generalization object detection has been given. We give a definition and taxonomy as follows:
Z1
Y1
OOD generalization object detection (OOD-OD): In object de- tection tasks, algorithms learn mapping function f to predict the category (y) and location interested objects in an image x. OOD generalization object detection tasks, training and test data pairs [ Y) are not necessarily drawn from the same distribution. This poses great challenges for existing machine learning methods as most methods are reliant on exploiting the correlation between X and Y. Due to the distribution change, the correlation might not be generalizable. More specifically, we depict the causal data gen- erating process in Figure 6 When the data X are given, the causal features Zn and the non-causal features Zz are given. The causal
X
Z2
Y2
Figure 6: The causal influ- ence among the concerned variables.
features reliably determine the location and categories of interested objects in the input images. The non-causal features are irrelevant features for predictions. An intuitive example is that if we want to recognize dog in an image, the causal features are dogs' shapes. The non-causal features are the environment features, such as weather or captured time of the image. We improve the definitions in (Ye et al. 202T) and propose the following mathematical definitions for Z₁ and Ze given overall semantic features 2
2 E Z, p(z) ((2))((())))))))) AAA E Y2 p(ul=)) ( 32 E Y2 p(U22)) = (12l22) (1)
2 E Z2 :(2) ((2))_(=))>>) OVV E Y2 p(u122) + (v)l)) v3 Y2 :(2222)) # (v2l22) (2)
where p is the training distribution and is the test distribution. Since Z₁ is the stable and reliable predictor for the category and location of objects, there are two kinds of shifts intuitively because of the discrepancy of Z2 in training and test distribution. Diversity shift stems from the first kind of features in Z2 since the diversity of data embodied by novel features not shared by the environ- ments; whereas correlation shift is caused by the second kind of features in Ze which is spuriously correlated with some Y1 or Y2. Based on this, we partition Ze into two subsets:
S = (z Zo (zz)) g(z) 0} T
lz Zz)p(z) q(z) + 0}
Definition A.1. (Definition of Diversity shift and Correlation shift for OOD-OD.) Given S and T defined in Equation 3 the definitions of diversity shift and correlation shift are given follows:
1
Däiversity = 2 Js
pp(z) -(()) |
dz
(4)
/ Vp(z) ((2)
Dearrelation
Σ
p(un. 3212) g(vi. 3212) dz
can be seen that both Daiversity and Deumelation are within the range of [0, 1). Daiversity measures the support difference non-causal features for object detection. While Deumelation gauges the variations of conditional probabilities of the object category Y given non-causal features and object locations. indicator for spurious correlations existing in datasets. The proposed definition
This serves first provides a quantitative way for measuring the distributional shifts for OOD-OD to the best of knowledge. We leave it for future work to compute the numerical values of shifts given object detection dataset.
A.2 PROOF
Proposition A.2. For any probability functions p and q of training distribution and testing distribu- tion. Daiversity and D Deorrelation are inclusively bounded between 0 and 1.
We no category label (Y:) shifts for simplicity and without loss of generality
16

---

Under review as a conference paper at ICLR 2023
Table 6: Illustration of the settings Sim2real. gam and sim indicate the training set and validating set from original 10K (Johnson-Roc Honsoon-Roberso et al., 2016) while city uain and city val are from original Cityscapes (Cordts et al. 2016). Quantity indicates the number of images. Total counts the total number of training and testing domains respectively.
Setting
Split
Train
Test
Quantity
Total
simarain
8000
8500
SiMaal
1000
part-sim-part-real
city tain
V
2975
3975
city val
V
500
Simarain
V
8000
all-sim-all-real
simaal
V
1000
9000
city train
V
2975
3475
city val
500
Proof. Obviously, both Daiversity and Deonelation are positive. triangle inequality as followed:
Then, we prove the upper bound by the
Daiversity
2 S p([))) (()) dz S 1 [p(2) + [())]d= <
2 /
(6)
Similarly, we have the following inequality:
2 / Vp(z) ((2)
1
Deomelltion
p(y,, 32|=) v2|=)) dz
v1.82
$ 2
1
/ Vp(z). .g(2)
Σ
Ip(1 3212) (311 12 d2
91.3/2
2 / 2/p(z) d(=)d== / Vp(z) 9(3) 5
1
The second inequality is due triangle inequality.
A.3 IMPLEMENTATION DETAILS
To evaluate the object detection algorithms, we use the models and the pre-trained weights provided by mmdetection (Chen et al., 2019) For domain generalization algorithms on OOD generalization object detection, derive the implementations using Faster R-CNN (Ren et al. (0115) with ResNet- 50 FPN backbone (He et al. 2016) from torchvision. The whole network is optimized by Stochastic Gradient Descent with learning rate 0.02, momentum 0.9 and weight decay 0.0005.
A.4 FURTHER RESULTS
Task complexity. To analyse the IID condition on CtrlShift, which indicates both Deotrelation and Daiversity equal zero, we propose a hyper-parameter task complexity to measure the difficulty of the task. The difficulty adjusted by using a percent novel data in the testing set addition to the original training data. The experimental results are shown in Figure 7 The generalization ability of each algorithm drops with the increase of task complexity.
Sim2real benchmark. The training set of the Sim2real results reported in the main manuscript com- prises the training data from Sim10K (Johnson- Roberson et al.) (2016) and the validating data from Cityscapes (Cordts al.) (2166). while the testing set comprises the training data from Cityscapes (Cordts et al.) (016)) and the validating data from Sim10K Johnson-Roberson et al. (016)) (noted by cert-sim-paat--ee
more details can be found in Table A.4) We reported the experimental results on si---immll-eess in Table 7 and Table g
Full results. Table 10
12 and are the full experimental results of Table and including AP, Ap AP7s. AP,, AP m and API evaluation metrics.
17

---

Under review as a conference paper at ICLR 2023
Table The experimental results of object detection algorithms on the aillsim-alll-eet of Sim2real. Mem (GB)f and Inf time (fps)f are from mmdetection (Chen et al. 2019).
Detector
Backbone
Memi
fpsi
AP
Faster R-CNN (Ren et al. 2015)
X-101
10.3
9.4
35.6
RetinaNet (Lin al. 201 77
X-101
10.0
8.7
38.0
Mask R-CNN (He et al. 2017
X-101
10.7
8.0
36.7
CornetNet (Law Deng 2018
Hourglass 104
13.9
4.2
21.6
YOLOv3 (Redmon & Farhadi 2018
DarkNet-53
7.4
48.1
28.2
FCOS (Tian et al.; 2019)
X-101
10.0
9.7
37.9
Cascade R-CNN (Car & Vasconcelos
2019
X-101
10.7
40.5
MS R-CNN (Huañg el al.
2019
R-X101
11.0
8.0
35.7
Libra R-CNN (Pang al. 2019
X-101
10.8
8.5
35.3
DH R-CNN (Wu et al. 2019
R-50
6.8
9.5
33.8
VarifocalNet (Zhang al. 2020a
X-101
42.3
Sparse R-CNN (Sun et al. 2020)
R-101
40.3
Deformable (Zhu et al. 2021)
R-50
37.4
YOLOX (Ge et al. 2021)
YOLOX-X
28.1
36.4
Table 8: The experimental results of domain generalization algorithms the all-sim-all-real of Sim2real.
Algorithm
hyper-parameters
AP
ERM (Vapnik
1998
32.8
IB-ERM (Ahuja
2021
Ai 100
18.3
IRM AAroovsyy el all
2019)
Airm 1
32.7
MMD et 2018b)
7mmd
33.2
CORAL (Sun & Saenko
2016
7mmm =
32.5
VREx (Krueger et al. (021)
Aprez
32.4
GS (Pezeshki et al. (0200)
Xreg 0.1
31.4
IGA (Koyama Yamaguchi 2021
penalty 1000
33.4
GroupDRO (Sagawa et al. 2019
Ngroupdro 0.01
31.9
ERM
IRM
42.5
MMD
40.0
CORAL
37.5
VREx
35.0
GS
IGA
GroupDRO
27.5
0.2
0.4
0.6
0.8
1.0
Figure 7: X-axis task complexity a. Each block indicates the AP(%).
18

---

Under review as a conference paper at ICLR 2023
Table 9: Experimental results of detectors Sim2real.
Detector
Backbone
AP
APs0
AP75
AP,
APm
APt
Faster R-CNN
X-101
52.1
73.6
55.2
22.3
58.6
82.3
RetinaNet
X-101
49.3
72.2
51.0
16.8
56.2
81.6
Mask R-CNN
X-101
52.9
74.2
56.4
23.2
60.0
82.7
CornetNet
Hourglass 104
27.8
40.2
28.7
6.6
34.4
46.7
YOLOv3
DarkNet-53
38.2
62.7
40.0
10.2
42.2
70.0
FCOS
X-101
50.8
72.1
53.1
21.4
56.6
82.1
Cascade R-CNN
X-101
53.1
74.1
56.2
23.2
59.2
83.6
MS R-CNN
R-X101
52.5
73.7
56.0
23.0
59.5
82.3
Libra R-CNN
X-101
51.4
72.1
55.0
22.0
59.7
81.5
DH R-CNN
R-50
52.2
73.4
56.1
23.6
59.2
81.2
VarifocalNet
X-101
56.1
75.3
59.4
25.9
63.9
85.9
Sparse R-CNN
R-101
54.1
76.8
57.7
26.7
59.3
81.9
DETR
R-50
37.8
63.9
38.3
10.2
38.0
72.1
Deformable
R-50
53.3
78.0
57.0
25.6
59.7
80.2
Swin Transformer
Swin-B
58.4
78.5
63.5
31.4
65.5
84.2
YOLOX
YOLOX-x
51.1
70.3
53.7
19.3
58.4
84.4
Table 10: Experimental results of detectors on Weather.
Detector
Backbone
AP
AP50
AP75
AP,
APm m
APP
Faster R-CNN
X-101
35.3
56.1
36.3
14.5
37.5
49.5
RetinaNet
X-101
34.7
54.3
35.8
12.8
36.9
49.4
Mask R-CNN
X-101
34.8
55.2
35.8
14.9
36.9
47.9
CornetNet
Hourglass 104
29.7
46.1
30.6
12.2
34.3
38.4
YOLOv3
DarkNet-53
27.0
47.9
26.1
8.7
29.4
40.5
FCOS
X-101
35.6
55.7
36.4
14.8
37.9
48.8
Cascade R-CNN
X-101
36.3
56.4
37.8
15.1
38.2
51.7
MS R-CNN
R-X101
35.5
56.1
36.6
14.6
37.4
50.2
Libra R-CNN
X-101
34.9
54.8
36.3
14.0
37.4
49.5
R-CNN
R-50
35.3
55.7
36.5
14.5
37.2
50.5
VarifocalNet
X-101
38.7
59.3
39.8
16.0
41.0
54.1
Sparse R-CNN
R-101
36.3
57.5
37.1
15.6
38.5
50.8
DETR
R-50
23.5
41.7
22.5
4.9
23.2
40.1
Deformable
R-50
35.1
56.7
35.3
14.7
37.2
49.8
Swin Transformer
Swin-B
36.6
56.2
38.2
15.8
39.7
51.1
YOLOX
YOLOX-X
33.1
51.7
34.1
11.5
35.4
47.6
A.5 FURTHER DISCUSSION
High accuracy in IID may be the results of over-fitting since the spurious correlation exists in both the training and the testing distribution. Recently proposed methods, such as Deformable DETR, Sparse R-CNN and Swin Transformer, significantly outperform classic Faster R-CNN IID dataset, however, have similar performance on OOD datasets. We can conclude that these meth- ods make prediction based on spurious correlated representations existed in IID data and existed in OOD data.
Defined by our theory in Appendix A. diversity shift is embodied by the novel features not shared by environments while correlation shift is caused by the shared non-causal features. Experiment results in Section 5.2 show that VREx and IRM obtain better OOD generalization ability against correlation shift than diversity shift, and it is challenge to tackle diversity shift.
Current study starts to explore the impact of model architectures on OOD performance. NAS-OoD (?) applies neural architecture search strategy to find the architecture with optimal OOD general- ization ability and the results demonstrate that model architecture can significantly influence OOD accuracy. Some detectors have relatively consistent performance OOD, such VarifocalNet,
19

---

Under review as a conference paper at ICLR 2023
Table 11: Experimental results of detectors on Scene.
Detector
Backbone
AP
50
AP75
AP,
APm
AP1
Faster R-CNN
X-101
36.0
56.3
38.0
15.9
38.8
54.7
RetinaNet
X-101
35.8
56.
37.4
13.5
39.5
55.9
Mask R-CNN
X-101
35.8
56.2
37.5
15.7
38.2
54.3
CornetNet
Hourglass 104
30.8
46.1
32.6
13.8
32.7
45.9
YOLOv3
DarkNet-53
28.2
49.7
27.1
10.0
32.3
44.2
FCOS
X-101
36.5
56.7
38.7
15.9
39.1
54.8
Cascade R-CNN
X-101
36.7
56.7
38.1
15.0
39.3
58.9
R-CNN
R-X101
35.7
56.1
37.2
15.3
38.4
54.4
Libra R-CNN
X-101
35.5
55.6
36.8
15.9
38.3
54.9
R-CNN
R-50
35.2
55.3
36.9
15.9
38.2
54.7
VarifocalNet
X-101
39.0
58.8
40.8
18.1
41.4
59.4
Sparse R-CNN
R-101
36.4
57.4
37.4
18.4
39.1
56.0
DETR
R-50
24.4
42.8
23.8
6.3
25.3
47.4
Deformable
R-50
35.3
55.8
36.2
15.3
37.9
54.6
Swin Transformer
Swin-B
32.0
50.1
33.7
14.7
36.7
48.3
YOLOX
YOLOX-X
34.8
54.1
36.2
13.4
37.7
53.4
Table 12: Experimental results of detectors on Time.
Detector
Backbone
AP
AP
AP15
AP,
APm
API
Faster R-CNN
X-101
26.4
45.2
26.3
9.7
25.2
39.7
RetinaNet
X-101
24.6
43.1
24.3
7.8
23.5
37.8
Mask R-CNN
X-101
26.8
45.8
26.5
9.5
25.8
39.8
CornetNet
Hourglass 104
22.5
37.8
22.3
8.6
23.4
29.6
YOLOv3
DarkNet-53
19.2
34.5
18.7
5.1
18.3
31.3
FCOS
X-101
24.8
42.5
24.4
9.2
23.6
36.2
Cascade R-CNN
X-101
24.1
41.0
23.6
8.7
21.9
37.6
MS R-CNN
R-X101
25.5
43.4
25.3
23.8
39.0
Libra R-CNN
X-101
25.5
43.7
25.2
8.9
24.6
39.2
DH R-CNN
R-50
25.5
43.9
25.4
24.9
37.6
VarifocalNet
X-101
27.4
45.6
27.0
10.4
26.0
39.3
Sparse R-CNN
R-101
28.2
48.3
27.7
10.6
27.2
41.4
DETR
R-50
15.8
30.9
13.7
3.2
14.7
28.2
Deformable
R-50
26.1
46.2
25.1
10.1
25.4
39.0
Swin Transformer
Swin-B
28.9
47.6
29.4
10.5
28.6
42.3
YOLOX
YOLOX-X
29.4
48.0
29.2
9.4
28.2
44.0
20

---

Under review as a conference paper at ICLR 2023
Table 13: Generalization performance of detectors with OOD algorithms.
Algorithm
Dataset
AP
30
AP
AP,
APm
APt
ERM
44.1
64.0
47.1
13.8
41.2
68.4
IB-ERM
A 0.1
42.5
64.4
44.9
12.6
39.2
67.0
IRM
Airm 0.1
42.3
64.9
45.0
12.7
39.4
66.1
MMD
7mmd
42.9
64.8
45.4
13.2
39.8
67.2
CORAL
Tmmd 10
42.6
64.6
45.3
12.8
39.7
66.8
Sim2real
VREx
Avrez
43.1
65.4
45.4
13.0
40.0
67.4
GS
Xreg 10
40.3
64.2
42.5
11.5
37.6
65.3
IGA
Apenalty =
42.9
64.7
45.6
13.7
40.0
66.6
GroupDRO
Ngroupdro
42.8
65.0
45,4
12.8
39.9
67.1
RSC
Arsc 10
41.0
63.6
43.2
10.8
38.6
64.9
ERM
24.4
47.8
21.8
9.5
29.3
41.9
IB-ERM
A 0.1
24.7
47.7
21.9
9.7
29.6
41.8
IRM
Arm 0.1
24.9
48.3
22.1
9.7
29.9
41.9
MMD
7mmd
24.1
47.2
21.5
9.4
29.0
41.6
CORAL
Tomd 10
24.4
47.3
21.9
9.7
29.2
41.6
Weather
VREx
Xprez
24.7
48.4
21.8
9.6
29.6
41.4
GS
Xveg 10
19.9
39.2
17.5
8.0
24.6
35.4
IGA
Apenalty
24.6
47.9
21.7
9.9
29.6
41.2
GroupDRO
Agroupdro 1
24.5
47.8
21.6
9.6
29.2
41.3
RSC
Arsc 10
8.7
17.6
7.5
3.3
11.1
17.9
ERM
24.1
46.7
21.2
9.7
29.4
46.5
IB-ERM
A 0.1
23.4
45.6
20.4
9.5
28.6
45.7
IRM
Aim 0.1
23.2
45.8
20.3
9.4
28.4
45.0
MMD
7mmd
23.9
46.4
21.2
9.4
29.2
47.5
CORAL
Tomm 10
22.8
44.5
20.1
9.4
27.8
45.4
Scene
VREx
Aprez
24.3
47.0
21.6
9.7
29.7
46.6
GS
Xveg 10
18.4
35.9
16.4
8.0
23.7
38.0
IGA
Apenalty =
24.3
47.6
21.0
10.2
29.3
45.9
GroupDRO
Agroupdro 1
23.6
46.1
21.1
10.1
29.0
45.7
RSC
Arsc 10
8.8
17.9
7.4
3.5
11.6
21.5
ERM
17.7
37.5
14.6
6.6
19.5
32.0
IB-ERM
A 0.1
18.2
38.3
15.2
6.8
20.1
32.1
IRM
Xirm 0.1
17.5
37.5
14.2
6.6
19.3
30.6
MMD
Ammd
18.4
38.4
15.4
7.0
20.2
31.4
CORAL
1mmd 10
17.7
37.6
14.6
6.4
19.5
30.7
Time
VREx
Aprez
18.0
38.3
14.9
6.8
19.9
31.2
GS
Xveg 10
15.2
32.4
12.5
5.6
17.4
27.7
IGA
Apenalty = 1
18.6
39.1
15.4
7.2
20.4
32.0
GroupDRO
Ngroupdro 1
18.1
38.1
15.1
6.9
20.1
31.4
RSC
Arsc 10
7.0
15.0
5.7
2.5
8.2
14.7
21

---

Under review as a conference paper at ICLR 2023
the proposed varifocal loss Zhang et al. 2020a can be considered to improve OOD generalization ability.
22


Reviews:
[{"review_id": "vUeZZTQ37G", "reviewer": "Reviewer_aJ6J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DetectBench - a benchmark for Out-of-Distribution (OoD) Object Detection, as most works mainly focus on OoD image classification. After introducing a new train-test split on existing datasets and newly proposed dataset CtrlShift, the paper benchmarks a wide range of detectors (Faster R-CNN, DETR, etc.) with different backbones, self-implemented OoD image classification algorithms (ERM, RSC, GroupDRO, etc.), and different detectors with same backbone (Faster R-CNN, RetinaNet, DETR) on these datasets. The main findings are that IID object detection algorithms are marginal on OoD object detection, and the authors propose a new synthetic dataset, which can be used to simulate the two types of distribution shifts based on user’s inputs.\n\nObject detection is an important task for many applications, including safety-critical ones such as self-driving cars and health care. Accordingly, improving the ability of object detectors to deal with unseen distribution shifts is highly relevant. This benchmark is thus well-motivated.", "summary_of_the_review": "I agree the proposed benchmark is well-motivated and the task is important for the research community. However, with the findings above, I currently give the paper a borderline reject score.\n\nUpdate: My concerns are not entirely addressed after the rebuttal. I will keep the rejection score for this submission, due to unconvincing experimental results (without cross-validation), weak diversity shift benchmark, the limited novelty of the findings, missing related works, and the proposed benchmarks overlapping with previous works.", "strength_and_weaknesses": "Strong points:\n\n1. The paper is well written.\n2. Details of the datasets are provided.\n3. The authors run the benchmark on both two kinds of distribution shifts, proposed by [1].\n4. The authors introduce a new synthetic dataset CtrlShift, of which API can be modified to simulate the two kinds of shifts mentioned above. \n5. This would be a good contribution to the community.\n6. The authors extensively benchmark and run experiments on a wide range of detection algorithms.\n7. Some algorithms from OoD image classification are also adapted to DetectBench.\n\nNevertheless, there are some concerns about the paper that make the findings not convincing. I am happy to raise the score if the authors can address the concerns.\n\nWeak points:\n\n1. The train-test split is still a bit weak. For example, in DomainBed [2], for PACS dataset they divide it into four domains. Using the training-domain splitting strategy, they train the models on three selected domains and test on the other. This is to determine if the distribution shift is actually present, or it is because a model or architecture simply performs better on a specific domain. In this paper, the authors split the Weather in BDD100K into six domains, but only select two domains to train and the rest four domains to test. What would happen if we simply flip the train-test split? Will the ranking still hold?\n2. Missing the potential impact of the small-scale dataset on big models. Due to the dataset size, Sim2real has fewer data compared to BDD100K. This could make DETR models underperform compared to Faster R-CNN, as transformer models are generally data-hungry due to their size [5].\n3. Missing justification for choosing a strong backbone for Faster R-CNN and other detectors, but a regular backbone for DETR. In the paper, the authors choose ResNeXt-101-64x4d for Faster R-CNN and other models but use ResNet50 for DETR and Deformable DETR. Backbone network plays a vital role in the performance, which has been observed in both IID Object Detection [3] and OoD classification [4]. I think it is better to use the same backbone in this table so that we can measure the effect of different detectors.\n4. Missing results in (strong) Diversity Shift datasets. Diversity shift datasets such as Watercolor2k, Comic1k, Clipart1k [7] are missing from the paper. In the classification benchmark, the diversity shift datasets (PACS, VLCS, etc.) are much harder to imporve than correlation shift datasets (NICO, CelebA). From my experience, all OoD classification algorithms achieve very high accuracy by simply using ResNet50 as backbone on Correlation shift datasets, making gains of algorithms on this shift extremely minor.\n5. The findings are limited in novelty and not very convincing. Section 5.1 highlights some findings in the paper, but I do not find two points very “surprising” nor convincing:\n5.1) Finding 1 (The enormous achievements of object detection on IID datasets are marginal on the OOD condition): It is widely known that IID is a strong assumption in real-world use cases, and most models are sensitive to OoD tasks. This is not limited to only image classification, but also object detection, which has already been observed in BDD100K paper [6].\n5.2) Finding 3 (The generalization inconsistency between classification and object detection of domain generalization algorithms happens among different detectors): As pointed out above in Weak points 2 and 3, Sim2real is a relatively small dataset for DETR (8.5k images), which possibly means results in Table 5 are not reflecting an accurate performance of DETR. Based on Table 3, the reviewer thinks Deformable DETR is a more suitable choice, and the conclusion “the degeneration of domain generalization algorithms has little relevance to the detectors” is probably a bit overclaimed.\n\nMinor fixes:\n1. Figure 4 X-axis is a bit hard to read. The authors may consider using vertical labels.\n2. Finding 3 in Section 5.1, “Table 4.2” should be “Table 5”\n\nReferences\n\n[1] Ye, Nanyang, et al. “Ood-bench: Benchmarking and understanding out-of-distribution generalization datasets and algorithms.” In CVPR (2021).\n\n[2] Gulrajani, Ishaan, and Lopez-Paz, David. “In search of lost domain generalization.” In ICLR (2020).\n\n[3] Zhu, Xizhou, et al. “Deformable DETR: Deformable transformers for end-to-end object detection.” In ICLR (2021).\n\n[4] Angarano, Simone, et al. “Back-to-bones: Rediscovering the role of backbones in domain generalization.” arXiv preprint arXiv:2209.01121 (2022).\n\n[5] Wang, Wen, et al. “Towards Data-Efficient Detection Transformers.” In ECCV (2022).\n\n[6] Yu, Fisher, et al. “Bdd100k: A diverse driving video database with scalable annotation tooling.” arXiv preprint arXiv:1805.04687 2.5 (2018): 6.\n\n[7] Inoue, Naoto, et al. “Cross-domain weakly-supervised object detection through progressive domain adaptation.” In CVPR (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WT7kgic_In", "reviewer": "Reviewer_m6Le", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is a benchmark for out of distribution generalization in object detection. The authors create a benchmark to evaluate out of distribution generalization algorithms specifically for the task of object detection (object classification and bounding box regression), using multiple well known datasets in specific settings, while also introducing a new synthetic dataset that can be used to study correlation and diversity shift.\n\nThe contributions are:\n- A new benchmark DetectBench to evaluate OOD generalization in object detectors (authors call this OOD-OD).\n- A new dataset CtrlShift to study diversity and attribute shift in OOD-OD.\n- A good evaluation and important insights on the effect of diversity and attribute shift on OOD generalization algorithms in object detectors, signaling the object detectors are more affected by diversity shift than attribute shift. This is an important conclusion that can guide future object detector and OOD-OD method design to improve OOD generalization performance.\n- The evaluation also indicates that standard OOD generalization algorithms (designed for classification), do not transfer successfully to object detection, indicating the need for more specialized OOD generalization algorithms for this task.", "summary_of_the_review": "I believe that this paper should be accepted. There are only minor issues, there is a clear gap in the state of the art that this paper fills, a benchmark/dataset for out of distribution generalization for object detection tasks, and the paper does an excellent initial evaluation. This paper has good insights for future research, mainly that OOD generalization algorithms designed for classification fail to improve performance in object detection, and that object detectors are more sensitive to diversity than correlation shifts.\n", "strength_and_weaknesses": "\nStrengths\n- The paper is very well written and it is clear, ideas flow clearly, I have no further remarks about writing.\n- An OOD generalization benchmark for object detection makes sense, as there is no dataset/benchmark specifically for this task, and the paper evaluation is of good quality, by combining standard object detectors with OOD generalization algorithms, to see if they work.\n- It is a good idea to have test OOD generalization for object detection, as the authors argue, OOD generalization algorithms are not usually evaluated for more complex tasks like object detection, they usually are only evaluated for classification datasets.\n- The proposed benchmark uses a selection of public datasets, covering multiple kinds of shifts, and additionally a synthetic dataset (CtrlShift) is proposed which has diversity and correlation shifts made in a controlled way, allowing the evaluation of the effect of both shifts combined. \n- The design of the DetectBench benchmark seems sensible, it contains multiple sub-benchmarks where train and test sets contain non-intersecting sets of conditions, like weather, scene types, time, and simulation/reality, which make a simple test set that is out of distribution with respect of the training set.\n- I believe that the evaluation and initial benchmark results presented in the paper are correct. There is a good selection of OOD generalization algorithms to test, and of object detectors (Faster R-CNN, DETR, RetinaNet) for the main results, which makes the overall conclusions robust.\n- Two major conclusions of this benchmark are that improvements to OOD generalization in classification settings do not transfer to improvements to OOD generalization in object detection, as tested on Faster R-CNN like on Table 4, and on sim2real for Faster R-CNN, DETR, and RetinaNet, and measured by mAP in the OOD setting. In some cases the performance actually decreases instead of increase, and that in the OOD setting, diversity shift seems to affect OOD generalization algorithms much more than attribute shift. These two conclusions are good contributions to the state of the art.\n- While the main results (Table 4) are presented for Faster R-CNN on the whole DetectBench benchmark, there are additional results on the sim2real sub-benchmark using Faster R-CNN, DETR, and RetinaNet about OOD generalization algorithms, producing a more robust conclusion that OOD generalization algorithms fail to improve performance in object detectors.\n- The design of CtrlShift, while simple, seems to do the job, as Figure 3 shows, there is correlation shift produced by changing the color of a car model, and for diversity shift there is simulated snow added to the environment, and both shifts can be combined to study their joint effect.\n\nWeaknesses\n- I think the only weakness in this paper is the name of the Benchmark, DetectBench is very generic, sounds like any detection benchmark, it should have OOD and generalization (or similar abbreviations) in the name, to distinguish it from OOD detection benchmarks and standard OOD generalization benchmarks. Please find a good name that includes all of this.\n\nMinor Issues\n- Figure 5 and related heatmaps would be better presented with the axis labeled in the figure itself, instead of relying on the caption. Just as Figure 3, Figure 5 should have axis labels for diversity and correlation shift. Additionally to this, Figure 5 should have the origin at the bottom left instead the top left, which would make the figure easier to interpret.\n- In Table 3, I think it would be good to also provide the original mAP of these detectors, in order to put the benchmarked numbers into context, so the reader can easily see if mAP is lower than expected or not.\n- I believe that in Table 2, simulation and reality are flipped in the sim2real description (last row, third column).\n- Could you provide a bit more information on CtrlShift? I see that there are 2000 images, but how are these distributed according to the two kinds of shifts?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Rxnc9LFjrY", "reviewer": "Reviewer_Sn2d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Building on top of BDD100k, Cityscapes and Sim10k, the proposed DetectBench in this work is an object detection benchmark for evaluating the OOD generalization performance of object detectors. The benchmark consists of 4 types of domain shifts, i.e., sim2real, weather, scene and time. The authors compared a number of object detectors on DetechBench, showing their yet poor OOD generalization performance. Using OOD generalization techniques developed for classification tasks, the object detection performance remains unsatisfactory, indicating OOD generalization of object detection is an underexplored task. ", "summary_of_the_review": "This work targets an important problem, but the usefulness, novelty and effectiveness of the proposed benchmark are not convincing.", "strength_and_weaknesses": "Strength: \nOOD generalization is an important task and is underexplored for object detection algorithms. This paper targets at this important problem. The authors proposed a OOD benchmark for object detection and compared a number of object detectors, empirically showing that existing solutions still struggle at dealing with domain shifts.\n\nWeakness:\n1. DetectBench does not differ much from previous benchmarks in this area. For example, [1] uses the eight shared categories in Cityscapes, Foggy Cityscape and BDD100k, and the Car category on these three datasets and additionally SIM 10k and KITTI. [2] also uses BDD100K, Cityscapes, Sim10K and KITTI as four domains, and train on three of them and test on the last one. The proposed CtrlShift dataset is different to these benchmarks, but is also limited to only two-dimension shift, and is not a very comprehensive new dataset with a large syn2real gap.\n\n2. While considering a good number of object detectors, the authors mainly evaluate their performance on the constructed benchmarks. Beyond the general observation that these methods are not robust under domain shifts, there is no further insightful discussion, e.g., correlation between ID and OOD performance, different generalization behaviors under different shifts, how the model architecture would affect ID and OOD performance. More importantly, there generally miss a good justification that DetectBench is a qualified benchmark that allows cross-compare different methods. Especially, it is already shown by many previous works that current object detectors fail at generalizing to OOD scenes [1] [2] [3]. \n\n3. The compared OOD generalization baselines are not adapted to the task of object detection. All the OOD generalization methods are only used for the classification branch of the object detectors. The localization branch is not considered. It is important that this new benchmark should also investigate these OOD generalization techniques developed for object detection.\n\n[1] Domain-Invariant Disentangled Network for Generalizable Object Detection\n[2] Towards Domain Generalization in Object Detection\n[3] OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LtXdEiSH6lO", "reviewer": "Reviewer_JACh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works as a benchmark of out-of-distribution object detection (OOD-OD), which evaluates the detector's generalization ability to a different data (open-world) domain. While many OOD methods have been proposed to amortize the domain gap, DetectBench, consisting of four datasets, shows the current OOD generalization algorithms no longer work on complex object detection tasks. This paper leads to an in-depth discussion on the progress of OOD work, and provides a foundation for OOD-OD research.", "summary_of_the_review": "This work is a benchmark of out-of-distribution object detection. Although it has limited technique contribution, it is marginally above the threshold because the problem is novel and less studied.  ", "strength_and_weaknesses": "[Strength]\n1. This paper works on an interesting and practical problem of OOD generalization of object detection models. The train-test discrepancy exists in many CV problems, and object detection has been well-studied and applied in research and industry. This DetectBench can work as a fundamental benchmark to study the OOD problem in object detection.\n\n2. The benchmark covers multiple conditions in train-test discrepancy: weather, scene, time, and simulation. They can fairly work together to evaluate the OOD progress\n\n3. Extensive experiments of 16 detectors and 12 OOD algorithms are measured on DetectBench. Those results are promising for the next generation of OOD-OD research.\n\n4. The author has several new assumptions from the empirical studies. For example, they claim that improvement in object detection does not always transfer to OOD object detection, and the success in OOD is also not always consistent between classification and object detection, \n\n[Weakness]\n1. The technical contribution is relatively limited, for example, a better OOD-OD method can be proposed/suggested given the benchmark result. \n\n2. Although it is sufficient to have more than ten OOD algorithms to draw the conclusion, but the re-implementation of a different problem may affect the observation. For example, using only 3 possible values for the hyper-parameter in all the methods may not be enough. \n\n3. The synthetic CtrlShift dataset is very fine-grained on a single object class with a single instance per image, therefore, it is lack of generalization. A better motivation is also supposed to explain the importance of the 'car' class in object detection, and why car color and snow level are selected for the only domain shift options. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject