Towards Unsupervised Model Selection for Domain Adaptive Object Detection
Authors: Hengfu Yu
Jinhong Deng
Wen Li
Lixin Duan
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain Adaptive Object Detection
Unsupervised Model Selection
Primary area: machine_vision
TL;DR: An effective model selection approach without using target labels  for domain adaptive object detection.

Abstract: Evaluating the performance of deep models in new scenarios has drawn increasing attention in recent years due to the wide application of deep learning techniques in various fields. However, while it is possible to collect data from new scenarios, the annotations are not always available. Existing Domain Adaptive Object Detection (DAOD) works usually report their performance by selecting the best model on the validation set or even the test set of the target domain, which is highly impractical in real-world applications. In this paper, we propose a novel unsupervised model selection approach for domain adaptive object detection, which is able to select almost the optimal model for the target domain without using any target labels. Our approach is based on the flat minima principle, i.e., models located in the flat minima region in the parameter space usually exhibit excellent generalization ability. However, traditional methods require labeled data to evaluate how well a model is located in the flat minima region, which is unrealistic for the DAOD task. Therefore, we design a Detection Adaptation Score (DAS) approach to approximately measure the flat minima without using target labels. We show via a generalization bound that the flatness can be deemed as model variance, while the minima depend on the domain distribution distance for the DAOD task. Accordingly, we propose a Flatness Index Score (FIS) to assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters and a Prototypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively represent the degree of flat minima and evaluate the model generalization ability on the target domain. We have conducted extensive experiments on various DAOD benchmarks and approaches, and the experimental results show that the proposed DAS correlates well with the performance of DAOD models and can be used as an effective tool for model selection after training. The code will be released at https://github.com/HenryYu23/DAS.

Towards Unsupervised Model Selection for Domain Adaptive Object Detection
Hengfu Yu"
Jinhong Deng"
Wen Lit
Lixin Duan
University of Electronic Science and Technology of China
hfyu@std. uestc. edu.
cn, en, (jhdengvision, liwenbmu, 1xduan)@gmail.. com
Abstract
Evaluating the performance deep models new scenarios has drawn increasing attention in recent years due to the wide application of deep learning techniques in various fields. However, while it is possible to collect data from new scenarios, the annotations not always available. Existing Domain Adaptive Object Detection (DAOD) works usually report their performance by selecting the best model on the validation set even the test set of the target domain, which is highly impractical in real-world applications. selection approach for domain adaptive object detection, which is able to select almost the optimal model for the target domain without using any target labels. Our approach is based the flat minima principle, i.e., models located in the flat min- ima region in the parameter space usually exhibit excellent generalization ability. However, traditional methods require labeled data to evaluate how well model is located the flat minima region, which is unrealistic for the DAOD task. There- fore, we design Detection Adaptation Score (DAS) approach to approximately measure the flat minima without using target labels. We show via a generalization bound that the flatness be deemed model variance, while the minima depend on the domain distribution distance for the DAOD task. Accordingly, we propose a Flatness Index Score (FIS) assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters and Prototypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively represent the degree of flat minima and evaluate the model generalization ability on the target domain. We have conducted extensive experiments on various DAOD benchmarks and approaches, and the experimental results show that the proposed DAS correlates well with the performance of DAOD models and can be used as an effective tool for model selection after training. The code will be released at httpttt://// // //gi github.......... com wtt./oom//e./eeee Humm/yyyyyy2/DA
this paper, we propose a novel unsupervised model
Introduction
With the explosion of deep neural networks 22 56
object detection 24
47
5 has
achieved promising results and shown great potential in many downstream tasks such as autonomous
driving [2] (53), video understanding [55]6], 6), robot navigation 41], etc. However, the well-trained object detection models frequently face previously unseen domains in real-world scenarios and often suffer from dramatic performance degradation when being deployed in a novel domain [[]. This is because the training set (i.e,, source domain) and the test set (i.e., target domain) have distinct domain distributions. To address this problem, Domain Adaptive Object Detection (DAOD) 50 14 38
Equual contribution.
I The corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

images
annotations
last checkpoint
mWP
41.98
Bols
41.98
(+0.00)
predictions
Detection Model
1.00
45.02
(+3,04)
c.a0
47.83
DAS (Ouns)
(+5.85)
Prototypical Distance Ratio
0.50
Flatness Index Score
DAS
20
(a)
(b)
Figure 1: (a) The performance of classic DAOD method AT [38 on Real-to-Art (P2C) adaptation task during training. It suffers from performance degradation as the training goes on. The proposed
DAS outperforms previous unsupervised model evaluation methods and selects desirable checkpoints without accessing any labels in the target domain. (b) The motivation of the work. We propose a Detection Adaptation Score including Prototypical Distance Ratio (PDR) score and Flatness Index Score (FIS) to evaluate the model performance in an unsupervised way. It can be a good substitute metric for using annotations for DAOD model evaluation.
has been proposed to transfer the knowledge from the labeled source domain to an unlabeled target domain by leveraging adversarial training or pseudo-labeled approaches.
Although effective, these DAOD methods [I 50
38 4 evaluate the detector performance and conduct model selection relying on the labeled target data, which is usually unavailable and impractical for woal-world domain adaptation scenarios. Due to the natural complexity of object detection, DAOD methods that leverage adversarial training and self-training techniques are often unstable and prone to overfitting to the target domains. As shown in Fig. | (a), DAOD methods usually suffer from a performance drop during training (marked purple in Fig. (a)). These issues limit the application of the DAOD model in real-world scenarios. Therefore, it is urgent and desirable to develop an unsupervised model selection method for DAOD, as shown in Fig.
(b).
Regarding the unsupervised model selection, several seminal works [22] 59, 60 attempt to evaluate the models from different aspects. For example, TS [59] examines the spatial uniformity of the unsupervised domain adaptive classifier, as well as the transfcrability and discriminability of deep representation. ATC [22 learns threshold on the confidence of the model and predicts accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. In object detection, however, the evaluation involves not only classification but also precise localization of objects within an image. This crucial distinction renders these methods ineffective in fully assessing an object detection model. A recent work (60) proposes BoS metric to evaluate the generalization of the detection model by measuring the stability of the box under feature dropout. However, it does not consider the domain discrepancy between the source and the target domain, making the metric unreliable for DAOD model selection (see Fig. (a) and our experiments in Sec. (.2))
In this paper, we propose novel Detection Adaptation Score (DAS) to evaluate the DAOD models without accessing any target labels, enabling select almost the optimal model for the target domain in unsupervised way. The proposed DAS is based on the flat minima principles, i.e., models that are located in the flat minima region in the parameter space exhibit better generalization ability than that in sharp minima region [1 7. However, the traditional flat minima search method requires labeled data to evaluate how well a model is located in the flat minima region, which is unrealistic for DAOD tasks. Therefore, we investigate how measure the flat minima approximately without using target labels. We derive a generalization error bound that shows the flatness can be deemed as model variance, while the minima depend on the domain distribution distance for the DAOD tasks. Therefore, we first propose Flatness Index Score (FIS) to assess the flatness by observing the classification and localization fluctuation before and after perturbations of model parameters. Then, we propose Prototypical Distance Ratio (PDR) score to seek the minima models by measuring the transferability and discriminability of the models in the target domain. To this end, the proposed DAS can effectively represent the degree of flat minima for DAOD models and evaluate the model performance on the target domain. To evaluate the effectiveness of the proposed DAS, we conduct extensive experiments on public DAOD benchmarks, including weather adaptation, real-to-art, and synthetic-to-reat adaptation. The experimental results show that the proposed metric can effectively evaluate the performance of DAOD models without annotating the target domain.

---

The contributions of our work can be summarized as follows:
With the pressing need for the application of DAOD real-world scenarios, to our best knowledge, are the first to evaluate the DAOD models without using target labels.
We propose novel Detection Adaptation Score (DAS) by seeking the flat minima without using any target labels to evaluate the performance of DAOD models on the target domain. A Flatness Index Score (FIS) and Prototypical Distance Ratio (PDR) score are proposed to meet the requirements of flatness and minima, respectively.
We have conducted extensive experiments on several DAOD benchmarks and approaches. Our DAS benefits from selecting the optimal checkpoints of model parameters to avoid negative transfer or pseudo-label error accumulation. The experimental results validate the effectiveness of the proposed DAS.
2 Related Work
is fundamental task in computer vision that involves identifying and locating multiple objects within an image or video. The object detection approaches can be roughly divided into two categories: one-stage and two-stage. The one-stage
Object Detection. Object detection [23 47 54 46 39
methods [54] 46 39 [9] directly estimate the categories and the location of the objects, such as FCOS [54], CenterNet (191, and YOLO series [46]. The two-stage methods [3 47 first generate region proposals for objects and then classify the category and regress the bounding box coordinates based on the proposal features, e.g., Faster RCNN [47) and Cascade RCNN 33. Recently, end- to-end object detection model based on Transformer (i.e., DEtection TRansformer, DETR) [5] has been proposed eliminate the complex anchor generation and post-processing operations such as non-maximum suppression (NMS). Many successive works [67 [3] further improve the training efficiency and accuracy of DETR. With the strong representation of deep neural networks, the object detection model has achieved promising results in many object detection benchmarks. However, these models usually suffer from performance degradation because of the domain discrepancy between the training and testing domains.
Domain Aappiie Object Detection. Domain Adaptive Object Detection (DAOD) 50
4 38 15
8,66 16 30 655 aims to transfer knowledge from the labeled source domain to an unlabeled target domain. Previous works be roughly categorized into two aspects: domain alignment and self- training. The domain alignment [1] 50 & 66) directly minimizes the domain discrepancy between the source and target domains. They minimize the feature distribution mismatch via adversarial
training [11 50 66 16),
prototype alignment (58), graph matching 335 36 37]. etc. The self-training
approaches 14
15 10 follow Mean Teacher (MT) framework and generate pseudo-labels
from the teacher network to supervise the training of the student network. UMT 14) leverages the style transfer algorithm to climinate the MT model bias towards the source domain. Adaptive Teacher (AT) (38) combines adversarial training and self-training to improve the accuracy on the target domain. CMT 4 introduces contrastive learning to improve the instance feature representation. HT [15 reveals that the consistency between classification and localization is crucial for pseudo-label generation and proposes reweight strategy based on the harmony measure between the classification and localization. While effective, these approaches [I] 50 4
35 36 evaluate the model performance relying on labeled target data, which is not always available in real-world scenarios. Therefore, we propose Detection Adaptation Score (DAS) to evaluate the model performance in an unsupervised manner, enabling the application of DAOD models in real-world scenarios.
38
15
16
Unsupervised Model Selection (UMS) for UDA. Unsupervised Model Selection for UDA evaluates model performance in the target domain without involving annotations. Some previous works 22 45
44] (2.22) ( 26 55) seek to predict model performance in OOD scenarios. PS (28), ATC [22) leverage
the prediction confidence, and 49 48 estimate from the perspective of entropy. DEV 61 estimates
and decreases the target risk by embedding adapted feature representation while validation. TS [59] examines the spatial uniformity of the classifier, as well as the transferability and discriminability of deep representation. However, they mainly focus on classification tasks. For object detection, the BoS (60) estimates the detection performance via the stability of bounding boxes with feature dropout but does not consider the domain discrepancy, which is vital for DAOD methods. To this end, we introduce the Detection Adaptation Score (DAS), which consists of a Flatness Index Score (FIS) and Prototypical Distance Ratio (PDR) score by seeking flat minima model in the target domain.
3

---

3 Method
In the DAOD task, we are given labeled source domain, which includes images annotated with bounding boxes and class labels, and an unlabeled target domain containing only unlabeled images. Let us denote D.- = (() N. drawn from data distribution P. as the labeled source domain and D. {-f] ز= drawn from data distribution P. the unlabeled target domain. Distributions P. and Pj are related but different domains, i.e., (Ps A Pt). In other words, they have distinct domain shifts. And y = {bb )=)) where by E R4 and ci E {I,...,, K} are the bounding box and corresponding category for each object, and my is the total number of objects in an image af. The goal of the DAOD approach to learn an object detection model that performs well on the target domain from both the labeled source and unlabeled target domains. In this work, we consider the model selection for DAOD approaches. In particular, there are M models F = { } from different epochs or iterations. Our goal is to propose proper metric for model evaluation in unsupervised manner that can reflect the detection performance (i.e., mAP) of the DAOD models.
In the following, we first introduce the domain adaptation generalization bound with flat minima in Sec. 3.J] Then, we will introduce our Detection Adaptation Score (DAS) in detail, which consists of a Flatness Index Score (FIS) in Sec. 3.2 and Prototypical Distance Ratio (PDR) score in Sec. 3.3
3.1 Domain Adaptation Generalization Bound with Flat Minima
In this paper, we propose a novel unsupervised model selection approach for the DAOD task, which can almost select the optimal checkpoint for the target domain. Our approach is based on the assumption that flat minima exhibit better model generalization than sharp minima, which has been evidenced by many literatures (7. The model parameters at flat minima will have smaller changes of loss values within its neighborhoods than sharp minima. To find flat minima, traditional methods require labeled data, which is unrealistic for DAOD, As the target labels are unavailable for model evaluation in DAOD scenarios, we cannot directly find the flat minima. To this end, we derive a generalization error bound as follows:
Theorem 1. Given any 8 2 O, exist hypothesis h E H where H is the hypothesis set, O2 denotes the parameters of h. Given any hypothesis h E (h'W E H. 10 Oall2 S T) which is located in the neighborhood of h with radius T > O, the following generalization error bound holds with at least probability of 6,
(1) where dis(S, T) is the distribution mismatch between the source domain S and target domain T. (h) is the risk of on the source domain. Si is a constant term. Proof is provided the appendix.
5 S - ET(h)) + Es(h)) - (S +)
The generalization bound shows that in addition to the constant term Ω the risk for the target hypothesis ht' around by h with b-ball radius can be bounded by three terms: the flatness, i.e., the difference between the original h and neighborhood h of h, the eπrоr the source domain D., and the distribution distance between the source and target domains. Usually, the error on source Es (h) can be minimized with the labeled source samples. To this end, one minimize the first and third terms to find the flat minima.
From the analysis in Theorem I the flatness can be deemed as model variance, while the minima depends on the domain distribution distance for DAOD task. Accordingly, we propose a Flatness Index Score (FIS) to assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters, and a Protorypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively represent the degree of flat minima and evaluate the model generalization ability on the target domain.
3.2 Flatness Index Score
In this subsection, we introduce the Flatness Index Score (FIS), which assesses the flatness of the model parameters by measuring the variance in outputs before and after parameter perturbations. For object detection, we calculate the variance both classification and localization predictions. For the property of minima on the target domain, we introduce it later in Sec. 3.3

---

Specifically, let y denote the radius of the parameter space. We can then obtain the neighbor model
f(; by adding a perturbation Á the original parameter 6, i.e., H' < 0 + A. We control y of the perturbation as a constant (i.e.. |||| 7), thus the neighbor model [(; 0*) lies on fixed radius
sphere of the original model f):
We measure the prediction correspondence between the original and neighbor models. Given an input target domain image Ii, the original model predicts {(b,, py) as the neighbor model
predicts {(b,, Dj)] j where by is the bounding box and Py is the probability vector of the j-th instance. We use df (f; 8)) to represent the matching cost of the j-th and j'-th object in two model's predictions
image Ię- As the model predictions has results from both regression and classification branches, the matching cost djy contains the divergence of bounding boxes and classification probability vectors follows:
dz = (L P,-) ToU byy)
where IoU is the intersection over union of two boxes, KL the KL-divergence over two probability vectors. The smaller the djy is, the better the two object predictions match. Then the Flatness Index Score (FIS) on the original model parameter 0 with radius 7 defined as:
n!
FIS = -Eg- -Eg--04 -0+
RR [ming n
where n²" min n1} is the smaller number of the predicted objects from the original and neighbor models. (j) is the j-th object's one-to-one corresponding index leading to the minimized matching
cost when n4 I n/. otherwise FIS = -8 ABee [ming Ej dotnyyl. Hungarian Algorithm is applied to select the best-matched pairs.
3.3 Prototypical Distance Ratio
To facilitate the search for flat minima, we explore methods identify minima regions in the target domain. In DAOD, the model with better transferability and discriminability would perform well on the target domain. There are many methods evaluate the domain distance between the source and target domains, for example, Maximum Mean Discrepancy (MMD) (25) and Proxy A-distance (PAD) (.. However, simply measuring the image feature distribution is not feasible to reflect the transferability of the detection model. Meanwhile, the traditional discriminability metrics such as entropy 48), and mutual information 333 also fail effectively correlate the model performance. In this paper, we consider the class prototype distance of instances in images across domains to evaluate the transferability and discriminability of the DAOD models. The class prototype of instances is the center of specific class and aggregates the instance information from the samples. In unsupervised domain adaptation, prototype-base domain alignment comprehensively studied in the literature [58, 57 4 and attempts to narrow the distance between prototypes of the same categories of two domains, showcasing the effectiveness of prototype alignment.
Now, we show how to leverage prototype distance to effectively evaluate the transferability and discriminability of the DAOD models. Our prototype is calculated based on the instance feature, e.g., the proposal feature in Faster RCNN. In particular, we denote the instance feature as Fij E Idd for the j-th instance in the i-th image and the final classification probability vector as Pij. The prototype of the k-th class Ps Idd for the target domain can be caleulated softly as follows:
P F6
Ps = E EE,,D,
P.
1=1
where nf is the number of proposals in the i-th image. The source prototypes can be obtained similarly. Finally, we have the source and target domain prototypes ps E kR xd and pT E Kxd
An ideal DAOD network aligns the features of the same category between domains while increasing the feature distances between different categories. We propose Prototypical Distance Ratio (PDR) score to evaluate this ability with the category-win wise prototypes. The prototype distance of the same
5

---

category across domains can be defined as follows:
dintra traco(M((S (P)))
(5)
K
where M(Ps, PT) E RRXK is the category-wise distance matrix between ps and pT Denote Ma (P, P') as the distance between Ps and Pic.
Similarly, we could obtain the inter-category prototype distances as follows:
dinter = =8(PS )²)) (0()))(pp)_))))) PS) 5(PT,PT))
(6)
where P(P,P")= Kーー Eisk' (P. is function to calculate distance among different categories. Combine the intra-category distance and inter-category distance into our Prototypical Distance Ratio (PDR) score as follows:
PDR diuter duuter/dintras
The proposed PDR score can be an effective metric to evaluate the transferability and discriminability of the DAOD models. The higher PDR score indicates that the instance features of detection models have better properties with large inter-category distances and small intra-category distances.
3.4 Detection Adaptation Score
For different checkpoints in training session, simply normalize the FIS and PDR separately by min-max normalization. The DAS a combination of FIS and PDR as follows.
DAS, FTS, + APDR, where is the index of the checkpoint. FIS, and PDR, are the t-th FIS and PDR score normalized among checkpoints. À is trade-off parameter that controls the contribution of PDR, for DAS,
4 Experiments
4.1 Experiment Setup
Benchmarks. We follow previous works [0 38 and evaluate the effectiveness of the proposed DAS on the following adaptation scenarios:
Real-to-Art Adaptation (P2C): In this scenario, we test our proposed method with domain shift between the real image domain and the artistic image domain. Following [50], choose the PASCAL VOC 2007/2012 and Clipart lk as the source and target domains, respectively. PASCAL VOC [20] widely used benchmark the object detection community. We use the 2007 and 2012 versions of PASCAL VOC that contain about 15k images with instance annotations for 20 object classes. Clipart1k (31 contains 500 train images and 500 testing images in clipart style, annotated with bounding boxes for the same 20 object categories as in the PASCAL. VOC.
Weather Adaptation (C2F): In real-world scenarios, such autonomous driving systems, object detectors may encounter various weather conditions. We study the adaptation from normal to foggy weather. In particular, source and target domains, respectively. Cityscapes 13 contains a diverse set of urban street scenes captured from 50 cities and 2, 975 training images and 500 validation images, annotated for object classes. Foggy Cityscapes (51] is a variant of the Cityscapes dataset where synthetic fog is added to the images to simulate adverse weather conditions. The annotations remain consistent with the original Cityscapes dataset. Note that we choose the worst foggy level (i.e., 0.02) of Foggy Cityscapes in the experiment following (Л
the Cityscapes and Foggy Cityscapes the
Synthetic-to-Rest Adaptation (S2C): Synthetic images offer an alternative solution for addressing the data collection and annotation issues. However, there is distinct distribution mismatch between synthetic images to real images. To adapt the synthetic to the real scenes, we utilize Sim 10k as the source domain and Cityscapes as the target domain. Sim 10k (32 consists of 10,000 synthetic images generated from simulation environment, with annotations for car bounding boxes. Because only the car is annotated in both domains, report the AP of "car" in the validation set of Cityscapes.

---

Table 1: The detection performance (mAP in %) comparison between the last checkpoints, the checkpoints selected by DAS, and the oracle checkpoints.
Benchmark
Real-to-Art Art Adaptation
Weather Adaptation Last Ours Imp.f Oracle
Synthetic-to-Rer Adaptation
DAOD
Last Ours Imp.t Oracle
Last Ours Imp.t
Oracle
DAF
15.72 17.16
17.49
28.27 29.01 +0.74 29.17
43.05 +2.04
45.09
MT
34,25 35.27 +1.02 35.75
45.74 +0.00 47.51
54.85 55.41 +0.56
55.41
AT
41,98 47.83 +5.85 47.83
48.16 49.26 +1.10 49.26
28.35 47.44 +19.09
47.44
CMT
40.11 40.11 +0.00 40.82
48.46 49.15 +0.69 49.36
42.35 48.81 +6,46
55,53
Table 2: Comparison of different unsupervised model evaluation methods on real-to-art adaptation (P2C), weather adaptation (C2F), and synthetic-to-real adaptation (S2C).
Real-to-Art
Weather Adaptation MT
Synthetic-to-Raal
Benchmark
Adaptation AT
Adaptation MT
Average
Methods
DAF MT'
CMT
DAF
AT
CMT
DAF
AT
CMT
Last Ckpt.
15.72 34,25 41.98
40.11
28.27
45.74
48.16
48.46
43.05
54.85 28.35 42.35
39.27
PS
15.72 34.25 43.93
39.93
28.27
46.95 48.23
48.46
42.68
55.29 32.56 47.18
40.29 (+1.02)
ES
17.49 34.25 43.93
39.93
26.77 46.95 47.98
48.86
42.68
54.85 30.57 42.99
39.77 (40.50)
ATC (th=.3)
15.72 34.25
43.93
39.93
28.27
46.95
48.23
48.46
42.68
55.29 32.56 47.18
40.29 (+1.02)
ATC (th=.4)
15.72 34.25
41.98
40.11
28.27
46.95
48.23
48.46
42.68
55.29 32.56 47.18
40.14 (+0.87)
ATC (th=.95)
17.32 34.25
43.93
39,93
29.01
46.95
48.23
48.46
42.68
54.85
32.56
47.18
40.45
(+118)
FD
13.50 31.22
45.18
40.11
28.27 45.74
48.69
40.48
43.05
55.41 47.44
42.35
40.12 (40.85)
14.17 33.62
45.02
40.03
28.27
46.95
48.69
49.36
42.56
54.76
30.57
47.18
40.10 (+0.83)
BoS
13.83 34.25
43.93
40.11
19.07
42.21
44.91
43.10
44.18
55.29
31.59
54.53
38.92 (-0.35)
DAS (ours)
17.16
35.27
47.83
40.11
29.01
45.74
49.26
49.15
45.09
55.41
47.44
48.81
42.52 (+3.25)
Oracle
17.49
35.75
47.83
40.82
29.17
47.51
49.26
49.36
45.09
55.41
47.44
55.53
43.39
DAOD Frameworks. In this work, we test our method on four classical DAOD models in the two main DAOD streams (i.e., adversarial training and self-training): DA Faster RCNN (DAF) [T]. Mcan- Teacher (MT) [52]. Adaptive Teacher (AT) (8). Contrastive Mcan-Teacher (CMT) 4. DAF [I]] typical DAOD framework extending the Faster RCNN architecture by incorporating domain adaptation techniques. It employs adversarial training to align the feature distribution across domains both image and instance levels. MT [52) leverages a teacher-student paradigm to generate pseudo labels to provide extra supervision for the student model on the target domain. The teacher model an exponential moving average (EMA) of the student model and thus can provide more accurate pseudo-labels for the student model. This approach helps in leveraging unlabeled data by enforcing consistency between the predictions of the teacher and student models. AT 38 improves upon the Mcan-Teacher framework by employing an adversarial learning module to align the feature distributions across two domains, reducing domain bias and improving pscudo-label quality. CMT [4] enhances the Mean-Teacher framework with contrastive learning techniques by encouraging similar instances to be closer in the feature space while pushing the features of dissimilar instances apart.
Unsupervised Model Selection Baselines. Existing unsupervised model selection methods for UDA tasks are mainly based on classification tasks, such as Prediction Score (PS) 228, Entropy Score (ES) 48 Average Threshold Confidence (ATC) [22]. Transfer Score (TS) (59 We reproduce these approaches on the classification branch of object detection models. Besides, we also use the Fréchet Distance (FD) 177 to measure the domain distance on backbone features as a compared method. Bounding Box Stability (BoS) is introduced to tackle unsupervised model evaluation problems specifically for object detection networks. It evaluates the model generalization of the detection model by probing the bounding box stability under feature dropout.
Implement Details. Following previous works [1]. we choose one of the representative detection frameworks Faster RCNN as our base detector. We followed the instructions from the released code of the DAOD methods and reproduced their results. The hyperparameters, learning rates, and optimizers are set according to the default configurations provided in the original papers. We set our hyperparameters À and y = 1 in all experiments. which perform well on all the benchmarks. Our implementation is built upon the Detectron2 detection framework. We have added detailed implementation in the appendix.
7

---

47.5
00
45.0
45.0
0.25
(1hmm
mAP
45.0
42.5
-0.05
42.5
42.5
0.20
-0.90
PCC: 0.49
PCC: 0.05
PCC: 0.49
37.5
37.5
37.5
10
15
20
10
15
20
10
15
20
Checkpoint nnex
Checkpoint Index
Checkpoint Index
(a)
(b)
(c)
47.5
45.0
1.04
45.0
0.83
45.0
1.50
DXS
42.5
42.5
1.02
0.82
1.00
40.0
PCG 0.38
PCC - 34
PCC: 0.86
37.5
1.00
37.5
0.81
37.5
10
10
15
20
30
3.5
20
Checkpaint (d)
Checkpoint Index
Checkpoint
(e)
Figure 2: The comparison of different unsupervised model evaluation methods for DAOD. The experiments are conducted on real-to-art adaptation (P2C) using AT. Note that the directions of all scores are unified.
ALunsip-0.
A
1.5
Aussip-1.0
unsup=1.4
Average
1.0
dis00.0
A
A.dom0.1
0.5
Add6s0112 125
0.5
47.00
47.25 50 47.75
48.25
48.50 48.75
46.5
47.0
47
48.0
48.5
mAP
mAP
(a)
(b)
Figure 3: Hyperprameter tuning on AT [38 using our DAS. (a) Adis that controls the weight of adversarial loss from domain discriminator, (b)Aunsup that controls the weight of unsupervised loss.
4.2 Main Results
Checkpoint Selection after Training. Checkpoint selection after training is pivotal for the application of DAOD approaches in real-world scenarios since the DAOD models usually suffer from negative transfer and overfitting the target domain during training. For example, on the real-to-art adaptation (P2C), AT 38] drops 5.85% from the highest mAP (47.83%) to the last checkpoint (41.9.). The detailed results on checkpoint selection, with the highest DAS, after training are listed in Table [ Compared with the last checkpoint, the proposed method works well on almost all the DAOD methods. For example, the proposed method for AT (38] achieves 5.85%, 1.1%, and 19.09% improvements terms of mAP on real-to-art, weather, and synthetic-to-real adaptations, respectively. These experimental results demonstrate that the proposed method can choose reliable checkpoint and thus avoid the negative transfer and overfitting on the target domain during training.
Morcover, we compare our method with the recent unsupervised model evaluation methods in Table 2 showing that our method consistently outperforms all the compared baselines. For instance, our methods outperform recent works TS [59] and BoS [60] by 2.42% and 3.60% mAP on average, respectively. These results demonstrate the effectiveness of the proposed method choosing the optimal checkpoints. We also present the correlation between unsupervised model evaluation metrics and the ground truth mAP (i.e., using the annotations in the target domain to evaluate the model) in Fig. 2 It is clearly shown that the previous methods give higher scores as the training goes on and fail to correlate with the performance DAOD checkpoints. contrast, the proposed DAS score correlates well with the performance of DAOD checkpoints (i.e. 0.86 PCC).

---

Table 3: The comparison of different methods for hyperparameter tuning on Weather Adaptation.
Hyperparam.
PS
ES
ATC th..4
FD
TS
BoS
DAS (ours)
mAP
47.99
47.99
47.99
48.69
47.99
47.99
48.69
Adis
PCC
-0.346
-0.399
0.348
0,631
0.007
0.024
0,865
mAP
47.56
47.87
47.56
48.69
48.57
48.09
48.69
Aunsup
PCC
0.273
0.453
0.298
0.751
0.557
0.659
0.938
Table 4: Ablation study of the proposed DAS Table 5: The hyperparameter sensitivity of the method. The results averaged from DAOD proposed method on real-to-art adaptation. benchmarks and approaches.
mAP
PCC
PDR
FIS
mAP
PCC
Last Checkpoint
33.02
Last Checkpoint
39.27
0.1
34.76
0.748
0.5
35.08
0.842
41.74
0.48
Ours
1.0
35.09
0.854
Ours
42.14
0.64
2.0
35.05
0.821
42.52
0.67
10.0
35.05
0,729
Hyperparameter Tuning for DAOD Methods. Domain adaptation methods can be highly sensitive to the hyperparameters. Inappropriate hyperparameter selection would limit the transfer performance and lead to negative transfer, which even leads models perform below the source-only ones. Therefore, the validation of hyperparameters is important problem in DAOD, yet researchers have unfortu- nately overlooked it. We evaluate our DAS in hyperparameters tuning by leveraging the weights of adversarial loss (denoted Adis) for the domain discriminator and of unsupervised loss (denoted as Aunsup) for self-training in AT [88. For a model with a specific hyperparameter setting, we obtain the last three checkpoints of its training process and calculate their average performance to represent the model. The final DAS of the specific model is defined as the average DAS of these obtained checkpoints. The experimental results are conducted on the weather adaptation and summarized in Fig. 3 From Fig. 3 we can observe that the model with higher DAS presents better improvement after adaptation. The proposed DAS can correlate the detection performance without any labels on the target domain. We further compare our DAS with the unsupervised model evaluation methods and summarize the results in Table 3 The proposed DAS consistently outperforms the previous baselines, which demonstrates the effectiveness of the proposed method in hyperparameter tuning. In a nutshell, our DAS can be a good indicator to guide the hyperparameter tuning for DAOD methods, thus avoiding the negative model optimization due to the inappropriate hyperparameter selection.
4.3 Further Analysis
Ablation Study. We conduct the ablation study of the proposed method by isolating DAS into separate metrics. In particular, we conduct experiments on three benchmarks and use four DAOD methods including DAF [1, MT [52], AT (38), and CMT 4 We summarize the results in Table 4 The experimental results indicate that the proposed PDR score and FIS evaluate the model performance without labels effectively. In particular, the PDR score and FIS select checkpoints with mAp 42.14% and 41.74%. PCC 0.64 and 0.48, respectively. DAS combines them and further improves their performance to 42.52% mAP and 0.67 PCC, demonstrating the synergy effect among them.
Hyperparameter Sensitivity. We investigate the sensitivity of the hyperparameter À controlling the weight of PDR for DAS on real-to-art adaptation (P2C). The results are summarized in Table A = 1.0 achieves the best average results. From a wide range of X, DAS has relatively stable results.
5 Conclusion
In this work, we propose a novel metric named Detection Adaptation Score (DAS) by seeking flat minima to evaluate the domain adaptive object detection models without involving any target labels. The proposed DAS consists of a Flatness Index Score (FIS) and a Prototypical Distance Ratio (PDR) score and can find flat minima of DAOD models in unsupervised way. Extensive experiments have been conducted on public DAOD benchmarks for several classical DAOD methods. Experimental results indicate that the proposed DAS correlates well with the performance of the detection model and thus can be used for checkpoint selection after training. The proposed method fosters the application of DAOD methods in the real-world scenario. We hope that our work will inspire researchers and contribute to advancing research in DAOD.

---

Acknowledgments and Disclosure of Funding
This work supported by the National Natural Science Foundation of China (No. 62176047), the Sichuan Science and Technology Program (No. 2022YFS0600). Sichuan Natural Science Foundation (No. 2024NSFTD004),, and the Fundamental Research Funds for the Central Universities Grant (No. ZYGX202IYGLH2008,
References
[1] Shai Ben-David, John Blitzer. Koby Crammer, Alex Kulesza. Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains, Machine learning, 79:151-175, 2010.
[2] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint lrXiv:1604.07311 2016.
[3] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, pages 1154-162,, 2018.
[4] Shengcao Cao, Dhiraj Joshi, Liang- Yan Gui, and Xuonio Wang. Contrastive mean teacher for domain adaptive object detectors. In CVPR, pages
2023.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229. Springer, 2020.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.
[7] Junbum Cha. Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. NeurlPS, 34:22405-22418, 2021.
[8] Chaoqi Chen, Zehiao Zheng, Xinghao Ding, Yue Huang, and Qi Dou. Harmonizing transferability and discriminabiliiy for adapting object detectors. In CVPR, pages 8869-8878, 2020.
[9] Jiefeng Chen, Frederick Liu, Besim Avei, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self- self-training ensembles. Advances in Neural Information Processing Systems, 34: 101114990199999 2021.
[10] Meilin Chen, Weijie Chen, Shicai Yang. Song, Xinchao Wang, Lei Zhang, Yunfeng Yan, Donglian Qi. Yueting Zhuang, Di Xie, et al. Learning domain adaptive object detection with probabilistic teacher. ICML, 2022.
[11] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In CVPR, pages 3339-33448, 2018.
[12] Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional networks for text classification. arXiv preprint 16060:1606.01781, 2016.
[13] Marius Cordts, Mohamed Omran. Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban understanding. In CVPR, pages 3213-3223. 2016.
[14] Jinhong Deng, Wen Li, Yuhua Chen, and Lixin Duan. Unbiased mean teacher for cross-domain object detection. CVPR, pages 4091-4101, 2021.
[15] Jinhong Deng, Dongli Xu, Wen Li, and Lixin Duan. Harmonious teacher for cross-domain object detection. In CVPR, pages 23829-23838, 2023.
[16] Jinhong Deng, Xiaoyue Zhang, Wen Li, Lixin Duan, and Dong Xu. Cross-domain detection transformer based on spatial awwre and semantic-awee aware token alignment. TMM, 2023.
[17] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In CVPR, pages 15069-15078, 2021.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2020.
10

---

[19] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi. Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In ICCV, pages 6369-6566 6578, 2019.
[20] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. IJCV, 88(2)303-3333 2010.
[21] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur, Sharpness-awand aware minimization for efficiently improving generalization. arXiv preprint arXtv:2010.01442,, 2020.
[22] Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanic Sedghi. Lever- aging unlabeled data to predict outtof-distributiinn performance. ICLR, 2022.
[23] Ross Girshick. Fast r-cna. In ICCV, pages 1440-1448, 2015.
[24] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Region- based convolutional networks for accurate object detection and segmentation. 7-PAMI, 38(1):142 158, 2015.
[25] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Berhhard Schölkopf, and Alex Smola. A kemel method for the ceessmmll-probeene Advances in neural information processing systems, 19, 2006.
[26] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEF/CVE international conference on computer vision, pages 1144,114,,2 2021.
[27] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770 1700-777777 2016.
[28] Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ICLR, 2016.
[29] Lukas Hoyer, Dengxin Dai, and Lue Van Gool. Daformer: strategies for domain-adaplis semantic segmentation. In CVPR, 2022.
Improving network architectures and training
[30] J. Huang, Y. Lu. S. Y. Lin, Y. Xie, and Y. Y. Lin. Aqt: Adversarial query transformers for domain adaptive object detection, In IJCAI, pages 972-979, 2022.
[31] Naoto Inoue. Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly- supervised object detection through progressive domain adaptation. In CVPR, pages 5001-5009, 2018.
[32] Matthew Johnson-Roherson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? ICRA, pages 746-753. IEFF, 2017.
[33] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 6966)066133, 2004.
[34] Shuang Li, Fangrui Lv. Binhui Xie, Chi Harold Liu, Jian Liang, and Chen Qin. Bi-classifier determinacy maximization for unsupervised domain adaptation. In AAAI, volume 35, pages 8455- 8464, 2021.
[35] Wuyang Li, Xinyu Liu, Xiwen Yao, and Yixuan Yuan. Scan: Cross domain object detection with semantic conditioned adaptation. In AAAI, 2022.
[36] Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma: Semantic-complete graph matching for domain adaptive object detection. In CVPR, pages 5291-5300, 2022.
[37] Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. T-PAMI, 2023.
[38] Yu-Jhe Li, Xiaoliang Dai, Chih- Yao Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He, Kris Kitani, and Peter Vajda. Cross-domain adaptive teacher for object detection. In CVPR, pages 7581-7590, 2022.
[39] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yane Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, pages 21-37. Springer, 2016.
[40] Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepputtis, Joseph Campbell, and Katia Sycara. Characterizing out-of-distriutiion error via optimal transport. NeurlPS, 17:22 2023.
[41] Raul Artalr Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. transactions robotics. 31(5):1147-1133 2015.
11

---

[42] David Nistér, Oleg Naroditsky, and James Bergen. Visual odometry. In CVPR. IEEE, 2004.
[43] Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong- Wah Ngo, and Tao Mei. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2239-2247, 2019.
[44] Ru Peng, Qiuyang Duan, Haobo Wang, Jiachen Ma. Yanbo Jiang, Yongjun Tu, Xiu Jiang, and Junbo Zhao. Came: Contrastive automated model evaluation. In ICCV, pages 20121-20132, 2023.
[45] Ru Peng, Heming Zou, Haobo Wang, Yawen Zeng, Zenan Huang, and Junbo Zhao. Energy-based automated model evaluation. arXiv preprint urXiv:2401.12689 2024.
[46] Joseph Redmon. Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR. pages 779-788, 2016.
[47] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn; Towards real-time time object detection with region proposal networks. NewrIPS, 28, 2015.
[48] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In CVPR, pages 8050-80555 8058, 2019.
[49] Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In ICCV, pages 9184 9193, 2021.
[50] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong- weang distribution alignment for adaptive object detection. In CVPR, pages 6956-69660 2019.
[51] Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Semantic foggy scene understanding with synthetic data. IJC,, 126(9):933-992, 2018.
[52] Antti Tarvainen and Harri Valpola. Mcan teachers are hetter role models; Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS, 30, 2017.
[53] Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, and Raquel Urtasun. Multinet: Real-time joint semantic reasoning for autonomous driving. In /V. pages 1013- -020. IEEE, 2018
[54] Z Tian, C Shen, H Chen, and T He. Fcos: Fully convolutional one-stage object detection. arXiv preprint arXiv:1904. 1904.0135, 2019.
[55] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4492 2015.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 30, 2017.
[57] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsuper- vised domain adaptation. In International conference on machine learning, pages 5423-5432. PMLR, 2018.
[58) Minghao Xu, Hang Wang, Binghing Ni, Qi Tian, and Wenjun Zhang. Cross-domain detection via graph-induced prototype alignment. In CVPR, pages 12355-12364 2020.
[59] Jianfei Yang, Hanjie Qian, Yuecong Xu, and Lihua Xie. Can evaluate domain adaptation models without target-domain labels? a metric for unsupervised evaluation domain adaptation. ICLR, 2024.
[60] Yang Yang. Wenhai Wang. Zhe Chen, Jifeng Dai, and Liang Zheng. Bounding box stability against feature dropout reflects detector generalization across environments. ICLR, 2024.
[61] Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In ICMI. pages 7124-7133. PMIR, 2019.
[62] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-o--distribution error with the projection norm. In JCML, pages 25721-25746. PMLR, 2022.
[63] Hao Zhang, Feng Li. Shilong Liu, Lei Zhang, Hang Su. Jun Zhu, Lionel M Ni, and Heung- Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR, 2023.
[64] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML, pages 7404-7413, PMLR, 2019.
12

---

[65] Wenzhang Zhou, Heng Fan, Tiejian Luo, and Libo Zhang. Unsupervised domain adaptive detection with network stabilityy analysis. In ICCV, pages 6986-6995, 2023.
[66] Xinge Zhu, Jiangmiao Pang, Ceyuan Yang. Jianping Shi, and Dahua Lin. Adapting object detectors via selective doosa- alignment. In CVPR. pages 687-696, 2019.
[67] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
13

---

A
Appendix
In this appendix, we first give the proof Theorem in Sec. A.I And then provide more implemen- tation details in Sec. A.2 and experimental results in Sec. A.3 Finally, discuss the limitations and societal impact of our work.
A.1 Proofs of Theorem
Theorem 1. Given any 8 2 exist hypothesis h H where H. is the hypothesis set, 0, denotes the parameters of h. Given any hypothesis h {h' H. 10, 0x S 7) located in the neighborhood with radius 0, the following generalization error bound holds with at least a probability of - ò,
ET(K') S (1T (h)
- ET()) Es(h) - (S +) Ω2
(9)
where dis(S, T) is the distribution mismatch between the source domain S and target domain T. (s( the risk of h on the source domain. f a constant term.
Proof. Given any 8 2 O, exist hypothesis It E H where H is a hypothesis set, On denotes the model parameters of h. Give any hypothesis h' E [h'll0>- - 02 ST). T >> is the radius of the neighborhood. The following generalization bound holds with at least probability of 8-
ET(N') = (T() Er() + (hh)
(10)
$ - -rr())))rr())++0
According to the domain adaptation theory from
34
64
we have the following inequality:
(h) S Es(h) dis(S,
(11)
where the Es(h) indicates the source enror and dis(S, T) denotes the distribution mismatch between the source domain and the target domain T. In this way, we replace the last term Er(h) in Eq. (IO) with Eq, (T). leading to the following error bound:
EThh') s (Er('') ET(h) + (h) - di T)
(12)
Till now, we prove the inequality in Theorem |
A.2 More Implementation Details
Training and Evaluating Details. In our experiment, the DAOD models are trained according to the default setting specified in their original papers and the open-source codebases. For DA Faster RCNN, we train it on the three benchmarks with a VGG-16 16 [12] backbone pre-trained on ImageNet, using batch size of 4. For Mean Teacher, Adaptive Teacher, and Contrastive Mean Teacher, we train the models with pre-trained ResNet- 101 (27 backbone on ImageNet for the real-to-art adaptation setting, and with preiteri VGG-16 16 [2] backbone for the weather adaptation and synthetic-to-ret adaptation settings. The batch size of AT, MT, and CMT are set to 8. Training is conducted on four RTX 3090 GPUs or A 100 GPUs according to the computational requirements of different DAOD methods. For all the results, we report the mean Average Precision (mAP) at the loU threshold of 0.5.
Detection Adaptation Score. For the Flatness Index Score (FIS). we perturb the detector model by adding a normalized random direction for all the model parameters including the backbone and detection head. For the Prototypical Distance Ratio (PDR) score, we use the features of the region proposals and the corresponding predicted probability (including background) from the classification branch of the detection head to aggregate the information. In our implementation, L2 distance is used as the prototype distance.
A.3 More Experimental Results
Top3 Checkpoint Selection Results after Training We summarize the top3 checkpoint selection results in Table 6 We can observe that with more loose selection condition, our method still outperforms other unsupervised model selection methods by large margin. These results further demonstrate the effectiveness of the proposed DAS in select optimal checkpoints for DAOD methods.
14

---

Last Checkporre Orace
1.79
Last Checkpolit
200
Checkpoint
Cracke
17.5
DAS (Dun)
DAS
(Dun)
4
map Offorence
12.5
1.00
2-
6.73
7.
D4F
MT
CNT DAF
HT
AT
CMT
DAF
MT
CNT
Framewores (a)
Framesorts (b)
Framesorks
(c)
Figure 4: The performance gap among the last checkpoint, our DAS, and Oracle checkpoint, (a) real-to-art adaptation, (b) the weather adaptation, and (c) the synthetic-to-real adaptation.
Performance Gap between Last Checkpoint and Oracle Model. Fig. we compare the DAS selection results between the last checkpoint and the Oracle checkpoint (i.e., using the ground-truth labels the target domain to evaluate the model performance) of various frameworks and benchmarks. It shows that there is significant performance gap between them, making it urgent to design an effective method for unsupervised model selection in DAOD scenarios. DAS by seeking the flat minima of the DAOD models. From Fig. 4 DAS shows promising unsupervised model selection results. While our method has made progress compared to others, there is still room for further improvement.
this paper, we propose can observe that the proposed
More Visual Correlation Results. We show the correlation comparison results in Fig. 5 Junder the real-to-art adaptation using MT [52]. Overall, our DAS correlates well with the detection performance and outperforms the previous method. It is worth noting that the compared method cannot figure out the overfitting issue while our method successfully reflects it.
More Unsupervised Model Selection Results. The DAS select models other DA methods and tasks. 1) We also conducted our method on SIGMA+ [37] on weather adaptation, which minimizes the domain gap by graph matching. DAS chooses checkpoint 41.7% mAP (which the oracle), while the last checkpoint reaches 39.5% mAP. 2) For prototype-based DAOD methods. The DAS includes FIS and PDR derived from the generalization bound. It can evaluate DAOD models from different aspects. Some existing works like GPA (58) use the prototype- prototye--bssed based alignment method to minimize the domain gap between domains. However, the prototype estimation in existing works only utilizes samples in a mini-batch during the model training. In contrast, our DAS uses the entire dataset estimate the prototypes, which is more robust and has better generalization ability. To verify this, we experimented on Synthetic-to-Ret Reall adaptation for GPA (58) The DAS chooses a checkpoint 45. 8% mAP (the oracle 47.0%) while the last checkpoint reaches 43.1%. Our proposed method still works when DAOD frameworks also optimize the prototype- based distance. 3) For other DA tasks. Our method can also work on other DA tasks, such as semantic segmentation. We conducted our DAS on the well-known DAFormer (29] method on GTA5 to Cityscapes Adaptation. It is shown that our DAS can choose better checkpoint the model with an average mloU of 64.2% while last checkpoint of the model only achieves 60.9% and the oracle checkpoint is at 65.9%.
A.4 Limitation
Although our method outperforms many unsupervised model evaluation methods in many DAOD benchmarks and methods, it still has considerable room for improvement. There are certain scenarios where our proposed method faces limitations, such as the inability to consistently select the best checkpoint and lack of sufficient correlation between the proposed DAS and the ground truth detection performance of DAOD models on the target domain. To address these limitations, we could consider incorporating a more fine- mne-grained distribution alignment metric evaluate the distance across domains. On the other hand, we can explore methods that extend beyond the constraints of labeled data by leveraging few-shot labeled data the target domain to help model evaluation. Another limitation is that our method requires the entire dataset of the source and target domains to calculate the evaluation score. would beneficial develop a more data-efficient approach that can effectively evaluate the performance of DAOD models while minimizing the data requirements.
15

---

Table 6: The top3 selection results of different unsupervised model evaluation methods on real-to-oa adaptation (P2C), weather adaptation (C2F), and synthetic real adaptation (S2C).
Rcal-to-Art
Weather
Synthetie-to-Ret Real
Benchmark
Adaptation
Adaptation
Adaptation MT
Average
Methods
DAF
MT
AT
CMT
DAF
MT
AT
CMT
DAF
AT
CMT
Last Ckpt.
15.72 34.25 41.98
40.11
28.27
45.74
48.16
48.46
43.05
54.85
28.35 42.35
39.27
PS
17.32 34.25 43.93
40.11
29.17
47.51
48.69
48.46
43.05
55.29
32.56 50.43
40.90
ES
17.49 34.25 43.95
40.11
29.01
46.95
48.69
49.15
44.33
55.29 32.56 44.66
40.54
ATC (th=.3
17.32 34.25 43.93
40.11
29.17
47.51
48.69
48.46
43.05
55.29
32.56
50.43
40.90
ATC (th=.4)
17.49 34.25 43.93
40.11
29.17
47.51
48.69
48.46
43.05
55.29
32.56
50.43
40.91
ATC (th=.95)
17.32
34.25
43.93
40.11
29.01
46.95
48.69
48.49
43.05
55.29
32.56
50.43
40.84
FD
14.70
33.93
45.18
40.11
29.17
47.51
48.69
48.46
45.09
55.41
47.44
44.66
41.70
17.16
34.92
45.94
40.05
29.17
47.51
48.69
49.36
44.73
55.14
31.17
50.43
41.19
BoS
16.38
34.25
43.93
40.11
22.01
44.74
49.26
48.31
45.07
55.29
32.56
55.21
40.59
DAS (ours)
17.49
35.27
47.83
40.11
29.01
47.51
49.26
49.15
45.09
55.41
47.44
50.43
42.83
0.30
35
35
35
20,00
34
34
ATC (thm
34
-25.00
-0.90
0.20
PCC 943
PCC:
PCC 2222
31
31
31
5
10
15
15
10
15
Checkpeint ndex
Checkpoint ndex
Checkpeint Index
(a)
(b)
(c)
0.84
1.02
35
35
34
34
1.00
33-
33
32
0082
32
0.50
P.C:0.27
0.27
31
31
10.00
10
15
Checkasint Index
Checkpoint Index
Checkpoint
(d)
(e)
(f)
Figure 5: More comparisons on different unsupervised model evaluation methods for DAOD. The experiments are conducted on real-to-art adaptation (P2C) using MT. The direction all the scores is unified.
A.5 Societal Impact
Our Detection Adaptation Score (DAS) method has the potential to positively impact various down- stream systems, such as autonomous driving and embodied AI. These systems frequently encounter previously unseen domains in real-world scenarios, and our method effectively addresses the unsuper- vised evaluation problem of domain adaptive object detection methods. However, it important to acknowledge that the application of method may also introduce potential negative impacts. For example, when applied to surveillance videos and medical images, privacy concerns may arise. is essential to understand that these issues not inherent the technology itself but rather depend on the responsible and ethical use by human beings.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, clearly express the contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitation part discuss the limitations the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide enough implementation details for reproducing the results of the proposed method in the main paper and the appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

reproducible, they Answer: [Yes]
Justification:
The code will available on the website mentioned in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nip. http:c/cpplllggguuieeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide adequate training and test details in the main paper and supple- mentary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Due to the complex of object detection, training the model requires much computational resources and training time. We do not report error bars following previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information about the computer resources in supplementary
materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide discussion about societal impacts and negative societal impacts of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our work does not a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide details for the assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "sCDyjKG1Sh", "reviewer": "Reviewer_ood2", "summary": "This paper delves into an unsupervised evaluation problem in Domain Adaptation Object Detection (DAOD). To solve this problem, this paper proposes a method based on the flat minima principle, named Detection Adaptation Score (DAS), which can measure the flat minima without using target labels. Specifically, the proposed method is composed of a flatness index score (FIS) and a prototypical distance ratio (PDR) to assess the flatness and measure the transferability and discriminability of the models. Experiments validate the effectiveness of the proposed method.", "questions": "See the weaknesses, especially the forth question.", "limitations": "Limitations had been discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThis paper is well motivated. And this paper is the first work to evaluate DAOD without using target labels.\n-\tThe proposed method is effective and reasonable.", "weaknesses": "-\tIn line 49, the authors should explain why these methods fail to evaluate the object detection model.\n-\tIn Eq.2, since KL term and IoU term are two metrics to evaluate the matching costs of two different tasks, object classification and box regression, why there is no a balance coefficient, such as d=KL-\\lambda IoU. If \\lambda=1 is the best?\n-\tThe proposed FIS is easy coming to mind, which uses parameter perturbation to evaluate flatness minima without using target labels. And the proposed PDR to use prototype-based domain alignment is also used as a training method in the existing prototype-based domain alignment for DAOD, such as [1].\n-\tThe proposed FIS and PDR are used for unsupervised evaluation. However, from another perspective, these methods can also be used as unsupervised training methods. If these kinds of methods are used for unsupervised training, can they be used as unsupervised evaluation methods as the same time? For example, if the benchmark DAOD framework is a prototype-based domain alignment method, can PDR, a prototype-based domain alignment evaluation method, serves as an unsupervised evaluation? As I can see in line 261, this paper only uses adversarial training or self-training methods for unsupervised domain adaptive training? More different kinds of benchmark DAOD framework are necessary to evaluate the scope of the proposed evaluation methods.\n\n[1] Cross-domain detection via graph-induced prototype alignment. CVPR 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ood2 for the valuable feedback and insightful comments. We appreciate the reviewer's positive comments regarding motivation and method. We now clarify the reviewer's concerns as follows.\n\n**Q1**: Explain why these methods fail to evaluate the object detection model in line49.\n\n**A1**: The methods referenced in line 49 rely on classifier-specific properties like predicted confidence and entropy, which are tailored for classification tasks. In object detection, however, the evaluation involves not only classification but also the precise localization of objects within an image. This crucial distinction renders these methods ineffective in fully assessing an OD model. By incorporating your suggestion into our paper, we will explicitly clarify this difference between classification-centric evaluation methods and the multifaceted demands of OD evaluation. \n\n**Q2**: In Eq.2, since KL term and IoU term are two metrics to evaluate the matching costs of two different tasks, object classification and box regression, why there is no a balance coefficient, such as d=KL-\\lambda IoU. If \\lambda=1 is the best?\n\n**A2**: Thank you for regarding the potential absence of a balance coefficient in Eq. (2) to weigh the KL divergence and IoU metrics. We do not deliberately add a balance coefficient following the previous works (e.g., DETR) to combine the classification and localization cost. As suggested by the reviewer, we conducted experiments on AT weather adaptation to assess the impact of tuning the balance coefficient. The results are shown in the table below. Our findings suggest that the parameter adjustment can actually influence performance, while setting the balance coefficient to 1 is a fairly good choice, indicating a balanced consideration of classification and localization within our framework.\n\n| $\\lambda$ |0|0.1|1|10|100|\n|-|-|-|-|-|-|\n| mAP |48.2|49.3|49.3|48.7|48.5|\n\n**Q3**: The proposed FIS is easy coming to mind, which uses parameter perturbation to evaluate flatness minima without using target labels. And the proposed PDR to use prototype-based domain alignment is also used as a training method in the existing prototype-based domain alignment for DAOD, such as [1].\n\n**A3**: We would like to highlight our paper is essentially different from previous works. We address the unsupervised model selection (UMS) problem for DAOD. Previous works [1] design effective methods for addressing the domain gap for object detection, while we address the UMS problem for DAOD through a flat minima principle, i.e., models that locate the flat minima region in the parameter space usually exhibit excellent generalization ability. Different from the traditional flat minima methods that require labeled data to evaluate how well a model is located at the flat minima region, we show via the generalization bound that the flatness can be deemed as model variance, while the minima depends on the domain distribution distance for the DAOD task. To this end, although some strategies in our paper may seem to be similar to existing works, the task and motivation are distinct from the existing works.\n\n**Q4**: If these kinds of methods are used for unsupervised training, can they be used as unsupervised evaluation methods as the same time? For example, if the benchmark DAOD framework is a prototype-based domain alignment method, can PDR, a prototype-based domain alignment evaluation method, serves as an unsupervised evaluation? \n\n**A4**: To effectively evaluate the DAOD models, we designed a DAS including FIS and PDR, which is derived from a generalization bounding for flat minima. Thus, they evaluate the DAOD models in different aspects. Additionally, existing work [1] use the prototype-based alignment method to minimize the domain gap between domains. The prototype estimation in existing work only utilizes the samples in a mini-batch during the model training. In contrast, we use the entire dataset to estimate the prototypes, which is more robust and has better generalization ability. To verify this, we conduct experiment results on Sim10k to Cityscapes (S2C) for [1]. The experimental results are presented below. It is shown that the proposed method still works when DAOD frameworks also optimize the prototype-based distance, i.e., choosing a checkpoint with 44.8% mAP (our PDR) while the last checkpoint is 43.1% mAP. Moreover, our DAS with FIS and PDR can further improve the performance, demonstrating the effectiveness of the FIS. The FIS is proposed by us in this paper, and we have not found any DAOD methods using it as a training method.\n\n|Method|Setting|Last|PDR|DAS|Oracle|\n|-|-|-|-|-|-|\n|GPA|S2C|43.1|44.8(+1.7)|45.8(+2.7)|47.0|\n\n\n**Q5**: As I can see in line 261, this paper only uses adversarial training or self-training methods for unsupervised domain adaptive training? \n\n**A5**: We evaluate DAOD methods with adversarial training or self-training because they are classical paradigms and achieve promising results in DAOD scenarios. To demonstrate the generalization of the proposed DAS method to other DAOD frameworks, we evaluate our method on GPA[1] and SIGMA++[2] which minimize the domain gap by prototype-based domain alignment and graph matching, respectively. The results are shown in the below table, we can see that our method is able to select better checkpoints for those two DAOD methods (2.7% and 2.2% improvements, respectively) beyond adversarial learning and self-training paradigms. \n\n|Method|Setting|Last|Ours|Imp.$\\uparrow$|Oracle|\n|-|-|-|-|-|-|\n|GPA|S2C|43.1|45.8|2.7|47.0|\n|SIGMA++|C2F|39.5|41.7|2.2|41.7|\n\n* We attempted but could not reproduce the GPA results on C2F using the released code. This problem is found by other researchers in the issue of the repo. Thus we choose the setting of S2C.\n\n[1] Cross-domain detection via graph-induced prototype alignment. CVPR 2020.\n\n[2] SIGMA++: Improved semantic-complete graph matching for domain adaptive object detection. TPAMI 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRRr8isseN", "reviewer": "Reviewer_4YNt", "summary": "The paper introduces a new metric (\"DAS\") for unsupervised model selection in domain-adaptive object detection. DAS consists of two components: a flatness index scores that approximates the flatness of the loss landscape in the target domain by measuring prediction agreement across perturbed model parameters, and a prototypical distance ratio that measures the ratio of inter-class and intra-class feature dissimilarity. DAS is evaluated for model selection and hyperparameter optimization and correlates well with true model performance.", "questions": "How is PDR or d_inter defined for single-class benchmarks such as Sim10k?\n\nDAS seems to work consistently very well for AT, while performance with other frameworks is less consistent (very apparent in Figure 4). Do you know why this might be the case?\n\nIt might make sense to include the oracle checkpoint in the main evaluation, especially tables 1-3. This would make it easier to see how much of the gap to the oracle is closed.\n\nPlease clarify what is meant by \"ES\" in table 2.\n\nIt would be interesting to see analogues for Figures 2 and 5 for Cityscapes and Sim10k in the appendix.", "limitations": "Limitations and societal impact are adequately discussed in the appendix.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a significant step forward in the underexplored area of unsupervised model selection in domain adaptive object detection. This is evidenced by considerable gains in object detection performance on the target domain, compared to baseline approaches. Although concepts similar to the two main contributions (flatness index score & prototypical distance ratio) have previously been in explored in the literature, their combination and application to unsupervised model selection in domain adaptive object detection is novel. The overall structure of the paper is really clear, and the evaluation is extensive and informative.", "weaknesses": "The writing quality should be improved (e.g. general grammar, typos such as \"Faster RCCN\", sometimes inconsistent terminology such as \"Prototype Distance Ratio\" / Prototypical Distance Ratio\"). Apart from writing quality, my only significant concern is the apparent invariance of DAS performance towards lambda, which raises the question of whether the flatness index score and prototypical distance ratio measure different aspects in practice and the degree to which they are complementary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank Reviewer 4YNt for constructive comments and valuable feedback. We appreciate the recognition of our work including the support for the novelty, structure, and evaluation. Now, we address the reviewer's concerns as follows.\n\n**Q1**: The writing quality should be improved.\n\n**A1**:We sincerely thank your constructive comments. We have corrected the grammar issues, typos, and inconsistent terminology. We also carefully proofread the paper.\n\n**Q2**: Apart from writing quality, my only significant concern is the apparent invariance of DAS performance towards lambda, which raises the question of whether the flatness index score and prototypical distance ratio measure different aspects in practice and the degree to which they are complementary.\n\n**A2**: From the generalization bound theory for flat minima present in the paper, the FIS and PDR measure the flatness and the minimum, thus they are theoretically evaluating the models in two aspects. In Table 5, the selected model mAP doesn't vary too much with different $\\lambda$ setting. This might have contributed to when the model convergences to some degree, the two terms FIS and PDR represent have been both optimized well. Thus they both show relatively stable results on the mAP. This is why we use PCC between DAS and the model to measure the subtle differences. On the other hand, the PCC has obvious changes with the variation of $\\lambda$. This indicates the $\\lambda$ indeed changes the DAS and influences the correlation between the proposed DAS and ground truth performance. To examine the complementary, we have conducted an ablation study in Table 4 from the main paper. It is shown that FIS only achieves $0.48$ PCC. With the help of PDR, our DAS improved the PCC from $0.48$ to $0.67$, demonstrating the complementary between FIS and PDR.\n\n**Q3**: How is PDR or d_inter defined for single-class benchmarks such as Sim10k?\n\n**A3**: Although Sim10k only has a single class, the object detection model also considers the background class, thus the PDR or d_inter can be calculated. We will add more descriptions in the final paper to reduce the confusion.\n\n**Q4**: DAS seems to work consistently very well for AT, while performance with other frameworks is less consistent (very apparent in Figure 4). Do you know why this might be the case?\n\n**A4**: The main reason lies in that the performance gap of other frameworks between the last checkpoint and Oracle results is narrow, so it gives our method a relatively small improvement space. For example, the average performance gap of AT over three settings is 8.68%, while that of MT over three settings is 1.28%. In particular, a narrow performance gap gives less tolerance for potential uncertainty of the DAS estimation. Therefore, the improvements with other frameworks show less consistent improvement like AT. Nevertheless, the proposed method can still select better checkpoints in most cases and checkpoints that are at least comparable with last checkpoints in very few cases.\n\n\n**Q5**: It might make sense to include the Oracle checkpoint in the main evaluation, especially tables 1-3. This would make it easier to see how much of the gap to the oracle is closed.\n\n**A5**: We thank your constructive comments. We will add the orcale checkpoint in tables 1-3. We present Tables A1 and A2 in the attached PDF in the rebuttal to show the added version of Tables 1 and 2 in the main manuscript.\n\n**Q6**: Please clarify what is meant by \"ES\" in table 2.\n\n**A6**: ES[40] indicates the entropy score, i.e., using the entropy of the prediction from the classification branch to select the checkpoints. We have clarified this in the experiment section.\n\n**Q7**: It would be interesting to see analogues for Figures 2 and 5 for Cityscapes and Sim10k in the appendix.\n\n**A7**: We provide more examples on Sim10k to Cityscapes and Cityscapes to Foggy Citysacpes adaptation in Figures A1 and A2 in the attached PDF. Similar to Figure 2 in the main paper, the proposed DAS score correlates well with the performance of DAOD checkpoints compared with other baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qC8i3b944J", "reviewer": "Reviewer_jxDH", "summary": "This paper tackles the problem of model selection in unsupervised domain adaptation for object detection (DAOD). In DAOD, existing methods choose the models (checkpoints) using ground truth data on the target domain, which is impractical in real-world settings. To solve the problem, this paper proposes a model selection method called Detection Adaptation Score (DAS) based on the relationship between flat minima and generalization ability. The proposed DAS consists of two scores: Flatness Index Score (FIS) and Prototypical Distance Ratio (PDR). The FIS calculates the distance between the predictions before and after the perturbation of the model parameters. The PDR calculates the distance between the class prototypes on source and target domains. The experimental results demonstrate the positive correlation between the DAS and mAP. Also, the models chosen by the DAS outperform those chosen by other unsupervised model selection methods.", "questions": "See i), ii), and iii) in Weakness.\n\ni) Does the proposed method work well in other UDA tasks as well?\n\nii) Is it possible to derive the mathematical relationship between the difference in risks and the variance in outputs?\n\niii) Why do better transferability and discriminability lead to flat minima?", "limitations": "Limitations are adequately addressed in this paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "i) This paper is well-written and easy to follow. The motivation for tackling the task is discussed in Sec. 1, and related works are well-summarized in Sec. 2. The ideas and details of the proposed method are clearly described in Sec. 3.\n\nii) The proposed method is simple yet effective. In addition, it is practically beneficial for not only model selection but also hyper-parameter tuning on DAOD.\n\niii) The performance of the proposed method is good. The proposed method shows better performance than other unsupervised model selection methods in terms of the mAP of the chosen models. The ablation studies show that each of PDR and FIS contributes to better model selection.", "weaknesses": "i) The experiments in this paper show that the proposed method works well on model selection in DAOD. However, I think the proposed method is not limited to DAOD; that is, it can be directly applied to other tasks such as UDA for image classification and semantic segmentation (except IoU calculation in Eq. 2). I would like to see whether the proposed method works well those tasks as well.\n\nI have some concerns about the relationships between the flat minima principal and the proposed method as follows:\n\nii) The first term of the right-hand side in Eq. (1) is the difference between the risks (i.e., losses) of $h$ and its neighbor $h'$. In contrast, the proposed method calculates the variance in the outputs, not the risks. They are different measures, and the relationships between them are not addressed in this paper although there might be a correlation between them. This paper will be more convincing if mathematical relations between the difference in the risks and the variance in the outputs are derived.\n\niii) I understand that the PDR evaluates the transferability and discriminability because high PDR means that the prototypes of the same classes are well-aligned between the source and target domains. However, the relationship between better transferability (and discriminability) and flat minima is not clear in this paper. Why do better transferability and discriminability lead to flat minima?\n\niv) Although it was confirmed that each of FIS and PDR contributes to better model selection in terms of mAP, the evaluation of the correlations between each of FIS and PDR and flatness is missing. Similar to [7], the flatness can be evaluated as the change in risk using GT labels. This paper will be better if it provides the experiments to evaluate the correlations between each of FIS and PDR and flatness. The experiments can address my concerns ii) and iii) indirectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the Reviewer jxDH for your insightful comments. We appreciate the positive comments regarding the writing, method, and performance. Now we answer the question raised by the reviewer as follows.\n\n**Q1**: Does the proposed method work well in other UDA tasks as well?\n\n**A1** Yes. Our method can also work in other UDA tasks. To verify this, we conduct experiments on semantic segmentation and evaluate the well-known DAFormer method on GTA-5 to Cityscapes. The experimental results in terms of mIoU are shown below. It is shown that our DAS score can choose a better checkpoint of the model with an average mIoU of 64.2% while the last checkpoint of the model only achieves 60.9%. This demonstrates that our DAS method can also work effectively on other UDA tasks.\n\n| Method   | Last | Ours | Imp.$\\uparrow$ | Oracle |\n| -------- | ---- | ---- | -------------- | ------ |\n| DAFormer | 60.9 | 64.2 | 3.3            | 65.9   |\n\n**Q2**: Is it possible to derive the mathematical relationship between the difference in risks and the variance in outputs?\n\n**A2**: The variance in outputs serves as an upper bound for the difference in risks. To establish this relationship, let us define the error as $\\mathcal{E} _ {h}(o _ {h}, g) = \\left| o_ {h}-g \\right|$, where $o _ {h}$ is the output of the network $h$ and $g$ is the ground truth. Similarly, error of network $h'$ is denoted as $\\mathcal{E} _ {h'}(o _ {h'}, g) = \\left| o_ {h'}-g \\right|$. By applying the triangle inequality, we derive:\n$$\n\\left|\\mathcal{E} _ {h}(o_{h}, g) - \\mathcal{E} _ {h'}(o _ {h'}, g) \\right| = \\left|\\left| o_{h} - g \\right| + \\left| o _ {h'} - g \\right| \\right|  \\leq \\left| o _ {h} - o _ {h'} \\right|\n$$\nConsequently, minimizing the variance in outputs effectively constrains the difference in risks, providing a mechanism to bound the variability in predictions across networks. \n\nWe appreciate your insightful comments and will delve further into these mathematical underpinnings in our revised manuscript to illustrate the implications of variance in outputs on risk difference.\n\n**Q3**：Why do better transferability and discriminability lead to flat minima?\n\n**A3**: This is a good question. I would like to gently point out that variance in outputs primarily indicates **flatness**. If the labels are available, we can use the labels to calculate the error on the target domain so as to ensure the property of **minima**. Without using target labels, we estimate the generalization error on the target domain through a domain adaptation theory, where the target error is bounded by source error, domain distance, and a constant term. The transferability and discriminability are widely used to improve the performance on the target domain. With the both properties of flatness and minima on the target domain, we can obtain the flat minima without using target labels. Thanks for your valuable feedback, we will add more explanations to make it clearer in the final paper.\n\n**Q4**: The correlations between each of FIS and PDR and flatness with GT labels.\n\n**A4**: Thank the reviewer for pointing this out. We provide an experiment to verify the positive correlation between our FIS and PDR and flatness with GT labels. In particular, we show the correlation coefficient between them on AT weather adaptation. It shows that the proposed FIS is well correlated with the GT flatness with a coefficient of 0.64, demonstrating that our DAS score can be an appropriate estimation for flatness with GT labels. However, PDR represents the target error by assessing the transferability and discriminability thus it doesn't well correlate with the flatness of GT labels, as the coefficient is 0.45. It is by combining FIS and PDR that we could find the flat minima without accessing target labels while achieving accurate model selection for DAOD models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxI8hn6g7H", "reviewer": "Reviewer_9Lzv", "summary": "This submission provides a strategy to select an appropriate domain adaptive object detection model without access to labels of target domain. The basic premise is the minima flat to instruct the selection, because minima flat means better generalization. Experiment indicate the effect of the design based on their detection adaption score.", "questions": "1. It is similar to DG and therefore the proposed strategy is not novel athought the motivation sounds good.\n2. Necessity of such motivation. \n3. More experiments on other related tasks.", "limitations": "1. The motivation is interesting but does not well match real-world applications. Test labels should be seen even not too much. Otherwise, it is just a domain generalization scenario, which means the novelty of flat minima is limited.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is easy to read and the idea is clearly presented.\n\n2. The proposed strategy is based flat minima which is therefore rational.\n\n3. Experiments show the effectiveness of the proposed strategy.", "weaknesses": "1. This paper may neglect a fact that in real application, target labels are often known for testing the provided model, and in fact the training labels may not be accessible. So, I do not think this strategy is really necessary.\n\n2. This paper is essentially a DG domain generalizationt task, where target domain is unseen. So we can only select (train) a better model by using existing source domain data. Flat minima has also been used in DG method to instruct how to train a domain generalization model. \n\n3. The proposed selection strategy is orthogonal to the high-level task, such as object detection. Therefore, why focus only on the domain adaptive object detection task? More tasks should also be tested such as image classification, semantic segmentation and others.\n\n4. In experimental comparison, the authors compare the last checkpoint, which may not be fair. \n\n5. Lacking comparisons to many SOTA DAOD models.\n\n6. What is the computational overhead?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank Reviewer 9Lzv for the constructive comments. We appreciate the positive comments including clear presentation, rational strategy, and effective experiments. Following are the responses to the reviewer's concerns.\n\n**Q1**: This paper may neglect a fact that in real application, target labels are often known for testing the provided model, and in fact the training labels may not be accessible. So, I do not think this strategy is really necessary.\n\n**A1**: The goal of unsupervised domain adaptation (UDA) is to solve the scenario where the target domain labels are inaccessible, which are aligned with the real-world application. However, how to evaluate the UDA model is a long-standing problem in the UDA research community. The researcher in the UDA community uses the target domain label to select models, which is inevitable to overfit the test set. Meanwhile, it violates the assumption of unavailability of the target domain labels. The unsupervised model selection of UDA is also discussed in many previous works [16,51]. In our paper, we are the first attempt to address the unsupervised model selection for domain adaptive object detection. Moreover, because the annotation of object detection is hard to obtain, our method is meaningful for the real-world application. We believe our work would inspire researchers in the community and push the real-world application of DAOD models forward.\n\n**Q2**: This paper is essentially a DG domain generalization task, where target domain is unseen. So we can only select (train) a better model by using existing source domain data. Flat minima has also been used in DG method to instruct how to train a domain generalization model.\n\n**A2**: Our paper is intrinsically different from the DG task. Domain generalization addresses the issue where the target domain is not accessible, while our paper addresses the model selection method for DAOD. The flat minima is a general concept in machine learning and has been used in many fields. Different fields have different estimation methods. In domain generalization, flat minima are estimated using the source domain labels. However, the target domain labels are not available for domain adaptation. Therefore, we propose a novel method via a generalization bound to obtain the flat minima without using target labels to address the unsupervised model selection problem for DAOD. This is essentially different from the DG task.\n\n**Q3**: Experiments on other related tasks.\n\n**A3**: We thank the reviewer for the valuable feedback. To further verify the effectiveness of the proposed method, we conduct the experiments on the semantic segmentation. In particular, we evaluate the well-known domain adaptive semantic segmentation approach DAFormer. The qualitative results are shown below. It is shown that our DAS method can choose a better checkpoint of the model with an mIoU of 64.2% while the last checkpoint of the model only achieves 60.9%. This demonstrates that our DAS method still works effectively on domain adaptive semantic segmentation task. We will evaluate our method on more tasks in future research.\n\n| Method   | Last | Ours | Imp.$\\uparrow$ | Oracle |\n| -------- | ---- | ---- | -------------- | ------ |\n| DAFormer | 60.9 | 64.2 | 3.3            | 65.9   |\n\n**Q4**: In experimental comparison, the authors compare the last checkpoint, which may not be fair.\n\n**A4**: Due to the unavailability of target labels, choosing the last checkpoint without a specifically designed model selection is a good choice. To this end, we choose the last checkpoint as a baseline. We also compare with other model selection methods in Table 2 in the manuscript to demonstrate the effectiveness of the proposed method.\n\n**Q5**: Lacking comparisons to many SOTA DAOD models.\n\n**A5**: Thank the reviewer for the comments. Our method is orthogonal to existing DAOD models. To evaluate the effectiveness of the proposed method, we evaluate our method on classic DAOD methods. To further verify the generalization of our method on SOTA DAOD models, we evaluate the model on SIGMA++. The results are shown in the below table. We can see that the proposed method also works with SIGMA++ where we choose a checkpoint with $41.7%$ AP, while the last checkpoint is $39.5%$. We will include the new results in our final paper. We will evaluate our method on more SOTA DAOD models in future research. \n\n| Method  | Last | Ours | Imp.$\\uparrow$ | Oracle |\n| ------- | ---- | ---- | -------------- | ------ |\n| SIGMA++ | 39.5 | 41.7 | 2.2            | 41.7   |\n\n**Q6**: What is the computational overhead?\n\n**A6**: In the implementation, the calculation of DAS needs only two inferences for each target domain sample and one inference for each source domain sample. Thus one checkpoint requires just a few minutes to obtain its DAS. For example, we evaluate one model like AT using 2 minutes (on RTX 3090) while the model training time is usually around 30 hours. Therefore, the computational overhead is small compared with the model training. Once a model is selected by our method, the computational overhead is the same as the original detector.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7PEcbbRgg", "reviewer": "Reviewer_XtC2", "summary": "The paper introduces the Detection Adaptation Score (DAS), an unsupervised approach for selecting optimal models in Domain Adaptive Object Detection (DAOD) without target domain labels. DAS is based on the flat minima principle and includes the Flatness Index Score (FIS) and Prototypical Distance Ratio (PDR) to evaluate model generalization. Experiments on various benchmarks demonstrate DAS's effectiveness in checkpoint selection and hyperparameter tuning, outperforming existing methods. The perspective of enhancing model selection for DAOD is interesting and novel. It attempts to tackle an overlooked yet important limitation of existing DAOD methods and shows potential for other computer vision tasks.", "questions": "I didn't find major issues with the paper, and I only have some minor confusion, as listed below.\n\n1. Can the proposed method be extended to other baselines for DAOD, such as the FCOS-based method [1] and the DETR-based method [2]? It will be great to enhance the adaptability of the method further.\n\n2. Could the authors explain why DAS does not give obvious gains in some settings? For example, DAS gives no gains for CMT on Real-to-Art Adaptation in Table 1.\n\n3. Can the method be used for fine-grained tasks, such as segmentation?\n\n4. Consider giving a more comprehensive review of some advanced cross-domain object detection works [1-4].\n\n[1] Deng, J., Xu, D., Li, W., & Duan, L. (2023). Harmonious teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23829-23838).\n\n[2] Huang, W. J., Lu, Y. L., Lin, S. Y., Xie, Y., & Lin, Y. Y. (2022, July). AQT: Adversarial Query Transformers for Domain Adaptive Object Detection. In IJCAI (pp. 972-979).\n\n[3] Zhou, W., Fan, H., Luo, T., & Zhang, L. (2023). Unsupervised domain adaptive detection with network stability analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6986-6995).\n\n[4] Li, W., Liu, X., & Yuan, Y. (2023). Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 9022-9040.", "limitations": "No potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper writing is clear and easy to follow. The motivation is well described.\n2. The perspective about model selection for DAOD is interesting because it seems to be the first attempt in this field. Besides, this can be extended to other communities since most tasks need to decide which checkpoint is optimal for practical usage without labels.\n3. The proposed method can be deployed for different DAOD methods and give moderate gains without sacrificing inference speed.", "weaknesses": "1. It seems that the proposed method is only tested on the Faster RCNN baseline.\n2. The performance gains on some detectors under some settings are limited.\n3. The method is only tested on object detection, lacking an analysis for more fine-grained tasks, e.g., instance segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the Reviewer XtC2 for your constructive comments and valuable feedback. We appreciate the recognition of our work on motivation and perspective. Now we answer the question raised by the reviewer as follows.\n\n**Q1**: It seems that the proposed method is only tested on the Faster RCNN baseline.\n\n**A1**: We thank the reviewer for the valuable comments. To further verify the effectiveness of the proposed DAS method on other detectors, we conduct experiments on SIGMA++[4] (FCOS-based) and  AQT[2] (DETR-based). The results on Cityscapes to Foggy Cityscapes are shown below table. It is shown that the proposed method also works with the FCOS-based, and DETR-based methods. In particular, our method chooses a SIGMA++ checkpoint with 41.7% map while the last checkpoint is 39.5% mAP, demonstrating the adaptability of the proposed method on other detectors. Although DAS gives a moderate gain compared with the last checkpoint of AQT, our method selected the optimal checkpoint. We will evaluate our method on other detectors in future research.\n\n| Method  | Last | Ours | Imp.$\\uparrow$ | Oracle |\n| ------- | ---- | ---- | -------------- | ------ |\n| SIGMA++ | 39.5 | 41.7 | 2.2            | 41.7   |\n| AQT     | 45.5 | 45.7 | 0.2            | 45.7   |\n\n**Q2**: Could the authors explain why DAS does not give obvious gains in some settings? For example, DAS gives no gains for CMT on Real-to-Art Adaptation in Table 1.\n\n**A2**: The main reason lies in that the performance gap between the last checkpoint and Oracle results is small, so give our method relatively small improvement space. For example, the average performance gap of AT over three settings is $8.68%$, while that of MT over three settings is $1.28%$. Nevertheless, the proposed method can select better checkpoints in most cases and at least the last checkpoints in very few cases.\n\n**Q3**: Can the method be used for fine-grained tasks, such as segmentation?\n\n**A3**: Yes, our method can be used for fine-grained tasks. To verify this, we conduct the experiment on the semantic segmentation. In particular, we evaluate the well-known domain adaptive semantic segmentation approach DAFormer[5]. The qualitative results on GTA5->Cityscapes are shown below. It showes that our DAS method can choose a better checkpoint of the model with an mIoU of 64.2% while the last checkpoint of the model only achieves 60.9%. This demonstrates that our DAS method still works effectively on domain adaptive semantic segmentation tasks.\n\n| Method   | Last | Ours | Imp.$\\uparrow$ | Oracle |\n| -------- | ---- | ---- | -------------- | ------ |\n| DAFormer | 60.9 | 64.2 | 3.3            | 65.9   |\n\n**Q4**: Consider giving a more comprehensive review of some advanced cross-domain object detection works [1-4].\n\n**A4**: Thanks for the valuable feedback. We will enhance the final paper by providing a more comprehensive review of some advanced cross-domain object detection works.\n\n[1] Deng, J., Xu, D., Li, W., & Duan, L. (2023). Harmonious teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23829-23838).\n\n[2] Huang, W. J., Lu, Y. L., Lin, S. Y., Xie, Y., & Lin, Y. Y. (2022, July). AQT: Adversarial Query Transformers for Domain Adaptive Object Detection. In IJCAI (pp. 972-979).\n\n[3] Zhou, W., Fan, H., Luo, T., & Zhang, L. (2023). Unsupervised domain adaptive detection with network stability analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6986-6995).\n\n[4] Li, W., Liu, X., & Yuan, Y. (2023). Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 9022-9040.\n\n[5] Hoyer L, Dai D, Van Gool L. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 9924-9935.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We would like to thank all the reviewers for their insightful comments and constructive feedback which have significantly enhanced the quality of our work. We appreciate that the reviewer acknowledges the advantages of our work: "**This paper is well motivated. The proposed method is effective and reasonable**"(Reviewer ood2), "**The paper presents a significant step forward in the underexplored area of unsupervised model selection in domain adaptive object detection**"(Reviewer 4YNt), "**The proposed method is simple yet effective**"(Reviewer jxDH), "**The proposed strategy is based flat minima which is therefore rational.**" (Reviewer 9Lzv), "**The perspective about model selection for DAOD is interesting**"(Reviewer XtC2).

On the other hand, we have diligently addressed all the concerns raised by the reviewers. In the attached PDF file, we also provide the tables with Orale results for easier to see how much of the gap to the Oracle is closed and more comparison of different unsupervised model evaluation methods for DAOD.

Best Regards,

Authors of Submission 7161.

Meta-review: Though the reviews are mixed, the majority of the reviewers recommend acceptance and emphasize the importance of the topic (unsupervised model selection for domain adaptation), which has been previously under-explored in the literature. The authors propose a novel method for addressing this which performs close to optimal when assuming access to labels in the target domain. The AC team appreciates the value of the contribution and thus agrees with the majority of the reviewers that the paper should be accepted.
Decision: Accept (poster)