Towards Unsupervised Model Selection for Domain Adaptive Object Detection

Authors: hengfu  yu
jinhong  deng
wen  li
lixin  duan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 17:25:50

Abstract: Evaluating the performance of deep models in new scenarios has drawn increasing attention in recent years due to the wide application of deep learning techniques in various fields. However, while it is possible to collect data from new scenarios, the annotations are not always available. Existing Domain Adaptive Object Detection (DAOD) works usually report their performance by selecting the best model on the validation set or even the test set of the target domain, which is highly impractical in real-world applications. In this paper, we propose a novel unsupervised model selection approach for domain adaptive object detection, which is able to select almost the optimal model for the target domain without using any target labels. Our approach is based on the flat minima principle, i.e., models located in the flat minima region in the parameter space usually exhibit excellent generalization ability. However, traditional methods require labeled data to evaluate how well a model is located in the flat minima region, which is unrealistic for the DAOD task. Therefore, we design a Detection Adaptation Score (DAS) approach to approximately measure the flat minima without using target labels. We show via a generalization bound that the flatness can be deemed as model variance, while the minima depend on the domain distribution distance for the DAOD task. Accordingly, we propose a Flatness Index Score (FIS) to assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters and a Prototypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively represent the degree of flat minima and evaluate the model generalization ability on the target domain. We have conducted extensive experiments on various DAOD benchmarks and approaches, and the experimental results show that the proposed DAS correlates well with the performance of DAOD models and can be used as an effective tool for model selection after training. The code will be released at https://github.com/HenryYu23/DAS.

Main paper: 
 introduction: 
 With the explosion of deep neural networks [27, 56, 18] , object detection [24, 47, 3, 54, 5] has achieved promising results and shown great potential in many downstream tasks such as autonomous driving [2, 53] , video understanding [55, 6] , robot navigation [42, 41] , etc. However, the well-trained object detection models frequently face previously unseen domains in real-world scenarios and often suffer from dramatic performance degradation when being deployed in a novel domain [11] . This is because the training set (i.e., source domain) and the test set (i.e., target domain) have distinct domain distributions. To address this problem, Domain Adaptive Object Detection (DAOD) [11, 50, 14, 38, 4] has been proposed to transfer the knowledge from the labeled source domain to an unlabeled target domain by leveraging adversarial training or pseudo-labeled approaches.Although effective, these DAOD methods [11, 50, 14, 38, 4] evaluate the detector performance and conduct model selection relying on the labeled target data, which is usually unavailable and impractical for real-world domain adaptation scenarios. Due to the natural complexity of object detection, DAOD methods that leverage adversarial training and self-training techniques are often unstable and prone to overfitting to the target domains. As shown in Fig. 1 (a), DAOD methods usually suffer from a performance drop during training (marked as purple in Fig. 1 (a)). These issues limit the application of the DAOD model in real-world scenarios. Therefore, it is urgent and desirable to develop an unsupervised model selection method for DAOD, as shown in Fig. 1 (b) .Regarding the unsupervised model selection, several seminal works [22, 59, 60] attempt to evaluate the models from different aspects. For example, TS [59] examines the spatial uniformity of the unsupervised domain adaptive classifier, as well as the transferability and discriminability of deep representation. ATC [22] learns a threshold on the confidence of the model and predicts accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. In object detection, however, the evaluation involves not only classification but also precise localization of objects within an image. This crucial distinction renders these methods ineffective in fully assessing an object detection model. A recent work [60] proposes a BoS metric to evaluate the generalization of the detection model by measuring the stability of the box under feature dropout. However, it does not consider the domain discrepancy between the source and the target domain, making the metric unreliable for DAOD model selection (see Fig. 1 (a) and our experiments in Sec. 4.2).In this paper, we propose a novel Detection Adaptation Score (DAS) to evaluate the DAOD models without accessing any target labels, enabling select almost the optimal model for the target domain in an unsupervised way. The proposed DAS is based on the flat minima principles, i.e., models that are located in the flat minima region in the parameter space exhibit better generalization ability than that in sharp minima region [21, 7] . However, the traditional flat minima search method requires labeled data to evaluate how well a model is located in the flat minima region, which is unrealistic for DAOD tasks. Therefore, we investigate how to measure the flat minima approximately without using target labels. We derive a generalization error bound that shows the flatness can be deemed as model variance, while the minima depend on the domain distribution distance for the DAOD tasks. Therefore, we first propose a Flatness Index Score (FIS) to assess the flatness by observing the classification and localization fluctuation before and after perturbations of model parameters. Then, we propose a Prototypical Distance Ratio (PDR) score to seek the minima models by measuring the transferability and discriminability of the models in the target domain. To this end, the proposed DAS can effectively represent the degree of flat minima for DAOD models and evaluate the model performance on the target domain. To evaluate the effectiveness of the proposed DAS, we conduct extensive experiments on public DAOD benchmarks, including weather adaptation, real-to-art, and synthetic-to-real adaptation. The experimental results show that the proposed metric can effectively evaluate the performance of DAOD models without annotating the target domain.The contributions of our work can be summarized as follows:• With the pressing need for the application of DAOD in real-world scenarios, to our best knowledge, we are the first to evaluate the DAOD models without using target labels.• We propose a novel Detection Adaptation Score (DAS) by seeking the flat minima without using any target labels to evaluate the performance of DAOD models on the target domain.A Flatness Index Score (FIS) and a Prototypical Distance Ratio (PDR) score are proposed to meet the requirements of flatness and minima, respectively.• We have conducted extensive experiments on several DAOD benchmarks and approaches.Our DAS benefits from selecting the optimal checkpoints of model parameters to avoid negative transfer or pseudo-label error accumulation. The experimental results validate the effectiveness of the proposed DAS. 
related work: 
 Object Detection. Object detection [23, 47, 54, 46, 39] is a fundamental task in computer vision that involves identifying and locating multiple objects within an image or video. The object detection approaches can be roughly divided into two categories: one-stage and two-stage. The one-stage methods [54, 46, 39, 19] directly estimate the categories and the location of the objects, such as FCOS [54] , CenterNet [19] , and YOLO series [46] . The two-stage methods [23, 47, 3] first generate region proposals for objects and then classify the category and regress the bounding box coordinates based on the proposal features, e.g., Faster RCNN [47] and Cascade RCNN [3] . Recently, an endto-end object detection model based on Transformer (i.e., DEtection TRansformer, DETR) [5] has been proposed to eliminate the complex anchor generation and post-processing operations such as non-maximum suppression (NMS). Many successive works [67, 63] further improve the training efficiency and accuracy of DETR. With the strong representation of deep neural networks, the object detection model has achieved promising results in many object detection benchmarks. However, these models usually suffer from performance degradation [11] because of the domain discrepancy between the training and testing domains.Domain Adaptive Object Detection. Domain Adaptive Object Detection (DAOD) [11, 50, 4, 38, 15, 8, 66, 16, 30, 65] aims to transfer knowledge from the labeled source domain to an unlabeled target domain. Previous works can be roughly categorized into two aspects: domain alignment and selftraining. The domain alignment [11, 50, 8, 66] directly minimizes the domain discrepancy between the source and target domains. They minimize the feature distribution mismatch via adversarial training [11, 50, 66, 16] , prototype alignment [58] , graph matching [35, 36, 37] , etc. The self-training approaches [14, 38, 4, 15, 10] follow a Mean Teacher (MT) framework and generate pseudo-labels from the teacher network to supervise the training of the student network. UMT [14] leverages the style transfer algorithm to eliminate the MT model bias towards the source domain. Adaptive Teacher (AT) [38] combines adversarial training and self-training to improve the accuracy on the target domain. CMT [4] introduces contrastive learning to improve the instance feature representation. HT [15] reveals that the consistency between classification and localization is crucial for pseudo-label generation and proposes a reweight strategy based on the harmony measure between the classification and localization. While effective, these approaches [11, 50, 4, 38, 15, 16, 35, 36] evaluate the model performance relying on labeled target data, which is not always available in real-world scenarios. Therefore, we propose a Detection Adaptation Score (DAS) to evaluate the model performance in an unsupervised manner, enabling the application of DAOD models in real-world scenarios.Unsupervised Model Selection (UMS) for UDA. Unsupervised Model Selection for UDA evaluates model performance in the target domain without involving annotations. Some previous works [22, 45, 40, 44, 62, 9, 26, 59 ] seek to predict model performance in OOD scenarios. PS [28] , ATC [22] leverage the prediction confidence, and [49, 48] estimate from the perspective of entropy. DEV [61] estimates and decreases the target risk by embedding adapted feature representation while validation. TS [59] examines the spatial uniformity of the classifier, as well as the transferability and discriminability of deep representation. However, they mainly focus on classification tasks. For object detection, the BoS [60] estimates the detection performance via the stability of bounding boxes with feature dropout but does not consider the domain discrepancy, which is vital for DAOD methods. To this end, we introduce the Detection Adaptation Score (DAS), which consists of a Flatness Index Score (FIS) and a Prototypical Distance Ratio (PDR) score by seeking a flat minima model in the target domain. 
method: 
 In the DAOD task, we are given a labeled source domain, which includes images annotated with bounding boxes and class labels, and an unlabeled target domain containing only unlabeled images.Let us denote D s = {(x s i , y s i )} Ns i=1 drawn from a data distribution P s as the labeled source domain and D t = {x t j } Nt j=1 drawn from a data distribution P t as the unlabeled target domain. Distributions P s and P t are related but different domains, i.e., (P s ̸ = P t ). In other words, they have distinct domain shifts. Andy s i = {(b s ij , c s ij )| mi j=1}, where b s ij ∈ R 4 and c s ij ∈ {1, . . . , K} are the bounding box and corresponding category for each object, and m i is the total number of objects in an image x s i . The goal of the DAOD approach is to learn an object detection model that performs well on the target domain from both the labeled source and unlabeled target domains. In this work, we consider the model selection for DAOD approaches. In particular, there are M models F = {f l | M l=1 } from different epochs or iterations. Our goal is to propose a proper metric for model evaluation in an unsupervised manner that can reflect the detection performance (i.e., mAP) of the DAOD models.In the following, we first introduce the domain adaptation generalization bound with flat minima in Sec. 3.1. Then, we will introduce our Detection Adaptation Score (DAS) in detail, which consists of a Flatness Index Score (FIS) in Sec. 3.2 and a Prototypical Distance Ratio (PDR) score in Sec. 3.3.In this paper, we propose a novel unsupervised model selection approach for the DAOD task, which can almost select the optimal checkpoint for the target domain. Our approach is based on the assumption that flat minima exhibit better model generalization than sharp minima, which has been evidenced by many literatures [21, 7] . The model parameters at flat minima will have smaller changes of loss values within its neighborhoods than sharp minima. To find flat minima, traditional methods require labeled data, which is unrealistic for DAOD. As the target labels are unavailable for model evaluation in DAOD scenarios, we cannot directly find the flat minima. To this end, we derive a generalization error bound as follows: Theorem 1. Given any δ ≥ 0, exist hypothesis h ∈ H where H is the hypothesis set, θ h denotes the parameters of h. Given any hypothesis h ′ ∈ {h ′ |h ′ ∈ H, ∥θ h ′ -θ h ∥ 2 ≤ τ }, which is located in the neighborhood of h with radius τ > 0, the following generalization error bound holds with at least a probability of 1 -δ,EQUATIONwhere dis(S, T ) is the distribution mismatch between the source domain S and target domain T . E S (h) is the risk of h on the source domain. Ω is a constant term. Proof is provided in the appendix.The generalization bound shows that in addition to the constant term Ω, the risk for the target hypothesis h ′ around by h with τ -ball radius can be bounded by three terms: the flatness, i.e., the difference between the original h and neighborhood h ′ of h, the error on the source domain D s , and the distribution distance between the source and target domains. Usually, the error on source E S (h) can be minimized with the labeled source samples. To this end, one can minimize the first and third terms to find the flat minima.From the analysis in Theorem 1, the flatness can be deemed as model variance, while the minima depends on the domain distribution distance for DAOD task. Accordingly, we propose a Flatness Index Score (FIS) to assess the flatness by measuring the classification and localization fluctuation before and after perturbations of model parameters, and a Prototypical Distance Ratio (PDR) score to seek the minima by measuring the transferability and discriminability of the models. In this way, the proposed DAS approach can effectively represent the degree of flat minima and evaluate the model generalization ability on the target domain.In this subsection, we introduce the Flatness Index Score (FIS), which assesses the flatness of the model parameters by measuring the variance in outputs before and after parameter perturbations. For object detection, we calculate the variance of both classification and localization predictions. For the property of minima on the target domain, we introduce it later in Sec. 3.3.Specifically, let γ denote the radius of the parameter space. We can then obtain the neighbor model f (•; θ ′ ) by adding a perturbation ∆ to the original parameter θ, i.e., θ ′ ← θ + ∆. We control γ of the perturbation as a constant (i.e., ∥∆∥ = γ), thus the neighbor model f (•; θ ′ ) lies on a fixed radius sphere of the original model f (•; θ).We measure the prediction correspondence between the original and neighbor models. Given an input target domain image x i , the original model predicts{(b j , p j )| ni j=1 } as the neighbor model predicts {( bj , pj )| n ′ i j=1 }, where b j is the bounding box and p j is the probability vector of the j-th instance. We use d jj ′ (f γ (x i ; θ)) to represent the matching cost of the j-th and j ′ -th object in two model's predictions on image x i . As the model predictions has results from both regression and classification branches, the matching cost d jj ′ contains the divergence of bounding boxes and classification probability vectors as follows:EQUATION)where IoU is the intersection over union of two boxes, KL is the KL-divergence over two probability vectors. The smaller the d jj ′ is, the better the two object predictions match. Then the Flatness Index Score (FIS) on the original model parameter θ with radius γ is defined as:EQUATIONwheren ′′ i = min{n i , n ′ i }is the smaller number of the predicted objects from the original and neighbor models. σ(j) is the j-th object's one-to-one corresponding index leading to the minimized matching cost whenn i ≤ n ′ i , otherwise FIS = -E θ ′ ←θ+∆ [E xi∼Dt [min σ 1 n ′′ i n ′′ i j ′ =1 d σ(j ′ )j ′ ]]. Hungarian Algorithm is applied to select the best-matched pairs.To facilitate the search for flat minima, we explore methods to identify minima regions in the target domain. In DAOD, the model with better transferability and discriminability would perform well on the target domain. There are many methods to evaluate the domain distance between the source and target domains, for example, Maximum Mean Discrepancy (MMD) [25] and Proxy A-distance (PAD) [1] . However, simply measuring the image feature distribution is not feasible to reflect the transferability of the detection model. Meanwhile, the traditional discriminability metrics such as entropy [48] , and mutual information [33] also fail to effectively correlate the model performance. In this paper, we consider the class prototype distance of instances in images across domains to evaluate the transferability and discriminability of the DAOD models. The class prototype of instances is the center of a specific class and aggregates the instance information from the samples. In unsupervised domain adaptation, prototype-based domain alignment is comprehensively studied in the literature [58, 57, 43] and attempts to narrow the distance between prototypes of the same categories of two domains, showcasing the effectiveness of prototype alignment. Now, we show how to leverage prototype distance to effectively evaluate the transferability and discriminability of the DAOD models. Our prototype is calculated based on the instance feature, e.g., the proposal feature in Faster RCNN. In particular, we denote the instance feature as F ij ∈ R d for the j-th instance in the i-th image and the final classification probability vector as p ij . The prototype of the k-th class P k ∈ R d for the target domain can be calculated softly as follows:EQUATIONwhere n p i is the number of proposals in the i-th image. The source prototypes can be obtained similarly. Finally, we have the source and target domain prototypes P S ∈ R K×d and P T ∈ R K×d .An ideal DAOD network aligns the features of the same category between domains while increasing the feature distances between different categories. We propose a Prototypical Distance Ratio (PDR) score to evaluate this ability with the category-wise prototypes. The prototype distance of the same category across domains can be defined as follows:EQUATIONwhere M (P S , P T ) ∈ R K×K is the category-wise distance matrix between P S and P T . Denote M kk ′ (P, P ′ ) as the distance between P k and P ′ k ′ . Similarly, we could obtain the inter-category prototype distances as follows:EQUATIONwhere δ(P, P ′ ) =1 K 2 -K K k̸ =k ′ M kk ′ (P, P ′) is a function to calculate distance among different categories. Combine the intra-category distance and inter-category distance into our Prototypical Distance Ratio (PDR) score as follows:EQUATIONThe proposed PDR score can be an effective metric to evaluate the transferability and discriminability of the DAOD models. The higher PDR score indicates that the instance features of detection models have better properties with large inter-category distances and small intra-category distances.For different checkpoints in a training session, we simply normalize the FIS and PDR separately by min-max normalization. The DAS is a combination of FIS and PDR as follows.EQUATIONwhere t is the index of the checkpoint. FIS t and PDR t are the t-th FIS and PDR score normalized among all checkpoints. λ is a trade-off parameter that controls the contribution of PDR t for DAS t . 
experiment setup: 
 Benchmarks. We follow previous works [11, 50, 38] and evaluate the effectiveness of the proposed DAS on the following adaptation scenarios:• Real-to-Art Adaptation (P2C): In this scenario, we test our proposed method with domain shift between the real image domain and the artistic image domain. Following [50] , we choose the PASCAL VOC 2007/2012 and Clipart1k as the source and target domains, respectively. PASCAL VOC [20] is a widely used benchmark in the object detection community. We use the 2007 and 2012 versions of PASCAL VOC that contain about 15k images with instance annotations for 20 object classes. Clipart1k [31] contains 500 train images and 500 testing images in clipart style, annotated with bounding boxes for the same 20 object categories as in the PASCAL VOC. • Weather Adaptation (C2F): In real-world scenarios, such as autonomous driving systems, object detectors may encounter various weather conditions. We study the adaptation from normal to foggy weather. In particular, we use the Cityscapes and Foggy Cityscapes as the source and target domains, respectively. Cityscapes [13] contains a diverse set of urban street scenes captured from 50 cities and 2, 975 training images and 500 validation images, annotated for 8 object classes. Foggy Cityscapes [51] is a variant of the Cityscapes dataset where synthetic fog is added to the images to simulate adverse weather conditions. The annotations remain consistent with the original Cityscapes dataset. Note that we choose the worst foggy level (i.e., 0.02) of Foggy Cityscapes in the experiment following [11] . • Synthetic-to-Real Adaptation (S2C): Synthetic images offer an alternative solution for addressing the data collection and annotation issues. However, there is a distinct distribution mismatch between synthetic images to real images. To adapt the synthetic to the real scenes, we utilize Sim10k as the source domain and Cityscapes as the target domain.Sim10k [32] consists of 10,000 synthetic images generated from a simulation environment, with annotations for car bounding boxes. Because only the car is annotated in both domains, we report the AP of "car" in the validation set of Cityscapes. DAOD Frameworks. In this work, we test our method on four classical DAOD models in the two main DAOD streams (i.e., adversarial training and self-training): DA Faster RCNN (DAF) [11] , Mean-Teacher (MT) [52] , Adaptive Teacher (AT) [38] , Contrastive Mean-Teacher (CMT) [4] . DAF [11] is a typical DAOD framework extending the Faster RCNN architecture by incorporating domain adaptation techniques. It employs adversarial training to align the feature distribution across domains at both image and instance levels. MT [52] leverages a teacher-student paradigm to generate pseudo labels to provide extra supervision for the student model on the target domain. The teacher model is an exponential moving average (EMA) of the student model and thus can provide more accurate pseudo-labels for the student model. This approach helps in leveraging unlabeled data by enforcing consistency between the predictions of the teacher and student models. AT [38] improves upon the Mean-Teacher framework by employing an adversarial learning module to align the feature distributions across two domains, reducing domain bias and improving pseudo-label quality. CMT [4] enhances the Mean-Teacher framework with contrastive learning techniques by encouraging similar instances to be closer in the feature space while pushing the features of dissimilar instances apart.Unsupervised Model Selection Baselines. Existing unsupervised model selection methods for UDA tasks are mainly based on classification tasks, such as Prediction Score (PS) [28] , Entropy Score (ES) [48] , Average Threshold Confidence (ATC) [22] , Transfer Score (TS) [59] . We reproduce these approaches on the classification branch of object detection models. Besides, we also use the Fréchet Distance (FD) [17] to measure the domain distance on backbone features as a compared method. Bounding Box Stability (BoS) is introduced to tackle unsupervised model evaluation problems specifically for object detection networks. It evaluates the model generalization of the detection model by probing the bounding box stability under feature dropout.Implement Details. Following previous works [11] , we choose one of the representative detection frameworks Faster RCNN as our base detector. We followed the instructions from the released code of the DAOD methods and reproduced their results. The hyperparameters, learning rates, and optimizers are set according to the default configurations provided in the original papers. We set our hyperparameters λ = 1 and γ = 1 in all experiments, which perform well on all the benchmarks. Our implementation is built upon the Detectron2 detection framework. We have added a detailed implementation in the appendix. Checkpoint Selection after Training. Checkpoint selection after training is pivotal for the application of DAOD approaches in real-world scenarios since the DAOD models usually suffer from negative transfer and overfitting the target domain during training. For example, on the real-to-art adaptation (P2C), AT [38] drops 5.85% from the highest mAP (47.83%) to the last checkpoint (41.98%). The detailed results on checkpoint selection, with the highest DAS, after training are listed in Table 1 . Compared with the last checkpoint, the proposed method works well on almost all the DAOD methods. For example, the proposed method for AT [38] achieves 5.85%, 1.1%, and 19.09% improvements in terms of mAP on real-to-art, weather, and synthetic-to-real adaptations, respectively. These experimental results demonstrate that the proposed method can choose a reliable checkpoint and thus avoid the negative transfer and overfitting on the target domain during training. Moreover, we compare our method with the recent unsupervised model evaluation methods in Table 2 , showing that our method consistently outperforms all the compared baselines. For instance, our methods outperform recent works TS [59] and BoS [60] by 2.42% and 3.60% mAP on average, respectively. These results demonstrate the effectiveness of the proposed method in choosing the optimal checkpoints. We also present the correlation between unsupervised model evaluation metrics and the ground truth mAP (i.e., using the annotations in the target domain to evaluate the model) in Fig. 2 . It is clearly shown that the previous methods give higher scores as the training goes on and fail to correlate with the performance of DAOD checkpoints. In contrast, the proposed DAS score correlates well with the performance of DAOD checkpoints (i.e., 0.86 PCC). Hyperparameter Tuning for DAOD Methods. Domain adaptation methods can be highly sensitive to the hyperparameters. Inappropriate hyperparameter selection would limit the transfer performance and lead to negative transfer, which even leads models perform below the source-only ones. Therefore, the validation of hyperparameters is an important problem in DAOD, yet researchers have unfortunately overlooked it. We evaluate our DAS in hyperparameters tuning by leveraging the weights of adversarial loss (denoted as λ dis ) for the domain discriminator and of unsupervised loss (denoted as λ unsup ) for self-training in AT [38] . For a model with a specific hyperparameter setting, we obtain the last three checkpoints of its training process and calculate their average performance to represent the model. The final DAS of the specific model is defined as the average DAS of these obtained checkpoints. The experimental results are conducted on the weather adaptation and summarized in Fig. 3 . From Fig. 3 , we can observe that the model with higher DAS presents better improvement after adaptation. The proposed DAS can correlate the detection performance without any labels on the target domain. We further compare our DAS with the unsupervised model evaluation methods and summarize the results in Table 3 . The proposed DAS consistently outperforms the previous baselines, which demonstrates the effectiveness of the proposed method in hyperparameter tuning. In a nutshell, our DAS can be a good indicator to guide the hyperparameter tuning for DAOD methods, thus avoiding the negative model optimization due to the inappropriate hyperparameter selection.Ablation Study. We conduct the ablation study of the proposed method by isolating DAS into separate metrics. In particular, we conduct experiments on three benchmarks and use four DAOD methods including DAF [11] , MT [52] , AT [38] , and CMT [4] . We summarize the results in Table 4 . The experimental results indicate that the proposed PDR score and FIS evaluate the model performance without labels effectively. In particular, the PDR score and FIS select checkpoints with mAP 42.14% and 41.74%, PCC 0.64 and 0.48, respectively. DAS combines them and further improves their performance to 42.52% mAP and 0.67 PCC, demonstrating the synergy effect among them.Hyperparameter Sensitivity. We investigate the sensitivity of the hyperparameter λ controlling the weight of PDR for DAS on real-to-art adaptation (P2C). The results are summarized in Table 5 . λ = 1.0 achieves the best average results. From a wide range of λ, DAS has relatively stable results. 
conclusion: 
 In this work, we propose a novel metric named Detection Adaptation Score (DAS) by seeking flat minima to evaluate the domain adaptive object detection models without involving any target labels. The proposed DAS consists of a Flatness Index Score (FIS) and a Prototypical Distance Ratio (PDR) score and can find flat minima of DAOD models in an unsupervised way. Extensive experiments have been conducted on public DAOD benchmarks for several classical DAOD methods.Experimental results indicate that the proposed DAS correlates well with the performance of the detection model and thus can be used for checkpoint selection after training. The proposed method fosters the application of DAOD methods in the real-world scenario. We hope that our work will inspire researchers and contribute to advancing research in DAOD. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: Our work does not a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original papers.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this appendix, we first give the proof of Theorem 1 in Sec. A.1. And then provide more implementation details in Sec. A.2 and experimental results in Sec. A.3. Finally, we discuss the limitations and societal impact of our work.A.1 Proofs of Theorem 1 Theorem 1. Given any δ ≥ 0, exist hypothesis h ∈ H where H is the hypothesis set, θ h denotes the parameters of h. Given any hypothesis h ′ ∈ {h ′ |h ′ ∈ H, ∥θ h ′ -θ h ∥ 2 ≤ τ } located in the neighborhood of h with radius τ > 0, the following generalization error bound holds with at least a probability of 1 -δ,EQUATIONwhere dis(S, T ) is the distribution mismatch between the source domain S and target domain T . E S (h) is the risk of h on the source domain. Ω is a constant term.Proof. Given any δ ≥ 0, exist hypothesis h ∈ H where H is a hypothesis set, θ h denotes the model parameters of h. Give any hypothesis h ′ ∈ {h ′ ∥θ h ′ -θ h ∥ 2 ≤ τ }, τ > 0 is the radius of the neighborhood. The following generalization bound holds with at least a probability of 1 -δ,EQUATIONAccording to the domain adaptation theory from [34, 64] , we have the following inequality:EQUATIONwhere the E S (h) indicates the source error and dis(S, T ) denotes the distribution mismatch between the source domain S and the target domain T . In this way, we replace the last term E T (h) in Eq. ( 10) with Eq. ( 11), leading to the following error bound:EQUATIONTill now, we prove the inequality in Theorem 1.Training and Evaluating Details. In our experiment, all the DAOD models are trained according to the default setting specified in their original papers and the open-source codebases. For DA Faster RCNN, we train it on the three benchmarks with a VGG-16 [12] backbone pre-trained on ImageNet, using a batch size of 4. For Mean Teacher, Adaptive Teacher, and Contrastive Mean Teacher, we train the models with a pre-trained ResNet-101 [27] backbone on ImageNet for the real-to-art adaptation setting, and with a pre-trained VGG-16 [12] backbone for the weather adaptation and synthetic-to-real adaptation settings. The batch size of AT, MT, and CMT are set to 8. Training is conducted on four RTX 3090 GPUs or two A100 GPUs according to the computational requirements of different DAOD methods. For all the results, we report the mean Average Precision (mAP) at the IoU threshold of 0.5.Detection Adaptation Score. For the Flatness Index Score (FIS), we perturb the detector model by adding a normalized random direction for all the model parameters including the backbone and detection head. For the Prototypical Distance Ratio (PDR) score, we use the features of the region proposals and the corresponding predicted probability (including background) from the classification branch of the detection head to aggregate the information. In our implementation, L2 distance is used as the prototype distance.Top3 Checkpoint Selection Results after Training We summarize the top3 checkpoint selection results in Table 6 . We can observe that with more loose selection condition, our method still outperforms other unsupervised model selection methods by a large margin. These results further demonstrate the effectiveness of the proposed DAS in select optimal checkpoints for DAOD methods. Performance Gap between Last Checkpoint and Oracle Model. In Fig. 4 , we compare the DAS selection results between the last checkpoint and the Oracle checkpoint (i.e., using the ground-truth labels in the target domain to evaluate the model performance) of various frameworks and benchmarks. It shows that there is a significant performance gap between them, making it urgent to design an effective method for unsupervised model selection in DAOD scenarios. In this paper, we propose DAS by seeking the flat minima of the DAOD models. From Fig. 4 , we can observe that the proposed DAS shows promising unsupervised model selection results. While our method has made progress compared to others, there is still room for further improvement.More Visual Correlation Results. We show the correlation comparison results in Fig. 5 under the real-to-art adaptation using MT [52] . Overall, our DAS correlates well with the detection performance and outperforms the previous method. It is worth noting that the compared method cannot figure out the overfitting issue while our method successfully reflects it.More Unsupervised Model Selection Results. The DAS can select models on other DA methods and tasks. 1) We also conducted our method on SIGMA++ [37] on weather adaptation, which minimizes the domain gap by graph matching. DAS chooses a checkpoint at 41.7% mAP (which is the oracle), while the last checkpoint reaches 39.5% mAP. 2) For prototype-based DAOD methods. The DAS includes FIS and PDR derived from the generalization bound. It can evaluate DAOD models from different aspects. Some existing works like GPA [58] use the prototype-based alignment method to minimize the domain gap between domains. However, the prototype estimation in existing works only utilizes samples in a mini-batch during the model training. In contrast, our DAS uses the entire dataset to estimate the prototypes, which is more robust and has better generalization ability. To verify this, we experimented on Synthetic-to-Real adaptation for GPA [58] . The DAS chooses a checkpoint at 45.8% mAP (the oracle is 47.0%) while the last checkpoint reaches 43.1%. Our proposed method still works when DAOD frameworks also optimize the prototype-based distance. 3) For other DA tasks. Our method can also work on other DA tasks, such as semantic segmentation. We conducted our DAS on the well-known DAFormer [29] method on GTA5 to Cityscapes Adaptation. It is shown that our DAS can choose a better checkpoint of the model with an average mIoU of 64.2% while the last checkpoint of the model only achieves 60.9% and the oracle checkpoint is at 65.9%.Although our method outperforms many unsupervised model evaluation methods in many DAOD benchmarks and methods, it still has considerable room for improvement. There are certain scenarios where our proposed method faces limitations, such as the inability to consistently select the best checkpoint and a lack of sufficient correlation between the proposed DAS and the ground truth detection performance of DAOD models on the target domain. To address these limitations, we could consider incorporating a more fine-grained distribution alignment metric to evaluate the distance across domains. On the other hand, we can explore methods that extend beyond the constraints of labeled data by leveraging few-shot labeled data in the target domain to help model evaluation.Another limitation is that our method requires the entire dataset of the source and target domains to calculate the evaluation score. It would be beneficial to develop a more data-efficient approach that can effectively evaluate the performance of DAOD models while minimizing the data requirements. Our Domain Adaptation Scoring (DAS) method has the potential to positively impact various downstream systems, such as autonomous driving and embodied AI. These systems frequently encounter previously unseen domains in real-world scenarios, and our method effectively addresses the unsupervised evaluation problem of domain adaptive object detection methods. However, it is important to acknowledge that the application of our method may also introduce potential negative impacts. For example, when applied to surveillance videos and medical images, privacy concerns may arise. It is essential to understand that these issues are not inherent to the technology itself but rather depend on the responsible and ethical use by human beings.Answer: [Yes] Justification: The code will be available on the website mentioned in the abstract. Guidelines: • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We provide a discussion about societal impacts and negative societal impacts of the paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: Though the reviews are mixed, the majority of the reviewers recommend acceptance and emphasize the importance of the topic (unsupervised model selection for domain adaptation), which has been previously under-explored in the literature. The authors propose a novel method for addressing this which performs close to optimal when assuming access to labels in the target domain. The AC team appreciates the value of the contribution and thus agrees with the majority of the reviewers that the paper should be accepted.

Reviews: review: paper_summary: paper_summary: This paper delves into an unsupervised evaluation problem in Domain Adaptation Object Detection (DAOD). To solve this problem, this paper proposes a method based on the flat minima principle, named Detection Adaptation Score (DAS), which can measure the flat minima without using target labels. Specifically, the proposed method is composed of a flatness index score (FIS) and a prototypical distance ratio (PDR) to assess the flatness and measure the transferability and discriminability of the models. Experiments validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: -	In line 49, the authors should explain why these methods fail to evaluate the object detection model.
-	In Eq.2, since KL term and IoU term are two metrics to evaluate the matching costs of two different tasks, object classification and box regression, why there is no a balance coefficient, such as d=KL-\lambda IoU. If \lambda=1 is the best?
-	The proposed FIS is easy coming to mind, which uses parameter perturbation to evaluate flatness minima without using target labels. And the proposed PDR to use prototype-based domain alignment is also used as a training method in the existing prototype-based domain alignment for DAOD, such as [1].
-	The proposed FIS and PDR are used for unsupervised evaluation. However, from another perspective, these methods can also be used as unsupervised training methods. If these kinds of methods are used for unsupervised training, can they be used as unsupervised evaluation methods as the same time? For example, if the benchmark DAOD framework is a prototype-based domain alignment method, can PDR, a prototype-based domain alignment evaluation method, serves as an unsupervised evaluation? As I can see in line 261, this paper only uses adversarial training or self-training methods for unsupervised domain adaptive training? More different kinds of benchmark DAOD framework are necessary to evaluate the scope of the proposed evaluation methods.

[1] Cross-domain detection via graph-induced prototype alignment. CVPR 2020.
questions: questions: See the weaknesses, especially the forth question.
limitations: limitations: Limitations had been discussed.
paper_summary: paper_summary: The paper introduces a new metric ("DAS") for unsupervised model selection in domain-adaptive object detection. DAS consists of two components: a flatness index scores that approximates the flatness of the loss landscape in the target domain by measuring prediction agreement across perturbed model parameters, and a prototypical distance ratio that measures the ratio of inter-class and intra-class feature dissimilarity. DAS is evaluated for model selection and hyperparameter optimization and correlates well with true model performance.
strength_weakness: strength_weakness: The writing quality should be improved (e.g. general grammar, typos such as "Faster RCCN", sometimes inconsistent terminology such as "Prototype Distance Ratio" / Prototypical Distance Ratio"). Apart from writing quality, my only significant concern is the apparent invariance of DAS performance towards lambda, which raises the question of whether the flatness index score and prototypical distance ratio measure different aspects in practice and the degree to which they are complementary.
questions: questions: How is PDR or d_inter defined for single-class benchmarks such as Sim10k?

DAS seems to work consistently very well for AT, while performance with other frameworks is less consistent (very apparent in Figure 4). Do you know why this might be the case?

It might make sense to include the oracle checkpoint in the main evaluation, especially tables 1-3. This would make it easier to see how much of the gap to the oracle is closed.

Please clarify what is meant by "ES" in table 2.

It would be interesting to see analogues for Figures 2 and 5 for Cityscapes and Sim10k in the appendix.
limitations: limitations: Limitations and societal impact are adequately discussed in the appendix.
paper_summary: paper_summary: This paper tackles the problem of model selection in unsupervised domain adaptation for object detection (DAOD). In DAOD, existing methods choose the models (checkpoints) using ground truth data on the target domain, which is impractical in real-world settings. To solve the problem, this paper proposes a model selection method called Detection Adaptation Score (DAS) based on the relationship between flat minima and generalization ability. The proposed DAS consists of two scores: Flatness Index Score (FIS) and Prototypical Distance Ratio (PDR). The FIS calculates the distance between the predictions before and after the perturbation of the model parameters. The PDR calculates the distance between the class prototypes on source and target domains. The experimental results demonstrate the positive correlation between the DAS and mAP. Also, the models chosen by the DAS outperform those chosen by other unsupervised model selection methods.
strength_weakness: strength_weakness: i) The experiments in this paper show that the proposed method works well on model selection in DAOD. However, I think the proposed method is not limited to DAOD; that is, it can be directly applied to other tasks such as UDA for image classification and semantic segmentation (except IoU calculation in Eq. 2). I would like to see whether the proposed method works well those tasks as well.

I have some concerns about the relationships between the flat minima principal and the proposed method as follows:

ii) The first term of the right-hand side in Eq. (1) is the difference between the risks (i.e., losses) of $h$ and its neighbor $h'$. In contrast, the proposed method calculates the variance in the outputs, not the risks. They are different measures, and the relationships between them are not addressed in this paper although there might be a correlation between them. This paper will be more convincing if mathematical relations between the difference in the risks and the variance in the outputs are derived.

iii) I understand that the PDR evaluates the transferability and discriminability because high PDR means that the prototypes of the same classes are well-aligned between the source and target domains. However, the relationship between better transferability (and discriminability) and flat minima is not clear in this paper. Why do better transferability and discriminability lead to flat minima?

iv) Although it was confirmed that each of FIS and PDR contributes to better model selection in terms of mAP, the evaluation of the correlations between each of FIS and PDR and flatness is missing. Similar to [7], the flatness can be evaluated as the change in risk using GT labels. This paper will be better if it provides the experiments to evaluate the correlations between each of FIS and PDR and flatness. The experiments can address my concerns ii) and iii) indirectly.
questions: questions: See i), ii), and iii) in Weakness.

i) Does the proposed method work well in other UDA tasks as well?

ii) Is it possible to derive the mathematical relationship between the difference in risks and the variance in outputs?

iii) Why do better transferability and discriminability lead to flat minima?
limitations: limitations: Limitations are adequately addressed in this paper.
paper_summary: paper_summary: This submission provides a strategy to select an appropriate domain adaptive object detection model without access to labels of target domain. The basic premise is the minima flat to instruct the selection, because minima flat means better generalization. Experiment indicate the effect of the design based on their detection adaption score.
strength_weakness: strength_weakness: 1. This paper may neglect a fact that in real application, target labels are often known for testing the provided model, and in fact the training labels may not be accessible. So, I do not think this strategy is really necessary.

2. This paper is essentially a DG domain generalizationt task, where target domain is unseen. So we can only select (train) a better model by using existing source domain data. Flat minima has also been used in DG method to instruct how to train a domain generalization model. 

3. The proposed selection strategy is orthogonal to the high-level task, such as object detection. Therefore, why focus only on the domain adaptive object detection task? More tasks should also be tested such as image classification, semantic segmentation and others.

4. In experimental comparison, the authors compare the last checkpoint, which may not be fair. 

5. Lacking comparisons to many SOTA DAOD models.

6. What is the computational overhead?
questions: questions: 1. It is similar to DG and therefore the proposed strategy is not novel athought the motivation sounds good.
2. Necessity of such motivation. 
3. More experiments on other related tasks.
limitations: limitations: 1. The motivation is interesting but does not well match real-world applications. Test labels should be seen even not too much. Otherwise, it is just a domain generalization scenario, which means the novelty of flat minima is limited.
paper_summary: paper_summary: The paper introduces the Detection Adaptation Score (DAS), an unsupervised approach for selecting optimal models in Domain Adaptive Object Detection (DAOD) without target domain labels. DAS is based on the flat minima principle and includes the Flatness Index Score (FIS) and Prototypical Distance Ratio (PDR) to evaluate model generalization. Experiments on various benchmarks demonstrate DAS's effectiveness in checkpoint selection and hyperparameter tuning, outperforming existing methods. The perspective of enhancing model selection for DAOD is interesting and novel. It attempts to tackle an overlooked yet important limitation of existing DAOD methods and shows potential for other computer vision tasks.
strength_weakness: strength_weakness: 1. It seems that the proposed method is only tested on the Faster RCNN baseline.
2. The performance gains on some detectors under some settings are limited.
3. The method is only tested on object detection, lacking an analysis for more fine-grained tasks, e.g., instance segmentation.
questions: questions: I didn't find major issues with the paper, and I only have some minor confusion, as listed below.

1. Can the proposed method be extended to other baselines for DAOD, such as the FCOS-based method [1] and the DETR-based method [2]? It will be great to enhance the adaptability of the method further.

2. Could the authors explain why DAS does not give obvious gains in some settings? For example, DAS gives no gains for CMT on Real-to-Art Adaptation in Table 1.

3. Can the method be used for fine-grained tasks, such as segmentation?

4. Consider giving a more comprehensive review of some advanced cross-domain object detection works [1-4].

[1] Deng, J., Xu, D., Li, W., & Duan, L. (2023). Harmonious teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23829-23838).

[2] Huang, W. J., Lu, Y. L., Lin, S. Y., Xie, Y., & Lin, Y. Y. (2022, July). AQT: Adversarial Query Transformers for Domain Adaptive Object Detection. In IJCAI (pp. 972-979).

[3] Zhou, W., Fan, H., Luo, T., & Zhang, L. (2023). Unsupervised domain adaptive detection with network stability analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6986-6995).

[4] Li, W., Liu, X., & Yuan, Y. (2023). Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7), 9022-9040.
limitations: limitations: No potential negative societal impact.

Comments: title: "Rating 3: Reject" may be biased
The setting may be useless and why not do general machine learning task
Response to Reviewer 4YNt
impractical and insufficient evaluation on the confidence
comment: We greatly appreciate your response and the recognition of our work. We will refine our work in accordance with your recommendations to further improve the final version.
Thank you for your timely and valuable feedback. We are grateful for your recognition of our work. We will carefully revise the final version based on your valuable feedback.
We sincerely appreciate the comments and the recognition of our work from reviewer XtC2. We are also committed to continuously advancing DAOD towards more practical applications.
Thanks for the timely response.

We would like to clarify that this work does not focus on replacing the target data in real-world applications. In contrast, our work focuses on how to achieve unsupervised model selection for domain adaptive object detection (DAOD). The goal of unsupervised domain adaptation (UDA) is an important and widely studied research problem in the machine learning community (which was also pointed out by Reviewer XtC2). We believe the unsupervised model selection approach proposed in this work would be interesting to researchers in the UDA field, especially to those working on DAOD. 

We appreciate the suggestions on testing on the domain generalization setting. We would like to clarify that 
DG is often treated as a different task from UDA, for which the models and benchmark datasets are different. While testing on the DG setting
is beyond the scope of this paper, we agree it is an interesting idea, and would like to explore this direction in the future. We sincerely look forward to your recognition on the motivation and contributions of this work.
Thank you for your prompt response and your recognition of our work. We greatly appreciate your time and effort in reviewing our submission. We will carefully incorporate your feedback to further enhance the quality of our final work.
**I agree with Reviewer 4YNt and believe the "Rating 3: Reject " may be biased.** The criteria for "Reject" include "technical flaws, weak evaluation, inadequate reproducibility, and/or incompletely addressed ethical considerations." However, this paper is technically sound, with no apparent flaws. Extensive experiments validate its effectiveness on standard benchmarks, and they provide sufficient reproducibility information without any potential ethical issues. **Therefore, the paper does not meet any of the "Reject" criteria.**

Additionally, I am puzzled by the reviewer's concerns about the "uselessness of the DAOD setting," as **there is ample literature supporting the practicality and validity, which has also been recognized by the ML community [1-4]**. I also believe that extending to DG is beyond the scope of this paper and should be considered for the journal-extended version as future work.

[1] Learning Domain-Aware Detection Head with Prompt Tuning NeurIPS 2023

[2] Learning Domain Adaptive Object Detection with Probabilistic Teacher ICML 2022

[3] Decoupled Adaptation for Cross-Domain Object Detection ICLR 2022

[4] Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection NeurIPS 2021
I would like to thank the authors for their careful rebuttal. Because my concerns were well addressed, I'm raising my rating to Borderline Accept.
Reviewer jxDH, do you have any additional questions or feedback?
Based on the author's responses to my questions and the additional information provided in the rebuttal, I'm raising my score to a strong accept.
Thanks for the rebuttals from the authors, which address my concerns to some extents. I tend to keep my original score, which leans to positive already.
From the feedback of the authors and the reviewer 4YNt, I may maintain the score because this strategy can not replace the target data for real application. If you would like to make sure your selected model is blind good to the real test scene (this test scene may not be unique), you should test your method in more target domains once your model is fixed. I believe that the performance for different target domains may not be good, which also demonstrate this work may be useless to some extent. In UDA or DG scenarios, the target domain or test data should be used, but for real application, more test data of different distributions should be used. 
Additionally, this submission should also be used for general machine learning problems by supposing test data to be unavailable, rather than UDA, a sub branch of ML.
Dear Reviewer ood2,

We hope this message finds you well. As the discussion period is ending soon, we would like to bring to your attention that we have submitted our response to your questions. We carefully consider your valuable feedback and suggestions, and hope the response well addressed your concerns.

We are truly thankful for the comments on our work.
If you find our responses to be satisfactory, we would greatly appreciate it if you could take this into account when considering the final score.

Best regards,

Authors of Submission 7161
Dear Reviewer jxDH,

We hope this message finds you well. As the discussion period is ending soon, we would like to bring to your attention that we have submitted our response to your questions. We carefully consider your valuable feedback and suggestions, and hope the response well addressed your concerns.

We are truly thankful for the comments on our work.
If you find our responses to be satisfactory, we would greatly appreciate it if you could take this into account when considering the final score.

Best regards,

Authors of Submission 7161
We thank the reviewer for the constructive comments. We agree with the comments that the target domain labels might be infeasible to obtain in many real-world domain adaptation cases and our method is useful for many real applications. Thank you again, we are happy to receive your comments to further improve the quality of our paper.
Thank you for your comments, and we would like to further clarify a few points as follows. 

First, it is true that testing with labeled data is the ideal way to evaluate a model. However, annotating target domain data is time-consuming and labor-intensive, and the target domain labels are often unavailable, especially when the target domain changes rapidly as pointed out by Reviewer 4YNt. Therefore, how to effectively evaluate the domain adaptation models in an unsupervised way is important for real-world applications. In our work, we address the UMS problem for DAOD through the flat minima principle. Different from the traditional flat minima methods that require labeled data to evaluate how well a model is located at the flat minima region, we show via the generation bound that the flatness can be deemed as model variance while the minima depends on the domain distribution distance for the DAOD task. We believe our work can inspire other researchers in the community and make the DAOD methods more practical in real-world applications. 

Second, regarding the experiments, we evaluate the proposed method on the widely-used DAOD benchmark datasets of multiple domain adaptation scenarios, including real-to-art adaptation, weather adaptation, and synthetic-to-real adaptation over serval representative DAOD methods. We also compare many model selection baselines and the proposed method outperforms these methods, which demonstrates the effectiveness of our method on unsupervised model selection. We appreciate your comments for further experiments on more target domain data and sincerely look forward to your specific suggestions for the experimental design. We would like to include these results in the final version to strengthen our work. 

We thank the reviewer for the opportunity to further clarify the motivation behind our work. We sincerely hope that our response can address the concerns from the reviewer and the reviewer can re-evaluate this paper based on our response.
> (1) In real application, for selecting or testing a model, some labeled target data from one or more target domains should be tested to guarantee the reliability of the deployed model. From this view, the proposed work may be useless.

I agree with you that target domain labels are necessary to give any kind of guarantees on the target domain. However, this might simply not be feasible, for example if the target domain is continuously shifting faster than labels can reasonably be obtained. In this setting, this method would select a best-effort model without any kind of guarantees, which is still useful for many real applications. I don't feel like the authors overclaim by suggesting guaranteed reliability on the target dataset.

> (2) From another view, if you think in real application the selected model "must be" reliable without need of any labeled target domain data, I think the current evaluation experiments are very insufficient. At least, you should test your selected model on more target domain data to make sure if your selected model is really reliable in real application even without using any labeled target data.

I'm not sure I understand how one would test the selected model on more target domain data. I feel like evaluating mAP on the target datasets, like the authors did, is the best one can do given the limited scope of existing UDA benchmark datasets.
Thanks for the authors' feedback. Although the authors would like to present a strategy for model selection, I do think this is impractical in real application and the evaluation is also insufficient to support the conclusion. 

(1) In real application, for selecting or testing a model, some labeled target data from one or more target domains should be tested to guarantee the reliability of the deployed model. From this view, the proposed work may be useless.

(2) From another view, if you think in real application the selected model "must be" reliable without need of any labeled target domain data, I think the current evaluation experiments are very insufficient. At least, you should test your selected model on more target domain data to make sure if your selected model is really reliable in real application even without using any labeled target data.
Thanks for your timely responses. We sincerely appreciate your constructive comments and valuable feedback, which enhance the quality of our paper. We will revise our paper in the final paper as you suggested.
Thank you for the clarification and effort. I have read all the author's responses and other reviews, which addressed my concerns and highlighted additional strengths and potential in tasks beyond DAOD, such as segmentation and DG. Therefore, I am happy to raise my score.