Alleviating the Equilibrium Challenge with Sample Virtual Labeling for Adversarial Domain Adaptation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 08:57:43

Abstract: Numerous domain adaptive object detection (DAOD) methods leverage domain adversarial training to align the features to mitigate domain gap, where a feature extractor is trained to fool a domain classifier in order to have aligned feature distributions. The discrimination capability of the domain classifier is easy to fall into the  local optimum due to the equilibrium challenge, thus cannot effectively further drive the training of feature extractor. In this work, we propose an efficient optimization strategy called \underline{V}irtual-label \underline{F}ooled \underline{D}omain \underline{D}iscrimination (VFDD), which revitalizes the domain classifier during training using \emph{virtual} sample labels.  Such virtual sample label makes the separable distributions less separable, and thus leads to a more easily confused domain classifier, which in turn further drives feature alignment. Particularly, we introduce a novel concept of \emph{virtual} label for the unaligned samples and propose the \emph{Virtual}-$\mathcal{H}$-divergence to overcome the problem of falling into local optimum due to the equilibrium challenge. The proposed VFDD is orthogonal to most existing DAOD methods and can be used as a plug-and-play module to facilitate existing DAOD models. Theoretical insights and experimental analyses demonstrate that VFDD improves many popular baselines and also outperforms the recent unsupervised domain adaptive object detection models.

Main paper: 
 introduction: 
 Object detection has shown great success in the deep learning era [2, 3, 30, 37, 38, 53] . However, these methods assumes that source domain and target domain have the identical data distribution, which is not applicable in real-world scenarios. In addition, collecting Figure 1 : Illustration of our motivation. In the case of Penrose triangle, even identical objects are challenging to ideally deceive domain classifiers (i.e., probabilities of domain classifier close to 0.5) due to variations in target angles, which in turn provides less driving power to the feature extractor for alignment and prevents effective optimization. large amounts of labeled samples is a time-consuming and laborious project. Unsupervised Domain Adaptation (UDA) [8, 12, 26, 34] serves as a promising solution to solve this problem by transferring knowledge from a labeled source domain to a fully unlabeled target domain.Regarding UDA, [1] conducted theoretical analysis, demonstrating that minimizing the domain divergence between the source and target domains can effectively lower the upper bound of the target domain error. Many UDA for object detection (DAOD) methods [5, 19, 46, 48, 51] have recently attempted to learn domain invariant feature representations within de facto detection frameworks, e.g., Faster RCNN [38] and Deformable DETR [54] . Inspired by the Generative Adversarial Networks (GANs) [16] , adversarial learning has been successfully applied for DAOD. The core idea behind adversarial learning methods involves training a domain classifier (i.e, D) to distinguish between the source (i.e, D 𝑠 , domain label=1) and target (i.e, D 𝑡 , domain label=0) domains, while simultaneously training the feature generator (i.e, G) to minimize the feature discrepancy between the domains, aiming to fool the discriminator in a minmax two-player game. Formally,EQUATIONWhere 𝑒𝑟𝑟 (𝑑 (𝑥)) denotes the prediction error of the domain classifier D and min 𝑑 ∈ D 𝑒𝑟𝑟 (𝑑 (𝑥)) means the minimum prediction error of an ideal domain classifier. It is worth noting that the domain divergence 𝑑 H (D 𝑠 , D 𝑡 ) is inversely proportional to the error rate of the domain classifier D. All of the existing DAOD methods generally believe that directly fooling the domain discriminator to minimize the H -divergence will help to align the domains. However, [31] claim that there is no guarantee that the two domains can be perfectly aligned due to the equilibrium challenge in adversarial learning.Taking the Penrose Triangle as an example in Figure 1 , simply due to different visual angles, the source domain may be recognized as a '⌞' while the target domain may be recognized as '△'. Consequently, the domain classifier cannot be impeccably deceived, leading to domain classification outcomes deviating significantly from the ideal equilibrium point 0.5. Such imperfect classification by the domain classifier in turn provides less driving power for alignment in the feature extractor, impeding effective optimization, even though there are still not aligned samples in the feature space. In other words, matching the feature distributions between domains inevitably results in sub-optimal risk due to the equilibrium challenge of adversarial learning, which makes it impossible to completely confuse the domain classifier during training. This unaligned phenomenon would be more severe in cross-domain object detection, given the complex combinations of various objects and the differentiated scene layouts between domains.To further verify the above observations, we conduct some experiments on real cross-domain object detection scenarios. Figure . 2(a) shows one representative example of the probability distribution of instance-level features extracted by DAF [5] on the task from Cityscapes [6] (i.e., domain label=1) to Foggy Cityscapes [40] (i.e., domain label=0). Ideally, the probabilities of instance-level features obtained by adversarial alignment are evenly distributed around 0.5. However, in Figure 2 (a), we can clearly observe that the predictions of source and target are distributed around 0.5260 and 0.4743 (by observing the horizontal and vertical coordinates), respectively. Therefore, a mechanism that could ideally fool the domain classifier to re-energize the adversarial training of feature generator is highly desired. This issue is rarely investigated in previous works, but it is a meaningful topic worthy of attention.In this paper, to solve the aforementioned problem, we propose a simple but effective strategy named Virtual-sample Fooled Domain Discrimination (VFDD) which aims to fool the domain classifier to re-energize the feature generator during the training. Particularly, as illustrated in Figure 3 , instead of fooling the domain classifier using real source and target samples, we propose to exploit virtual domain samples, where we copy the "unaligned" samples, i.e., those far from the equilibrium point 0.5, as virtual samples, in each mini-batch to be optimized/trained. The rational behind is that, for the VFDD adversarial alignment, using the same instance (i.e., virtual vs. real) features of the source or the target domains for alignment can naturally maximize the error rate of the domain classifier D, which enables the features domain-invariant in the course of the optimization. As a result, the detector will produce more confusing instance features. One evidence is shown in Figure . 1(b), in which we implement the VFDD on DAF [5] . We can see that the probability distributions of different domains are more similar, which means that using the VFDD can better bridge the domain gap. Besides, we provide a theoretical analysis of the proposed Virtual-H -divergence that it has a smaller upper-bound than the standard H -divergence. Therefore, the effectiveness of VFDD is guaranteed. The proposed VFDD is universal and can be quickly embedded in alignment-based unsupervised domain adaptive object detection In column 1, the horizontal axis corresponds to the probability distribution of the source domain, while the vertical axis represents that of the target domain. The intensity of the color gradient in the plot reflects the density of the distributions, with brighter colors indicating denser regions. In column 2, orange signifies the source domain and blue signifies the target domain. Clearly, the probability distributions of the source and target become closer to the ideal saddle point of 0.5 and the feature distributions of the source and target domains become more consistent with ours VFDD. methods. The main contributions and novelties of this paper are summarized as follows:•  
related work: 
 Classification Based UDA. Unsupervised domain adaptation (UDA) is extensively studied, which aims to apply model learned from labeled source domain samples to relevant and unlabeled target domain samples [18, 33, 45] . At present, the mainstream approaches for UDA tend to realize domain alignment by adversarial learning domain-invariant feature representations across domains. Domain Adversarial Neural Network (DANN) [14] is a representative work, where a domain classifier is connected to the feature extractor via a gradient reversal layer (GRL) [12] . Subsequently, CDAN [31] proposes an adversarial adaptation model for the discriminative information transmitted in the prediction of the classifier. GVB [9] learns the domain-invariant feature representations by applying the gradually vanishing bridge mechanism on the feature generator. DWL [50] dynamically weights the learning losses of alignment and discriminability by introducing the degree of alignment and discriminability. Nevertheless, these studies focus on the image classification and segmentation (a.k.a. pixel classification), rather than the task of object detection, which faces more challenges.Object Detection Based UDA. Driven by domain adaptation theory, researchers have proposed various approaches to alleviate the problem of domain gap in cross-domain object detection. DAF [5] proposes an adaptive Faster-RCNN [38] method, which achieves image-level and instance-level feature alignment by using adversarial gradient reversal for the first time. Building upon this, MAF [19] introduces a hierarchical adversarial feature alignment strategy that reduces domain disparity at different scales. HTCN [4] employ CycleGAN for data augmentation, generating intermediate domain images to facilitate model alignment between source and target domains. VDD [49] tackles the problem by disentangling domaininvariant and domain-specific representations using vector decomposition, while also exploring the extraction of instance-invariant features [48] . IDF [28] propose a non-adversarial domain discriminator to extract domain-specific features. Additionally, PTMAF [21] and PAATF [22] introduce additional constraints during the adversarial learning stage. Recently, regarding the Transformer object detector, existing adaptation techniques for DETR [3] predominantly rely on model-based approaches SFA [47] , aiming to reduce the distribution shift between different domains through sequence feature alignment. However, as the optimization of the min-max game in domain adversarial adaptation proceeds, domain classifier is inevitably prone to falling into local optimum due to the equilibrium challenge, which reduces the driving power for the feature alignment. In this work, we introduce the virtual source and virtual target samples to disrupt the domain classifier from being trapped in local optima, enabling the domain classifier to be perfectly fooled, thus reenergizing the feature extractor to acquire more domain-invariant features. 
our vfdd method: 
 Following the general formula of domain adaptive object detection, we denote the labeled source domain as D 𝑠 = {𝑥 𝑠 𝑖 , 𝑦 𝑠 𝑖 } 𝑛 𝑠 𝑖=1 with 𝑛 𝑠 samples covering 𝐶 classes, and the unlabeled target domain asD 𝑡 = {𝑥 𝑡 𝑖 } 𝑛 𝑡 𝑖=1with 𝑛 𝑡 samples that belong to the same 𝐶 classes. D 𝑠 and D 𝑡 share the same feature space and category space, but have different data distributions. DAOD aims to utilize the labeled data D 𝑠 and unlabeled data D 𝑡 to learn a deep model, which can accurately predict the class label of samples in the target domain.Adversarial learning has proven to be an effective method for domain alignment, starting from Domain Adversarial Neural Network (DANN) [14] . The basic idea is to use the feature generator G to generate features to trick the domain discriminator D. Then the domain classifier predicts whether the generated feature by G is from the source domain or the target domain. The training of domain alignment is achieved through the game between generator and domain classifier. The parameter 𝜃 𝑔 of generator G and the parameter 𝜃 𝑑 of domain discriminator D are optimized by the following domain alignment objective function.EQUATIONIn order to improve the classification performance of the target domain samples, we must first ensure that the classifier C can correctly classify the samples from the source domain. Thus, the supervised classification loss can be described as:EQUATIONwhere L 𝑐𝑒 is the standard cross-entropy loss function. During training stage, the existing methods usually jointly optimize the two objective functions (L 𝑑𝑜𝑚 and L 𝑐𝑙𝑠 ). The overall minimax objective function is:EQUATIONwhere 𝜃 𝑔 , 𝜃 𝑑 , 𝜃 𝑐 denote the parameters of feature generator, domain discriminator and category classifier, respectively. 𝜂 denotes a hyper-parameter for balancing the losses.During the training, the discrimination capability of the domain classifier is easy to fall into the local optimum due to the equilibrium challenge. This in turn negatively influences the effectiveness of the optimization towards feature alignment. Thus, with the adversarial optimization objectives as in Eq.4, the networks converge to states that are less than optimal. Even though the distributions are not fully aligned, the deteriorated discrimination capability of the domain classifier cannot further efficiently drive the feature alignment.We introduce the virtual source and virtual target samples for the unaligned samples to perfectly fool the domain classifier to re-energize the feature extractor to acquire more domain-invariant features, which then formulates the proposed VFDD technique. As described in Eq.1, the domain distance between source and target is inversely proportional to the error rate of the domain classifier D. Intuitively, larger domain prediction error means smaller domain discrepancy. Thus, in order to fully fool the domain, we propose the Virtual-H -divergence which aims at minimizing the domain discrepancy without information loss. Instead of directly calculating the distance between D 𝑠 and D 𝑡 via H -divergence (i.e., 𝑑 H (D 𝑠 , D 𝑡 ) in Eq.1 ), we introduce the concept of virtual domain samples and learn domain invariant features between virtual and real domain to implicitly align the features across source and target domains. Specifically, we introduce the virtual source domain D𝑠 with domain label 𝑑 = 0 instead of 𝑑 = 1 and the virtual target domain D𝑡 with domain label 𝑑 = 1 instead of 𝑑 = 0, by simply copying the feature matrices of real source D 𝑠 (𝑑 = 1) and real target D 𝑡 (𝑑 = 0) for easier manipulation. We propose to measure the distances between D 𝑠 and D𝑠 and the distances between the D 𝑡 and D𝑡 by:EQUATIONDue to the 𝑒𝑟𝑟 (•) refers to the cross entropy loss, the Virtual-Hdivergence can be written as:EQUATIONwhere ⊕ denotes the union operator. Because the features of the (S ⊕ T ) and ( Ŝ ⊕ T ) are identical except for the domain labels, the error rate of the domain classifier D is very large. This means the value of 𝑑 H (S ⊕ T , Ŝ ⊕ T ) is much smaller than 𝑑 H (S ⊕ S, T ⊕ T ). Formally,EQUATIONIn this way, such virtual sample makes the separable distributions less separable, and thus leads to a more easily confused domain classifier, which in turn further drives feature alignment. As revealed in Figure 2 (a) and (b), after applying VFDD, the domain-invariant information extraction capability of the feature extractor is stronger than the baseline. Based on the above novel Virtual-H -divergence, the expected error on target samples 𝜖 T is bounded as,EQUATIONwhere 𝜖 S is the expected error on the source domain, 𝜆 is the ideal joint hypothesis. In this way, the upper bound of the expected target error, i.e., 𝜖 T can be effectively reduced in our work. We show the derivation of Inequality 8 in the supplementary material.With the general DAOD (i.e., Eq. 4) described in Section 3.1 and the VFDD based Virtual-H -divergence strategy in Eq. 6, a balanced DAOD model can be trained and implemented.Measurement of Alignment for a Sample. To virtual copy the "unaligned" samples (i.e., as shown in Figure 3 (e)), we need to evaluate whether a sample is "unaligned" or not. As we know, when a sample is more aligned, it is harder for the domain classifier to identify its domain and has a higher uncertainty w.r.t. to the predicted domain label of this sample. Correspondingly, the entropy of domain classification of this sample is in general higher. Therefore, we measure how well a target sample is aligned with the source domain by simply using the entropy of domain classification, where 𝑝 denotes the probability of predicting a sample 𝑥, i.e., 𝑝 = D (G(𝑥). The larger the entropy value, the larger the ambiguity/uncertainty of the discriminator has when identifying which domain it belongs to, and the more aligned the sample is. Note that there may exist other more accurate or advanced measurement metrics, but this is not the focus of this paper and we use this simple one here.EQUATIONUnaligned Sample Selection and Virtual Copy. For a given sample, when it can be well distinguished by the domain classifier, it could be considered approximately unaligned. In other words, when the entropy of the domain classifier prediction is smaller than a threshould 𝜏, we define it as a "unaligned" sample and introduce a corresponding virtual sample (as shown in Figure 3 (e)). 𝜏 is a hyper-parameter that controls the strictness of the definition of "unalign". We will study its influence in the experiments.When to start VFDD. Intuitively, we could start our virtual copy strategy to fool the domain classifier whenever its discrimination capability begins to fall into local optimum. In general, at the early training stage, the optimization of the domain discriminator persistently improves its discrimination capability. It is thus unnecessary to enable the virtual copy strategy. Inspired by the popular learning rate (lr) adjustment algorithms [36] which adjust the learning rate if no improvement is seen for a "patience" number of epochs (where "patience" is usually set to 2 to 10) , we start VFDD if no improvement of the discrimination capability is seen for a "patience" number of epochs and we denote this hyper-parameter as 𝐾.Adversarial Training. As is done in previous adversarial domain adaptation methods, we perform mini-batch level optimization where a batch consists of both source and target domain samples. Once VFDD is activated, for each mini-batch, we first check whether each sample should be copied as a virtual sample and copy them if deemed so. Then the adversarial training is performed between the virtual and real domain labels. The novel domain alignment objective function for the unaligned samples can be expressed as:EQUATIONThus, the domain alignment objective function for the all source and target samples can be expressed as:EQUATIONwhere L 𝑎𝑑𝑣 (𝜃 𝑔 , 𝜃 𝑑 ) and L 𝑉 𝑎𝑑𝑣 (𝜃 𝑔 , 𝜃 𝑑 ) are calculated by the Eq.2 and Eq.10, respectively. The domain classifier is updated by maximizing the domain classification loss (i.e., adversarial loss as in Eq.11) over the updated source set and target set. Simultaneously, the feature extractor is trained to acquire more domain-invariant (confused) features. Note that the virtual copy strategy has no impact on object classification loss.With the thoughts above, the overall training objective is:EQUATION)The pseudo code is presented in the Algorithm 1, showing the training process of VDFF+DAOD. This endeavor will contribute to a more precise assessment of our research. The code will be made publicly available.Require: A source set D 𝑠 = {𝑥 𝑠 𝑖 , 𝑦 𝑠 𝑖 } 𝑛 𝑠 𝑖 , including the images and labels (i.e., bounding box coordinates and category labels). An unlabeled target training set D 𝑡 = {𝑥 𝑡 𝑖 } 𝑛 𝑡 𝑖 . Ensure: A detector adaptive to different domains. 1: Initialization. The iteration counter 𝑛, the total iteration number 𝑚. Take the samples from the source set D 𝑠 , including the image 𝑥 𝑠 𝑖 and label 𝑦 𝑠 𝑖 . Take the samples 𝑥 𝑡 𝑖 from the target set D 𝑡 .Feed the source sample and corresponding labels into the VFDD+DAOD part, and compute L 𝑐𝑙𝑠 in Eq.( 3) in our manuscript.Feed the source and target samples into the VFDD+DAOD part. Unaligned samples selection and virtual copy with the threshold 𝜏, then compute the adversarial learning loss L 𝑎𝑙𝑙 𝑎𝑑𝑣 in Eq.( 11) in our manuscript.Compute the total loss Eq.( 12) and gradient, and update the parameters of the VFDD+DAOD part.8:𝑛 = 𝑛 + 1 9: until 𝑛 = 𝑚 
experimental setup: 
 Datasets and Settings. We test our method on Cityscapes [7] , Foggy Cityscapes [41] , Sim10k [25] , and BDD100k [52] . These datasets cover diverse and challenging scenarios for domain adaptation tasks:• Weather Adaptation. In this scenario, we use Cityscapes as the source dataset, consisting of 2,975 training images and 500 evaluation images. The counterpart, known as Foggy Cityscapes, is derived from Cityscapes through a fog synthesis algorithm. These datasets enable us to assess the effectiveness of our method in adapting object detection models from clear weather to foggy conditions. Implementation Details. We benchmark our approach against cutting-edge domain adaptation methods across two categories: (1) The Faster RCNN series, including DAF [5] , MAF [19] , ATF [20] , HTCN [4] , UMT [10] , PAATF [22] , PTMAF [21] , and IDF [28] . (2) The Deformable DETR series, encompassing SFA [47] . In addition, we also compare our VFDD with the image classification task, and more details will be presented in the Supplementary.We validate the effectiveness and versatility of our method against various baselines, including DAF, MAF, IDF, and SFA. By default, we employ ImageNet pre-trained ResNet-50 [17] and VGG-16 [42] as CNN backbones in all experiments. Aligning with the Faster RCNN series, we train the network using the SGD optimizer with a momentum of 0.9 and a weight decay of 5 × 10 -4 . The initial learning rate is set to 1 × 10 -3 and is reduced to 1 × 10 -4 after 5 epochs. A total of 15 epochs are conducted, with a batch size of 2 maintained throughout. In line with the Deformable DETR series, we utilize the Adam optimizer [27] for training over 50 epochs. The learning rate is initialized at 2 × 10 -4 and reduced by a factor of 0.1 after 40 epochs. A batch size of 4 is employed consistently in all experiments. All these experiments are conducted using NVIDIA Tesla V100 GPUs.Weather Adaptation. To assess the reliability of object detectors under varying weather conditions, we conduct cross-domain transfers of models from Cityscapes to Foggy Cityscapes. The outcomes are presented in Table 1 . Notably, the application of VFDD yields marked improvements in mAP for various methods. Specifically, DAF, MAF, IDF, and SFA achieve mAP values of 31.9%, 36.8%, 43.5%, and 43.0%, respectively, after VFDD integration. Furthermore, the results demonstrate that VFDD notably amplifies the cross-domain performance of Deformable DETR, showcasing a substantial absolute mAP gain of 14.5% (28.5% vs. 43.0%). These promising results highlight the ability of our method to improved performance in unsupervised domain adaptation for object detection tasks. Leveraging economical yet precise simulation datasets has proven to elevate the performance of object detectors. Nevertheless, this approach introduces a significant challenge in the form of a substantial inter-domain gap. In the context of synthetic-to-real adaptation, we have evaluated the efficacy of VFDD, our proposed method, as outlined in Table 2 . We can see that our VFDD significantly outperforms the baseline DAF, MAF, IDF, and SFA by 2.4%, 1.9%,1.2% and 2.1%,respectively.Scene Adaptation. In real-world applications, such as autonomous driving, scene layouts are dynamic and subject to frequent changes. Consequently, model adaptability to scene variations becomes pivotal. VFDD, our proposed method, underscores its effectiveness in scene adaptation, as evidenced in Table 3 , where it achieves state-of-the-art results (SFA+VFDD 31.4%). Notably, performance improvements are observed across all categories. Detection Results. We present visualizations of SFA+VFDD outcomes on three target domain datasets: Foggy Cityscapes, Cityscapes, and BDD100k. These visuals are accompanied by ground truth and previous SOTA methods. In Figure 4 (a), Row 1 (Cityscapes to Foggy Cityscapes), SFA+VFDD exhibits enhanced recall and more accurate classification in scenarios with dense fog occlusion. In Row 2 (Sim10k to Cityscapes), our approach even mitigates label misalignment (car vs. truck) to a certain extent without supervision. In Row 3 (Cityscapes to BDD100k), our method effectively classifies and locates objects, even under heavy occlusion or challenging small sizes. These visuals match quantitative findings, confirming VFDD's effectiveness in mitigating domain shifts within the UDA Transformer detector. t-SNE Distribution Results. For clear analysis, we do experiments on our scheme VFDD for design choices study on the weather adaptation.Influence of Threshold 𝜏. As described in Section 3.3, we employ a hyper-parameter 𝜏 as the threshold to determine whether a sample is "unaligned". We study its influence in Figure 5 (a). We can see that a superior performance is achieved when 𝜏 ranges from 0.63 to 0.69 on two schemes. Similar trends are observed on other datasets. We set 𝜏 = 0.68 on all datasets. Influence of 𝐾. As described in Section 3.3, we start our VFDD if the domain classifier fall into the local optimum. illustrates the impact of 𝐾. We find optimal performance when 𝐾 falls between 4 and 6. If 𝐾 is too small, assessing improvement becomes unreliable due to noise sensitivity. Conversely, if 𝐾 is too large, the optimization strategy's full potential cannot be harnessed.Comparison with RADA RADA [24] utilizes dynamic domain label adjustments for the 'well-aligned' samples to re-energize the domain classifier. Conversely, our VFDD uses a unique strategy by giving 'unaligned' samples virtual labels to deceive the domain classifier. This encourages the feature generator to produce features that are indistinguishable between domains, leading to better alignment. As depicted in Figure 6 , VFDD outperforms various baseline methods, exhibiting positive outcomes across diverse categories. Conversely, RADA occasionally yields negative outcomes. This unequivocally demonstrates VFDD's superiority in enhancing learning and adaptation across domains compared to RADA.Experimental Results on Classification To showcase the effectiveness of VFDD, we extend our experimentation to the image classification datasets of Digits, as presented in Table 4 . The table reveals that, under the optimization coordination of VFDD, the classification accuracy of MCD has reached 96.3%, 96.5%, and 95.1% on tasks 𝑀 → 𝑈 , 𝑈 → 𝑀, and 𝑆 → 𝑀 respectively. This demonstrates that VFDD is a powerful auxiliary tool to help to achieve better object Classification performance. Further, we can observe that our method VFDD can be easily plugged and played in the existing alignment-based UDA methods to enhance their recognition performance on image classification task.Convergence Analysis We present the convergence curves of test accuracy with respect to the number of iterations on tasks of 𝑈 𝑆𝑃𝑆 → 𝑀𝑁 𝐼𝑆𝑇 as shown in Figure 7 M → U U → M S → M Average DANN [13] 80.3 77.8 73.5 77.2 DRCN [15] 91.8 73.7 82.0 82.5 CoGAN [29] 91.2 89.1 --ADDA [43] 89.4 90.1 76.0 85.2 CAT [11] 90.6 80.9 98.1 89.9 TPN [35] 92.1 94.1 93.0 93.1 CDAN [32] 93.9 96.9 88.5 93.1 CyCADA [23] 95.6 96.5 90.4 94.2 MCD [39] 94. MCD combined with VFDD. Clearly, as training progresses, the MMD value gradually decreases, indicating a diminishing disparity between the source and target domains. Under the influence of VFDD, the MMD value is further diminished compared to the baseline, highlighting the efficacy of VFDD in narrowing the gap between domains. This outcome also validates the effectiveness of our proposed Virtual-H -divergence (i.e., Eq. 6). 
conclusion: 
 In this paper, we propose an effective optimization strategy for adversarial domain adaptation, termed as Virtual-sample Fooled Domain Discrimination (VFDD), which aims to prevent the domain classifier from getting stuck in local optima and simultaneously enhance feature alignment. We achieve this by copying the "unaligned" samples as virtual domain samples, which encourages the exploration of virtual domain labels, disrupting potential local optima entrapment of the domain classifier, and revitalizing the feature extractor in a dynamic manner. Extensive experiments across diverse benchmarks and various networks have consistently showcased the potency and broad applicability of our VFDD strategy. These results underscore its effectiveness in achieving robust and generalized adversarial domain adaptation. We aim to inspire further research to infuse greater dynamism into adversarial domain adaptation, thereby enhancing training effectiveness. 
 
 Appendix:  


Decision: The paper receives consistent weak acceptance ratings from all the reviewers. The paper presents a valid and technically sound solution for the adversarial domain adaptation. However, two of the reviewers reveal the novelty concern about the method. I read into the paper and find the similar concern, e.g., when adversarial learning goes to a stable state, which usually means the source and target domain are sort of well aligned. It would be good in the final version, the authors can address this concern, e.g., by pivoting the domain label the continuous adaptation can further improve the alignment or classification performance.

Reviews: review: paper_summary: paper_summary: This paper proposes an optimization strategy called Virtual Label Fooled Domain Discrimination for improving Domain Adaptive Object Detection (DAOD) methods, typically which are using the domain adversarial training approach. This method aims to address the equilibrium challenge during the training of the feature extractor and the domain classifier. Specifically, the authors propose to identify the unaligned samples for each mini-batch and assign virtual labels to them in order to confuse the classifier. The authors claim by this approach, the discriminator will be more likely not to fall into suboptimal points during training, which will lead to an improved feature extractor and more aligned feature space for source domain and target domain. The authors evaluate their methods on several datasets and show better results than previous methods.
strength_weakness: strength_weakness: 1. The motivation is interesting. The idea of using virtual labels to confuse the classifier to improve the feature extractor might have contribution to the community.
2. The authors demonstrate the effectiveness of the methods on several datasets and compare with a variety of previous methods, which can be supportive.
limitations: limitations: 1. The novelty of the approach might be limited. Although effective, the methods are straightforward. It is similar to applying some data augmentation methods during the training session. One can think of some variations and comparisons from this perspective, which might impact the novelty of the proposed method.
2. Some parts of this method can be further explored. The authors mention that how to select the unaligned samples is not the focus of the paper, however, given the simplicity of the proposed methods, some more exploration on this part could also be beneficial.
paper_summary: paper_summary: The paper aims to prevent the  domain classifier from being trapped in local optima by introducing virtual samples. The author also  provide a theoretical analysis of the proposed Virtual–H-divergence that it has a smaller upper-bound than the standard H-divergence.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow. The motivation is well-expressed with some preliminary experiments.
2. Theoretical proof is also provided to validate the proposed method.
3. Extensive experiments are provided.
limitations: limitations: 1.The methods somehow share the same spirit as existing methods.[1] Although the problem is different, the high-level idea involves creating virtual samples based on current target and source samples to augment the dataset and achieve better accuracy.
2. The improvement in performance using the proposed methods does not seem significant.
Minor:
1. In line 165, the authors may refer to the wrong figure.
2. The captions of the figures are too long, it would be better to shorten them.
[1]Pairwise adversarial training for unsupervised class-imbalanced domain adaptation
paper_summary: paper_summary: This paper addresses the equilibrium challenge in Domain Adaptive Object Detection (DAOD), where the domain classifier's discrimination capability degrades over time, hindering effective feature alignment. To solve this, the authors propose a novel optimization strategy called Virtual Label Fooled Domain Discrimination (VFDD), which uses virtual sample labels to rejuvenate the domain classifier, facilitating better feature alignment. This method introduces virtual labels for "unaligned" samples to confuse the domain classifier and improve feature alignment by making the distributions less separable. VFDD is effective when integrated with existing DAOD methods, as demonstrated through theoretical insights and extensive experiments on benchmarks such as Cityscapes, Foggy Cityscapes, Sim10k, and BDD100k. The experimental results show significant improvements in cross-domain object detection tasks, and the method's effectiveness is further validated through t-SNE plots and qualitative results. VFDD's ability to enhance feature alignment and domain adaptation makes it a valuable addition to current DAOD techniques.
strength_weakness: strength_weakness: The paper addresses the equilibrium challenge in Domain Adaptive Object Detection (DAOD) where the discrimination capability of the domain classifier degrades, hindering effective feature alignment. The authors propose a novel optimization strategy called Virtual Label Fooled Domain Discrimination (VFDD), which uses virtual sample labels to rejuvenate the domain classifier, facilitating better feature alignment. VFDD can be integrated with existing DAOD methods to enhance their performance.
1.	The paper presents an innovative optimization strategy, Virtual Label Fooled Domain Discrimination (VFDD), to tackle the equilibrium challenge in adversarial domain adaptation. This novel approach of using virtual sample labels to confuse the domain classifier is a good contribution to the field. 
2.	The authors provide a strong theoretical basis for VFDD, introducing the concept of Virtual-H divergence to explain why their method can overcome local optima issues. This theoretical insight adds depth to the proposed solution and helps understand its effectiveness. The presented illustrations and formulas clearly describe the current problems in DAOD, while providing a sufficiently detailed explanation and description of the proposed methods.
3.	The paper includes comprehensive experimental results across various benchmarks such as Cityscapes, Foggy Cityscapes, Sim10k, and BDD100k. These experiments demonstrate the robustness and general applicability of VFDD, showing consistent performance improvements over existing methods.
limitations: limitations: 1.	In line 162, you mention part (b) of Figure 1, but your Figure 1 does not have a part (b). Are you referring to part (b) of Figure 2? If so, please correct this. Additionally, in Figure 2, you mention that the distributions of the source and target domains are similar, but I don't think this figure demonstrates the similarity in distributions. I suggest creating separate feature distribution plots for the source and target domains.
2.	The description of the VFDD technique in section 3.2 is too abstract and does not provide readers with a clear understanding. I suggest incorporating specific examples from the dataset to clarify:
(1)	What are V-Source (V-target) samples? Which samples in the dataset can serve as V-source (V-target) samples? Are there any boundary cases where samples cannot serve as V-source (V-target) samples?
(2)	For lines 395-401, I recommend using graphical illustrations with specific examples from the dataset to explain.
3.	The paper mentions the threshold τ as a crucial parameter for determining whether a sample is "unaligned" (Equation 9). However, the experimental section lacks a detailed sensitivity analysis of τ across different datasets and tasks. The threshold τ is briefly discussed in Figure 5(a) (Design Choices), but the exploration is limited to a narrow range of values. A more comprehensive analysis would involve varying τ across a wider range and reporting its effect on the performance metrics systematically.
4.	The paper provides a clear analysis of all the illustrations. For instance, the explanation of Figure 1 is clear and connects well with the narrative in the text, and Figure 3 is well-explained with step-by-step details, making the concept of VFDD understandable. However, some improvements can provide a clearer and more comprehensive understanding of the figures and their implications, thereby strengthening the overall narrative.
a)	For Figure 5, provide a more detailed interpretation of the trends observed in the influence of threshold τ and patience parameter K. Explain how different values impact the performance and why certain values might be optimal.
b)	For Figure 6, expand the explanation to discuss specific performance improvements of VFDD over RADA. Highlight any notable observations and potential reasons for the differences in performance.
5.	The scalability of VFDD to larger datasets or more challenging domain gaps is not thoroughly evaluated: 
a)	Understanding how VFDD performs as the size and complexity of data and models increase is crucial for assessing its practical applicability in real-world scenarios. The paper does not include an in-depth analysis of scenarios where VFDD might fail or perform sub-optimally.
b)	In my opinion, there are particularly challenging scenarios of domain gaps in medical image analysis. Can the proposed method work effectively under larger domain gaps? For instance, the following articles [1]-[2] analyze the adaptation to significant domain gaps in medical scenarios.
[1] Shin, H., Kim, H., Kim, S., Jun, Y., Eo, T. and Hwang, D., 2023. SDC-UDA: volumetric unsupervised domain adaptation framework for slice-direction continuous cross-modality medical image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7412-7421).
[2] Fan, J., Liu, D., Chang, H., Huang, H., Chen, M. and Cai, W., 2023. Taxonomy adaptive cross-domain adaptation in medical imaging via optimization trajectory distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 21174-21184).
paper_summary: paper_summary: In domain adaptive object detection, traditional methods often struggle with domain gaps due to the limited discrimination capability of domain classifiers, which tend to reach a local optimum. To overcome this, the proposed Virtual-label Fooled Domain Discrimination (VFDD) strategy introduces an innovative approach using virtual sample labels to rejuvenate the domain classifier's effectiveness during training. This makes the distributions less separable, enhancing the classifier's confusion and promoting better feature alignment. VFDD, which is compatible with most existing DAOD methods as a plug-and-play module, has been shown to enhance baseline models and outperform recent unsupervised domain adaptive object detection models through theoretical insights and experimental validation.
strength_weakness: strength_weakness: 1. The authors' experiments are good. They have demonstrated their method's effectiveness in various scenarios, such as Synthetic to Real Adaptation and weather adaptation. The authors' comparisons using different cases are decent.
   
2. Several motivations/claims are justified using suitable experiments/visualizations (like Fig 2).

3. The proposed methodology also betters the SOTA concerning difficult
   situations like heavy occlusion and small size of objects, thus enhancing
   the applicability of the methodology.

4. Overall, the paper is well-written and easy to follow.
limitations: limitations: The manuscript's explanation of how the Virtual-label Fooled Domain Discrimination (VFDD) enhances feature alignment lacks clarity, which is crucial for understanding the efficacy of this method in domain adaptation challenges. A detailed step-by-step breakdown of the VFDD mechanism should be provided to improve the manuscript, explicitly showing how virtual labels contribute to improved feature alignment across domains.

Furthermore, the study's selection of baselines for comparison appears to rely on somewhat dated methodologies. In a rapidly evolving field like machine learning, it is imperative to accurately compare new techniques against the most recent and relevant baselines to establish their effectiveness and innovation. The authors should consider incorporating more current state-of-the-art methods that utilize similar or competing approaches to domain adaptation.

An F-divergence analysis would add depth to the evaluation of VFDD, offering insights into how different the distributions are before and after applying VFDD. 

Comparing the performance of a Vision Transformer (ViT)--based backbone with that of other architectures like CNNs in the VFDD setup could offer significant insights. Since ViTs are known for their efficacy in handling different types of data distributions due to their attention mechanisms, evaluating VFDD with a ViT backbone could reveal additional benefits or limitations of the approach. 

Lastly, providing qualitative analysis concerning the overlap of the domains post-application of VFDD would be beneficial. This could include visualizations of feature spaces before and after applying VFDD.