Decoupled Adaptation for Cross-Domain Object Detection

1 INTRODUCTION . The object detection task has aroused great interest due to its wide applications . In the past few years , the development of deep neural networks has boosted the performance of object detectors [ 33 ; 15 ; 41 ] . While these detectors have achieved excellent performance on the benchmark datasets [ 11 ; 31 ] , object detection in the real world still faces challenges from the large variance in viewpoints , object appearance , backgrounds , illumination , image quality , etc . Such domain shifts have been observed to cause significant performance drop [ 8 ] . Thus , some work uses domain adaptation [ 39 ] to transfer a detector from a source domain , where sufficient training data is available , to a target domain where only unlabeled data is available [ 8 ; 44 ] . This technique successfully improves the performance of the detector on the target domain . However , the improvement of domain adaptation in object detection remains relatively mild compared with that in object classification . The inherent challenges come from three aspects . Data challenge : what to adapt in the object detection task is unknown . Instance feature adaptation in the object level ( Figure 1 ( a ) ) might confuse the features of the foreground and the background since the generated proposals may not be true objects and many true objects might be missing ( Figure 5 ) . Global feature adaptation in the image level ( Figure 1 ( b ) ) is likely to mix up features of different objects since each input image of detection has multiple objects . Local feature adaptation in the pixel level ( Figure 1 ( c ) ) can alleviate domain shift when the shift is primarily low-level , yet it will struggle when the domains are different at the semantic level . Architecture challenge : while the above adaptation methods introduce domain discriminators and gradient reverse layers [ 12 ] into the detector architecture to encourage domaininvariant features , the discriminability of features might get deteriorated [ 6 ; 5 ] , which will greatly influence the localization and the classification of the detectors . Besides , where to place these modules in the detection architecture has a great impact on the final performance but is a little tricky . Therefore , the scalability of these methods to different detection architectures is not so satisfactory . Task challenge : object detection is a multi-task learning problem , consisting of both classification and localization . Yet previous adaptation algorithms mainly explored the category adaptation , and it ’ s still difficult to obtain an adaptation model suitable for different tasks at the same time . To overcome these challenges , we propose a general framework – D-adapt , namely Decoupled Adaptation . Since adversarial alignment directly on the features of the detector might hurt its dis- Published as a conference paper at ICLR 2022 Back bone ROI RPN Detector Back bone ROI RPN Detector Back bone ROI RPN Detector ; source ortarget ? Global Discriminator ; Local Discriminator G R L features G R L features features Instance Discriminator ; G R L ( a ) Instance adapt Back bone ROI RPN Detector Back bone ROI RPN Detector Back bone ROI RPN Detector ; source ortarget ? Global Discriminator ; Local Discriminator G R L features G R L features features Instance Discriminator ; G R L ( b ) Global adapt Back bone ROI RPN Detector Back bone ROI RPN Detector Back bone ROI RPN Detector ; source ortarget ? Global Discriminator ; Local Discriminator G R L features G R L features features Instance Discrim nator ; G R L ( c ) Local adapt Back bone ROI RPN Detector bbox labels Image Back bone Head D Category Adaptor Back bone Head AdvHead BBox Adaptor category labels proposals foreground proposals : data flow : tensor flow ( d ) Decouple adapt criminability ( architecture challenge ) , we decouple the adversarial adaptation from the training of the detector by introducing a parameter-independent category adaptor ( see Figure 1 ( d ) ) . To tackle the task challenge , we introduce another bounding box adaptor that ’ s decoupled from both the detector and the category adaptor . To tackle the data challenge , we propose to adjust the object-level data distribution for specific adaptation tasks . For example , in the category adaptation step , we encourage the input proposals to have IoU1 close to 0 or 1 to better satisfy the low-density separation assumption , while in the bounding box adaptation step , we encourage the input proposals to have IoU between 0.5 and 1 to ease the optimization of the bounding box localization task . The contributions of this work are summarized as three-fold . ( 1 ) We introduce D-adapt framework for cross-domain object detection , which is general for both two-stage and single-stage detectors . ( 2 ) We propose an effective method to adapt the bounding box localization task , which is ignored by existing methods but is crucial for achieving superior final performance . ( 3 ) We conduct extensive experiments and validate that our method achieves state-of-the-art performance on four object detection tasks , and yields 17 % and 21 % relative improvement on Clipart1k and Comic2k . 2 RELATED WORK . Generic domain adaptation for classification . Domain adaptation is proposed to overcome the distribution shift across domains . In the classification setting , most of the domain adaptation methods are based on Moment Matching or Adversarial Adaptation . Moment Matching methods [ 51 ; 36 ] align distributions by minimizing the distribution discrepancy in the feature space . Taking the same spirit as Generative Adversarial Networks [ 16 ] , Adversarial Adaptation [ 12 ; 37 ] introduces a domain discriminator to distinguish the source from the target , then the feature extractor is encouraged to fool the discriminator and learn domain invariant features . However , directly applying these methods to object detection yields an unsatisfactory effect . The difficulty is that the image of object detection usually contains multiple objects , thus the features of an image can have complex multimodal structures [ 20 ; 59 ; 5 ] , making the image-level feature alignment problematic [ 59 ; 20 ] . Generic domain adaptation for regression . Most domain adaptation methods designed for classification do not work well on regression tasks since the regression space is continuous with no clear decision boundary [ 22 ] . Some specific regression algorithms are proposed , including importance weighting [ 55 ] or learning invariant representations [ 40 ; 38 ] . RSD [ 7 ] defines a geometrical distance for learning transferable representations and disparity discrepancy [ 58 ] proposes an upper bound for the distribution distance in the regression problems . Yet previous methods are mainly tested on simple tasks while this paper extends domain adaptation to the object localization tasks . Domain adaptation for object detection . DA-Faster [ 8 ] performs feature alignment at both image-level and instance-level . SWDA [ 44 ] proposes that strong alignment of the local features is more effective than the strong alignment of the global features . Hsu et al . [ 20 ] carries out center- 1The Intersection-over-Union between the proposals and the ground-truth instance . aware alignment by paying more attention to foreground pixels . HTCN [ 5 ] calibrates the transferability of feature representations hierarchically . Zheng et al . [ 60 ] proposes to extract foreground regions and adopts coarse-to-fine feature adaptation . ATF [ 19 ] introduces an asymmetric tri-way approach to account for the differences in labeling statistics between domains . CRDA [ 54 ] and MCAR [ 59 ] use multi-label classification as an auxiliary task to regularize the features . However , although the auxiliary task of outputting domain-invariant features to fool a domain discriminator in most aforementioned methods can improve the transferability , it also impairs the discriminability of the detector . In contrast , we decouple the adversarial adaptation and the training of the detector , thus the adaptors could specialize in transfer between domains , and the detector could focus on improving the discriminability while enjoying the transferability brought by the adaptors . Self-training with pseudo labels . Pseudo-labeling [ 30 ] , which leverages the model itself to obtain labels on unlabeled data , is widely used in self-training . To generate reliable pseudo labels , temporal ensembling [ 29 ] maintains an exponential moving average prediction for each sample , while the mean-teacher [ 50 ] averages model weights at different training iterations to get a teacher model . Deep mutual learning [ 57 ] trains a pool of student models with supervisions from each other . FixMatch [ 48 ] uses the model ’ s predictions on weakly-augmented images to generate pseudo-labels for the strongly-augmented ones . Unbiased Teacher [ 35 ] introduces the teacher-student paradigm to Semi-Supervised Object Detection ( SS-OD ) . When some image-level labels exist , the performance can be further improved by encoding correlations between coarse-grained and fine-grained classes [ 56 ] , employing noise-tolerant training strategies [ 13 ] , or learning a mapping from weaklysupervised to fully-supervised detectors [ 24 ] in SS-OD . Recent works [ 21 ; 25 ; 26 ] utilize selftraining in cross-domain object detection and take the most confident predictions as pseudo labels . MTOR [ 3 ] uses the mean teacher framework and UMT [ 10 ] adopts distillation and CycleGAN [ 61 ] in self-training . However , self-training suffers from the problem of confirmation bias [ 1 ; 4 ] : the performance of the student will be limited by that of the teacher . Although pseudo labels are also used in our proposed D-adapt , they are generated from adaptors that have independent parameters and different tasks from the detector , thereby alleviating the confirmation bias of the overly tight relationship in self-training . 3 PROPOSED METHOD . In supervised object detection , we have a labeled source domain Ds = { ( Xis , Bis , Yis ) } ns i=1 , where Xis is the image , B i s is the bounding box coordinates , and Y i s is the categories . The detector G det is trained with Ldets , which consists of four losses in Faster RCNN [ 42 ] : the RPN classification loss Lrpncls , the RPN regression loss L rpn reg , the RoI classification loss Lroicls and the RoI regression loss Lroireg , Ldets = E ( Xs , Bs , Ys ) ∈DsL rpn cls + L rpn reg + Lroicls + Lroireg . ( 1 ) In cross-domain object detection , there exists another unlabeled target domain Dt = { Xit } nt i=1 that follows different distributions from Ds . The objective of Gdet is to improve the performance on Dt . 3.1 D-ADAPT FRAMEWORK . To deal with the architecture challenge mentioned in Section 1 , we propose the D-adapt framework , which has three steps : ( 1 ) decouple the original cross-domain detection problem into several subproblems ( 2 ) design adaptors to solve each sub-problem ( 3 ) coordinate the relationships between different adaptors and the detector . Since adaptation might hurt the discriminability of the detector , we decouple the category adaptation from the training of the detector by introducing a parameter-independent category adaptor ( see Figure 1 ( d ) ) . The adaptation is only performed on the features of the category adaptor , thus will not hurt the detector ’ s ability to locate objects . To fill the blank of regression domain adaptation in object detection , we need to perform adaptation on the bounding box regression . Yet feature visualization in Figure 6 ( c ) reveals that features that contain both category and location information do not have an obvious cluster structure , and alignment might hurt its discriminability . Besides , the common category adaptation methods are also not effective on regression tasks [ 22 ] , thus we decouple category adaptation and the bounding box adaptation to avoid their interfering with each other . Section 3.2 and 3.3 will introduce the design of category adaptor and box adaptor in details . In this section , we will assume that such two adaptors are already obtained . To coordinate the adaptation on different tasks , we maintain a cascading relationship between the adaptors . In the cascading structure , the later adaptors can utilize the information obtained by the previous adaptors for better adaptation , e.g . in the box adaptation step , the category adaptor will select foreground proposals to facilitate the training of the box adaptor . Compared with the multi-task learning relationship where we need to balance the weights of different adaptation losses carefully , the cascade relationship greatly reduces the difficulty of hyper-parameter selection since each adaptor has only one adaptation loss . Since the adaptors are specifically designed for cross-domain tasks , their predictions on the target domain can serve as pseudo labels for the detector . On the other hand , the detector generates proposals to train the adaptors and higher-quality proposals can improve the adaptation performance ( see Table 5 for details ) . And this enables the self-feedback relationship between the detector and the adaptors . For a good initialization of this self-feedback loop , we first pre-train the detector Gdet on the source domain with Ldets . Using the pre-trained Gdet , we can derive two new data distributions , the source proposal distribution Dprops and the target proposal distribution Dpropt . Each proposal consists of a crop of the image x2 , its corresponding bounding box bdet , predicted category ydet and the class confidence cdet . We can annotate each source-domain proposal xs ∈ Dprops with a ground truth bounding box bgts and category label y gt s , similar to labeling each RoI in Fast RCNN [ 14 ] , and then use these labels to train the adaptors . In turn , for each target proposal xt ∈ Dpropt , adaptors will provide category pseudo label yclst and box pseudo label b reg t to train the RoI heads , Ldett = E ( Xt , bdett , yclst , bregt ) ∈Dpropt L roi cls ( Xt , b det t , y cls t ) + Lroicls ( Xt , b reg t , y cls t ) +Ifg ( yclst ) · Lroireg ( Xt , bdett , b reg t ) , ( 2 ) Algorithm 1 : D-adapt Training Pipeline . input : Source domainDs and target domainDt , number of iterations T output : Cross-domain object detector Gdet initialize the object detector Gdet by optimizing with Ldets ; for t← 1 to T do generate proposalsDprops andD prop t for each sample inDs andDt by Gdet ; for each mini-batch inDprops andD prop t do train the category adaptor Gcls ; end generate category label for each proposal inDpropt ; generate foreground proposalsDfgs andD fg t fromDprops andD prop t ; for each mini-batch inDfgs andD fg t do train the bounding box adaptor Greg ; end generate bounding box label for each proposal inDfgt ; train the object detector Gdet by optimizing with Ldett ; end where Ifg is a function that indicates whether it is a foreground class . Note that regression loss is activated only for foreground anchors . After obtaining a better detector by optimizing Equation 2 , we can generate higher-quality proposals , which facilitate better category adaptation and bounding box adaptation . This process can iterate multiple times and the detailed optimization procedures are summarized in Algorithm 1 . Note that our D-adapt framework does not introduce any computational overhead in the inference phase , since the adaptors are independent of the detector and can be removed during detection . Also , D-adapt does not depend on a specific detector , thus the detector can be replaced by SSD [ 34 ] , RetinaNet [ 32 ] , or other detectors .

Summary: The authors propose a tailored method, named Decoupled Adaptation (D-adapt), for cross-domain object detection. The conventional domain adaptation techniques hurt the discriminability of the detector and ignores the adaptation on bounding box regression. In order to avoid hurting the discriminability of the detector, the authors propose to decouple the adaptation modules from the detector, which achieves the parameter independence. In addition, the authors propose the module for adapting the bounding box regression. Due to the decoupling property, the proposed D-adapt framework is compatible to many off-the-shelf detectors. The experimental results are very promising.