Decouple and Reconstruct: Mining Discriminative Features for Cross-domain Object Detection

1 INTRODUCTION . Along with the advances in deep convolutional neural networks ( Krizhevsky et al. , 2012 ) , remarkable progresses have been achieved in various visual tasks ( He et al. , 2016 ; Long et al. , 2015 ; Ren et al. , 2017 ) . Such outstanding achievements heavily rely on the assumption that training and test data are drawn from an independent identical distribution . Nevertheless , due to the changes of environmental condition in real-world , a phenomenon known as “ domain shift ” occurs and challenges this assumption , thus leading to a significant performance decay . Although collecting more training data can alleviate such problems to some extent , it is non-trivial because of the high cost of image annotation . An appealing solution to alleviate the shift is introducing domain adaptation ( DA ) methodology , which aims at bridging the gap between domains by learning invariant representations . In this paper , we focus on the unsupervised scenarios , called unsupervised domain adaptation ( UDA ) , where the training set is formed from labeled source data and unlabeled target data . Recently , a variety of UDA methods have been well-studied for classification ( Tzeng et al. , 2014 ; Ganin & Lempitsky , 2015 ; Saito et al. , 2018 ; Wang et al. , 2020 ; Li et al. , 2021b ) and semantic segmentation ( Hoffman et al. , 2018 ; Zou et al. , 2018 ; Dong et al. , 2020 ; Li et al. , 2020 ; Jaritz et al. , 2021 ) , which are mostly based on minimizing domain discrepancy measurement or adversarial training . These strategies have made great progress but are hard to be applied to object detection directly because of the local nature of detection tasks ( Zhu et al. , 2019 ) . To address the adaptation dilemma for detection tasks , instance-level feature alignment has been widely used in Chen et al . ( 2018b ) ; Saito et al . ( 2019 ) ; Zheng et al . ( 2020 ) . Most of them strive to handle the relation between local regions by calibrating cross-channel and spatial information . However , it may implicitly drive networks to pay more attention on shared attributes and ignore the domain-specific feature . As shown in Figure 1 , DA reduces the domain shift by feature alignment between source and target domain , which highlights the common components at the cost of domain-specific features . We review the feature alignment of domain adaptation and suppose that image-level style and instancelevel content play a vital role in detection tasks . The style information should be globally consistent within its own domain while local homogeneous content information between different domains ought to be analogous , and vice versa . Building upon the above findings , we propose a novel adaptation framework for cross-domain object detection , which supplements the domain-specific representation by restructuring the content and style information . Specifically , we adopt the Domain Adaptive Instance Normalization ( DAdaIN ) on shallow layers to generate synthesis features , which contributes on encoding the domain-specific and domain-invariant features in source and target domain . Furthermore , Global Style Alignment ( GSA ) module and Local Content Alignment ( LCA ) module are exploited to regularize the consistency of style and content between multiple domains . GSA consists of two sub-modules , i.e. , style similarity alignment and style consistency alignment , which are adopted in multi-layer to align global style representation . Moreover , LCA is proposed to calibrate the content information of local regions , which enhances the model ’ s response to foreground objects of various styles . In summary , the main contributions of this work are three-fold : • We review the feature alignment in domain adaptation and point out that it may implicitly drives networks to focus more on public component and ignore the domain-specific feature . Furthermore , we suppose that global style and local content informations play a vital role in detection tasks . • We propose a novel adaptation framework for cross-domain object detection . A style-aware feature fusion method and two plug-and-play adaptation modules are designed to reposition the focus of the model to domain-specific features by restructuring the style and content of feature . • We conduct extensive experiments in adaptation between both similar and dissimilar domains to validate the effectiveness of our approach . The compelling experimental results and visualizations demonstrate that our method significantly boost the performance of existing Domain Adaptive Faster R-CNN detectors under various domain-shift scenarios . 2 RELATED WORK . 2.1 DOMAIN ADAPTATION . Domain adaptation ( Tzeng et al. , 2014 ; Gong et al. , 2015 ; Sun & Saenko , 2016 ; Zou et al. , 2018 ) with convolution neural network has emerged as a popular paradigm to boost the performance of trained model when training and test data do not satisfy the assumption of independent identical distribution . They strive to learn domain-invariant features , and along this line the methods can be roughly divided into two categories : statistical matching and adversarial learning . The common idea of statistical matching ( Fernando et al. , 2014 ; Gong et al. , 2015 ; Zellinger et al. , 2017 ; Peng et al. , 2018 ) is to measure and minimize the distance between latent feature space of different domains by appropriate metrics . The methods ( Chang et al. , 2019 ; Chen et al. , 2018a ; Russo et al. , 2017 ; Fu et al. , 2020 ; Xu et al. , 2020 ) based on adversarial learning aim to learning domain-invariant feature by construct minimax optimization with the domain classifier . In addition , Li et al . ( 2018 ) proposed to modulate the statistics in all BN layers across the network . And He & Zhang ( 2019 ) applied a progressive curriculum-based adversarial training to boost fine-grained visual categorization . Besides , Jin et al . ( 2021 ) designed a SNR module to ensure both generalization and discrimination capability . Despite the great success of the methods mentioned above , most of them are hard to be applied to object detection directly because of the local nature of detection tasks . 2.2 DA FOR OBJECT DETECTION . Recently , an increasing number of researches ( Chen et al. , 2018b ; Zhu et al. , 2019 ; Yang et al. , 2020 ; Zhao et al. , 2020 ) have been proposed for cross-domain object detection . Domain Adaptive Faster R-CNN ( Chen et al. , 2018b ) proposed to learn domain-invariant features in image-level and instance-level by adversarial learning . Furthermore , Saito et al . ( 2019 ) applied strong-weak hierarchical feature alignment to alleviate non-transferable features , and Zhu et al . ( 2019 ) repositioned the focus of adaptation from global to local and introduces an additional module to reconstruct the patches . For the sake of more accurate feature alignment , Xu et al . ( 2020 ) integrated an image-level multi-label classifier upon the detection backbone to pay more attention on the foreground instances and re-weight the proposals to automatically hunt for the hard samples . Similarly , Zheng et al . ( 2020 ) adopted the attention mechanism to highlight the importance of the foreground regions , and then align instances with the same category between source and target domain by maintaining global prototypes . Besides , Chen et al . ( 2020 ) hierarchically calibrated the transferability of feature representations in images/instance/local-region level , aiming to harmonize transferability and discriminability . And Vibashan et al . ( 2021 ) employed category-wise discriminators to ensure categoryaware feature alignment . The above methods enable the network to extract domain-invariant features in virtue of refined feature alignment , and enhance the transferablility of the model . However , how to make better use of target-like domain-specific details is under-explored . 2.3 STYLE TRANSFER . Style transfer ( Ulyanov et al. , 2016 ; Yoo et al. , 2019 ; Huang & Belongie , 2017 ; Johnson et al. , 2016 ) has been proven efficient on the transformation of the object appearance . In general , style transfer is considered as a texture conversion issue . Some previous methods ( Denton et al. , 2015 ; Frigo et al. , 2016 ) typically rely on low-level statistics but fail to extract the semantic image content . Gatys et al . ( 2016 ) pioneered the separation of image content information from its style and introduced a new algorithm to perform image style transfer . Furthermore , Ulyanov et al . ( 2017 ) propose a light-weight feed-forward network . Surprisingly , Ulyanov et al . ( 2016 ) achieves remarkable improvement by replacing BN with IN . Along this line , Dumoulin et al . ( 2016 ) ; Huang & Belongie ( 2017 ) go further and design alternative modules , which improve the quality and diversity of image stylization . And Kotovenko et al . ( 2021 ) review the stylization process and concentrate on optimizing parameterized brushstrokes . All these methods motivate us to design a complementary module for mining target-like details in domain adaptation . While previous works ( Chen et al. , 2020 ; Rodriguez & Mikolajczyk , 2019 ) employed CycleGAN to generate interpolation samples , they faced a critical limitation that the performance is greatly determined by image quality . In this paper , we propose to manipulate the content and style information simultaneously by feature reconstruction during the training phase , aiming to exploit more domain-specific features in an end-to-end fashion . 3 METHOD . 3.1 PROBLEM FORMULATIONS . Unsupervised domain adaptation for object detection can boil down to the analysis of data distributions in two domains : a source domain with adequate annotated image data and a target domain with unlabeled images . Formally , we denote source domain as DS = { ( xsi , ysi ) } Ns i=1 } , where xsi and ysi = ( b s i , c s i ) represent the i-th sample and label ( class and bounding box ) respectively . Similarly , target domain is defined as DT = { ( xti } Nt i=1 } , where Nt is the amount of target samples . DS and DT have the identical categories tags c but are drawn from diverse scenarios with heterogeneous data distribution . Our destination is to generalize the detector from labeled DS to unlabeled DT . 3.2 FRAMEWORK OVERVIEW . The overview of proposed framework is illustrated in Figure 2 . It consists of three main components : DAdaIN , GSA and LCA . Previous works ( Hsu et al. , 2020 ; Chen et al. , 2020 ) exploited synthetic data by CycleGAN to narrow the domain gap , which can not be trained in an end-to-end fashion and is sub-optimal for tackling the domain shift . In this paper , we design a style-aware feature fusion module called DAdaIN and insert it into the detector during the training phase aiming to improve the domain invariance of the model itself . Moreover , Global Style Alignment ( GSA ) and Local Content Alignment ( LCA ) are introduced to regularize the style and content information between multiple domains respectively . We suppose that image-level style and instance-level content play a vital role in detection tasks . Extensive experiments verify the effectiveness of our method . Our framework effectively increases the learning ability of the model for domain-specific features . Meanwhile , the proposed module can be flexibly used as a supplement to existing SOTA methods and improve the generalization of the model without any modification to the inference process . 3.3 STYLE-AWARE FEATURE FUSION . With the increase of domain gap , existing UDA methods usually exhibit instability or adaptability degradation . It can be intuitively understood as the result of domain-specific features missing . We suppose that low-level variations , e.g . color , texture and edges , constitute a unique style which reflects the difference of feature distributions between different domains . Inspired by Huang & Belongie ( 2017 ) , we adopt the style-aware feature fusion method and design Domain Adaptive Instance Normalization ( DAdaIN ) to provide extra target-like details . Our key insight is that while it is difficult to extract discriminative features in target domain , it is feasible to directly assign the underlying details to the model via feature style transfer , so as to alleviate the inconsistency between training and testing . Specifically , we denote fi ∈ RN×C×H×W as the feature map extracted by the detector in domain i ∈ { DS , DT } , where H × W and C are the spatial dimensions and the number of feature map channels , respectively . DAdaIN adaptively computes the affine parameters from the target feature ft to scale and shift the source feature fs : DAdaIN ( fs , ft ) = σt ( fs − µs σs ) + µt = σt σs fs + ( 1− σt σs ) µs + ( µt − µs ) ( 1 ) where µi ∈ RN×C and σi ∈ RN×C represent the mean and standard deviation of each channel for each sample fi . From Eq.1 , we can derive that the output feature of DAdaIN depends on the feature statistics ( µ , σ ) computed across spatial dimensions . In order to enrich the diversity of synthesis feature and maintain the stability during training phase , we introduce κ1 to control the translation scale between source and target domains . Meanwhile , we use κ2 to balance the translation bias caused by the scale of parameter κ1 . The DAdaIN is adopted in the feature space of intermediate domain DM . Eq.1 with translation parameters can be re-written as : DAdaIN ( fs , ft ; κ1 , κ2 ) = κ1 σt σs fs + ( 1− κ1 σt σs ) µs + κ2 ( µt − µs ) ( 2 ) where κ1 and κ2 obey the normal distribution N ( 1 , 1 ) . Finally , the style-aware synthesis feature fm can be defined as following : fm = λ DAdaIN ( fs , ft ) + ( 1− λ ) fs ( 3 ) where λ is a hyper-parameter to control the degree of feature style transfer . In all experiments , we set λ to 1 . Technically , we integrate DAdaIN module after relu4 2 of VGG16 , then fuse the style and content informations to generate the synthesis features fm . It is worth noting that , all subsequent work is based on fm . Moreover , since the role of DAdaIN is to introduce domain-specific details during training phase , it is feasible to remove such module for inference without affecting the performance . By doing so , we can generate the synthesis features which encode the discriminative information of target domain in an end-to-end fashion . Besides , since the content of synthesis feature is shared with the source domain , the label information of the source domain can be better used .

Summary: The paper introduces a method to tackle the unsupervised domain adaptation problem, focusing on the object detection problem. Towards the final objective, an architecture building on the style transfer concepts is proposed. The main idea is to build upon the AdaIN ideas to encode the domain-specific and domain-invariant features in the source and target domain. Two additional modules are exploited to regularize the consistency of style and content between multiple domains. Experimental results on 3 benchmark UDA setups are conducted. Similar/better results than existing approaches are obtained.