Decouple and Reconstruct: Mining Discriminative Features for Cross-domain Object Detection

1 INTRODUCTION . Along with the advances in deep convolutional neural networks ( Krizhevsky et al. , 2012 ) , remarkable progresses have been achieved in various visual tasks ( He et al. , 2016 ; Long et al. , 2015 ; Ren et al. , 2017 ) . Such outstanding achievements heavily rely on the assumption that training and test data are drawn from an independent identical distribution . Nevertheless , due to the changes of environmental condition in real-world , a phenomenon known as “ domain shift ” occurs and challenges this assumption , thus leading to a significant performance decay . Although collecting more training data can alleviate such problems to some extent , it is non-trivial because of the high cost of image annotation . An appealing solution to alleviate the shift is introducing domain adaptation ( DA ) methodology , which aims at bridging the gap between domains by learning invariant representations . In this paper , we focus on the unsupervised scenarios , called unsupervised domain adaptation ( UDA ) , where the training set is formed from labeled source data and unlabeled target data . Recently , a variety of UDA methods have been well-studied for classification ( Tzeng et al. , 2014 ; Ganin & Lempitsky , 2015 ; Saito et al. , 2018 ; Wang et al. , 2020 ; Li et al. , 2021b ) and semantic segmentation ( Hoffman et al. , 2018 ; Zou et al. , 2018 ; Dong et al. , 2020 ; Li et al. , 2020 ; Jaritz et al. , 2021 ) , which are mostly based on minimizing domain discrepancy measurement or adversarial training . These strategies have made great progress but are hard to be applied to object detection directly because of the local nature of detection tasks ( Zhu et al. , 2019 ) . To address the adaptation dilemma for detection tasks , instance-level feature alignment has been widely used in Chen et al . ( 2018b ) ; Saito et al . ( 2019 ) ; Zheng et al . ( 2020 ) . Most of them strive to handle the relation between local regions by calibrating cross-channel and spatial information . However , it may implicitly drive networks to pay more attention on shared attributes and ignore the domain-specific feature . As shown in Figure 1 , DA reduces the domain shift by feature alignment between source and target domain , which highlights the common components at the cost of domain-specific features . We review the feature alignment of domain adaptation and suppose that image-level style and instancelevel content play a vital role in detection tasks . The style information should be globally consistent within its own domain while local homogeneous content information between different domains ought to be analogous , and vice versa . Building upon the above findings , we propose a novel adaptation framework for cross-domain object detection , which supplements the domain-specific representation by restructuring the content and style information . Specifically , we adopt the Domain Adaptive Instance Normalization ( DAdaIN ) on shallow layers to generate synthesis features , which contributes on encoding the domain-specific and domain-invariant features in source and target domain . Furthermore , Global Style Alignment ( GSA ) module and Local Content Alignment ( LCA ) module are exploited to regularize the consistency of style and content between multiple domains . GSA consists of two sub-modules , i.e. , style similarity alignment and style consistency alignment , which are adopted in multi-layer to align global style representation . Moreover , LCA is proposed to calibrate the content information of local regions , which enhances the model ’ s response to foreground objects of various styles . In summary , the main contributions of this work are three-fold : • We review the feature alignment in domain adaptation and point out that it may implicitly drives networks to focus more on public component and ignore the domain-specific feature . Furthermore , we suppose that global style and local content informations play a vital role in detection tasks . • We propose a novel adaptation framework for cross-domain object detection . A style-aware feature fusion method and two plug-and-play adaptation modules are designed to reposition the focus of the model to domain-specific features by restructuring the style and content of feature . • We conduct extensive experiments in adaptation between both similar and dissimilar domains to validate the effectiveness of our approach . The compelling experimental results and visualizations demonstrate that our method significantly boost the performance of existing Domain Adaptive Faster R-CNN detectors under various domain-shift scenarios . 2 RELATED WORK . 2.1 DOMAIN ADAPTATION . Domain adaptation ( Tzeng et al. , 2014 ; Gong et al. , 2015 ; Sun & Saenko , 2016 ; Zou et al. , 2018 ) with convolution neural network has emerged as a popular paradigm to boost the performance of trained model when training and test data do not satisfy the assumption of independent identical distribution . They strive to learn domain-invariant features , and along this line the methods can be roughly divided into two categories : statistical matching and adversarial learning . The common idea of statistical matching ( Fernando et al. , 2014 ; Gong et al. , 2015 ; Zellinger et al. , 2017 ; Peng et al. , 2018 ) is to measure and minimize the distance between latent feature space of different domains by appropriate metrics . The methods ( Chang et al. , 2019 ; Chen et al. , 2018a ; Russo et al. , 2017 ; Fu et al. , 2020 ; Xu et al. , 2020 ) based on adversarial learning aim to learning domain-invariant feature by construct minimax optimization with the domain classifier . In addition , Li et al . ( 2018 ) proposed to modulate the statistics in all BN layers across the network . And He & Zhang ( 2019 ) applied a progressive curriculum-based adversarial training to boost fine-grained visual categorization . Besides , Jin et al . ( 2021 ) designed a SNR module to ensure both generalization and discrimination capability . Despite the great success of the methods mentioned above , most of them are hard to be applied to object detection directly because of the local nature of detection tasks . 2.2 DA FOR OBJECT DETECTION . Recently , an increasing number of researches ( Chen et al. , 2018b ; Zhu et al. , 2019 ; Yang et al. , 2020 ; Zhao et al. , 2020 ) have been proposed for cross-domain object detection . Domain Adaptive Faster R-CNN ( Chen et al. , 2018b ) proposed to learn domain-invariant features in image-level and instance-level by adversarial learning . Furthermore , Saito et al . ( 2019 ) applied strong-weak hierarchical feature alignment to alleviate non-transferable features , and Zhu et al . ( 2019 ) repositioned the focus of adaptation from global to local and introduces an additional module to reconstruct the patches . For the sake of more accurate feature alignment , Xu et al . ( 2020 ) integrated an image-level multi-label classifier upon the detection backbone to pay more attention on the foreground instances and re-weight the proposals to automatically hunt for the hard samples . Similarly , Zheng et al . ( 2020 ) adopted the attention mechanism to highlight the importance of the foreground regions , and then align instances with the same category between source and target domain by maintaining global prototypes . Besides , Chen et al . ( 2020 ) hierarchically calibrated the transferability of feature representations in images/instance/local-region level , aiming to harmonize transferability and discriminability . And Vibashan et al . ( 2021 ) employed category-wise discriminators to ensure categoryaware feature alignment . The above methods enable the network to extract domain-invariant features in virtue of refined feature alignment , and enhance the transferablility of the model . However , how to make better use of target-like domain-specific details is under-explored . 2.3 STYLE TRANSFER . Style transfer ( Ulyanov et al. , 2016 ; Yoo et al. , 2019 ; Huang & Belongie , 2017 ; Johnson et al. , 2016 ) has been proven efficient on the transformation of the object appearance . In general , style transfer is considered as a texture conversion issue . Some previous methods ( Denton et al. , 2015 ; Frigo et al. , 2016 ) typically rely on low-level statistics but fail to extract the semantic image content . Gatys et al . ( 2016 ) pioneered the separation of image content information from its style and introduced a new algorithm to perform image style transfer . Furthermore , Ulyanov et al . ( 2017 ) propose a light-weight feed-forward network . Surprisingly , Ulyanov et al . ( 2016 ) achieves remarkable improvement by replacing BN with IN . Along this line , Dumoulin et al . ( 2016 ) ; Huang & Belongie ( 2017 ) go further and design alternative modules , which improve the quality and diversity of image stylization . And Kotovenko et al . ( 2021 ) review the stylization process and concentrate on optimizing parameterized brushstrokes . All these methods motivate us to design a complementary module for mining target-like details in domain adaptation . While previous works ( Chen et al. , 2020 ; Rodriguez & Mikolajczyk , 2019 ) employed CycleGAN to generate interpolation samples , they faced a critical limitation that the performance is greatly determined by image quality . In this paper , we propose to manipulate the content and style information simultaneously by feature reconstruction during the training phase , aiming to exploit more domain-specific features in an end-to-end fashion . 3 METHOD . 3.1 PROBLEM FORMULATIONS . Unsupervised domain adaptation for object detection can boil down to the analysis of data distributions in two domains : a source domain with adequate annotated image data and a target domain with unlabeled images . Formally , we denote source domain as DS = { ( xsi , ysi ) } Ns i=1 } , where xsi and ysi = ( b s i , c s i ) represent the i-th sample and label ( class and bounding box ) respectively . Similarly , target domain is defined as DT = { ( xti } Nt i=1 } , where Nt is the amount of target samples . DS and DT have the identical categories tags c but are drawn from diverse scenarios with heterogeneous data distribution . Our destination is to generalize the detector from labeled DS to unlabeled DT . 3.2 FRAMEWORK OVERVIEW . The overview of proposed framework is illustrated in Figure 2 . It consists of three main components : DAdaIN , GSA and LCA . Previous works ( Hsu et al. , 2020 ; Chen et al. , 2020 ) exploited synthetic data by CycleGAN to narrow the domain gap , which can not be trained in an end-to-end fashion and is sub-optimal for tackling the domain shift . In this paper , we design a style-aware feature fusion module called DAdaIN and insert it into the detector during the training phase aiming to improve the domain invariance of the model itself . Moreover , Global Style Alignment ( GSA ) and Local Content Alignment ( LCA ) are introduced to regularize the style and content information between multiple domains respectively . We suppose that image-level style and instance-level content play a vital role in detection tasks . Extensive experiments verify the effectiveness of our method . Our framework effectively increases the learning ability of the model for domain-specific features . Meanwhile , the proposed module can be flexibly used as a supplement to existing SOTA methods and improve the generalization of the model without any modification to the inference process . 3.3 STYLE-AWARE FEATURE FUSION . With the increase of domain gap , existing UDA methods usually exhibit instability or adaptability degradation . It can be intuitively understood as the result of domain-specific features missing . We suppose that low-level variations , e.g . color , texture and edges , constitute a unique style which reflects the difference of feature distributions between different domains . Inspired by Huang & Belongie ( 2017 ) , we adopt the style-aware feature fusion method and design Domain Adaptive Instance Normalization ( DAdaIN ) to provide extra target-like details . Our key insight is that while it is difficult to extract discriminative features in target domain , it is feasible to directly assign the underlying details to the model via feature style transfer , so as to alleviate the inconsistency between training and testing . Specifically , we denote fi ∈ RN×C×H×W as the feature map extracted by the detector in domain i ∈ { DS , DT } , where H × W and C are the spatial dimensions and the number of feature map channels , respectively . DAdaIN adaptively computes the affine parameters from the target feature ft to scale and shift the source feature fs : DAdaIN ( fs , ft ) = σt ( fs − µs σs ) + µt = σt σs fs + ( 1− σt σs ) µs + ( µt − µs ) ( 1 ) where µi ∈ RN×C and σi ∈ RN×C represent the mean and standard deviation of each channel for each sample fi . From Eq.1 , we can derive that the output feature of DAdaIN depends on the feature statistics ( µ , σ ) computed across spatial dimensions . In order to enrich the diversity of synthesis feature and maintain the stability during training phase , we introduce κ1 to control the translation scale between source and target domains . Meanwhile , we use κ2 to balance the translation bias caused by the scale of parameter κ1 . The DAdaIN is adopted in the feature space of intermediate domain DM . Eq.1 with translation parameters can be re-written as : DAdaIN ( fs , ft ; κ1 , κ2 ) = κ1 σt σs fs + ( 1− κ1 σt σs ) µs + κ2 ( µt − µs ) ( 2 ) where κ1 and κ2 obey the normal distribution N ( 1 , 1 ) . Finally , the style-aware synthesis feature fm can be defined as following : fm = λ DAdaIN ( fs , ft ) + ( 1− λ ) fs ( 3 ) where λ is a hyper-parameter to control the degree of feature style transfer . In all experiments , we set λ to 1 . Technically , we integrate DAdaIN module after relu4 2 of VGG16 , then fuse the style and content informations to generate the synthesis features fm . It is worth noting that , all subsequent work is based on fm . Moreover , since the role of DAdaIN is to introduce domain-specific details during training phase , it is feasible to remove such module for inference without affecting the performance . By doing so , we can generate the synthesis features which encode the discriminative information of target domain in an end-to-end fashion . Besides , since the content of synthesis feature is shared with the source domain , the label information of the source domain can be better used .

Summary: This paper proposes a novel adaptation framework to balance transferability and discriminability for cross-domain object detection. Concretely, they focus on domain-specific features thus manage to reconstruct the style and content of features by AdaIN and some alignment techniques. Extensive experiments show the performance improvement by proposed modules.