CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

1 INTRODUCTION . Deep neural network have achieved remarkable success in a wide range of application scenarios but it still suffers poor generalization performance to other new domain because of the domain shift problem ( Csurka , 2017 ; Zhao et al. , 2020 ; Zhang et al. , 2020 ; Oza et al. , 2021 ) . To handle this issue and avoid the expensive laborious annotations , lots of research efforts ( Bousmalis et al. , 2017 ; Kuroki et al. , 2019 ; Wilson & Cook , 2020 ; VS et al. , 2021 ) are devoted on Unsupervised Domain Adaptation ( UDA ) . The UDA task aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain . In UDA , most approaches focus on aligning distributions of source and target domain and learning domain-invariant feature representations . One kind of such UDA methods are based on category-level alignment ( Kang et al. , 2019 ; Zhang et al. , 2019 ; Jiang et al. , 2020 ; Li et al. , 2021b ) , which have achieved promising results on public UDA datasets using deep convolution neural networks ( CNNs ) . The fundamental problems in category-level based alignment is the production of pseudo labels for samples in target domain to generate the input source-target pairs . However , the current CNNs-based methods are not robust to the generated noisy pseudo labels for accurate domain alignment ( Morerio et al. , 2020 ; Jiang et al. , 2020 ) . With the success of Transformer in natural language processing ( NLP ) ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ) and vision tasks ( Dosovitskiy et al. , 2020 ; Han et al. , 2020 ; He et al. , 2021 ; Khan et al. , 2021 ) , it is found that cross-attention in Transformer is good at aligning different distributions , even from different modalities e.g. , vision-to-vision ( Li et al. , 2021e ) , vision-to-text ( Tsai et al. , 2019 ; Hu & Singh , 2021 ) and text-to-speech ( Li et al. , 2019 ) . And we find that it is robust to noise in pseudo labels to some extent . Hence , in this paper , we apply transformers to the UDA task to take advantage of its robustness to noise and super power for feature alignment to deal with the problems as described above in CNNs . In our experiment , we conclude that even with noise in the labeling pair , the cross-attention can still work well in aligning two distributions , thanks to the attention mechanism . To obtain more accurate pseudo labels , we designed a two-way center-aware labeling algorithm for samples in the target domain . The pseudo labels are produced based on the cross-domain similarity matrix , and a center-aware matching is involved to weight the matrix and weaken noise into the tolerable range . With the help of pseudo labels , we design the cross-domain transformer ( CDTrans ) for UDA . It consists of three weight-sharing transformer branches , of which two branches are for source and target data respectively and the third one is the feature alignment branch , whose inputs are from source-target pairs . The self-attention is applied in the source/target transformer branches and crossattention is involved in the feature alignment branch to conduct domain alignment . Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously . In summary , our contributions are three-fold : • We propose a weight-sharing triple-branch transformer framework , namely , CDTrans , for accurate unsupervised domain adaptation , taking advantage of its robustness to noisy labeling data and great power for feature alignment . • To produce pseudo labels with high quality , a two-way center-aware labeling method is proposed , and it boosts the final performance in the context of CDTrans . • CDTrans achieves the best performance compared to state-of-the-arts with a large margin on VisDA-2017 ( Peng et al. , 2017 ) and DomainNet ( Peng et al. , 2019 ) datasets . 2 RELATED WORK . 2.1 TRANSFORMER FOR VISION . Transformer is proposed in ( Vaswani et al. , 2017 ) to model sequential data in the field of NLP . Many works have shown its effectiveness for computer-vision tasks ( Han et al. , 2020 ; Khan et al. , 2021 ; Li et al. , 2021d ; Han et al. , 2021b ; Yu et al. , 2021 ; Li et al. , 2021c ; Yang et al. , 2021 ) . Pure Transformer based models are becoming more and more popular . For example , ViT ( Dosovitskiy et al. , 2020 ) is proposed recently by feeding transformer with sequences of image patches ; Touvron et al . ( Touvron et al. , 2021 ) propose DeiT that introduces a distillation strategy for transformers to help with ViT training ; many other ViT variants ( Yuan et al. , 2021a ; Wang et al. , 2021 ; Han et al. , 2021a ; Chen et al. , 2021 ; Ranftl et al. , 2021 ; Liu et al. , 2021 ) are proposed from then , which achieve promising performance compared with its counterpart CNNs for both image classification and downstream tasks , such as object detection ( Liu et al. , 2021 ) , semantic segmentation ( Yuan et al. , 2021b ) and object ReID ( He et al. , 2021 ) . For multi-modal based networks , there are several works ( Tsai et al. , 2019 ; Li et al. , 2021e ; Hu & Singh , 2021 ) that apply cross-attention for multi-modal feature fusion , which demonstrates that attention mechanism is powerful at distilling noise and feature alignment . This paper adopts cross-attention in the context of pure transformers for UDA tasks . 2.2 UNSUPERVISED DOMAIN ADAPTATION . There are mainly two levels for UDA methods : domain-level ( Tzeng et al. , 2014 ; Long et al. , 2015 ; Ghifary et al. , 2016 ; Tzeng et al. , 2017 ; Bousmalis et al. , 2017 ; Hoffman et al. , 2018 ) and categorylevel ( Saito et al. , 2018 ; Kang et al. , 2019 ; Du et al. , 2021 ; Li et al. , 2021a ) . Domain-level UDA mitigates the distribution divergence between the source and target domain by pulling them into the same distribution at different scale levels . The commonly used divergence measures include Maximum Mean Discrepancy ( MMD ) ( Gretton et al. , 2006 ; Tzeng et al. , 2014 ; Long et al. , 2015 ) and Correlation Alignment ( CORAL ) ( Sun et al. , 2016 ; Sun & Saenko , 2016 ) . Recently , some works ( Saito et al. , 2018 ; Du et al. , 2021 ; Li et al. , 2021a ) focus on the fine-grained category-level label distribution alignment through an adversarial manner between the feature extractor and two domain-specific classifiers . Unlike coarse-grained alignment at the domain scale , this approach aligns each category distribution between the source and target domain data by pushing the target samples to the distribution of source samples in each category . Obviously , the fine-grained alignment results in more accurate distribution alignment within the same label space . Although the adversarial approach achieves new improvements by fusing fine-grained alignment operations of source and target samples at the category level , it still does not solve the problem of noisy samples in the wrong category . Our method adopts Transformers for category-level UDA to solve the noise problem . 2.3 PSEUDO LABELING . Pseudo labeling ( Lee et al. , 2013 ) is first introduced for semi-supervised learning and gains popularity in domain adaptation tasks . It learns to label unlabeled data using predicted probabilities and performs fine-tuning together with labeled data . In terms of using pseudo labeling for domain adaptation tasks , ( Long et al. , 2017 ; 2018 ) adopt pseudo labels to conduct conditional distribution alignment ; ( Zhang et al. , 2018 ; Choi et al. , 2019 ) use pseudo labels as a regularization for domain adaptation ; Zou et al . ( 2018 ) designs a self-training framework by alternately solving pseudo labels ; Caron et al . ( 2018 ) propose a deep self-supervised method by generating pseudo labels via k-means cluster to progressively train the model ; Liang et al . ( 2020 ) develop a self-supervised pseudo labeling method to alleviate the effects of noisy pseudo labels . Based on Liang et al . ( 2020 ) , in this work , we propose a two-way center-aware labeling algorithm to further filter the noisy pseudo pairs . 3 THE PROPOSED METHOD . We first introduce the cross attention module and analyze its robustness to the noise in Section 3.1 . Then the two-way center-aware labeling method is presented in Section 3.2 . With the produced pseudo labels as inputs , our cross-domain transformer ( CDTrans ) is proposed in Section 3.3 , consisting of three weight-sharing transformers . 3.1 THE CROSS ATTENTION IN TRANSFORMER . 3.1.1 PRELIMINARY . Vision Transformer ( ViT ) ( Dosovitskiy et al. , 2020 ) has achieved comparable or even superior performance on computer vision tasks . One of the most important structures in ViT is the selfattention module ( Vaswani et al. , 2017 ) . In ViT , an image I ∈ RH×W×C is reshaped into a sequence of flattened 2D patches x ∈ RN× ( P 2·C ) , where ( H , W ) is the resolution of the original image , C is the number of channels , ( P , P ) is the resolution of each image patch , and N = HW/P 2 is the resulting number of patches . For self-attention , the patches are first projected into three vectors , i.e . queries Q ∈ RN×dk , keys K ∈ RN×dk and values V ∈ RN×dv . dk and dv indicates their dimensions . The output is computed as a weighted sum of the values , where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key . The N patches serve as the inputs for the self-attention module , and the process can be formulated as below . The self-attention module aims to emphasize relationships among patches of the input image . Attnself ( Q , K , V ) = softmax ( QKT√ dk ) V ( 1 ) The cross-attention module is derived from the self-attention module . The difference is that the input of cross-attention is a pair of images , i.e . Is and It . Its query and key/value are from patches of Is and It respectively . The cross-attention module can be calculated as follows : Attncross ( Qs , Kt , Vt ) = softmax ( QsK T t√ dk ) Vt ( 2 ) where Qs ∈RM×dk are queries from M patches of image Is , and Kt ∈RN×dk , Vt ∈RN×dv are keys and values from N patches of image It . The output of the cross-attention module holds the same length M as the number of the queries . For each output , it is calculated by multiplying Vt with attention weights , which comes from the similarity between the corresponding query in Is and all the keys in It . As a result , among all patches in It , the patch that is more similar to the query of Is would hold a larger weight and contribute more to the output . In other words , the output of the cross-attention module manages to aggregate the two input images based on their similar patches . So far , many researchers have utilized the cross-attention for feature fusion , especially in multimodal tasks ( Tsai et al. , 2019 ; Li et al. , 2019 ; Hu & Singh , 2021 ; Li et al. , 2021e ) . In these works , the inputs of the cross-attention module are from two modalities , e.g . vision-to-text ( Tsai et al. , 2019 ; Hu & Singh , 2021 ) , text-to-speech ( Li et al. , 2019 ) and vision-to-vision ( Li et al. , 2021e ) . They apply the cross-attention to aggregate and align the information from two modalities . Given its great power in feature alignment , we propose to use the cross attention module to solve the unsupervised domain adaptation problem .

Summary: This paper proposes a weight-sharing triple-branch transformer framework, or CDTrans for unsupervised domain adaptation. A two-way center-aware labeling method is proposed to provide better pseudo-labels. SOTA performances were achieved via the proposed method.