CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

1 INTRODUCTION . Deep neural network have achieved remarkable success in a wide range of application scenarios but it still suffers poor generalization performance to other new domain because of the domain shift problem ( Csurka , 2017 ; Zhao et al. , 2020 ; Zhang et al. , 2020 ; Oza et al. , 2021 ) . To handle this issue and avoid the expensive laborious annotations , lots of research efforts ( Bousmalis et al. , 2017 ; Kuroki et al. , 2019 ; Wilson & Cook , 2020 ; VS et al. , 2021 ) are devoted on Unsupervised Domain Adaptation ( UDA ) . The UDA task aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain . In UDA , most approaches focus on aligning distributions of source and target domain and learning domain-invariant feature representations . One kind of such UDA methods are based on category-level alignment ( Kang et al. , 2019 ; Zhang et al. , 2019 ; Jiang et al. , 2020 ; Li et al. , 2021b ) , which have achieved promising results on public UDA datasets using deep convolution neural networks ( CNNs ) . The fundamental problems in category-level based alignment is the production of pseudo labels for samples in target domain to generate the input source-target pairs . However , the current CNNs-based methods are not robust to the generated noisy pseudo labels for accurate domain alignment ( Morerio et al. , 2020 ; Jiang et al. , 2020 ) . With the success of Transformer in natural language processing ( NLP ) ( Vaswani et al. , 2017 ; Devlin et al. , 2018 ) and vision tasks ( Dosovitskiy et al. , 2020 ; Han et al. , 2020 ; He et al. , 2021 ; Khan et al. , 2021 ) , it is found that cross-attention in Transformer is good at aligning different distributions , even from different modalities e.g. , vision-to-vision ( Li et al. , 2021e ) , vision-to-text ( Tsai et al. , 2019 ; Hu & Singh , 2021 ) and text-to-speech ( Li et al. , 2019 ) . And we find that it is robust to noise in pseudo labels to some extent . Hence , in this paper , we apply transformers to the UDA task to take advantage of its robustness to noise and super power for feature alignment to deal with the problems as described above in CNNs . In our experiment , we conclude that even with noise in the labeling pair , the cross-attention can still work well in aligning two distributions , thanks to the attention mechanism . To obtain more accurate pseudo labels , we designed a two-way center-aware labeling algorithm for samples in the target domain . The pseudo labels are produced based on the cross-domain similarity matrix , and a center-aware matching is involved to weight the matrix and weaken noise into the tolerable range . With the help of pseudo labels , we design the cross-domain transformer ( CDTrans ) for UDA . It consists of three weight-sharing transformer branches , of which two branches are for source and target data respectively and the third one is the feature alignment branch , whose inputs are from source-target pairs . The self-attention is applied in the source/target transformer branches and crossattention is involved in the feature alignment branch to conduct domain alignment . Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously . In summary , our contributions are three-fold : • We propose a weight-sharing triple-branch transformer framework , namely , CDTrans , for accurate unsupervised domain adaptation , taking advantage of its robustness to noisy labeling data and great power for feature alignment . • To produce pseudo labels with high quality , a two-way center-aware labeling method is proposed , and it boosts the final performance in the context of CDTrans . • CDTrans achieves the best performance compared to state-of-the-arts with a large margin on VisDA-2017 ( Peng et al. , 2017 ) and DomainNet ( Peng et al. , 2019 ) datasets . 2 RELATED WORK . 2.1 TRANSFORMER FOR VISION . Transformer is proposed in ( Vaswani et al. , 2017 ) to model sequential data in the field of NLP . Many works have shown its effectiveness for computer-vision tasks ( Han et al. , 2020 ; Khan et al. , 2021 ; Li et al. , 2021d ; Han et al. , 2021b ; Yu et al. , 2021 ; Li et al. , 2021c ; Yang et al. , 2021 ) . Pure Transformer based models are becoming more and more popular . For example , ViT ( Dosovitskiy et al. , 2020 ) is proposed recently by feeding transformer with sequences of image patches ; Touvron et al . ( Touvron et al. , 2021 ) propose DeiT that introduces a distillation strategy for transformers to help with ViT training ; many other ViT variants ( Yuan et al. , 2021a ; Wang et al. , 2021 ; Han et al. , 2021a ; Chen et al. , 2021 ; Ranftl et al. , 2021 ; Liu et al. , 2021 ) are proposed from then , which achieve promising performance compared with its counterpart CNNs for both image classification and downstream tasks , such as object detection ( Liu et al. , 2021 ) , semantic segmentation ( Yuan et al. , 2021b ) and object ReID ( He et al. , 2021 ) . For multi-modal based networks , there are several works ( Tsai et al. , 2019 ; Li et al. , 2021e ; Hu & Singh , 2021 ) that apply cross-attention for multi-modal feature fusion , which demonstrates that attention mechanism is powerful at distilling noise and feature alignment . This paper adopts cross-attention in the context of pure transformers for UDA tasks . 2.2 UNSUPERVISED DOMAIN ADAPTATION . There are mainly two levels for UDA methods : domain-level ( Tzeng et al. , 2014 ; Long et al. , 2015 ; Ghifary et al. , 2016 ; Tzeng et al. , 2017 ; Bousmalis et al. , 2017 ; Hoffman et al. , 2018 ) and categorylevel ( Saito et al. , 2018 ; Kang et al. , 2019 ; Du et al. , 2021 ; Li et al. , 2021a ) . Domain-level UDA mitigates the distribution divergence between the source and target domain by pulling them into the same distribution at different scale levels . The commonly used divergence measures include Maximum Mean Discrepancy ( MMD ) ( Gretton et al. , 2006 ; Tzeng et al. , 2014 ; Long et al. , 2015 ) and Correlation Alignment ( CORAL ) ( Sun et al. , 2016 ; Sun & Saenko , 2016 ) . Recently , some works ( Saito et al. , 2018 ; Du et al. , 2021 ; Li et al. , 2021a ) focus on the fine-grained category-level label distribution alignment through an adversarial manner between the feature extractor and two domain-specific classifiers . Unlike coarse-grained alignment at the domain scale , this approach aligns each category distribution between the source and target domain data by pushing the target samples to the distribution of source samples in each category . Obviously , the fine-grained alignment results in more accurate distribution alignment within the same label space . Although the adversarial approach achieves new improvements by fusing fine-grained alignment operations of source and target samples at the category level , it still does not solve the problem of noisy samples in the wrong category . Our method adopts Transformers for category-level UDA to solve the noise problem . 2.3 PSEUDO LABELING . Pseudo labeling ( Lee et al. , 2013 ) is first introduced for semi-supervised learning and gains popularity in domain adaptation tasks . It learns to label unlabeled data using predicted probabilities and performs fine-tuning together with labeled data . In terms of using pseudo labeling for domain adaptation tasks , ( Long et al. , 2017 ; 2018 ) adopt pseudo labels to conduct conditional distribution alignment ; ( Zhang et al. , 2018 ; Choi et al. , 2019 ) use pseudo labels as a regularization for domain adaptation ; Zou et al . ( 2018 ) designs a self-training framework by alternately solving pseudo labels ; Caron et al . ( 2018 ) propose a deep self-supervised method by generating pseudo labels via k-means cluster to progressively train the model ; Liang et al . ( 2020 ) develop a self-supervised pseudo labeling method to alleviate the effects of noisy pseudo labels . Based on Liang et al . ( 2020 ) , in this work , we propose a two-way center-aware labeling algorithm to further filter the noisy pseudo pairs . 3 THE PROPOSED METHOD . We first introduce the cross attention module and analyze its robustness to the noise in Section 3.1 . Then the two-way center-aware labeling method is presented in Section 3.2 . With the produced pseudo labels as inputs , our cross-domain transformer ( CDTrans ) is proposed in Section 3.3 , consisting of three weight-sharing transformers . 3.1 THE CROSS ATTENTION IN TRANSFORMER . 3.1.1 PRELIMINARY . Vision Transformer ( ViT ) ( Dosovitskiy et al. , 2020 ) has achieved comparable or even superior performance on computer vision tasks . One of the most important structures in ViT is the selfattention module ( Vaswani et al. , 2017 ) . In ViT , an image I ∈ RH×W×C is reshaped into a sequence of flattened 2D patches x ∈ RN× ( P 2·C ) , where ( H , W ) is the resolution of the original image , C is the number of channels , ( P , P ) is the resolution of each image patch , and N = HW/P 2 is the resulting number of patches . For self-attention , the patches are first projected into three vectors , i.e . queries Q ∈ RN×dk , keys K ∈ RN×dk and values V ∈ RN×dv . dk and dv indicates their dimensions . The output is computed as a weighted sum of the values , where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key . The N patches serve as the inputs for the self-attention module , and the process can be formulated as below . The self-attention module aims to emphasize relationships among patches of the input image . Attnself ( Q , K , V ) = softmax ( QKT√ dk ) V ( 1 ) The cross-attention module is derived from the self-attention module . The difference is that the input of cross-attention is a pair of images , i.e . Is and It . Its query and key/value are from patches of Is and It respectively . The cross-attention module can be calculated as follows : Attncross ( Qs , Kt , Vt ) = softmax ( QsK T t√ dk ) Vt ( 2 ) where Qs ∈RM×dk are queries from M patches of image Is , and Kt ∈RN×dk , Vt ∈RN×dv are keys and values from N patches of image It . The output of the cross-attention module holds the same length M as the number of the queries . For each output , it is calculated by multiplying Vt with attention weights , which comes from the similarity between the corresponding query in Is and all the keys in It . As a result , among all patches in It , the patch that is more similar to the query of Is would hold a larger weight and contribute more to the output . In other words , the output of the cross-attention module manages to aggregate the two input images based on their similar patches . So far , many researchers have utilized the cross-attention for feature fusion , especially in multimodal tasks ( Tsai et al. , 2019 ; Li et al. , 2019 ; Hu & Singh , 2021 ; Li et al. , 2021e ) . In these works , the inputs of the cross-attention module are from two modalities , e.g . vision-to-text ( Tsai et al. , 2019 ; Hu & Singh , 2021 ) , text-to-speech ( Li et al. , 2019 ) and vision-to-vision ( Li et al. , 2021e ) . They apply the cross-attention to aggregate and align the information from two modalities . Given its great power in feature alignment , we propose to use the cross attention module to solve the unsupervised domain adaptation problem .

Summary: This submission proposes a transformer framework for unsupervised domain adaptive classification tasks. In this submission, they conduct an exploration about cross attention layer and found that the cross attention layer is robust to pseudo label noise. Inspired by this, they construct a three branches architecture in this submission, which includes a source transformer, target transformer, a source-target transformer. Due to the robustness of the cross attention layer, the source-target transformer acts as a teacher to guide the other two branches.