Year: 2021

Abstract:
Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. Most existing UDA methods focus on learning domain-invariant feature representation, either from the domain level or category level, using convolution neural networks (CNNs)-based frameworks. One fundamental problem for the category level based UDA is the production of pseudo labels for samples in target domain, which are usually too noisy for accurate domain alignment, inevitably compromising the UDA performance. With the success of Transformer in various tasks, we find that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task. Specifically, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples. Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and source-target domain alignment, respectively. Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously. The proposed method is dubbed CDTrans (cross-domain transformer), and it provides one of the first attempts to solve UDA tasks with a pure transformer solution. Experiments show that our proposed method achieves the best performance on public UDA datasets, e.g. VisDA-2017 and DomainNet.

heading: 1 INTRODUCTION
text: Deep neural network have achieved remarkable success in a wide range of application scenarios but it still suffers poor generalization performance to other new domain because of the domain shift problem (Csurka, 2017; Zhao et al., 2020; Zhang et al., 2020; Oza et al., 2021). To handle this issue and avoid the expensive laborious annotations, lots of research efforts (Bousmalis et al., 2017; Kuroki et al., 2019; Wilson & Cook, 2020; VS et al., 2021) are devoted on Unsupervised Domain Adaptation (UDA). The UDA task aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. In UDA, most approaches focus on aligning distributions of source and target domain and learning domain-invariant feature representations. One kind of such UDA methods are based on category-level alignment (Kang et al., 2019; Zhang et al., 2019; Jiang et al., 2020; Li et al., 2021b), which have achieved promising results on public UDA datasets using deep convolution neural networks (CNNs). The fundamental problems in category-level based alignment is the production of pseudo labels for samples in target domain to generate the input source-target pairs. However, the current CNNs-based methods are not robust to the generated noisy pseudo labels for accurate domain alignment (Morerio et al., 2020; Jiang et al., 2020).
With the success of Transformer in natural language processing (NLP) (Vaswani et al., 2017; Devlin et al., 2018) and vision tasks (Dosovitskiy et al., 2020; Han et al., 2020; He et al., 2021; Khan et al., 2021), it is found that cross-attention in Transformer is good at aligning different distributions, even from different modalities e.g., vision-to-vision (Li et al., 2021e), vision-to-text (Tsai et al., 2019; Hu & Singh, 2021) and text-to-speech (Li et al., 2019). And we find that it is robust to noise in pseudo labels to some extent. Hence, in this paper, we apply transformers to the UDA task to take advantage of its robustness to noise and super power for feature alignment to deal with the problems as described above in CNNs.
In our experiment, we conclude that even with noise in the labeling pair, the cross-attention can still work well in aligning two distributions, thanks to the attention mechanism. To obtain more accurate pseudo labels, we designed a two-way center-aware labeling algorithm for samples in the target domain. The pseudo labels are produced based on the cross-domain similarity matrix, and a center-aware matching is involved to weight the matrix and weaken noise into the tolerable range. With the help of pseudo labels, we design the cross-domain transformer (CDTrans) for UDA. It consists of three weight-sharing transformer branches, of which two branches are for source and target data respectively and the third one is the feature alignment branch, whose inputs are from source-target pairs. The self-attention is applied in the source/target transformer branches and crossattention is involved in the feature alignment branch to conduct domain alignment. Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously. In summary, our contributions are three-fold:
• We propose a weight-sharing triple-branch transformer framework, namely, CDTrans, for accurate unsupervised domain adaptation, taking advantage of its robustness to noisy labeling data and great power for feature alignment.
• To produce pseudo labels with high quality, a two-way center-aware labeling method is proposed, and it boosts the final performance in the context of CDTrans.
• CDTrans achieves the best performance compared to state-of-the-arts with a large margin on VisDA-2017 (Peng et al., 2017) and DomainNet (Peng et al., 2019) datasets.

heading: 2 RELATED WORK

heading: 2.1 TRANSFORMER FOR VISION
text: Transformer is proposed in (Vaswani et al., 2017) to model sequential data in the field of NLP. Many works have shown its effectiveness for computer-vision tasks (Han et al., 2020; Khan et al., 2021; Li et al., 2021d; Han et al., 2021b; Yu et al., 2021; Li et al., 2021c; Yang et al., 2021). Pure Transformer based models are becoming more and more popular. For example, ViT (Dosovitskiy et al., 2020) is proposed recently by feeding transformer with sequences of image patches; Touvron et al. (Touvron et al., 2021) propose DeiT that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al., 2021a; Wang et al., 2021; Han et al., 2021a; Chen et al., 2021; Ranftl et al., 2021; Liu et al., 2021) are proposed from then, which achieve promising performance compared with its counterpart CNNs for both image classification and downstream tasks, such as object detection (Liu et al., 2021), semantic segmentation (Yuan et al., 2021b) and object ReID (He et al., 2021). For multi-modal based networks, there are several works (Tsai et al., 2019; Li et al., 2021e; Hu & Singh, 2021) that apply cross-attention for multi-modal feature fusion, which demonstrates that attention mechanism is powerful at distilling noise and feature alignment. This paper adopts cross-attention in the context of pure transformers for UDA tasks.

heading: 2.2 UNSUPERVISED DOMAIN ADAPTATION
text: There are mainly two levels for UDA methods: domain-level (Tzeng et al., 2014; Long et al., 2015; Ghifary et al., 2016; Tzeng et al., 2017; Bousmalis et al., 2017; Hoffman et al., 2018) and categorylevel (Saito et al., 2018; Kang et al., 2019; Du et al., 2021; Li et al., 2021a). Domain-level UDA mitigates the distribution divergence between the source and target domain by pulling them into the same distribution at different scale levels. The commonly used divergence measures include Maximum Mean Discrepancy (MMD) (Gretton et al., 2006; Tzeng et al., 2014; Long et al., 2015) and Correlation Alignment (CORAL) (Sun et al., 2016; Sun & Saenko, 2016). Recently, some works (Saito et al., 2018; Du et al., 2021; Li et al., 2021a) focus on the fine-grained category-level label distribution alignment through an adversarial manner between the feature extractor and two domain-specific classifiers. Unlike coarse-grained alignment at the domain scale, this approach aligns each category distribution between the source and target domain data by pushing the target samples to the distribution of source samples in each category. Obviously, the fine-grained alignment results in more accurate distribution alignment within the same label space. Although the adversarial approach achieves new improvements by fusing fine-grained alignment operations of source and target samples at the category level, it still does not solve the problem of noisy samples in the wrong category. Our method adopts Transformers for category-level UDA to solve the noise problem.

heading: 2.3 PSEUDO LABELING
text: Pseudo labeling (Lee et al., 2013) is first introduced for semi-supervised learning and gains popularity in domain adaptation tasks. It learns to label unlabeled data using predicted probabilities and performs fine-tuning together with labeled data. In terms of using pseudo labeling for domain adaptation tasks, (Long et al., 2017; 2018) adopt pseudo labels to conduct conditional distribution alignment; (Zhang et al., 2018; Choi et al., 2019) use pseudo labels as a regularization for domain adaptation; Zou et al. (2018) designs a self-training framework by alternately solving pseudo labels; Caron et al. (2018) propose a deep self-supervised method by generating pseudo labels via k-means cluster to progressively train the model; Liang et al. (2020) develop a self-supervised pseudo labeling method to alleviate the effects of noisy pseudo labels. Based on Liang et al. (2020), in this work, we propose a two-way center-aware labeling algorithm to further filter the noisy pseudo pairs.

heading: 3 THE PROPOSED METHOD
text: We first introduce the cross attention module and analyze its robustness to the noise in Section 3.1. Then the two-way center-aware labeling method is presented in Section 3.2. With the produced pseudo labels as inputs, our cross-domain transformer (CDTrans) is proposed in Section 3.3, consisting of three weight-sharing transformers.

heading: 3.1 THE CROSS ATTENTION IN TRANSFORMER

heading: 3.1.1 PRELIMINARY
text: Vision Transformer (ViT) (Dosovitskiy et al., 2020) has achieved comparable or even superior performance on computer vision tasks. One of the most important structures in ViT is the selfattention module (Vaswani et al., 2017). In ViT, an image I ∈ RH×W×C is reshaped into a sequence of flattened 2D patches x ∈ RN×(P 2·C), where (H,W ) is the resolution of the original image, C is the number of channels, (P, P ) is the resolution of each image patch, and N = HW/P 2 is the resulting number of patches. For self-attention, the patches are first projected into three vectors, i.e. queries Q ∈ RN×dk , keys K ∈ RN×dk and values V ∈ RN×dv . dk and dv indicates their dimensions. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. The N patches serve as the inputs for the self-attention module, and the process can be formulated as below. The self-attention module aims to emphasize relationships among patches of the input image.
Attnself (Q,K,V ) = softmax( QKT√ dk )V (1)
The cross-attention module is derived from the self-attention module. The difference is that the input of cross-attention is a pair of images, i.e. Is and It. Its query and key/value are from patches of Is and It respectively. The cross-attention module can be calculated as follows:
Attncross(Qs,Kt,Vt) = softmax( QsK T t√
dk )Vt (2)
where Qs ∈RM×dk are queries from M patches of image Is, and Kt ∈RN×dk ,Vt ∈RN×dv are keys and values from N patches of image It. The output of the cross-attention module holds the same length M as the number of the queries. For each output, it is calculated by multiplying Vt with attention weights, which comes from the similarity between the corresponding query in Is and all the keys in It. As a result, among all patches in It, the patch that is more similar to the query of Is would hold a larger weight and contribute more to the output. In other words, the output of the cross-attention module manages to aggregate the two input images based on their similar patches.
So far, many researchers have utilized the cross-attention for feature fusion, especially in multimodal tasks (Tsai et al., 2019; Li et al., 2019; Hu & Singh, 2021; Li et al., 2021e). In these works, the inputs of the cross-attention module are from two modalities, e.g. vision-to-text (Tsai et al., 2019; Hu & Singh, 2021), text-to-speech (Li et al., 2019) and vision-to-vision (Li et al., 2021e). They apply the cross-attention to aggregate and align the information from two modalities. Given
its great power in feature alignment, we propose to use the cross attention module to solve the unsupervised domain adaptation problem.

heading: 3.1.2 ROBUSTNESS TO NOISE
text: As mentioned above, the input of the cross-attention module is a pair of images, which usually comes from two domains, and the cross-attention module aims to align these two images. If label noise exists, there would be false positive pairs in the training data. Images in the false positive pairs would have dissimilar appearance, and forcibly aligning their features would inevitably injure the training and compromise the performance. We assume that the dissimilar patches in false positive pairs are more harmful to the performance than the similar patches. In the cross-attention module, two images are aligned based on their patch similarity. As shown in Fig. 1a, the cross-attention module would assign a low weight to the dissimilar patches in false positive pairs. Thus it weakens the negative effects of the dissimilar patches on the final performance to some extent.1
To further analyze this issue, an experiment is carefully designed. Specifically, we randomly sample true positive pairs from source and target domain in VisDA-2017 dataset (Peng et al., 2017) as the training data. Then we manually replace the true positive pairs with random false positive pairs to increase the noise, and watch the changes of the performance as shown in Fig. 1b. The x-axis indicates the rate of false positive pairs in the training data, and the y-axis shows the performance of different methods on the UDA task. The red curve represents the results by aligning pairs with the cross-attention module, while the green curve is that without cross-attention, i.e. to directly train the target data with the label of corresponding source data in the pair. It can be seen that the red curve achieves a much better performance than the green one, which implies the robustness of the cross-attention module to the noise.
We also provide another baseline shown as the blue curve in Fig. 1b, which is to remove the false positive pairs from the training data and train the cross-attention with only true positive pairs. Without the noisy data, this baseline can be considered as the upper bound to our methods. We can see the red curve is very close to the blue curve, and both of them are much better than the green one. It further implies that the cross-attention module is robust to the noisy input pair.

heading: 3.2 TWO-WAY CENTER-AWARE PSEUDO LABELING

heading: 3.2.1 TWO-WAY LABELING
text: To build the training pairs for the cross-attention module, an intuitive method is that for each image in the source domain, we manage to find the most similar image from the target domain. The set PS
1The patches in true positive pairs, either similar and dissimilar, would bring no noise to the final performance, which is out of the discussion of our paper.
of selected pairs is:
PS = {(s, t)|t = min k d(fs,fk),∀k ∈ T, ∀s ∈ S} (3)
where S, T are the source and target data respectively. d(fi,fj) means the distance between features of image i and j. The advantage of this strategy is to make full use of source data, while its weakness is obvious that only a part of target data is involved. To eliminate this training bias from target data, we introduce more pairs PT from the opposite way, consisting of all the target data and their corresponding most similar images in the source domain.
PT = {(s, t)|s = min k d(ft,fk),∀t ∈ T, ∀k ∈ S} (4)
As a result the final set P is the union of two sets, i.e. P = {PS ∪ PT }, making the training pairs include all the source and target data.

heading: 3.2.2 CENTER-AWARE FILTERING
text: The pairs in P are built based on the feature similarities of images from both domains, thus the accuracy of the pseudo labels of pairs is highly dependent on the feature similarities. Inspired by Liang et al. (2020), we find that the pre-trained model of the source data is also useful to further improve the accuracy. Firstly, we send all the target data through the pre-trained model and obtain their probability distributions δ on the source categories from the classifier. Similar to Liang et al. (2020), these distributions can be used to compute initial centers of each category in the target domain by weighted k-means clustering:
ck =
∑ t∈T δ
k t ft∑
t∈T δ k t
(5)
where δkt indicates the probability of image t on category k. Pseudo labels of the target data can be produced via the nearest neighbor classifier:
yt = argmin k d(ck,ft) (6)
where t ∈ T and d(i, j) is the distance of features i and j. Based on the pseudo labels, we can calculate new centers:
c′k = ∑ t∈T 1(yt = k)ft∑ t∈T 1(yt = k)
(7)
In Liang et al. (2020), Eq. 6 and 7 could be updated for multiple rounds, and we only adopt one round in our paper. The final pseudo labels are then used to refine the selected pairs. Specifically, for every pair, if the pseudo label of the target image is consistent with the label of the source image, this pair would be kept for our training, otherwise it will be discarded as a noise.

heading: 3.3 CDTRANS: CROSS-DOMAIN TRANSFORMER
text: The framework of the proposed Cross-domain Transformer (CDTrans) is shown in Fig. 2, which consists of three weight-sharing transformers. There are three data flows and constraints for the weight-sharing branches.
The inputs of the framework are the selected pairs from our labeling method mentioned above. The three branches are named as source branch, target branch, source-target branch. As shown in Fig. 2, the source and target images in the input pair are sent to source branch and target branch respectively. In these two branches, the self-attention module is involved to learn the domain-specific representations. And the softmax cross-entropy loss is used to train the classification. It is worth noting that all three branches share the same classifier due to the same label of two images.
The cross-attention module is imported in the source-target branch. The inputs of the source-target branch are from the other two branches. In the N -th layer, the query of the cross-attention module comes from the query in the N -th layer of the source branch, while the keys and values are from
those of the target branch. Then the cross-attention module outputs aligned features which are added with the output of the (N−1)-th layer.2
The features of the source-target branch not only align distributions of two domains, but are robust to the noise in the input pairs thanks to the cross-attention module. Thus we use the output of the source-target branch to guide the training of the target branch. Specifically, the source-target branch and target branch are denoted as teacher and student respectively. We consider the probability distribution of the classifier in source-target branch as a soft label that can be used to further supervise the target branch through a distillation loss (Hinton et al., 2015):
Ldtl = ∑ k qk log pk (8)
where qk and pk are the probabilities of category k from the source-target branch and the target branch respectively.
During inference, only the target branch is used. The input is an image from testing data, and only the target data flow is triggered, i.e. the blue lines in Fig. 2. Its output of the classifier is utilized as the final predicted labels.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS AND IMPLEMENTATION
text: The proposed method is verified on four popular UDA benchmarks, including VisDA-2017 (Peng et al., 2017), Office-Home (Venkateswara et al., 2017), Office-31 (Saenko et al., 2010) and DomainNet (Peng et al., 2019). The input image size in our experiments is 224×224. Both the DeiT-small and DeiT-base (Touvron et al., 2021) are adopted as our backbone for fair comparison. We use the Stochastic Gradient Descent algorithm with the momentum of 0.9 and weight decay ratio 1e-4 to optimize the training process. The learning rate is set to 3e-3 for Office-Home, Office-31 and DomainNet, 5e-5 for VisDA-2017 since it can easily converge. The batch size is set to 64.

heading: 4.2 COMPARISON TO SOTA
text: We compare our method with state-of-the-art methods on UDA tasks, including MinEnt (Grandvalet et al., 2005), DAN (Long et al., 2015), DANN (Ganin & Lempitsky, 2015), ADR (Saito et al., 2017), CDAN+E (Long et al., 2018), CDAN+BSP (Chen et al., 2019), CDAN+TN (Wang et al., 2019), rRGrad+CAT (Deng et al., 2019), MCD (Saito et al., 2018), SWD (Lee et al., 2019), AFN (Xu
2The addition operation is not included in the 1st layer.
et al., 2019), MSTN+DSBN (Chang et al., 2019), SAFN (Xu et al., 2019), SAFN+ENT (Xu et al., 2019), CADA-P (Kurmi et al., 2019), BNM (Cui et al., 2020), DCAN (Li et al., 2020), SHOT (Liang et al., 2020), MCC (Jin et al., 2020), ATDOC-NA (Liang et al., 2021), CGDM (Du et al., 2021) and TVT (Yang et al.). The results are shown in Table 1, 2, 3 and 4.
For Office-Home, Office-31 and DomainNet, as most of the methods use ResNet-50 as their backbones, we provide results with DeiT-small as our backbone for a fair comparison, which has a comparable model size as ResNet-50, but we also show the results using DeiT-base. And for VisDA-2017, we adopt the DeiT-base backbone for fair comparisons, where other methods utilize ResNet-101 for their results.
The “Baseline-S/B” indicates directly training a DeiT-small/DeiT-base on the source domain and testing on the target domain. The baseline shows a competitive result even compared to other SoTA methods on most datasets. It demonstrates that Transformers has better generalization ability over ConvNets. We also provide some insights about why transformers can generalize well from source domain to target domain in the supplementary materials. To further eliminate the unfairness of using different backbones, we reproduce the results of SHOT and CGDM (marked as “*”), and replace their backbones with DeiT-base as the same as ours, denoted as “-B*”.
From Table 1, 2, 3 and 4, it can be seen that our method outperforms the baseline with a large margin on all four datasets, e.g. nearly 21% on VisDA. With our improvements, the new Transformer with cross-attention module shows a much better generalization power, and achieves the best performance on VisDA-2017 compared to other SoTAs methods. It further implies the effectiveness of our method on the UDA task.
Taking a closer look at the results, for the hard categories, such as “person” in VisDA-2017 dataset, the baseline is very low, which indicates the initial model of our method has a poor classification ability on this category, leading to the pseudo labels with more noise. Even with
such a poor baseline and poor quality of pseudo labels, our method can still achieve a much higher performance boost (from 10.3% to 88.6%). It suggests that our method has a great robustness to the labeling noise and can overcome the noise problem to some extent. In Table 4, although the average performance of our methods outperforms other approaches by a large margin, it is noticed that the results of qdr-to-others is much lower due to the low baseline. It is because the Transformer is weak in shape modeling caused by its hard patch partition (Naseer et al., 2021).
We can see that TVT achieves a better result on Office-Home and Office-31. Because TVT utilizes ViT (Dosovitskiy et al., 2020) as backbone which is pretrained on ImageNet21K. While the pretrained model of our CDTrans and other UDA methods are trained on ImageNet1K.

heading: 4.3 ABLATION STUDY

heading: 4.3.1 DIFFERENT PSEUDO LABELING
text: We have conducted experiments on different pseudo labeling methods to verify their influence on the final performance. The results on VisDa-2017 are listed in Table. 5. RPLL (Zheng & Yang, 2021) and MRKLD+LRENT (Zou et al., 2019) are two commonly used pseudo-label generation methods, we reproduce their pseudo-label generations on our baseline to compare with our proposed pseudo labeling method. Recs, Rect means the recall of the selected training pairs in the source and target
data, while Prec represents the accuracy of the pairs. “One-way-source” and “One-way-target” denote only using the pair set PS in Eq. 3 or PT in Eq. 4 for training. “Two-way” indicates results of using the union of Ps and Pt without the center-aware strategy. “Tw+Ca” implies our two-way center-aware labeling method, and “Groundtruth” means all training pairs are from groundtruthes.
By looking atRecs andRect in Table. 5, it can be found that the one-way methods have an apparent bias on either source or target data, and its results are lower than the two-way method. By comparing “Two-way” and “Tw+Ca”, we can conclude that although the center-aware method filters the training pair and slightly reduces the recall, it largely improves the precision and leads to a better final performance. We also find that our two-way center-aware labeling method achieves a very high result, not only better than other pseudo-label generation methods, but also very close to the upper bound trained with groundtruth pairs.

heading: 4.3.2 DIFFERENT LOSSES
text: As there are three losses in our method, we conduct another experiment to verify the effectiveness of each loss on VisDa-2017, as shown in Table. 6. “cls” in Ls+t denotes that we replace the distillation loss with a classification loss for the source-target branch. We can see that the 3rd row with both Ls and Lt having classification loss achieves a better result than the first row where only Ls has the cls loss, which means the target branch with the pseudo labels is helpful to improve the UDA result. With the addition of “cls” in Ls+t, the performance is further improved, which demonstrates the advantages of using the cross-attention module for feature alignment. Using “dtl” instead of “cls” on the source-target branch can further improve the results, showing the effectiveness of our distillation loss.

heading: 5 CONCLUSION
text: In this paper, we tackle the problem of unsupervised domain adaptation by introducing the crossattention module into Transformer in a novel way. We propose a new network structure CDTrans which is a pure transformer-based structure with three branches, and we also propose to generate high-quality pseudo labels using a two-way center-aware labeling method. Training CDTrans using the generated high-quality pseudo labels yields a robust solution and also achieves state-of-the-art results on four popular UDA datasets, outperforming previous methods by a large margin. We believe that transformer-based approaches will have great potential in the UDA community, and our work, as one of the first attempts along this direction, has pushed forward the frontiers and shed lights for future research.

Review summaries:
In this work, the authors propose a method for domain adaptation by introducing a new way of generating pseudo labels and a cross-transformer with classification and distillation losses. The authors used a cross-transformer where the queries come from the source domain and the values and keys come from the target domain, and try to minimize the output distribution difference between the cross branch and the target branch by a distillation loss. Since the transformers compare source and target in a patch-based manner, the authors find it is more robust to false-positive pairs. The authors do not focus on learning domain-invariant feature representation and mainly solve the domain adaptation problem by pseudo labeling. In four public datasets, the proposed method showed outstanding performance.

This paper proposes a weight-sharing triple-branch transformer framework, or CDTrans for unsupervised domain adaptation. A two-way center-aware labeling method is proposed to provide better pseudo-labels. SOTA performances were achieved via the proposed method.

This submission proposes a transformer framework for unsupervised domain adaptive classification tasks. In this submission, they conduct an exploration about cross attention layer and found that the cross attention layer is robust to pseudo label noise. Inspired by this, they construct a three branches architecture in this submission, which includes a source transformer, target transformer, a source-target transformer. Due to the robustness of the cross attention layer, the source-target transformer acts as a teacher to guide the other two branches.