CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
Authors: Tongkun Xu
Weihua Chen
Pichao WANG
Fan Wang
Hao Li
Rong Jin
Venue: ICLR 2022 Poster (2022)

Abstract: Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. Most existing UDA methods focus on learning domain-invariant feature representation, either from the domain level or category level, using convolution neural networks (CNNs)-based frameworks. One fundamental problem for the category level based UDA is the production of pseudo labels for samples in target domain, which are usually too noisy for accurate domain alignment, inevitably compromising the UDA performance.  With the success of Transformer in various tasks, we find that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task. Specifically, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples. Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and source-target domain alignment, respectively. 
Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously. The proposed method is dubbed CDTrans (cross-domain transformer), and it provides one of the first attempts to solve UDA tasks with a pure transformer solution. Experiments show that our proposed method achieves the best performance on public UDA datasets, e.g. VisDA-2017 and DomainNet. Code and models are available at https://github.com/CDTrans/CDTrans.

Published as a conference paper at ICLR 2022
CDTRANS: CROSS-DOMAIN TRANSFORMER FOR UN- SUPERVISED DOMAIN ADAPTATION
Tongkun Xu 12 Weihua Chen' Pichao Wang', Fan Wang's Hao Li', Rong Jin'
Allibaba Group. *Shandong University
wowoonnnnnnn00nmmmmo.. com, kugang, www.aalbaaa--icc........ com
(pichao. wang, fan. 1ihao. 1h, J jinrong. frlealibbbaaaaaaaa com
ABSTRACT
Unsupervised domain adaptation (UDA) aims to transfer knowledge Iearned from a labeled source domain to a different unlabeled target domain. Most existing UDA methods focus learning domain-invariant feature representation, either from the domain level or category level, using convolution neural networks (CNNs)-based frameworks. One fundamental problem for the category level based UDA is the production of pseudo labels for samples in target domain, which are usually too noisy for accurate domain alignment, inevitably compromising the UDA performance. With the success of Transformer in various tasks, we find that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task. Specifically, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples. Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and sarce-target domain alignment, respectively. Such design explicitly enforces the framework to learn discriminative domain-speciff and domain-invariant representations simultaneously. The proposed method dubbed CDTrran (cross-domain transformer), and it provides one of the first attempts to solve UDA tasks with pure transformer solution. Experiments show that our proposed method achieves the best performance on public UDA datasets, e.g. VisDA-2017 and DomainNet. Code and models are available at https: http://tb.cco///rrraa///////////
INTRODUCTION
Deep neural network have achieved remarkable success in a wide range of application scenar-
ios (Wang et al. 2022; Qian al. 2021 Yigi Jiang 2022 Tan et al. 2019 Chen et al. 2021b
Jiang et al.) 2021: Chen et al.j 2017) 7) but still suffers poor generalization performance to other
new domain because of the domain shift problem (Csurka 2017: Zhao et al. 2020; Zhang et al.
2020 Oza et al. 2021). To handle this issue and avoid the expensive laborious annotations, lots of research efforts (Bousmalis al. 2017 Kuroki al. 2019 Wilson & Cook, 2020 VS al., 2021 are devoted on Unsupervised Domain Adaptation (UDA). The UDA task aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. In UDA, most approaches focus aligning distributions of source and target domain and learning domain- invariant feature representations. One kind of such UDA methods are based on category-
alignment (Kang et al.) 2019) Zhang et al. 2019 Tiang et al. 2020 Li et al.) 20211)) which have achieved promising results on public UDA datasets using deep convolution neural networks (CNNs). The fundamental problems in category-level based alignment is the production of pseudo labels for samples in target domain to generate the input source-target pairs. However, the current CNNs-based methods are not robust to the generated noisy pseudo labels for accurate domain alignment (Moreio 2020 Jiang et al.) 2020).
et al.
With the success of Transformer in natural language processing (NLP) Vaswani et al. 2017 Devlin
et al.
2018) and vision tasks (Dosovitskiy et al. 2020 Han et al. 2020 He et al. 202 Khan et al.
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2022
2021]. J. it is found that cross-attention in Transformer is good at aligning different distributions, even from different modalities e.g.. vision-to-vision (Li et al., 2021e). vision-to-text (Tsai et al. 2019 Hu & Singh 202T) and text-to-specch (Li et al. 2019). And we find that it is robust to noise in pseudo labels to some extent. Hence, in this paper, we apply transformers to the UDA task to take advantage of its robustness to noise and super power for feature alignment to deal with the problems
as described above in CNNs.
In our experiment, we conclude that even with noise in the labeling pair, the cross-attention can still work well in aligning two distributions, thanks to the attention mechanism. To obtain more accurate pseudo labels, we designed a two-way center-aware labeling algorithm for samples in the target domain. The pseudo labels are produced based on the cross-domain similarity matrix, and center-aware matching is involved to weight the matrix and weaken noise into the tolerable range. With the help of pseudo labels, we design the cross-domain transformer (CDTraans) for UDA. It consists of three weight-sharing transformer branches, of which two branches are for source and target data respectively and the third one is the feature alignment branch, whose inputs are from source-target pairs. The self-attention is applied in the source/target transformer branches and cross- attention is involved in the feature alignment branch to conduct domain alignment. Such design explicitly enforces the framework to learn discriminative domain-specific and invarininvariant representations simultaneously. In summary, our contributions three-fold:
We propose a weight-sharing triple-branch transformer framework, namely, CDTrans, for accurate unsupervised domain adaptation, taking advantage of its robustness to noisy labeling data and great power for feature alignment.
To produce pseudo labels with high quality, two-way center-aware labeling method is proposed, and it boosts the final performance in the context of CDTrans.
CDTrans achieves the best performance compared to ttate-of-the-arts with a large margin VisDA-2017 (Peng al. 2017) and DomainNet (Peng et al. 2019) datasets.
RELATED WORK
2.1 TRANSFORMER FOR VISION
Transformer is proposed in (Vaswani et al.) 2017) to model sequential data in the field of NLP. Many works have shown its effectiveness for computer-vision tasks (Han et al. 2020 Khan et al,
2021 Li et al., 2021d] Han et al. 202 Yu et al., 202 Li et al. 2021c Yang et al. 202T Qian et al. (022). Pure Transformer based models are becoming more and more popular. For example, VIT (Dosovitskiy et al. 2020) is proposed recently by feeding transformer with sequences of image patches; Touvron et al. (Tovvron et al. 2021) propose DeiT' that introduces a distillation strategy for transformers to help with VIT training: many other ViT variants (Yuan al. 2021a
Wang et al. 2021 Han et al. 2021a Chen et al.) 2021a Ranftl et al. [022]] Liu et al. [021] are proposed from then, which achieve promising performance compared with its counterpart CNNs for both image classification and downstream tasks, such as object detection (Liu et al. (02T).
semantic segmentation Yuuan et al., 202Tb) and object ReID (He et al., 202T). For multi modal based networks, there are several works (Tsai et al. 2019 Li et al. 2021e Hu & Singh (02T)) that apply cross-attention for multi-mode feature fusion, which demonstrates that attention mechanism is powerful at distilling noise and feature alignment. This paper adopts cross-attention in the context of pure transformers for UDA tasks.
2.2 UNSUPERVISED DOMAIN ADAPTATION
domain-level TZeng et al. 2014 Long et al. 2015 Ghifary et al. 2016 Tzeng et al.) 2017, Bousmalis et al. 2017 Hoffman et al. (018) and category- level (Saito et al., 2018 Kang et al. [2119] (0099/DD. Du et al. 2021 Li et al. 2021a). Domain-level UDA mitigates the distribution divergence between the source and target domain by pulling them into the same distribution at different scale levels. The commonly used divergence measures include Maximum Mean Discrepancy (MMD) (Gretton et al. |2066] Tzeng et al. 2014 Long et al., 2015) and Correlation Alignment (CORAL) (Sun et al.) 2016 Sun & Sacnko 2016). Recently, some
There are mainly two levels for UDA methods:
works (Saito et al. 2018 Du et al. 2021 Li et al.
2021a) focus the fine-grained category-level
2

---

Published as a conference paper at ICLR 2022
label distribution alignment through an adversarial manner between the feature extractor and two domain-specifit classifiers. Unlike coarse-graine coarse-grained alignment at the domain scale, this approach aligns each category distribution between the source and target domain data by pushing the target samples to the distribution of source samples in each category. Obviously, the fine-grained alignment results in more accurate distribution alignment within the same lahel space. Although the adversarial approach achieves new improvements by fusing fine-grained alignment operations of source and target samples at the category level, it still does not solve the problem of noisy samples in the wrong category. Our method adopts Transformers for category-leve UDA to solve the noise problem.
2.3 PSEUDO LABELING
Pseudo labeling (Lee et al. 2013) is first introduced for semi-supervised learning and gains popularity in domain adaptation tasks. It learns to label unlabeled data using predicted probabilities and performs fine-tuning together with labeled data. In terms of using pseudo labeling for domain adaptation tasks, (Long et al. 2017, 2018) adopt pseudo labels to conduct conditional distribution alignment; (Zhang al. 2018, Choi et al. 2019) use pseudo labels regularization for domain adaptation: Zou ct al. (2018 18) designs a sell-training framework by alternately solving pscudo labels; Caron (2018) propose a deep self- supervied method by generating pseudo labels via k-means cluster to progressively train the model: Liang et al. (2020) develop a superviue pseudo labeling method to alleviate the effects of noisy pseudo labels. Based on Liang et al. (2020), in this work, we propose a two-way center-aware labeling algorithm to further filter the noisy pseudo pairs.
THE PROPOSED METHOD
We first introduce the cross attention module and analyze its robustness to the noise in Section 3.1
Then the two-way center-aware labeling method is presented in Section 3.2 With the produced pseudo labels as inputs, our cross-domain transformer (CDTrans) is proposed in Section 3.3 consisting of three seight-sharing transformers.
3.1
THE CROSS ATTENTION IN TRANSFORMER
3.1.1 PRELIMINARY
Vision Transformer (ViT) (Dosovitskiy et al., 2020 has achieved comparable or even superior performance on computer vision tasks. One of the most important structures in ViT is the self- attention module Vaswani et al. 2017). ). In ViT, an image I E H xC' is reshaped into a sequence of flattened 2D patches E TR AX(PPC)) where (H, W) is the resolution of the original image, C the number of channels, (P, P) is the resolution of each image patch, and N HW/P² is the resulting number of patches. For self-attention, the patches are first projected into three vectors, i.e. queries Q E IR Ndd, keys K E Nxd and values V E N de and d, indicates their weighted of the values, where the weight assigned to
dimensions. The output is computed each value is computed by a compatibility function of the query with the corresponding key. The N patches serve the inputs for the self-attention module, and the process can be formulated as below. The self-attentin attentiin module aims to emphasize relationships among patches of the input image.
QKT
get (, V) = softmar
V
(1)
dd
The cross-atte attention module is derived from the self-attention module. The difference is that the input of cross-attention is pair of images, i.e. I, and I, Its query and key/value are from patches of I, and I, respectively. The cross-attention module can be calculated as follows:
Q.KT
Attn (Qo, K,. Vi) = =softmant
VV,
(2)
Vdk
where Qs dxda are queries from M patches of image L,, and K, E Nxdx, V, Nxd, keys and values from N patches of image I, The output of the cross-attention module holds the same length M as the number of the queries. For each output, it is calculated by multiplying Vi
3

---

Published as a conference paper at ICLR 2022
100
accuracy
0.12
0.03
0.13
0.85
0.03 0.16 0.29
0.19 0.18 0.18
cros poos-ettt wiD coneratt tou pos palns
0.35 0:18 0.21 0.36 0.44
0.47
the rate of false positive pains
(b)
Figure 1: (a): The heatmap the pros-aattention weights for false positive pair (Car vs. Truck). (b): The changes of UDA performance by the ratio of false positive pairs. The red/green curves represent the model with and without the cross-attention module. The blue curve means that only true positive pairs are involved in the cross- coss-attention module.
with attention weights, which comes from the similarity between the corresponding query in I, and all the keys in 4, As result, among all patches in It the patch that is more similar to the query of I. would hold a larger weight and contribute more to the output. other words, the output of the cross-attention module manages to aggregate the input images based their similar patches.
So far, many researchers have utilized the cross-attention for feature fusion, especially in multi- modal tasks (Tsai et al.) 2019: Li al. 2019: Hu & Singh/ 2021 Li et al., (202ee). In these works, the inputs of the cross-attention module are from two modalities, e.g. vision-to-text (T'ai et al., 2019 Hu & Singh (021), text-to-speech (Li et al., 2019) and vision-to-vision (Li al. (021e). They apply the cross-attention to aggregate and align the information from two modalities. Given its great power in feature alignment, propose to use the cross attention module to solve the unsupervised domain adaptation problem.
3.1.2 ROBUSTNESS TO NOISE
As mentioned above, the input of the cross-attention module is a pair of images, which usually comes from two domains, and the cross-attention module aims to align these two images. If label noise exists, there would be false positive pairs in the training data. Images in the false positive pairs would have dissimilar appearance, and forcibly aligning their features would inevitably injure the training and compromise the performance. We assume that the dissimilar patches in false positive pairs more harmful to the performance than the similar patches. In the cross-attention module, two images are aligned based on their patch similarity. As shown in Fig. Ial the cross-attention module would assign low weight the dissimilar patches in false positive pairs. Thus it weakens the negative effects of the dissimilar patches on the final performance to some extent.
To further analyze this issue, an experiment is carefully designed. Specifically, we randomly sample true positive pairs from source and target domain in VisDA-2017 dataset (Peng et al. 2017) as the training data. Then we manually replace the true positive pairs with random false positive pairs to increase the noise, and watch the changes of the performance as shown in Fig. [b The x-axis indicates the rate of false positive pairs in the training data, and the y-axis shows the performance of different methods the UDA task. The red curve represents the results by aligning pairs with the cross-attention module, while the green curve that without cross-attention, i.e. to directly train the target data with the label of corresponding source data in the pair. It can be curve achieves a much better performance than the green one, which implics the robustness of the cross-attention module the noise. We also provide another baseline shown the blue curve in Fig. [b] which to remove the false positive pairs from the training data and train the cross-attention with only true positive pairs. Without the noisy data, this baseline can be considered as the upper bound to our methods. We can see the red curve is very close to the blue curve, and both of them much better than the green one. further implies that the cross-altention module is robust the noisy input pair.
that the red
'The patches in true positive pairs, either similar and dissimilar, would bring no noise to the final performance, which is out of the discussion of paper.
4

---

Published as a conference paper at ICLR 2022
3.2 TwO-Wa WAY CENTER-AWARE PSEUDO LABELING
3.2.1 Two-WA WAY LABELING
To build the training pairs for the cross-attention module, an intuitive method is that for each image in the source domain, manage find the most similar image from the target domain. The set IPs of selected pairs is:
Pg = {(s, min minddffa fc), E E
(3)
where S, T are the source and target data respectively. d(f:: f2) means the distance between features of image and j. The advantage of this strategy to make full use source data, while its weakness obvious that only part of target data is involved. To climinate this training bias from target data, we introduce more pairs Pr from the opposite way, consisting of all the target data and their corresponding most similar images in the source domain.
T {(s a)
min dft, Jk), T, E E
(4)
As result the final set the union of two sets, i.e. {P's U Pr), making the training pairs include all the source and target data.
3.2.2 CENTER-AWARE FILTERING
The pairs in P are built based on the feature similarities of images from both domains, thus the accuracy of the pseudo labels pairs is highly dependent on the feature similarities. Inspired by Liang al. (2020), we find that the pre-trained model of the source data is also useful further improve the accuracy. Firstly, we send all the target data through the pre-trained model and obtain their probability distributions d on the source categories from the classifier. Similar Liang et al. (2020). these distributions can be used to compute initial centers of each category in the target domain by weighted k-means clustering:
Σ1E 8 ft
(5)
Σier 85
where 8 indicates the probability image / on category Pseudo labels of the target data can be produced via the nearest neighbor classifier:
arg min d(ck: ft)
(6)
where / E T and d(i,j) is the distance of features and j. Based the pseudo labels, we can calculate new centers:
c Σver 1 k)fft Ever 1 = kt)
In Liang et (020)) Eq. 6 and could be updated for multiple rounds, and we only adopt one round in our paper. The final pseudo labels are then used to refine the selected pairs. Specifically, for every pair, if the pseudo label of the target image is consistent with the label of the source image, this pair would be kept for our training, otherwise it will be discarded as a noise.
3.3 CDTRANS: CROSS-DOMAIY TRANSFORMER
The framework of the proposed Cross-domain Transformer (CDTrans) is shown in Fig. 2 which consists of three weight-sharing transformers. There are three data flows and constraints for the weight-sharing branches.
The inputs of the framework are the selected pairs from our labeling method mentioned above. named source branch, target branch, source-target branch. As shown
The three branches in Fig. [ the source and target images in the input pair are sent to source branch and target branch respectively. In these two branches, the self-attention module is involved to learn the domain-specific
5

---

Published as a conference paper at ICLR 2022
Sell-Attention
Cross-Attention
Classifier
Add de Norn
Add Norr
Add & Norm
Food Forward
Forward
Food Forwand
Self-Artention & Cress-Attention
Noro
Norti
Mali Aneetion
Aneaties
Self-Attention & Cross-Atentiin
Ve
02
Kr
W.
We
Wg
Wy
We
W.
The proposed CDTrans framework. It consists of three weight-sharing transformers fed by inputs from the selected pairs using the two-way center-aware labeling method. Cross-entropy adopted to source branch (Hş) and target branch (Hr), while the distillation loss is applied between source-targe branch (Hs-T) and Hr.
Figure 2:
representations. And the softmax cross-entropy loss is used to train the classification. It is worth noting that all three branches share the same classifier due the same label of two images.
The eross-attention module is imported in the source-target branch. The inputs of the source-tare target branch are from the other two branches. In the --th layer, the query of the cross-attention module comes from the query in the N-th layer of the source branch, while the keys and values are from those of the target branch. Then the gross-attention module outputs aligned features which are added with the output of the (N -V-1): layer") 2
The features of the source-target branch not only align distributions of two domains, but are robust to the noise in the input pairs thanks to the cross-attention module. Thus we use the output of the source-target branch to guide the training of the target branch. Specifically, the source-target branch and target branch are denoted as teacher and student respectively. We consider the probability distribution of the classifier in source-target branch as a soft label that can be used to further supervise the target branch through distillation loss (Hinton et al. 2015):
Lau =
Σ log
(8)
where aa and pa are the probabilities of category & from the source-target branch and the target branch respectively.
During inference, only the target branch is used. The input is an image from testing data, and only the target data flow is triggered, ie. the blue lines in Fig. 2 Its output of the classifier is utilized as the final predicted labels.
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION
The proposed method is verified on four popular UDA benchmarks, including VisDA-2017 (Peng ct al.) 2017). Office-Home (Venkateswara et al.] 2017). Office-31 (Saenko et al., 2010 and DomainNet (Peng et al., 2019). The input image size in our experiments is 224x224. Both the DeiT-small and -bas- (Touvron et al. 2021) are adopted as our backbone for fair comparison. We use the Stochastic Gradient Descent algorithm with the momentum of 0.9 and weight decay ratio le-4 to optimize the training process. The learning rate is set to 3e-3 for Office-Homee Office-31 and DomainNet, 5e-5 for VisDA-2017 since it can easily converge. The batch size is set to 64.
Thhe addition operation is included in the 1st layer.

---

Published as a conference paper at ICLR 2022
Method
plane
bcycl
bus
car
horse
knife
meycl
person
plant
skthrd
truck
Avg.
ResNet-50
55.1
53.3
61.9
59.1
80.6
17.9
79.7
31.2
81.0
26.5
73.5
8.5
52.4
DANN
81.9
77.7
82.8
44.3
81.2
29.5
65.1
28.6
51.9
54.6
82.8
7.8
57.4
MinEnt
80.3
75.5
75.8
48.3
77.9
27.3
69.7
40.2
46.5
46.6
79.3
16.0
57.0
MCD
87.0
60.9
83.7
64.0
88.9
79.6
84.7
76.9
88.6
40.3
83.0
25.8
71.9
SWD
90.8
82.5
81.7
70.5
91.7
69.5
86.3
77.5
87.4
63.6
85.6
29.2
76.4
CDAN+E
85.2
66.9
83.0
50.8
84.2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73.9
BNM
89.6
61.5
76.9
55.0
89.3
69.1
81.3
65.5
90.0
47.3
89.1
30.1
70.4
MSTN+DSBN
94.7
86.7
76.0
72.0
95.2
75.1
87.9
81.3
91.1
68.9
88.3
45.5
80.2
CGDM
93.7
82.7
73.2
68.4
92.9
94.5
88.7
82.1
93.4
82.5
49.2
82.3
CGDM
92.8
85.1
76.3
64.5
91.0
93.2
81.3
79.3
92.4
83.0
85.6
44.8
80.8
SHOT
94.3
88.5
80.1
57.3
93.1
93.1
80.7
80.3
91.5
89.1
86.3
58.2
82.9
SHOT*
95.5
87.5
80.1
54.5
93.6
94.2
80.2
80.9
90.0
89.9
87.1
58.4
82.7
TVT
92.9
85.6
77.5
60.5
93.6
98.2
89.4
76.4
93.6
92.0
91.7
55.7
83.9
Baseline-B B
97.7
48.1
86.6
61.6
78.1
63.4
94.7
10.3
87.7
47.7
94.4
35.5
67.1
CGDM-B B
96,3
87.1
86.8
83.5
92.2
98.3
91.6
78.5
96.3
48.4
89.4
39.0
82.3
SHOT-B*
97.9
90.3
86.0
73.4
96.9
98.8
94.3
54.8
95.4
87.1
93.4
62.7
85.9
Ours-B 97.1
90.5
82,4
77.5
96.6
96.1
93.6
88.6
97.9
86.9
90.3
62.8
88.4
Table Comparison with SoTA methods on VisDA-2017 "S/B" implies the DeiT-small/Deii-baa backbone respectively. indicates the results are reproduced by ourselves. implies its pretrained model is trained on ImageNet21K instead of ImageNet The best performance is marked as bold.
Method
Ar-CCI Ar
PP
Ar
Re CI
>Ar
CI
PPr
CI
Re
Pr
Ar Pr
+CI
Pr
Re
Re
+Ar
Ar
Re
CI-
Re
Pr
Avg.
ResNet-50
44.9
66.3
74.3
51.8
61.9
63.6
52.4
39.1
71.2
63.8
45.9
77.2
59.4
MinEnt
51.0
71.9
77.1
61.2
69.1
70.1
59.3
48.7
77.0
70.4
53.0
81.0
65.8
CDAN+E
54.6
74.1
78.1
63.0
72.2
74.1
61.6
52.3
79.1
72.3
57.3
82.8
68.5
DCAN
54.5
75.7
81.2
67.4
74.0
76.3
67.4
52.7
80.6
74.1
59.1
83.5
70.5
BNM
56.7
77.5
81.0
67.3
76.3
77.1
65.3
55.1
82.0
73.6
57.0
84.3
71.1
ATDOC-NA
58.3
78.8
82.3
69.4
78.2
78.2
67.1
56.0
82.7
72.0
58.2
85.5
72.2
SHOT
57.1
78.1
81.5
68.0
78.2
67.4
54.9
82.2
73.3
58.8
84.3
71.8
SHOT*
56.2
77.6
81.6
67.5
78.2
54.0
82.0
72.5
58.8
84.5
71.6
TVT
74.9
86.8
89.5
82.8
88.0
79.8
71.9
90.1
85.5
74.6
90.6
Baseline-S
55.6
73.0
79.4
70.6
72.9
76.3
67.5
51.0
81.0
74.5
53.2
82.7
69.8
Ours-S
60.6
79.5
82.4
75.6
81.0
82.3
72.5
56.7
84.4
77.0
59.1
85.5
74.7
Baseline-B
61.8
79.5
84.3
75.4
78.8
81.2
72.8
55.7
84.4
78.3
59.3
86.0
74.8
CGDM-B*
67.1
83.9
85.4
77.2
83.3
83.7
74.6
64.7
85.6
79.3
69.5
87.7
78.5
SHOT-B*
67.1
83.5
85.5
76.6
83.4
83.7
76.3
65.3
85.3
80.4
66.7
83.4
78.1
Ours-B
68.8
85.0
86.9
81.5
87.1
87.3
79.6
63.3
88.2
82.0
66.0
90.6
80.5
Table 2: Comparison with SoTA methods on Office-Home.
The best performance is marked as bold.
4.2
COMPARISON TO SOTA
We compare our method with state-of off-the-ar methods on UDA tasks, including MinEnt (Grandvalet 2005), DAN (Long et al., 2015), DANN (Ganin & Lempitsky (215)). CDAN+E (Long al.
et al.
2018
CDAN+BSP (Chen et al., 2019), CDAN+TN (Wang et al. 2019), rRGrad+CAT (Deng
al.
2019), MCD (Saito et al. 2018), SWD (Lee et al. 2019), MSTN+DSBS (Chang al. 2019)
SAF
+ENT (Xu et al. 2019). BNM (Cui al. 2020). DCAN (Li et al. 2020). SHOT (Liang et al.,
2020
ATDOC-NA (Liang et al. 2021 CGDM (Du et al. 202 and TVT (Yang al.). The results and
are shown in Table
For Office-Home, Office-31 and DomainNet, most of the methods use ResNet-50 their backbones, we provide results with DeiT-small as our backbone for a fair comparison, which has comparable model size as ResNet-50, but we also show the results using DeiT-base. And for VisDA-2017, we adopt the DeiT-base backbone for fair comparisons, where other methods utilize ResNet- for their results.
The "Baseline-S/B" indicates directly training a DeiT-small/Deii--eae on the source domain and testing on the target domain. The baseline shows a competitive result even compared other SoTA methods on most datasets. It demonstrates that Transformers has better generalization ability over ConvNets. We also provide some insights about why transformers can generalize well from source domain to target domain in the supplementary materials. To further eliminate the unfairness of using different backbones, we reproduce the results SHOT and CGDM (marked as "*"), and replace their backbones with DeiT--baee the same as ours, denoted "-B*"

---

Published as a conference paper at ICLR 2022
MCD clp
inf
put
qdr
rel
skt
Ave.
CDAN
clp
lnf
pnt
qdr
rel
skt
Avg.
BNM
clp
inf
qdr
rel
skt
Avg.
dp
25,4
3.3
44.631.2
24.0
dp
13.5
28.3
9.3
43.8
30.2
25.0
cpp
12.1
33.1
6.2
40.2
28.5
inf
24.1
24.0
1.6
39.
35.2
20.9
inf
18.9
21.4
1.9
36.3
20.0
inf 26.6
28.5
2.4
38.5
18.1 22.8
pnt
31.1
14.8
1.7
28.1
23.7
pat
29.6
14,4
45.2
24.2444442
242
pnt 39,9
12.2
344
54.5
qdr
8.5
2.1
4.6
7.9
7.1
6.0
qdr
11.8
1.2
4.0
944
9.5
7.2
qdr
17.8
1.0 3.6
9.2 8.3 8.0
rel
39.417.841.
1.5
25.2
rel
36.4
18.3
40.9
3.4
24.6
24.7
rel
48.6 13.2
3.6
33.9
29.8
skt
37.3 12.1226274
34.5
23.1
38.2
14.7 33.9
36.6
26.1
skI
54,9 12.8423
51.3
33.3
Avg.
28.1 12.5 24.5 24
34.1
20.5
Avg.
27.0
25.7
5.1
22.3
Avg.
37.6 10.6 30. 4.2 27.9
SWD
clp
inf put gdr
rel skt
Avg.
CGDM
clp
inf
Qdr
rel Avg.
skt
Base-S
clp inf put Qdr
rel skt Avg.
clp
14.73199 45.14 45.3
27.7
clp
36.9
13.3 36,9 30.7
clp 36.8
21.2 15.3 36,9 37.3
inf
22,9
24.2
33.2
20.0
27.8
28.2
48.2
22.5
26.2
39.4 54 52.1 32.6 33.3
pat
15.6
46.1
30.7
26.0
put
37.7
14.5
59.4
33.5 30.0
put
47.121.7
5.7 60,2
qdr
15.5
2.2
6.4
11.1
10.2
9.1
qdr
14,9
1.5
6.2
10.9
10.2 8.7
qdr
25.0
3.3 10.4
18.8 14.0
rel
41.2
18.1 44.2
4.6
31.6
27.9
rel
49.4
20.8
4.8
38.2
32.0
rel 54.8
23,9 52.6 7.4
40.1 35.8
sk1
44.2 15.2 37.3 10.3 44.7
30.3
50.1
16.5
55.6
35.4
55.6 48 14,9 55.7
37.5
Avg.
31.5 13.1 28.8
64 36
26.1
23.6
Avg.
36.0
32.0 7.1
45.5
28.3
27.2
Avg.
43.9 37.9 9.7 34.5 32.2
Ours-S clp
inf put
qdr rel
skt
Avg.
Base-B
pnt inf clp
odr
vel skt Avg.
Ours-B clp inf Qdi rel skt Ave.
clp
25.3
63.3 44.3
clp
24.2
13.39333333 30.7 40.6
cip $7.0
29.4 26.0 12.68 48.7
inf
47.6
48.3 9.9 62.8 41.1 41.9
43.5
44.9 6.5 58.8 37.6 38.3
54.4 69. 68.56.944444444 48.4 48.4
pat
55.424.5
11.7 67.4
put
52.823.3
64.6 38.4
put 62,927.4
53.1
qdr
36.6
5.3 19.3 33.8 22.7 23.5
qdr
31.8
6.1 15.6 23.4 18.9 19.2
qdr 44.6
8.9 29.0 28.5
rel
28. 56.8 12.8
47.2 41.3
rel
58.9 56.7 9.1
45.0
39.2
rel
66.231.66 16.2
45.6
sk1
64.3 26.1 53.2 23.9
46.7
60.022.1
66.6
41.6
69.0 29,2
51.5
Avg.
53.1 21.9 46.0 16.3
59.7 42.4 89.9
Avg.
49.4 20.2 42.9 10.9 54.5 39.3 36.2
20.2
Avg. 59.9 52.2 19.6 48.4
Table 4: Comparison with So methods on DomainNet. "Base" is the Baseline.
From Table 2 3 and it can be seen that our method outperforms the baseline with a large margin on all four datasets, e.g. nearly 21% on VisDA. With our improvements, the new Transformer with cross-attention module shows much better generalization power, and achieves VisDA-2017
Method
A-D A-
D- >
D- W
W
>
W
WD+
Avg
ResNet-50
68.9
68.4
62.5
96.7
60.7
99.3
76.
DANN
79.7
82.0
68.2
96.9
67.4
99.1
82.2
CDAN+E
92.9
94.1
71.0
98.6
69.3
100.
87.7
+RGrad+CAT
90.8
94.4
72.2
98.0
70.2
100.
87.6
SAFN+ENT
90.7
90.1
73.0
98.6
70.2
99.8
87.1
CDAN+BSP
93.0
93.3
73.6
98.2
72.6
100.
88.5
CDAN+TN
94.0
95.7
73.4
98.7
74.2
100.
89.3
the best performance compared to other SoTAs methods. It further implies the effectiveness of method on the UDA task.
SHOT
94.0
90.1
74.7
98.4
74.3
99.9
88.6
SHOT*
93.8
91.8
74.8
98.2
74.1
99.8
88.8
TVT*
96.4
96.4
84.9
99.4
86.1
100.
93.8
Baseline-S
87.6
86.9
74.9
97.7
73.5
99.6
86.7
Ours-S
94.6
93.5
78.4
98.2
78.0
99.6
90.4
Taking closer look at the results, for the hard categories, such as "person" in VisDA-2017 dataset, the baseline is very low, which indicates the initial model of our method has poor classification ability on this category, leading to the pseudo labels with more noise. Even with such a poor baseline and poor quality of pseudo labels, our method can still achieve much higher performance boost (from 10.3% to 88.65 88.6%). It suggests that our method has a great robustness to the labeling noise and can overcome the noise problem to some extent.
Baseline-B B
90.8
90.4
76.8
98.2
76.4
100.
88.8
C
94.6
95.3
78.8
97.6
81.2
99.8
91.2
SHOT-B#
95.3
94.3
79.4
99.0
80.2
100.
91.4
Ours-B
97.0
96.7
81.1
99.0
81.9
100.
92.6
Table 3: Comparison with SoTA methods on Office-31.
The best performance is marked bold.
We can see that TVT achieves a better result on Office-Home and Office-31. Because TVT utilizes ViT (Dosovitskiy et al.) (2020) backbone which is pretrained on ImageNet2IK. While the pretrained model of our CDTrans and other UDA methods are trained on ImageNetIK.
4.3 ABLATION STUDY
4.3.1
DIFFERENT PSEUDO LABELING
We have conducted experiments on different pseudo labeling methods to verify their influence on the final performance. The results on VisDa-2017 are listed in Table. 5 RPLL (Zheng & Yang 2021) and MRKLD+LRENT (Zou et al. 2019) are two commonly used pseudo-label generation methods, we reproduce their pseudo-label generations on our baseline to compare with our proposed pseudo labeling method. Rec,, Rec, means the recall of the selected training pairs in the source and target data, while Prec represents the accuracy of the pairs. "One-way woo-uccce and WOne-way-targer denote only using the pair set IPs in Eq. or Pr in Eq. 4 for training. "Two-way" indicates results
8

---

Published as a conference paper at ICLR 2022
Pseudo labels
Rec, Rec, Prec
plane hcyel hus
horse knife meyel person
plant
skthrd train
truck
Avg.
One-way-source
100. 6.6 90.6
96.1 52.7 85.5 69.6 95.0 90.2
95.1
66.6
88.8
54.6
95.4
29.5
76.6
One-way-target
8.0 100. 76.3
98.2 32.0 87.7 84.1 95.5
89.9
98.3
66.8
95.7
57.5
95.6
22.0
76.9
Two-way
100. 100. 81,8
97.5 49.6 88.7 73.9 94.6
85.8
96.6
58.6
93.3
63.6
94.8
27.9
77.1
Tw + Ca
97.8 94.8 91.3
98.1 86.9 87.9 80.9 97.9
97.3
96.8
85.3
97.6
83.2
94.0
54.4
88.4
RPLL
98.4
63.4 85.8 68.8 97.0 95.4
97.77
59.3
96.2
57.2
96.2 48.1
80.3
MRKLD+LRENT
97.8
77.3 64.4 94.6 93.9
93.3
77.5
93.1
74.9
92.6
59.0
83.3
Groundtruth
100. 100. 100.
97.9 89.1 92.3 91.9 98.4 97.2 97.5
86.8
98.6
90.7
96.3 60.0
91.5
Table Comparison among different pseudo labeling methods on VisDa-2017. Rec,, Rec, express the recall of pseudo labels in source and target data, while Prec represents the accuracy of the pairs. sann-way-souuee/arr denotes only using the source/target pair set for training. "Tw+Ca" implies the proposed way center-aware labeling method.
T.
lotte
b
plane
beycl
bus
horse
knife
meycl
person
plant
sktbrd
train
truck
Avg.
cls
97.7
48.1
86.6
61.6
78.1
63.4
94.7
10.3
87.7
47.7
94.4
35.5
67.1
cls
98.3
85.0
88.0
76.3
98.1
96.1
96.9
61.1
97.2
85.5
94.6
54.9
86.0
cls
cls
98.3
87.4
89.1
77.3
98.0
97.4
95.4
69.5
97.1
86.3
95.3
49.5
86.7
cls
cls
cls
98.2
88.4
88.0
76.8
98.2
97.2
95.6
80.1
97.1
84.7
94.5
54.1
87.7
cls
drl
cls
98.0
86.9
87.9
80.9
97.9
97.3
96.8
85.3
97.6
83.2
94.0
54.4
88.4
Table 6: Comparison among different losses on VisDa-2017. La and Lu4z represent the loss used
in source, target and source+target branches respectively. cls and dil imply the classification loss and the distillation loss.
of using the union of P. and IP, without the center-aware strategy. "Tw+Ca" implies our two-way center-aware labeling method, and "Groundtruth" means all training pairs are from groundtruthes.
By looking at Rec, and Rec, in Table. 5 can be found that the one-way methods have an apparent bias on either source target data, and its results lower than the two- two-way method. By comparing "Two-way" and "Tw+Ca", we can conclude that although the center-aware method filters the training pair and slightly reduces the recall, it largely improves the precision and leads to better final performance. We also find that our two-way center-aware labeling method achieves a very high result, only better than other pseudo-label generation methods, but also very close to the upper bound trained with groundtruth pairs.
4.3.2 DIFFERENT LOSSES
As there are three losses in our method, we conduct another experiment to verify the effectiveness of each loss on VisDa-2017, as shown Table. "cls" in Le+: denotes that we replace the distillation loss with a classification loss for the source-target branch. We can see that the 3rd row with both Lg and Lg having classification loss achieves a better result than the first row where only Lg has the cls loss, which means the target branch with the pseudo labels is helpful to improve the UDA result. With the addition of "cls" in Ly++s the performance further improved, which demonstrates the advantages of using the cross-attention module for feature alignment. Using "dtl" instead of "cls" on the source-target branch can further improve the results, showing the effectiveness of our distillation loss.
5 CONCLUSION
In this paper, we tackle the problem of unsupervised domain adaptation by introducing the cross- attention module into Transformer novel way. We propose a new network structure CDTrans which is pure transformer- high-quality pseudo labels using a two-way center-aware labeling method. Training CDTrans using the generated high-quality pseudo labels yields robust solution and also achieves state-of-the-art results on four popular UDA datasets, outperforming previous methods by a large margin. We believe that transformer-batt based approaches will have great potential in the UDA community, and our work, as one of the first attempts along this direction, has pushed forward the frontiers and shed lights for future research.
structure with three branches, and we also propose to generate

---

Published as a conference paper at ICLR 2022
REFERENCES
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR,
2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specing batch normalization for unsupervised domain adaptation. CVPR, 2019.
Boyu Chen, Peixia Li, Baopu Li, Chuming Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Psvit: Better vision transformer via token pooling and attention sharing. arXiv preprint arXiv:2108.03428, 2021a.
Kai Chen, Weihua Chen, Tao He, Rong Du, Fan Wang, Xiuyu Sun, Yuchen Guo, and Guiguang Ding. Tagperson: A target-aware generation pipeline for person re-identification. arXiv preprint arXiv:2112.14239 2021b.
Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadruplet network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 403 2017.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In ICML, 2019.
Jaehoon Choi, Minki Jeong, Taekyung Kim, and Changick Kim. Pseudo-labeling curriculum for unsupervised domain adaptation. arXiv preprint arXiv: 0w..//9880000 00262, 2019.
Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:1702.05374 2017.
Shuhao Cui, Shuhui Wang. Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In CVPR, 2020.
Zhijie Deng, Yucen Luo, and Jun Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. In ICCV, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.0480) 040.00000000 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020.
Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu. Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In CVPR, 2021.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconsractionclaasffcattion networks for unsupervised domain adaptation. In ECCV, 2016.
Yves Grandvalet, Yoshua Bengio, al. Semi-supervised learning by entropy minimization. CAP. 367:281-296, 2005.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kerel method for the two-sample-problem NIPS, 2006.
10

---

Published as a conference paper at ICLR 2022
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint
arXiv:2012.12556. 2020.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. transformer. arXiv preprint arXiv:2103.00112 2021a.
Transformer in
Liang Han, Pichao Wang, Zhaozheng Yin, Fan Wang, and Hao Li. Context and structure mining network for video object detection. IJCV, 2021b.
Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. arXiv preprint arXiv:2102.04376. 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint S503::5500.023.11 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified
transformer. arXiv preprint arXiv:2102. 10772, 2021.
Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaci. Implicit class-conditioned domain alignment for unsupervised domain adaptation. In ICML, 2020.
Yiqi Jiang, Weihua Chen, Xiuyu Sun, Xiaoyu Shi, Fan Wang, and Hao Li. Exploring the quality of gan generated images for person re-identification. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 4156-4155, 2021.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv preprint arXiv:2101.00166, 2021.
Seiichi Kuroki, Nontawat Charoenphakdee, Han Bao, Junya Honda, Issei Sato, and Masashi Sugiyama. Unsupervised domain adaptation based on source-guided discrepancy. AAAI, 2019.
Chen-Yu Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, 2019.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi- remi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICML, 2013.
Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu. Cross-domain adaptive clustering for semi- supervised domain adaptation. In CVPR, 202
Naihan Li, Shujic Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In AAAI, 2019.
Shuai Li, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Category dictionary guided unsupervised domain adaptation for object detection. In AAAI, 2021b.
Shuang Li, Chi Liu, Qiuxia Lin, Binhui Xie, Zhengming Ding, Gao Huang, and Jian Tang. Domain conditioned adaptation network. In AAAI, volume 34, pp. 11386-11333 11393, 2020.
Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, and Pichao Wang. Lifting transformer for 3d human pose estimation in video. arXiv preprint arXiv:2103. /4304, 2021c.
Xiangyu Li, Yonghong Hou, Pichao Wang, Zhimin Gao, Mingliang Xu, and Wanqing Li. Transformer guided geometry model for flow-based unsupervised visual odometry. Neural Computing and Applications, 2021d.
Xiangyu Li. Yonghong Hou, Pichao Wang, Zhimin Gao, Mingliang Xu, and Wanqing Li. Trear: Transformer-based rgh-d egocentric action recognition. T-CDS, 2021e.

---

Published as a conference paper at ICLR 2022
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. Domain adaptation with auxiliary target domain-oriented classifier. In CVPR, pp. 16632-16642. 2021.
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICMI,, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In JCML, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. NeurIPS, 2018.
Pietro Morerio, Riccardo Volpi, Ruggero Ragonesi, and Vittorio Murino. Generative pseudo-label refinement for unsupervised domain adaptation. In WACV, 2020.
Poojan Oza, Vishwanath A Sindagi, Vibashan VS, and Vishal M Patel. Unsupervised domain adaption of object detectors: A survey. arXiv preprint a3Xiv:2105.13502, 2021.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint
2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Yichen Qian, Zhiyu Tan, Xiuyu Sun, Ming Lin, Dongyang Li, Zhenhong Sun, Hao Li, and Rong Jin. Learning accurate entropy model with global reference for image compression. 2021.
Yichen Qian, Xiuyu Sun, Ming Lin, Zhiyu Tan, and Rong Jin. Entroformer: A transformer-based entropy model for learned image compression. arXiv preprint arXiv:2202.05499 2022.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. arXiv preprint arXiv:2103.134/ 2021.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.
Zhiyu Tan, Xuecheng Nie, Qi Qian, Nan Li, and Hao Li. Learning to rank proposals for object detection. In ICCV, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pp. 10347-10357, 2021.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ACL, 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 141..3474 2014.
12

---

Published as a conference paper at ICLR 2022
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Vibashan VS, Vikram Gupta, Poojan Oza, Vishwanath A Sindagi, and Vishal M Patel. Mega-cda: Memory guided attention for category- aware unsupervised domain adaptive object detection. In CVPR, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122, 2021.
Ximei Wang, Ying Jin, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable normalization: Towards improving transferability of deep neural networks. 2019.
Yaohua Wang. Yaobin Zhang, Fangyi Zhang, Senzhang Wang, Ming Lin, YuQi Zhang, and Xiuyu Sun. Ada-nets: Face clustering via adaptive neighbour discovery in the structure space. arXiv preprint arXiv:2109.00 065,66666666666666 2022.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. TIST, 2020.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In iccv, pp. 1426-1435, 2019.
Guanglei Yang. Hao Tang, Zhun Zhong, Mingli Ding, Ling Shao, Nicu Sebe, and Elisa Ricci. Transformer- source-free domain adaptation. arXiv preprint 1rXiv:2105.14/333 2021.
Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsupervised domain adaptation. arXiv preprint 108.00088..
Junyan Wang Xiuyu Sun Ming Lin Hao Li Yiqi Jiang, Zhiyu Tan. Giraffedet: A heavy-neck paradigm for object detection. arXiv preprint arXiv:2202.04255, 2022.
Zitong Yu, Xiaobai Li, Pichao Wang, and Guoying Zhao. Transrppg: Remote photoplethysmogra- phy transformer for 3d mask face presentation attack detection. IEEE SPL., 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11966, 2021a.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint arXiv:2106.1 13112, 2021b.
Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor-guided unsupervised domain adaptation for semantic segmentation. NIPS, 2019.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In CVPR, 2018.
Yabin Zhang, Bin Deng, Hui Tang, Lei Zhang, and Kui Jia. Unsupervised multi-class domain adaptation: Theory, algorithms, and practice. PAMI. 2020.
Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao Bichen Wu, Ravi Krishna, Sanjit A Seshia, et al. A review of single-
Joseph E Gonzalez, Alberto I. Sangiovanni- source deep unsupervised visual domain adaptation. T-NNLS, 2020.
Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. IJCV, 2021.
13

---

Published as a conference paper at ICLR 2022
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via dlass-balanced seaf-nrai In ECCV, 2018.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In ICCV, 2019.
14


Reviews:
[{"review_id": "Mgjg0nsCnVg", "reviewer": "Reviewer_3J2D", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-- an exploration about cross attention layer is conducted and it is discovered that that the cross attention layer is robust to pseudo label noise. \n\n-- a novel three branches architecture is proposed in this submission, and the source-target branch is utilized as a teacher to guide the other two branches during training. As far as I know, no similar work was proposed before on UDA classification tasks.\n\n-- the experimental results seem promising, which demonstrate the efficacy of this method.\n\nWeakness:\n-- I am curious about the memory cost and the model size of the propose method. And inference speed is also expected.", "summary_of_the_paper": "This submission proposes a transformer framework for unsupervised domain adaptive classification tasks. In this submission, they conduct an exploration about cross attention layer and found that the cross attention layer is robust to pseudo label noise. Inspired by this, they construct a three branches architecture in this submission, which includes a source transformer, target transformer, a source-target transformer. Due to the robustness of the cross attention layer, the source-target transformer acts as a teacher to guide the other two branches. ", "summary_of_the_review": "This submission proposes a transformer framework for unsupervised domain adaptive classification tasks. In this submission, they conduct an exploration about cross attention layer and found that the cross attention layer is robust to pseudo label noise. Inspired by this, they construct a three branches architecture in this submission, which includes a source transformer, target transformer, a source-target transformer. Due to the robustness of the cross attention layer, the source-target transformer acts as a teacher to guide the other two branches. \n\nAs far as I know, no similar work was proposed before on UDA classification tasks. This work is of novelty and the experimental results are strong enough to demonstrate the efficacy of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6MW0t-zgMrC", "reviewer": "Reviewer_1DYQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has the following strengths.\n- The problem addressed is interesting for the community. The study of pseudo labeling is under-explored compared with domain invariant feature learning.\n- The figures describing the different modules are clear and stimulating.\n- The method achieves outstanding performance on multiple datasets.\n\nMy main concerns are as follows:\n- Comparison with other methods is unfair. From the experiments, we can clearly see the superiority of the adopted Transformer-based backbone against ResNet. Baseline-B even surpasses most previous works under the source-only setting. However, most previous works compared (e.g, SHOT, CGDM) use ResNet as the backbone. To better show the effectiveness of the proposed method, it is better to adopt the domain adaptation techniques of other works with the same backbone.\n- The technical contribution of this work is limited. There are many works that have used cross transformers in computer vision and NLP tasks. However, not specifically in the UDA research landscape.", "summary_of_the_paper": "In this work, the authors propose a method for domain adaptation by introducing a new way of generating pseudo labels and a cross-transformer with classification and distillation losses. The authors used a cross-transformer where the queries come from the source domain and the values and keys come from the target domain, and try to minimize the output distribution difference between the cross branch and the target branch by a distillation loss. Since the transformers compare source and target in a patch-based manner, the authors find it is more robust to false-positive pairs. The authors do not focus on learning domain-invariant feature representation and mainly solve the domain adaptation problem by pseudo labeling. In four public datasets, the proposed method showed outstanding performance.", "summary_of_the_review": "Despite the concern on novelty, overall this paper lacks justification of their method to see whether the improvement is brought by the backbone change. The authors should compare other methods when replaced with the same backbone as the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1HrVyeY06lW", "reviewer": "Reviewer_GSc9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper is well-written and easy to follow.\n+ The methods achieve SOTA performances in various UDA benchmarks.\n+ It's interesting to see the performances of transformers in UDA.\n--------------------\n- The source only transformers already achieve good baseline UDA performances which already outperforms some of other previous works. But the authors did not provide insights about why transformers has better generalization ability over ConvNets.\n- Although the authors provide some ablation study over different pseudo-label generation methods, it seems it is just compared with the two-way's variants in source/target domain. The authors did not provide comparison with the pseudo-labeling methods as follows:\n1. Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation, IJCV 2020\n2. Confidence Regularized Self-Training, ICCV 2019\n3. Two-phase Pseudo Label Densification for Self-training based Domain Adaptation, ECCV 2020\nI suggest the authors to compare their methods to other pseudo-label generation method to see if the proposed pseudo-label generation improves over them.\n- Lack a relevant reference of transformer based UDA:\nTVT: Transferable Vision Transformer for Unsupervised Domain Adaptation, arxiv. \n(Although it is an arxiv paper, it's also good to include it as it provides better UDA performances on office-31/Home)", "summary_of_the_paper": "This paper proposes a weight-sharing triple-branch transformer framework, or CDTrans for unsupervised domain adaptation. A two-way center-aware labeling method is proposed to provide better pseudo-labels. SOTA performances were achieved via the proposed method.", "summary_of_the_review": "I hope the authors could provide some insights/intuition why transformer can generalize well from source to target. And the comparison with other pseudo-labeling approaches is still important to evaluate the contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes use of cross-attention transformers to extract invariant features for unsupervised domain adaptation. Combined with pseudo-label approaches, the proposed method achieves state-of-the-art performance, possibly because the transformer features are more robust to the noise. In addition, a two-way centre-aware labeling method is proposed to produce more reliable pseudo labels. 

However, there are some concerns raised by reviewers. After the discussion period, there is still a concern that is not completely unresolved. The comparison with existing methods might not be fair. It is possible that the performance gain is caused by the generally better representation of transformers, which has been shown in supervised image classification.

Overall, the paper is novel and interesting.  I would recommend acceptance of this paper given its impressive performance, but I highly suggest the authors add more ablation studies, for example, compare the proposed transformer and ResNet on a supervised classification task, to further confirm that the performance gain is solely because the transformer is more robust to label noise. The results can be updated in the supplementary. Also, as promised in the discussion, I hope the authors could release their code as soon as possible. This is because the backbone in this paper is totally new, it will be hard for other researchers to achieve SOTA results if they still use CNNs.
Decision: Accept (Poster)