Joint Attention-Driven Domain Fusion and Noise-Tolerant Learning for Multi-Source Domain Adaptation
Authors: Tong Xu
Lin Wang
Wu Ning
Chunyan Lyu
Kejun Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-source Unsupervised Domain Adaptation
Attention Mechanism
Noisy Label Learning

Abstract: Multi-source Unsupervised Domain Adaptation (MUDA) transfers knowledge from multiple source domains with labeled data to an unlabeled target domain.
Recently, endeavours have been made in establishing connections among different domains to enable feature interaction. However, these approaches essentially enhance category information and thus lack the transfer of the domain-specific information. Moreover, few research has explored the connection between pseudo-label generation and the framework’s learning capabilities, crucial for ensuring robust MUDA. In this paper, we propose a novel framework, which significantly reduces the domain discrepancy and demonstrates new state-of-the-art performance. In particular, we first propose a Contrary Attention-based Domain Merge (CADM) module to enable the interaction among the features so as to achieve the mixture of domain-specific information instead of focusing on the category information. Secondly, to enable the network to correct the pseudo labels during training, we propose an adaptive and reverse cross-entropy loss, which can adaptively impose constraints on the pseudo-label generation process. We conduct experiments on four benchmark datasets, showing that our approach can efficiently fuse all domains for MUDA while showing much better performance than the prior methods.

Published as a conference paper at ICLR 2023
JOINT ArTENTION-DRIVER DOMAIN FUSION AND NOISE-TOLERANT LEARNING FOR MULTI-SOURCE DOMAIN ADAPTATION
ABSTRACT
Multi-source Unsupervised Domain Adaptation (MUDA) transfers knowledge from multiple source domains with labeled data to an unlabeled target domain. Recently, endeavours have been made in establishing connections among different domains to enable feature interaction. However, as these approaches essentially enhance category information, they lack the transfer of the domain--pecific in- formation. Moreover, few research has explored the connection between pscudo- label generation and the framework's ' learning capabilities, crucial for ensuring ro- bust MUDA. In this paper, we propose a novel framework, which significantly re- duces the domain discrepancy and demonstrates new state-of-the-art performance. In particular, we first propose Contrary Attention-based Domain Merge (CADM) module to enable the interaction among the features so to achieve the mixture of dpmein--peccfic information instead of focusing on the category information. Secondly, to enable the network to correct the pseudo labels during training, we propose an adaptive and reverse cross-entropy loss, which can adaptively impose constraints on the pseudo-label generation process. We conduct experiments on four benchmark datasets, showing that our approach can efficiently fuse all do- mains for MUDA while showing much better performance than the prior methods.
INTRODUCTION
Deep neural networks (DNNs) have achieved excellent performance on various vision tasks under the assumption that training and test data come from the same distribution. However, different scenes have different illumination, viewing angles, and styles, which may cause the domain shift problem (Zhu et al. 2019 Tzeng et al.) 2017 Long et al., 2016). This can eventually lead to a significant performance drop on the target task.
Unsupervised Domain Adaptation (UDA) aims at addressing this issue by transferring knowledge from the source domain to the unlabeled target domain (Saenko et al. 2010). Early research has mostly focused on Single-source Unsupervised Domain Adaptation (SUDA), which transfers knowl- edge from one source domain to the target domain. Accordingly, some methods align the feature distribution among source and target domains (Tzeng et al. 2014) while some (Tzeng et al.) 2017) learn domain invariants through adversarial learning. Liang et al. (2020) use the label information to maintain the robust training process. However, data is usually collected from multiple domains in the real-world scenario. which arises a more practical task, i.e., Multi-source Unsupervised Domain Adaptation (MUDA) (Duan et al. 2012).
MUDA leverages all of the available data and thus enables performance gains; nonetheless, it in- troduces a new challenge of reducing domain shift between all source and target domains. For this, some research (Peng et al. 2019) builds their methods based on SUDA, aiming to extract com- mon domain-invarian features for all domains. Moreover, some works, e.g., Venkat et al. (202T); Zhou et al. (202I) focus on the classifier's predictions to achieve domain alignment. Recently, some approaches (Li et al. 2021 Wen et al. 2020) take advantage of the MUDA property to create con- nections for each domain. Overall, since the main challenge of MUDA is to eliminate the differences between all domains, there two main ways to achieve this. One is extract domain invariant features among all domains, i.c., filter domain domain-specific information for different domains. The other is by mixing domain-specific information from different domains that all domains share such

---

Published as a conference paper at ICLR 2023
Sourte=1
Feature Extractor
CADM
real label
h
Sourroo-2
hard-label
hag
Clesdifie
Sourro33
AA--CR
solt-labed
pacudo-label
M emoory
Overview of our proposed framework that consists of three primary elements: (1) The feature extractor extracts features from various domains. (2) CADM proposed to implement message passing and fuse features from different domains. The same domain is represented by the same color, while different shapes represent different classes. (3) we propose AR-CE loss and use the maintained memory to compute the soft label and pseudo label (hard label) for target domain.
Figure/ 1:
mixed information and thus fuse into one domain. Previous approaches have mostly followed the former, however, filtering the domain-specific information for multiple domains can be difficult and often results in losing discrimination ability. For the latter, few methods have been proposed to ad- dress MUDA this way, and there is a lack of effective frameworks to achieve such domain fusion, which is the main problem to be addressed by our proposed approach. Moreover, existing methods ignore the importance of generating reliable pseudo labels as noisy pseudo labels could lead the accumulation of prediction errors. Consequently, it is imperative to design a robust pseudo-label generation based on the MUDA framework.
In this paper, we propose novel framework that better reduces the domain discrepancy and shows the new state-of-the-ar (SoTA) MUDA performance, as shown in Fig. [ Our method enjoys two pivotal technical breakthroughs. Firstly, we propose Contrary Attention-based Domain Merge (CADM) module (Sec. 4.2). whose role is to perform the domain fusion. Self attention (Vaswani capture the higher-order correlation between features
et al. 2017 Dosovitskiy et al., (020)) and emphases more relevant feature information, e.g., the semantically closest information. Dif- ferently, our CADM proposes the contrary attention, enabling each domain pay more attention to semantically different domain-specific information of other domains. Then, by integrating these domain-specific information, each domain can achieve movement other domains, thus resulting in domain fusion. Secondly, to enable the network to correct the pseudo labels during training, we take the pseudo-label generation as the optimization objective of the network by proposing an adap- tive and reverse cross-entropy (AR-CE) loss (Sec. (.3).. It imposes the optimizable constraints for pseudo-label generation, enabling the network to correct pseudo labels that tend to be wrong and reinforce pseudo labels that tend to be correct.
We conduct extensive experiments on four benchmark datasets. The results show our method achieves new state-of-the-art performance and especially has significant advantages in dealing with the large-scale dataset. In summary, we have made the following contributions:
We propose a CADM to fuse the features of source and target domains, bridging the gap between different domains and enhancing the discriminability of different categories.
We propose loss function, AR-CE loss, to diminish the negative impact of the noisy pseudo label during training.
Our method demonstrates the new state-of-the- (SoTA) MUDA performance on multiple benchmark datasets.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
2.1 UNSUPERVISED DOMAIN ADAPTATION (UDA)
According to the number of source domains, UDA can be divided into Single-source Unsuper- vised Domain Adaptation (SUDA) and Multi-source Unsupervised Domain Adaptation (MUDA). In SUDA, some methods use the metrics, such as Maximum Mean Discrepancy (Long et al. 2017 al. 2018 Ma et al. 2019) learn the domain-invariant representations through adversarial learning. Other methods are based on the label information. Liang et al.] (2020) aligns domains implicitly by con- straining the distribution of prediction. Following Liang et al. (2020). Liang et al. (202T) introduces a label shifting strategy as a way to improve the accuracy of low confidence predictions. For MUDA, previous methods eliminate the domain shift while aligning all source domains by constructing mul- tiple source-target pairs, where (Peng et al., 2019 Zhu et al. 2019) are based on the discrepancy metric among different distributions and (Venkat et al. 202 Ren et al., (022) are based on the ad- versarial learning. Some methods (Venkat et al. 2021:
Wang et al.] 2018), define and optimize domain shift. Some methods (Sankaranarayanan
Nguyen et al. 2202 Li et al., [022 Nguyen et al. 22221) explore the relation between different classifiers and develop different agreements to achieve domain alignment. Some (Wen et al. 2020: Ahmed et al. 202I) achieve an adaptive trans-
fer by a weighted combination of source domains. Current methods (Wang al. 2020 Li al.,
2021) mainly construct connections between different domains to enable feature interaction and ex-
plore the relation of category information from different domains. Our approach also constructs connections between different domains; however, it mainly transfers domain-specific information to achieve the fusion of all domains.
2.2 ROBUST LOss FUNCTION UNDER NOISY LABELS
robust loss function critical for UDA because the unlabeled target domain requires pseudo labels, which are often noisy. Previous work (Ghosh et al., 2017) demonstrates that some loss functions such as Mean Absolute Error (MAE) are more robust to noisy labels than the commonly used loss functions such as Cross Entropy (CE) loss. (Wang al. 2019b) proposes the Symmetric Cross Entropy (SCE) combining the Reverse Cross Entropy (RCE with the CE loss. Morcover, Wang al. 2019a) shows that directly adjusting the update process of the loss by the weight variance is effective. Some methods (Venkat et al. 2021 Zhou et al. (021) mentioned in Section 2. focus exploring learning strategy handle prediction values that have high low confidence levels, respectively. Differently, we design an adaptive loss function for the network to learn robust pseudo-label generation by self-correction.
3 PROBLEM SETTING
For MUDA task, there are N source distributions and one target distribution, which can be denoted as (Px, (z. }) and {p( v)}. The labeled source domain images (Xayx Yey are obtained from the source distributions, where Xa; = { are the images in the source domain j and H |
represents the corresponding ground-truth labels. As for the unlabeled data in the target domain, there are target images X; = { from target distribution. In this paper, we uniformly set the number of samples in a batch as B, with b = N for every domain (including the target domain). For single sample v/,,, the subscript 8y represents the j-th source domain, while the superscript i indicates that it is the i-th sample in that domain.
y1,, (vi,
B
PROPOSED METHOD
4.1
OVERALL FRAMEWORK
The general MUDA pipeline consists of two phases: 1) pre-training on the source domain; 2) training when transferring to the target domain. The base structure of the model consists a feature extractor F(-) and classifier C(.). Previous approaches mostly use different extractors classifiers, unlike
3

---

Published as a conference paper at ICLR 2023
D5
EMA
MLP
hag
x
1c
91
52 Upe
4
4,
4
1
AvgO
A4
(N x
1/
(2 8
Contary Attertion
Figure 2: CADM review: Features from different domains are fused by CADM. For simplicity, we N, B, and D as 2, 6, and 5, respectively. Domain prototypes are mapped to Q, while all features are mapped to K and V. The domain style centroids are obtained by the contrary attention map A' and V, which is then updated by EMA. Finally, the fused feature h is obtained by MI.P.
domain, and therefore F and C are shared among domains. In the pre-training phase, we use the source domain data {.x,, J== for training to obtain the pre-trained F((-) and C(-). In the training phase for the target domain, we firstly feed the source domain data { Y,, N. fj=1 and target domain data X, into F-). to get the features and perform the feature fusion by CADM to get the fused feature h. Then, use the fused features of the target domain for the pseudo-label generation and feed the fused features of all domains into C()) to get prediction values. Finally, we compute the loss based on the labels. Since souree training is general, training in this paper typically refers to target domain training.
them, our approach expects achieve deep fusion of all domains, i.e., belonging
4.2 CONTRARY ATTENTION-BANE BASED DOMAIN MERGE MODULE
The proposed CADM enables the passing of domain-specific information achieve the deep fusion of all domains. For each domain, expect it receive domain- -ppeiff information from other domains by the information transfer to move towards other domains as shown in Fig. [ To achieve this goal, the transfer of semantically distinct domain-specifie information should be encouraged. is because receiving semantically close domain-specifff information primarily strengthens the orig- inal domain information and does not help eliminate the domain shift. Instead, receiving different domain-specite information can contribute to the mixing of domain-specift information and 'push' the domain to other domains, thus achieving domain fusion. As result, we construct contrary tention to focus on semantic differences among domains. The specific structure is shown in Fig, Specifically. features are first extracted from the samples via F, and then be mapped into the latent space to obtain query, key, and value feature:
Q [f,, K, Vl [F]
f. U
Ug RRDDD Uk Dx2D
f,
.. kv
(2)
where U, and Up. are learnable projection matrices, D is the feature dimension, f! is the feature obtained from the sample x/, and fa, is the domain prototype representing the domain information of $1. Let represent any domain, the domain prototype is obtained as follows:
f - Σ 5
(3)
Since the query of each domain and the key of all samples are obtained in Eq. 1 then we use the query feature qy in Q compute the correlation with the samples of different domains:
qg K T
A, = Softmax
Ag E l
(4)
VD
4

---

Published as a conference paper at ICLR 2023
We use the self-attention mechanism to model the correlation because of its powerful ability to cap- ture dependencies, and A, is the obtained attention map. The value in Ag reflects the correlation between the domain 9 and the samples from other domains, while, mentioned before, it is more meaningful to focus on semantically different domain-specific information. Consequently, we 're- verse' the obtained Ag to generate new attention map, which we refer to the contrary attention map:
I Ag A/ sum Ag)
Ag R
(5)
where 1 is all-one vector. The A' thus obtained can achieve the opposite effect of the previous Ag. where those features that are more different from the domain g in terms of domain- specifcc information are assigned higher weights.
Subsequently, we use A'g to perform the feature fusion for domain g, which allows the merging of information from features according to their degree of dissimilarity with domain g:
D g
A
(6)
Vi
99
GEg
i=
where "gg= represents the contrary attention weight between the prototype of g and the i-th feature in q. b is obtained by weighting features from different domains and summing them according to their discrepancies from g. Given its greater emphasis on the merging of knowledge with distinct styles, we call & the domain style centroid with respect to g. In addition, guarantee the robustness of gg we use exponential moving averages (EMA) to maintain it:
(Ø,) = (1 - (0,))(+++"() a²
where n denotes the n-th mini-batch. The recorded 4, will be used in the inference phase.
Finally, we integrate the domain style centroid 4 y into the original features. A fundamental MLP is used to simulate this process:
h MLP (IC) $g1).
where h is the fused feature and T is integrated into every feature g. thus creating the overall movement of domain. Through the process described above, the original features achieve the merg- ing of different domain-specifie information by integrating the domain style centroid. Meanwhile, to maintain well-defined decision boundary during such domain fusion, we use metric-based loss (Wen al., 2016) to optimize the intra-class distance. allows samples from the same categories to be close to each other and shrink around the class center.
Lor ΣΣ BEG
h
My; 12
(9)
where uy is the label of he and ni IIID denotes the y/-th class center. For the robustness of class center, we maintain memory M to store features at the class level and update them batch by batch during training, which is widely applied in UDA (Saito et al. (020). The class-level features can be regarded as the class center, and we then apply them in the computation of Loy. Specifically, traverse all the datasets extract features and obtain different class centers before the training process. In the target domain, we use the classification result as the category since there is no ground-truth label:
arg max Py (=,)==
N 14 Avg
Σ
h;).
(10)
EE
K.EX.
where py (z)) represents the confidence level of predicting x the j-th class. Then after each back- propagation step, we update the memory M with the following rule:
new (1-31)
1)=)) old 3h,
(11)
where le is the label of h and μy is the k-th class center in M. Under the supervision of Lers the features are able maintain intra-class compactness in domain movement.
5

---

Published as a conference paper at ICLR 2023
Overall, through CADM, establish connections among domains and enable the deep fusion of different domains. Eventually, with the Lor, the model is able to reach a tradeoff between domain merging and a well-defined decision boundary.
4.3 ADAPTIVE AND REVERSE CROSS ENTROPY LOSS
General UDA methods use the K K-means-like algorithm to generate pseudo labels for the entire unla- beled target dataset. However, it has been observed that the pseudo labels generated in this way are often misassigned due to the lack of constraints, which introduces noise and impairs performance. To address this issue, creatively propose to assign gradients to the generation process of pseudo labels and constrain it with a well-designed rule, so that the network can optimize both the predic- tion values and pseudo labels simultaneously. To begin, we expect obtain the pseudo labels with gradient flow. Since the dynamically updated memory M is introduced in Section 4.2) we can ob- tain both soft labels and hard labels by computing the distance between features and the class-level features in M in every batch:
q/y k)z)
(12)
K
nny
i arg max q(v k/(z),
(13)
where ((/[2z) the soft label and ğ is the hard label. The soft label makes the gradient backpropa- gation possible. Specifically, previous experimental observations (Venkat et al. [222]] Wang et al.)
2019b) demonstrate that the trained classifier has important implications, which is key to prevent- ing the model from collapsing in the unlabeled target domain. Therefore, based on the output of classifier, we establish constraints on the pseudo label by a reverse cross entropy loss:
K
Lrce (2x Σ (x( log q(y k|x),
(14)
where k represents the k-th class, and p&(ar) represents the classifier's output at the k-th class. As shown in the Eq. [4 unlike the cross entropy loss, which uses labels to supervise the output of the classifier, free uses the output of the classifier supervise the generated soft labels. Then the gradient can be back puok-propagated through the soft label, thus preventing the accumulation of errors from pseudo-label generation. However,
unfair to apply the same rule to those pseudo labels that tend to be accurate. An ideal situation is that the loss can be adaptively adjusted to correspond to different pseudo labels. Therefore, we propose the adaptive factor a(x) to achieve the adaptive adjustment of free, which is denoted as:
(,(x)
5, 0) = 1 1-Pg(2) (z)
(15)
i&0
a(x) (x. 1) log (x,
(16)
where is the generated hard label of the target domain sample We consider the classifie's output other than the class ý as a distribution and the probability of j-th class is denoted as (1 9) in Eq 15 The entropy of this distribution then serves the adaptive factor a(x). It is because that pseudo label essentially arises from the feature extracted by the network, so its accuracy is closely related to the ability of the network to distinguish this sample. Liang al. (2021) demonstrates that when the model has a good distinguishing ability for one sample, the output of classifier should be close to the oot vector. Consequently, the value of a(x), this case, larger than the case of poor discriminatory ability. Based on a(x) and free. we design an adaptive and reverse cross entropy loss:
(xea(x)
(a) exp (x(x)))))
(17)
B
CAR-CE
Σ
(arc
(18)

---

Published as a conference paper at ICLR 2023
The exponential form and used to amplify its effect. Since LAR-CE aims to focus the optimization of pseudo label, we freeze the gradient of the prediction values in Eq. [4 The back- propagation of CAR-CE CC eventually enable the optimization of pseudo-label generation.
Furthermore, the cross entropy loss is denoted as follows:
B
LCF
Σ
log Py, (24);
(19)
B
2m1
where y/ is the ground-truth label for source domain sample and is the hard pscudo label ý obtained by Eq. 13 when x; is in the target domain.
Ultimately, the loss function for the entire framework can be represented as follows:
L = LCE Lor + CAR-CE.
(20)
where Lax is the normal cross entropy loss, Lcr is for feature fusion, LAR-CE is used for noise tolerant learning, and the latter one item acts only on the target domain.
5 EXPERIMENTS
5.1 DATASETS AND IMPLEMENTATION
Office-31 (Saenko et al. 2010) is a fundamental domain adaptation dataset comprised of three sep-
arate domains: Amazon (A), Webcam (W), and DSLR (D). Office-Caltech (Gong et al. (2012) has
an additional domain Caltech-256 (C) than Office-31. Office-Home (Venkateswara et al., (017)) is
medium-size dataset which contains four domains: Art (Ar), Clipart (CI), Product (Pr), and Real- World (Rw). DomainNet (Peng et al. (2019) is the largest dataset available for MUDA. It contains different domains: Clipart (Clp), Infograph (Inf), Painting (Pnt), Quickdraw (Qdr), Real (Rel), and Sketch (Skt).
For comparison with our method, we introduce the SoTA methods MUDA, including DCTN
Xuu et al. 2018), MDSD (Peng et al. 2019), MFSAN (Zhu et al. 2019). MDDA (Zhao et al. 2020). DARN (Wen et al. (2200,, Lic-MSDA (Wang et al. (020)). SHOT++ (Liang et al.) 202 SImpAl Venkat et al. 2222)) T-SVDNet (Li et al. (021), CMSS (Ahmed et al. 2021) DEC SION (Ahmed et al. 2021), DAEL (Zhou al. 2021). PTMDA (Ren et al.j (022) and DINE (Liang al. 2022]. For Office-31, Office-Caltech and Office-Homa
we use an Imagenet (Deng al. 2009)
pre-trained ResNer-50 (He et al., 2016) as the feature extractor. As for the DomainNet, we use ResNet- 101 instead. In addition, bottleneck layer (consists of two fully-connected layers) is used as the classifier. We use the Adam optimizer with learning rate of x 10 and a weight decay of 5 x 10 We set the learning rate of extractor to ten times that of the classifier. The hyperparameters a and B are set to 0.05 and 0.005, respectively. All the framework based on the Pytorch (Paszke) al. 2019
5.2 RESULTS
We compare the ADNT with different approaches four datasets, and the results shown in the following tables. Note that the one pointed by the arrow is the target domain and the others the source domains. As shown in Table [ the experiments In particular, it is second only to PTMDA in the ResNee50-based approach and achieves the best results in all scenarios, except for transferring W and D method not only outperforms all current approaches, but also achieves an impressive 100% accuracy on the task of transferring to W and to D. In addition, our method achieves the best results on all tasks except transfer to A, and has significant advantage over the 1ONNeet-111baasee method. As shown Table 3 on the medium-sizes Office-Home dataset, our proposed ADNT is the best performer and has 0.9% lead over the best ResNee-50-based method. Furthermore, compared with the ResNet- based methods (DINE and 10 ). our approach achieves leading performance with fewer parameters and simpler model structure. As shown in Table ADNT also demonstrates its strength on the largest and most challenging dataset. The total result outperforms the current optimal method by 2.3%, which is highly impressive Nem Overall, our approach not
Office-31 show optimal results.
On the Office-Caltech dataset, our
7

---

Published as a conference paper at ICLR 2023
Classification accuracies (%) Office-31 dataset, " indicates that the method based on ResNet-101.
Table 1:
'abble 2: Classification accuracies (%) Office- Caltech dataset. indicates that the method is based on ResNet-101.
A.W
A.D
D.W
A.C.D
A.C.W
A.W.D
C.D.W. -A
Method
Avg
Method
Avg
DD
-W
-A
+W
+C
MDDA
99.2
97.1
56.2
84.2
DCTN
99.4
99.0
90.2
92.7
95.3
LIC-MSDA
99.6
97.2
56.9
84.6
M'
99.4
99.2
91.5
94.1
96.1
DCTN
99.3
98.2
64.2
87.2
SImpAlç0
99.3
99.8
92.2
95.3
96.7
MESAN
99.5
98.5
72.7
90.2
CMSS
99.3
99.6
96.6
93.7
97.2
SImpA1g0 50
99.2
97.4
70.6
89.0
SHOT+
100
99.4
96.5
96.2
98.0
DECISION
99.6
98.4
75.4
DECISION
99.6
100
95.9
95.9
DINE
99.2
98.4
76.8
91.5
DINE
98.9
98.5
95.2
95.9
97.1
PTMDA
100
99.6
75.4
91.7
PTMDA
100
100
96.5
96.7
98.3
ADNT (Ours)
100
99.6
74.4
91.3
ADNT (Ours)
100
100
97.6
96.3
98.5
Table 3:
Classification accuracies (%) on Office-Home dataset, " indicates that the method is based on ResNet-101.
Cl,Pr.Rw AAr
Ar,Pr,Rw +CI
Rr,CI, Prr
Ar,CI,Pr Rw
Method
Avg
MESAN
72.1
62.0
80.3
81.8
74.1
SImpAlse 50
70.8
56.3
80.2
81.5
72.2
SImpAl
73.4
62.4
81.0
82.7
74.8
DARN
70.0
68.4
82.7
83.9
76.3
SHOT+
73.1
61.3
84.3
84.0
75.7
DECISION
74.5
83.6
75.5
DINE
74.8
64.1
85.0
84.6
77.1
ADNT (Ours)
73.8
66.5
85.1
83.3
77.2
only achieves the best results, but also has better performance on large datasets with more domains, especially DomainNet. For tasks with degraded performance, in addition to gaps in model structure (compared ResNet-101), they mainly appear when transferring some less stylized domains (Amazon, Clipart, Quickdraw). Further analysis is presented in the Appendix.
Table 4: Classification accuracies (%) on NomainNet dataset.
Methods
Clp
Inf
Pnt
Qdr
+ Rel
Skt
Avg
DCTN
48.6=0.70
23.5++0.60
48.8+:0.60
7.2+
53.5++060
47.3++0.5
38.2
MDSD
58.6=0.53
26.0:10.9
52.310.55
6.310.58
62.7=0.51
49.5+10.6
42.6
MDDA
59.4=0.60
23.81:080
53.240060
12.5+0.60
61.8=1050
48.6:10.80
43.2
CMSS
64.2=0.20
28.0110.20
53.6110.0
16.010.10
63.4-0.20
53.8:10.40
46.5
T-SVDNet
66.110.4
25.0:10.8
54.3110.7
16.510.9
65.4-10.5
54.6110.6
47.0
LIC-MSDA
63.1+00.5
28.7+10.7
56.1+0.55
16.3+0.55
66.1+:0.6
53.8+0.6
47.4
DAEL
70.8=0.14
26.5+-0.13
57.440.28
12.2+0.70
65.0+0.23
60.6+0.25
48.7
PTMDA
66.0+0.3
28.5++0.2
58.4+00.4
13.0+0..
63.0-+024
54.1+0..
47.2
ADNT (ours)
69.0
28.210.41
60.5.10.45
16.3
68.7-10.63
63.5.10.99
51.0
5.3
ABLATION STUDY
To verify the effectiveness of the proposed components, we conduct ablation experiments on the Office-Home dataset, as shown in Table In order to determine whether the attention-driven mod- ule actually achieves domain fusion, we conduct experiments with only the domain fusion without Ler, which is shown '+ CADM (w/o Lcr)' in the table. Additionally, due the fact that our pro- posed contrary attention differs from general attention by paying attention to the fusion domain- specific information, comparison with general attention evaluate the viability of strategy.
8

---

Published as a conference paper at ICLR 2023
Table 5: Classification accuracies (%) of ablation study Office-Home dataset.
Cw, -Ar
Ar,Pr,Rw +CI
Rr, Pr
Ar.LPPr
Method
Avg
-R
Baseline
67.1
56.6
78.6
77.0
69.8
CR-CE
67.5
62.9
78.9
77.4
71.6
LAR-CE
69.2
63.1
81.2
78.6
73.0
ADM
69.7
60.2
81.3
79.5
72.7
CADM (w/o Lcr)
72.3
62.2
83.1
83.1
75.2
CADM (w/ Lcr)
73.1
64.2
84.3
83.7
76.3
CADM Lcr) + LAR-CE
73.8
66.5
83.3
77.2
(a)
(6)
Figure 3: t-sne: The results of feature visualization on the Offce-Home dataset, while performing Ar. CI, Rw to Pr. (a) (b) and (c) represent the result of Source-only, CADM without Lc, and CADM with fcr respectively. All source domains are represented in red, while target domain represented in blue.
Therefore, we replace the contrary attention map in the proposed CADM with a general attention map, i.e., without Eq. [5] The result of general attention shown '+ ADM' in Table The signif- icant performance improvement powerfully demonstrates the effectiveness of our proposed CADM. We also use t-sne (Van der Maaten & Hinton 2008) to visualize the results with and without CADM, as shown in Fig. 3 We can observe that with the designed CADM, the different distributions are mixed with each other, achieving higher degree of fusion. Meanwhile, the decision boundaries of different categories are more distinct.
Table verifies the effectiveness of our AR-CE loss. Additionally, we conduct experiments using only reverse cross entropy loss, denoted as ---CE-- As shown in the Table S CR-CF brings the performance improvement, and the adaptive factor able amplify this improvement. This demon- strates the positive and effective role of adaptive factor. Due to the space limit, additional analytical experiments, including the visualization the training process and analysis of hyperparameters, can be found in the appendix.
CONCLUSION
This paper proposes the ADNT, framework that combines the domain fusion module and robust loss function under noisy labels. Firstly, We construct a contrary iitention-basee domain merge module, which achieve mixture of different domain-specifee information and thus fuse features from different domains. In addition, we design a robust loss function to avoid the error accumula- tion of pseudo-label generation. Substantial experiments four datasets have shown the superior performance of ADN'T, which strongly demonstrates the effectiveness of our approach.

---

Published as a conference paper at ICLR 2023
REFERENCES
Sk Miraj Ahmed, Dripta Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy-Chowdhury. Unsupervised multi-source domain adaptation without access source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10103-10112, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 245. Icce, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang. Domain adaptation from multiple sources: A domain-dependent regularization approach. IEEE Transactions on neural networks and learning systems, 23(3):00--518, 2012.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Boqing Gong, Yuan Shi. Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and patlern recognition, pp- 2066-2073. IEEE, 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Keqiuyin Li, Jie Lu, Hua Zuo, and Guangquan Zhang. Dynamic classifier alignment for unsuper- vised multi-source domain adaptation. IEEE Transactions on Knowledge and Data Engineering, 2022.
Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9991-10000, 2021.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning. pp. 0028-6039. PMLR, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2021.
Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Dine: Domain adaptation from single and multiple Proceedings of the IEEE/CVF Conference on Computer Vision and
black-box predictors. Pattern Recognition, pp. 8003-8015 2022.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. arXiv preprint arXiv: TTtv:v660.0000 1602.04433, 2016.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning, pp. 2208 2217.227..2 PMLR, 2017.
Xinhong Ma, Tianzuu Zhang, and Changsheng Xu. Gean: Graph convolutional adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8266-8276, 2019.
Tuan Nguyen, Trung Le, He Zhao, Quan Hung Tran, Truyen Nguyen, and Dinh Phung. Most: Multi-source source domain adaptation via optimal transport for student-teacher learning. In Uncertainry in Artificial Intelligence, pp. 235-235. PMLR, 2021a.
10

---

Published as a conference paper at ICLR 2023
Van-Anh Nguyen, Tuan Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung. Stem: An approach to multi-source domain adaptation with guarantees. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pp.
2021b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buch E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL httpt/tssp//roccppciiiiii/ neur. wates.ccppaaee cc/pppper/219//ieee bcbb:228fee11122255f881122222221
pdf
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-1415, 2019.
Chuan-Xian Ren, Yong-Hui Liu, Xi-Wen Zhang, and Ke-Kun Huang. Multi-source unsupervised domain adaptation via pseudo target domain. IEFE Transactions on Image Processing. 31:2122- 2135, 2022.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference computer vision, pp. 213 116-226. Springer, 2010.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in neural information processing systems, 33:16282-16292, 2020.
Swami Sankaranarayanan, Yogesh Balaji, Carlos Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3503-8512, 2018.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.335 1412.347, 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 7167-7176, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Naveen Venkat, Jogendra Nath Kundu, Durgesh Kumar Singh, Ambareesh Revanur, and R Venkatesh Babu. Your classifier can secretly suffice multi-source domain adaptation. arXiv preprint arXiv:2103. 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning combine: Knowledge aggregation for multi-source domain adaptation. In European Conference on Computer Vision, pp. 727-744. Springer, 2020.
Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu. Visual domain adaptation with manifold embedded distribution alignment. In Proceedings of the 26th ACM international conference Multimedia, pp. 402-410, 2018.

---

Published as a conference paper at ICLR 2023
Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil M Robertson. Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude's ' variance matters. arXiv preprint arXiv: 193h:1903.12144. 2019a.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 322-330, 2019b.
Junfeng Wen, Russell Greiner, and Dale Schuurmans. Domain aggregation networks for multi- source domain adaptation. In International Conference on Machine Learning, pp. 10214-10224. PMLR, 2020.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3964 3973, 2018.
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 22975-12983, 2020.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions on Image Processing, 30:8008-8015 2021.
Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and clas- sifier for cross-domain classification from multiple sources. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 33, pp. 5989-5996, 2019.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ADDITIONAL EXPERIMENTS ABOUT CADM
Our proposed CADM aims achieve the domain fusion through the transfer of domain-specific in- formation, where the coefficients in the Contrary Attention Map A' have important role to focus on the semantically different domain-specifie information. To verify the action of contrary atten- tion, we perform the visualization by Grad-CAM. Specifically, we used Grad-CAM to visualize the weights A', as the column 'Painting' Fig. 9
reflecting the attention of other domains to that sample in Painting domain. Since our approach emphasizes attention to domain-specifie infor- mation from other domains, it is reasonable to focus on texture and background, etc., rather than content. As shown in Fig/2 the designed contrary attention is able to focus domain- specific information, such as texture, color, background, and other domain characteristics (e.g. handwritten text in Sketch), rather than content. This makes the domain fusion possible. Meanwhile, the visual- ization results on domain Quickdraw show that since this domain is composed of simple lines, there not enough information related texture, background, etc., and the attention can be limited. This also explains why our approach does not achieve the best performance on some tasks, except for the reason of different model structure (ResNet-50 and ResNet- 101). This is because domains such as in Office-31 I and Qdr in DomainNet lack sufficient domain information to support contrary attention. Nevertheless, our ADNT still has top performance on these domains, which reflects the robustness of our method.
Painting
RealWorld
Sketch
QuickDraw
Figure 4: The visualization about Grad-CAM. The first row shows the original image and the second row shows the visualization.
Furthermore, the weight value in A' actually reflects the correlation among domains (Here, we use the domain to refer the features in the domain). Therefore, the relation of different domains can be estimated by A'. In this paper, we expect the following domain relationship: Initially, they should have very different styles, and the weights A' will be quite uncertain. Then, as the optimization of the model parameters, different domains will gradually be mixed. Finally, all domains should have a consistent style and maintain balanced relationship. Therefore, the merging of all domains together means that there should be little difference among the final correlation coefficients in A'. So we calculate the standard deviation of A' during the training.
As shown in the Fig. during the training process, the standard deviation gradually becomes smaller and eventually tends to be stable, which indicates that the features from different domains are dis- tributed in the space with uniform state.
13

---

Published as a conference paper at ICLR 2023
1.330
0.900
Shitard 101
Suntard MorkenDorartt
3.9
0.002
Figure 5: The standard deviation A' on the Effice-Home dataset.
Figure 6: Visualization of the feature distribution during the training process in the transfer task Ar, CI, Rw to Pr. (a), (b), (c) and (d) represent the epoch 10, 20 and 30, respectively. To provide clearer illustration, we select features from the first ten classes of each domain, with different color representing different class. Solid circles represent all source domains, while the squares represent target domains. Best viewed enlarged.
However, the above results alone do not demonstrate that features from different domains are deeply fused because the previous effect in Fig. can also be achieved features are sufficiently separated from each other. Consequently, we map the fused features h into the low-dimensional space and visualize the distribution during the training process, divided by different domains and categories. As shown in Fig.(6] G at the start of training, the fused features are distributed haphazardly in space, and the network does not have capacity to process them. Then, different domains gradually move toward each other and merge together as the updating of model parameters. Finally, the features of the same category become more compact while preserving domain fusion. This powerfully demonstrates that through CADM, different domains embrace each other and achieve deep fusion. We make the model learn how to handle the fused features and show its excellent results.
A.2 ADDITIONAL. EXPERIMENTS ABOUT LAR-CE
First, shown in the Fig. 7 show the specific structure of the pseudo-label generation and the gradient back- pock-propagation. Through CAR-CE, the soft label generation can be optimized during gradient descent weaken the influence from the noise in the pseudo-labels.
Furthermore, the accuracy of the generated pseudo labels can reflect the effectiveness and stability of the pseudo-label generation. Therefore, we compare the average accuracy of pseudo labels generated with and without AR-CE loss Office-31, Office-Caltech and Office-Home, which is shown in Fig. 7 Specifically, we calculate the mean accuracy the final five epochs during training. It can be seen that after the introduction of the AR-CE loss, the model can derive noisy labels through self-correction and achieve the high accuracy of pseudo labels.
Finally, we have conducted additional experiments vertify the effect of adaptive factors. Specif- ically, we counted 1000 samples with correct and wrong pseudo labels separately and calculated their distribution entropy described in Sec. 4.3. The results shown in Fig. in the appendix. can be seen that the distribution entropy of the correct samples has values around 4, while the incorrect samples have smaller values, mostly around 2.5. This difference proves the feasibility of
14

---

Published as a conference paper at ICLR 2023
Memory
d(x, y)
argmaa
Softmax
soft-label
pseudo label
LR CE
CEB
Classifier
d(( Compute memory
Forward Propagation
Class Score
Batkward Propagation of Lap-cs
Batkward Propagation of fer
Figure 7: The specific structure of the pseudo-label generation in our ADNT, where the Class Score the output classifier.
Correct Label
Wrong Label
70
60
50
Number of Samples
40
30
20
10
0
1
3
4
5
Value of Entropy
Figure 8: Visualization of the value of the distribution entropy for different samples, where the verti- cal coordinate is the number of samples and the horizontal coordinate is the value of the distribution entropy.
15

---

Published as a conference paper at ICLR 2023
proposed adaptive factor. Although it is difficult be completely correct because the pseudo labels depend on the generation, our method can significantyy achieve the correction of pseudo-labo generation. As result, the impact of noise the model performance is reduced.
WHD ARCE hou
ARCE IC ABCCE
Annoney 15
A
-3
-V
Figure 9: Mean accuracies (%) pseudo labels on three datasets. (a), (b), (c) represent the result on Office-31, Office-Caltech and Office-Home respectively.
A.3 MODEL ARCHITECTURE
Since the feature extractor F() in our method is the pre-trained ResNet-50, in order to explore the performance of our method using larger network structure, we conducted experiments on Office- 31, Office-Caltechl and Office- Home using ResNet- 101 the feature extractor, as shown in Table The results show that the performance the network does improve somewhat, however, there tasks (especially some tasks that have close to 100% accuracy with ResNet-50).
drop in On the Office-Home dataset, there is a significant improvement. This indicates that data size is an important factor limiting the application of large model to our approach. Moreover, since we simply perform model replacement in this experiment, there is further room to optimize the way the larger model is combined with our proposed ADNT.
Table 6: Classification accuracies (%) on Office-31 dataset
A.W ++
A,D
D,W
Method
Avg
-A
ADNT (ResNet-50)
100
99.6
74.4
91.3
ADNT (ResNet-101)
99.7
99.3
76.1
91.7
Table Classification accuracies (%) on Office-Caltech dataset
A,C.D
A,C.W
A,W,D
C.D.W
Method
Avg
-W
+D
C
-A
ADNT (ResNet-50)
100
100
97.6
96.3
98.5
ADNT (ResNet-100)
99.8
100
98.1
96.5
98.6
Table Classification accuracies (%) on Office-Home dataset
Cl,Pr.Rw Ar
Ar.Pr.Ww +CI
Ar.Cl.ww +Pr
Ar.CI,Pr RR
Method
Avg
ADNT (ResNet-50)
73.8
66.5
85.1
83.3
77.2
ADNT (ResNet-101)
74.6
66.3
85.7
84.8
77.9
A.4 PARAMETER ANALYSIS
Exponential moving average (EMA) is used in the experiments to maintain the domain style centers and class-level features, so the selection of hyperparameters and requires additional analysis and
16

---

Published as a conference paper at ICLR 2023
verification. Therefore, we perform experimental comparison of their values and present them in Fig.
10
90
85
85
80
80
Accuracy (96
Accuracy
75
75
70
65
60
0.009
0.012
(a) a
(b) B
Figure 10: Sensitivity analysis of and 8
As can be seen, the value of the hyperparameter a is somewhat greater and the taken at an overall smaller value. This indicates that the domain style centroid is updated more vigorously and provides gradient sufficient for the backpropagation of network. Memory M, on the other hand, requires slower update to maintain network robustness because it involves the pseudo-label generation.
17


Reviews:
[{"review_id": "S6RbwvfMv0b", "reviewer": "Reviewer_2VJ6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel method named ADNT for tackling the multi-source unsupervised domain adaptation (MUDA) problem. Their method mainly contains two well-designed strategies: the contrary attention-based domain merge (CADM) module for domain feature fusion, and the adaptive and reverse cross entropy (AR-CE) loss for robust pseudo-label generation. They verified the large superiority of ADNT on four MUDA benchmarks.", "summary_of_the_review": "Although most of the two main strategies are inspired by other closely related researches, the method proposed in this paper is practical and effective, and can be used as a representative method in the field of MUDA.", "strength_and_weaknesses": "Strength:\n1) The elaborately designed CADM module is inspired by the cross-attention mechanism. By adding a metric-based loss for maintaining a clear decision boundary and a memory M for storing features, the domain feature fusion capability of this module has been further improved. This fusion strategy may generalize to other visual MUDA tasks, like object detection and semantic segmentation.\n2) The proposed new loss function AR-CE focuses on alleviating the influence of noisy pseudo-label generated from the target domain during training, which is inspired by the method SHOT++. By adding the reverse operation, the new loss can achieve pretty good results.\n3) The presented method ADNT has obtained SoTA performances on four common MUDA benchmarks.\n\nWeaknesses:\n1) As mentioned above, the two main strategies involved in the method are derived from the improvement of existing modules. This limits the innovation and novelty of the paper to a certain extent.\n2) The explanation of proposed loss AR-CE around Equations (15), (16) and (17) is ambiguous and difficult to understand. I think authors may need to rearrange the language to describe their design ideas.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qkq-cKWbxs", "reviewer": "Reviewer_eMFp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets transferring multi-source knowledge to the target domain. While the previous studies tried to utilize the generalized knowledge of the multiple source domains, the proposed algorithm focus on domain-specific information. To accomplish the extraction of the domain-specific information, the paper presents new modules of a contrary attention-based domain merge module and an adaptive and reverse cross-entropy loss. The contrary attention-based domain merge module can handle the interaction among the features for the domain-specific information. In addition, the adaptive and reverse cross-entropy loss works as a constraint for stable pseudo-label generation. The overall framework shows the state-of-the-art performance in the various experimental scenarios.", "summary_of_the_review": "Due to the weakly explained motivation of the proposed algorithm and the missing analysis of the performance drop, it becomes hard to understand the reasoning of the framework design and its state-of-the-art performance. In addition, the overall organization of the paper should be improved and proofread. The details can be referred to the weakness section.", "strength_and_weaknesses": "The visualization figures of the paper are helpful to understand the overall framework, and the experimental settings are reasonable to validate the algorithm. The state-of-the-art performance of the proposed algorithm confirms its validity, and several ablation studies verify the effectiveness of each component.\n\nHowever, I have several concerns and questions yet.\n\n1. Weak motivation for sharing the same domain-specific information\n\n   The authors insisted that filtering the domain-specific information for every domain is difficult and often results in losing discrimination ability. Of course, the filtering process can be challenging, but the approach looks more reasonable than the shared domain-specific information. Since the multiple domains would have different characteristics that are helpful to extend the distinguishability of the target domain, it seems more reasonable to extract the different domain-specific information from each domain. In addition, when we want to extract the shared domain-specific information from the multi-source, only the limited information would be extracted when the multiple sources are composed of dispersed domains. I hope to know the additional explanation for this problem.\n\n2. The organization of sampled batch\n\n In the first paragraph of section 3, the sampled batch contains \"b\" samples for each domain. Then, when we have a variety of source domains, the number of domain-wise samples becomes very limited. I hope to know the relationship between the number of domains and the performance of the proposed algorithm.\n\n3. Meaning of Eq.9\n\n  The training loss of Eq.9 seems contradictive with Eq.5. While Eq.5 makes the similar domain information connected less effectively, Eq.9 lets the scattered information be gathered again. Why do we need to consider the two contradictive losses in the integrated form? And what would be the effectiveness of the integration?\n\n4. Missing explanation of Table1, 2, 3\n\n  There is no explanation for Tables 1, 2, and 3. Since the results from Tables 1, 2, and 3 are from the main experiments, the related analysis is essential but missing. \n\n5. MIssing analysis for the performance drop\n\nIn Tables 3 and 4, there appear several performance drops compared to the state-of-the-art algorithms for some scenarios, but no related analysis is explained in the paper. We can catch the effectiveness of the proposed algorithm through the analysis of limitations.\n\n6. Several minor comments\n\n In Figure 4, the font size is too small\nAt the last of Eq. 6 and 7, commas should be followed to continue the sentences.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JFPMhZbJOee", "reviewer": "Reviewer_buhi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies multi-source domain adaptation, an interesting and important domain adaptation topic. Instead of learning domain-invariant feature representations, this paper proposes to transfer domain-specific information and also considers pseudo-label generation. Specifically, this paper proposes (1) a Contrary Attention-based Domain Merge (CADM) module to enable the interaction among the features so that domain-specific information can be mixed, and (2) an adaptive and reverse cross-entropy loss to correct the pseudo labels during training. Experiments are conducted on four benchmark datasets for classification.", "summary_of_the_review": "Interesting idea but unclear motivation, generally new components for multi-source domain adaptation, insufficient analysis on the results, and could be improved presentation.", "strength_and_weaknesses": "Strengths:\n1. The idea of transfering domain-specific information is interesting though I am not sure whether this makes sense.\n2. The proposed CADM and adaptive and reverse cross-entropy loss are relatively novel based on some existing techniques.\n3. The results on some adaptation settings are good and the ablation studies and visualizations are convincing.\n\nWeaknesses:\n1. The motivation is not clearly explained. Learning domain-invariant features and enhancing the category information are easy to understand for multi-source domain adaptation. However, this paper tries to transfer domain-specific information. Since they are domain-specific, how can they be transferred and why do they work for adaptation?\n2. As some methods claimed, different source domains and different examples in each source domain play different roles during the adaptation process. How is such information considered?\n3. It seems that one feature extractor F and one classifer C are firstly pre-trained. Since there are multiple domains, do you think simply combining all source samples and train one common F and C makes sense?\n4. The results on DomainNet are much better than the compared baselines, but the result on other three datasets are only marginally better or even inferior. Why? There is no analysis on the conditions that the proposed method work. \n5. The presentation needs to be revised and improved. For example, what does $D$ indicate (feature dimension in my understanding)? In Introduction, \"Secondly, To enable\"->\"Secondly, to enable\". The format of references, especially the names of conferences and journals, is inconsistent. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8Jb3hdiuEH", "reviewer": "Reviewer_javN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work concentrates on two problems in the MUDA task: a) sharing the same domain-specific information across domains to fuse multi-source data; b) generating reliable pseudo labels to alleviate the effect of noisy pseudo labels. Thus, they proposed the Contrary Attention-based Domain Merge modules to pass domain-specific information through EMA-updated domain style centroids. Moreover, a growing class center is applied to optimize intra-class distance. In order to generate reliable pseudo labels, gradients are assigned to the generation process of pseudo labels. Additionally, adaptive and reverse cross-entropy loss is proposed to ensure reliable pseudo-label generation. Extensive experiments were conducted to validate their contributions.", "summary_of_the_review": "The paper is a borderline paper for this conference.", "strength_and_weaknesses": "1. Figure 1 doesn't convey the pipeline and process of your model well, especially in the right part concerning the classifier and pseudo-label generation. Adding some sub-figures may be better.\n2. As you proposed to assign gradients to the pseudo label generation process, give a comparison of the traditional gradient-free pseudo generation process and your process in the form of the figure may be better. Or, you may point out the difference by giving annotations of the\narrows and lines in Figure 1.\n3. The adaptive and reverse cross-entropy loss depends on the correctness of generated hard labels. However, there is no guarantee that the hard labels are reliable. The adaptive factor based on the entropy of distribution may amplify the influence of the wrong assignment and mislead the model. Have you ever considered about this and what about your explanation?\n4. Is there any evidence to validate the contribution of reverse cross-entropy in the ablation study? This may validate the effectiveness of the adaptive factor.\n5. The annotations and explanations should be reconsidered. For example, if the adaptive and reverse cross-entropy loss is not added, does the method assign gradients to the pseudo-label generation process? If the gradients are assigned, what is the loss? reverse cross-entropy loss or others? This should be annotated clearly and added an explanation in the ablation study part.\n6. The ResNet50-based models achieve 100% on some experiments on Office-31, Office-Caltech dataset outperformed the methods which are based on ResNet101. What are the results if your use ResNet101 on ADNT on the Office-31, Office-Caltech, and Office-Home\ndataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject