Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
Authors: Zhongqi Yue
Qianru Sun
Hanwang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: unsupervised domain adaptation
transfer learning
TL;DR: Leverage target domain inherent distribution to remove spurious correlation in UDA.

Abstract: Domain Adaptation (DA) is always challenged by the spurious correlation between the domain-invariant features (e.g., class identity) and the domain-specific ones (e.g., environment) that does not generalize to the target domain. Unfortunately, even enriched with additional unsupervised target domains, existing Unsupervised DA (UDA) methods still suffer from it. This is because the source domain supervision only considers the target domain samples as auxiliary data (e.g., by pseudo-labeling), yet the inherent distribution in the target domain---where the valuable de-correlation clues hide---is disregarded. We propose to make the U in UDA matter by giving equal status to the two domains. Specifically, we learn an invariant classifier whose prediction is simultaneously consistent with the labels in the source domain and clusters in the target domain, hence the spurious correlation inconsistent in the target domain is removed. We dub our approach "Invariant CONsistency learning" (ICON). Extensive experiments show that ICON achieves the state-of-the-art performance on the classic UDA benchmarks: Office-Home and VisDA-2017, and outperforms all the conventional methods on the challenging WILDS 2.0 benchmark. Codes are in https://github.com/yue-zhongqi/ICON.

Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
Zhongqi Yue', Hanwang Zhang', Qianru Sun?
'Nayyang Technological University. "Singaporc Management University
yuez0003@ntu, hwwp::ttttuuuuuu...e.ee edu.sg, hanvangzhangôntu edu.sg, qianrueun@emi, edu.
Abstract
Domain Adaptation (DA) is always challenged by the spurious correlation be- tween domain-invariinn features (e.g., class identity) and domain-specific features (e.g., environment) that does not generalize to the target domain. Unfortunately, even enriched with additional unsupervised target domains, existing Unsuperyised DA (UDA) methods still suffer from it. This is because the source domain pervision only considers the target domain samples as auxiliary data (e.g., by pseudo-labeling), yet the inherent distribution in the target momainn-weers the valuable de-correlation clues hide-is disregarded. We propose to make the U UDA matter by giving equal status the two domains. Specifically, we learn invariant classifier whose prediction is simultancously consistent with the labels in the source domain and clusters in the target domain, hence the spurious correlation inconsistent in the target domain is removed. We dub our approach "Invariant CON- sisteney learning" (ICON). Extensive experiments show that ICON achieves the state-of-the ofttth-rrtttttt performance on the classic UDA benchmarks: OFFICE- HoMe and VIsDA-201 and outperforms all the conventional methods on the challenging WILDS 2.0 benchmark. Codes are in Appendix.
Introduction
Domain Adaptation (DA) is all about train- ing model in labelled source domain S (e.g., an autopilot trained in daytime), and
Source domain S
Target domain T
Samples
0
/
0
0
0
0
the model is expected to generalize in tar- get domain T' (e.g., the autopilot deployed nighttime), where T' and has signif- icant domain shift (e.g., day vs. night) [41].
0
0
/
Taain Accuracy: 90.6%
Teat Accuracy 90.2% 2%
E-SNE features
To illustrate the shift and generalization, we
introduce the classic notion of causal repre-
: any sample generated by x ( e), where 4 is the generator, e the causal feature determin- ing the domain-invarian class identity (e.g., road lane), and is the environmental fea- ture (e.g., lighting conditions). As the en- vironment is domain -ppeiffcc, the domain shift es * er results in Xs * XT, chal- lenging the pursuit of a domain-invariant class model. However, if model disentangles the causal feature by transforming each can generalize under arbitrary shift e. In Figure [ even though er = "colorful" significantly
sentation learning 220 49
Figure I: A digit classification model trained on red digits in generalizes to colorful digits T by disen- tangling digit shape (i.e., the causal feature c), as color in (i.e., es) is not discriminative. o: "0", "1". The classification boundary is rendered blue and red.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

e (digit shape)
ICON
e (digit color)
(o)
Figure 2: Red: S. Blue: T. @: "0", "1". (a) A failure case of existing lineups. The red line denotes the classifier trained in S. (b) Classifier can be corrected (black line) by respecting the BCE loss between S (red line) and T (blue line). (c) A failure case of minimizing the combined BCE losses in S and T. Due to the low variance of eş in each class, the optimal classifier w.z.t the BCE loss in S (red line) looks "flat", i.e., it has large weight on e 33 (d) Our ICON prevents such a failure case.
different from es "red", thanks to the con-discriminative "red" color in S, the model still easily disentangles e from ((c, es) and thus generalizes to digits of any color.
Unfortunately, in general, disentangling by using only S is impossible eş can be also discrimina- tive, ie., spuriously correlated with c. In Figure 2a] the color eş correlated with the shape c, ie., C₁ ="0" and C2 ="I" in S tend to have e1 dark and e2 light digit colors, respectively. As both с and es are discriminative, a classifier trained in S will inevitably capture both (see the red line in Figure 2ap. This leads to poor generalization when the correlation between ey and e is different, i.e. the colors of C1 =0" and C2 ="1" in T tend to be c₂ = light and e₁ = dark instead.
To avoid spurious correlations between eş and c, a practical setting called Unsupervised DA (UDA) introduces additional information about T through a set of unlabeled samples [TS] [3] 12.
There are 10 learn the common feature space of S and T that minimizes the classification error S. In Figure 2a their goal is to unify "0"s and "1"s in S and T. However, they suffer from the well-known misalignment issue 31 71], e-g., collapsing "0"s and "I"s in T to "I"s and "O's in S respectively also satisfies the training objective, but it generalizes poorly to T' 2) Self-training methods [72 38, (52) use classifier trained in S to pscudo-label samples in T, and jointly train the model with the labels in S and confident pseudo-labed in T., Yet, the pseudo-labels can be unreliable even with expensive threshold tweaking 332 29 e.g., the X areas in Figure 2a have confident but wrong pseudo-labels. In fact, the recent WILDS 2.0 benchmark [47] on real-world UDA tasks shows that both lineups even under-perform embarrassingly naive baseline: directly train in S and test in T.
two main lineups: 1) Domain-aligmmen
methods [58] 34
Notably, although both methods respect the idiosyncrasies in S (by minimizing the classification error), they fail to account for the inherent distribution in T, e.g., in Figure [a even though 7 is unlabelled, can still identify the two sample clusters in T' (enclosed by the blue circles), which provide additional supervision for the classifier trained in S. In particular, the classifier (red line) in Figure 2a breaks up the two clusters in T, showing that the correlation between color es and in S is inconsistent with the clusters affected by or in T, which implies that color e is the environmental feature. Hence, to make the U in UDA matter, we aim to learn classifier that is consistent with classification in and clustering in T:
Consistency. We use the Binary Cross Cross-Entropy (BCE) loss to encourage the prediction similarity of each sample pair from the same class of S or cluster of T, and penalize the pairs from different ones. In Figure 2b the red and blue lines denote the classifiers that minimize the BCE loss in S and T. respectively. By minimizing the combined loss, we learn the classifier consistent with the sample distribution in both domains (the upright black line), which predicts solely based on the causal feature e (i.e., disentangling from (cc,e)
Invariance. However, enforcing consistency is a tug-of-war between S and T, and the BCE loss in one domain can dominate the other to cause failures. Without loss of generality, we consider the case where S dominates in Figure 2c Due to the strong correlation between es and the classifier that minimizes the combined losses of BCE (black line) deviates from the desired upright position as in Figure 2b To tackle this, we aim to learn an invariant classifier [ that is simultaneously optimal w.e. the BCE loss in S and T. In Figure 2d] for each of the two losses, we color the candidate regions where the classifier is optimal (i.e., gradients se 0) with red and blue, respectively. In particular, by

---

S
model
dat-product
"cloc*
-cluster 24
loss
similarity in S.T
f
"bed"
Ys -cluster
bed dook
clock bed
BCE(S,F)
BCEIT /
0
cT
Invariance
Figure 3: A training example where the pair of samples in S is from different classes and that in T from the same cluster. The black arrows on the similarity bars denotes that minimizing the BCE losses in S and T will decrease the similarity in S (as + y2) and increase that in T (as = 82).
learning an invariant classifier lying in the intersection (purple region), we successfully recover the desired one in Figure 2b even when domain dominates.
Overall, term our approach as Invariant CONsisteney learning (ICON), Our contributions:
ICON is a novel UDA method that can be plugged into different self-training baselines. is simple to implement (Section 2), and disentangles the causal feature from $(c, e) that generalizes to T with a theoretical guarantee (Section 33
ICON significantly improves the current state-of-the-art classic UDA benchmarks 22 6 with ResNet-50 [9] as the feature backbone: 75.8% (+2.6%) accuracy averaging over the 12 tasks in
OFFICE-HOME [60), and 87.4% (+3.7%) mean-class accuracy in VISDA-2017 [43] (Section 5
Notably, as of NeurIPS 2023, ICON dominates the leaderboard of the challenging WILDS 2.0 UDA benchmark [47] (https:/ http:::/wwwwlee stanssttnford www.afor..dd//eeaeeeeeeeee edw/leaderboard/ which includes 8 large-scale
classification, regression and detection tasks on image, text and graph data (Section
2
Algorithm
UDA aims to learn a model with the labelled training data {x,, J. in the source domain S and unlabelled {M in the target domain T, where x; denotes the feature of the i-th sample extracted by a backbone parameterized by 0 (e.g., ResNet-50 [19),, and y; is its ground-truth label. We drop the subscript for simplicity when the context is clear. As shown in Figure our model includes the backbone 0, a classification head f, and cluster head where I and output the coftmax-normalizer probability for each class and cluster, respectively. Note that f and y have the same output dimension as the class numbers are the same in S and T. As discussed in Section [ the generalization objective of ICON the classification in S, and the clustering in I', identified by ICON is illustrated in Figure 3 and explained in details below.
learn an invariant f that simultaneously consistent with
2.1 Consistency with S
We use Binary Cross-Entropy (BCE) loss to enforce the consistency of with S, denoted as BCE(S, f). Specifically, BCE(S,)) is pair-wise loss given by:
BCE(S, = Exx,x,)~ (blog(0) + b)log(1 - ú)]
where b = 1(y = yハ) is a binary label indicating if the pair xx, is from the same class, and fxx)) (f is the predictive similarity. Intuitively, BCE is a form of contrastive loss that clusters sample pairs with the same label (by increasing the dot- dot-product similarity of their predictions) and pushes away pairs with different labels (by decreasing the similarity). Note that in practice, BCE loss is computed within each batch, hence there no N² prohibitive calculation.
Consistency with T
Unfortunately, BCE(T, /) cannot be evaluated directly, as the binary label b in Eq. 1 is intractable given unlabeled T. To tackle this, we cluster capture its inherent distribution and use cluster labels to compute BCE(T,

---

Clustering T. We adopt the rank-statistice, algorithm [18] as it can be easily implemented in mini-batch sampling manner for online clustering. The algorithm trains the cluster head g to group clustering algorithm is Section 5.4 Note that our g is trained from scratch, rather than initialized from Hence ! captures the distribution of 7 without being affected by that of S.
features whose top values have the same indices (details in Appendix). Ablation
trained in S [56 29 32
Computing (C f). After training g. we replace b 1 (y; - yj for BC in Eq. ① with b = 1 (argmax g (x,) argmax (X;)) for BCE(T' f), which compares the cluster labels of each sample pair. We emphasize that it is necessary to use pairwise BCE loss to enforce the consistency of with T. This is because cross-entrope loss is not applicable when the cluster labels in T are not aligned with the class indices in S.
2.3 Invariant Consistency (ICON)
ICON simultaneously enforces the consistency of f with S (i.e., the decision boundary of f separates the classes) and T (i.e., it also separates the clusters). The objective is given by:
Consistency
Supervision
min BCE(S, () + BCE(T, + CE(S, aL.t 8.f
s.t. f argmin BCE f) argmin a
(2)
Line 1. The BCE losses train be consistent with the pair-wise label similarity in S and T. CE loss in S trains I' to assign each sample its label (e.g., predicting a clock feature as "clock"), L the self-training loss with weight a, which leverages the learned invariant classifier to generate accurate pseudo-labels in T for additional supervision (details in Section 5.2).
Line 2. The constraint below requires the backbone 0 to elicit an invariant classifier / that simultane- ously minimizes the BCE loss S and T. This constraint prevents the case where one BCE loss is minimized at the cost of increasing the other to pursue a lower sum of the two (Figure
To avoid the challenging bi-level optimization, we use the following practical implementation:
min BCE(S, f) + BCE(T, CE(S, f) aLst + Varr (!BCE(S, f), (CE(T,f))), (3) R.f
where the variance term Var(-) ) is known as the REx loss 25 implementing the invariance constraint. The self-training weight and invariance weight / are later studied in ablation (Section 5.4).
Training and Testing. Overall, at the start of training, the backbone / is initialized from a pre-trained weight (e.g., on ImageNet (46)) and f, g are randomly initialized. Then ICON is trained by Eq. until convergence. Only the backbone 0 and the classifier f are used testing.
Theory
Preliminaries. We adopt the group-theoretic definition of disentanglement 20
justify ICON.
We start by introducing some basic concepts in group theory. G is group of semantics that generate data by group action, e.g., "turn darker" element g E G transforms x from white "0" to g as grey "0" (bottom-left to top-left in Figure 4p). A sketch of theoretical analysis is given below and interested readers are encouraged read the full proof in Appendix.
Definition (Generalization). Feature space X generalizes under the direct product decomposition G = G/H x H. if X has non-trivial G/H- "reeteeprenntt,o,n,, i.e., the action of each h H. corresponds to a trivial linear map in X (i.e., identity map), and the action of each g E G/H corresponds to a non-trivial one.
This definition formalizes the notion of transforming each sample {cc, e) to the causal feature c. Subgroup H. C G and quotient group G/H HH transform the environmental feature e and the causal c, respectively. If X' has a non-trivial 9 //H-represennation, samples in a class are transformed to the same feature different e (i.e., non-trivial). This definition indeed in accordance with the common view of good features (62). i.e., they enable zero-shot generalization is class-unique, and they are robust the
X' regardless of (i.e., representation of G/H), and different classes have

---

domain shift by discarding the environmental features from ((c, To achieve generalization, we need the following assumptions in ICON.
Assumption (Identificabilitty of g G/H).
1) 1) T, yi = yj iff x; {h o xj E H]:
2) There exists no linear map 1 X + IR such that 1(x) > 1(h o x), Vx E 0 E T and 1(g x) < l(gh x), Vg x E gh E T, where h + e E H.g + 9/H.
Assumption states that classes in are separated by clusters. This corresponds to the classic cluster- ing assumption, a necessary assumption for learn- ing with unlabeled data [59]. Figure 4 shows failure case of Assumption 1, where the intra-class distance (i.e., dotted double-arrow) is much larger than the inter-class one (i.e., solid double-arrow) in T, i.e., shape is less discriminative than color er. Hence the clusters in I are based on e, causing ICON to capture the wrong invariance. To help fulfill the assumption, UDA methods (and SSL in general) commonly leverage feature pre-traing pre-training and data augmentations. We also specify the num- ber of clusters as the class numbers in T,, which is necessary condition of this assumption.
(digit shape)
e (dight colon
(b)
Figure 4: Red: S, Blue: T.
"0", "I".
Black lines denote the invariant classifier satis- fying the ICON objective. (a) Assumption violated. (b) Assumption 2 is violated.
Assumption 2 states that is the only invariance between S and T. 4p illustrates failure case, where there exist two invariant classifiers: both the vertical one based on shape and the horizontal one based on color e separate the classes in S and clusters in T. Yet as unlabelled, there is no way for the model to determine which one is based on c. In practice, this assumption can be met by collecting more diverse unlabelled samples in T (e.g., darker "0"s and lighter "I"s s in Figure4b), or introducing additional priors on what features should correspond to the causal one.
Theorem. When the above assumptions hold, ICON (optimizing Eq. (3) learns backbone O mapping to a feature space X that generalizes under G/H x H. In particular, the learned f is the optimal classifier in S and T.
By transforming each sample $(c, e) in and T c E ; the classifier easily associate each c to its corresponding class label using the labelled S to reach optimal.
Related Work
Domain-Alignment methods 13
57 map samples in S and T to common feature space where
they become indistinguishable. Existing works either minimize the discrepancy between the sample distributions of and T' (36) 27], or learn features that deceive a domain discriminator [69 63 33)
However, they suffers from the misalignment issue under large domain shift from S to T 7
such shift in the support of the sample distribution
23 (e.g., Figure
Self-Tree Tee-Trainng methods [7 53 are the mainstream in cuni-supervised learning (SSL), and recently became promising alternative in UDA, which focuses generating accurate T pseudo-labels This lineup explores different classifier design, such as k-NN [51] a teacher-student network [11] or an exponential moving average model (30 Yet the pseudo-labels are error-prune due the spurious correlations in S, hence the model performance is still inconsistent 33
Alleviating Spurious Correlation has been studied in several existing works of UDA 56 32 29
They train a -pspccific classifier to generate pscudo-labels [56] (2]. However, their T-classifien. are still initialized based on a classifier trained in S, hence the spurious correlation in S are introduced to the T-classifiers right in the beginning. On the contrary, our ICON learns a cluster head T without supervision in S, which is solely for discovering the sample clusters in T. Hence, the spurious correlation in S inconsistent with the clusters in T can be removed by ICON (Section(3).
share a similar spirit with ICON, where they learn classifier using the labeled samples that consistent with the unlabelled sample similarities. However, they lack the invariant constraint in ICON, which can cause the failure cases (e.g., Figure 2c).
based SSL methods 16 42 22
5

---

Dataset
Orric HOMe
V5DA-2017
MILDCA
YAMELYON17
RMOW
POYOITYMAP
Sample
ebject image
diec image
scrad Itage
XnS image
whaee image
mokked graph
ofline Commant
producc ae nee
Label
12
142
boca
imanees
245
Proppitl f-0
2007-2013
Loliiied you
Example
ASamples
avarage & Brr
152 2017
123,000 203
302,435
883
119.000
2,045
250 543
200 US3
245 5012
Targat 7
uatioaa?
mot pholicx
phockss troes
ides hrom hospkal
Irasees Joom
Imagaa in 1966
images acicus The
43.73 scaltild
33334
Example
dary
Lei
ASamplen
B75
55,786
1200
600
175,206
281 1.336
42.
SIT 0RB
1,551515
265,201 201
Evaluation
average boG.
mann-chwo arruney
m1
000
wonstongion a02
commeeine
Pwrrsn
8cc.
preciion
berraetle
(ERUM)
Y0.4
75.3
47.0/39.2
90.61 82.0
60.8134
65
77. 75.0/51.0
-/20.3
89 99.6
72.0154.2
TCM (81)
CORAL [51]
10.7
75.0
40.5/27.8
90.4/77.9
58, M6.130
54 (:3
+/-
-/29.6
+1-
71.7/533
SENTRY [41]
DANN
72.0
76.7
48.5/51.9
86.0/68.4
57.0/34.6
50 '58
-I-
-120.4
-1-
71.7/53.3
Ecading Methods
CeT (29)
P [24]
80.6
47.3/30.3
01:3/67.7
60.9/3577
75./4229
00
71.//5.3
TeAlign [58]
NDO (es)
Notey Student (8A)
72.7
77.8
47.5/32.1
60:2/86.7 86.7
61.5/37 8
61/422
78 146.
-1275
-1-
-/-
(37)
MT-16ougs
FiMMaoh (50)
Nosked
[1]
73.0
46.3/31.0
81.3/71.0
68.8/ 52.1
-1-
40..7
71.91
ATDOG (20)
MCC-NWD HI
BWAY 88
ERM (ubelled
13.2
03.7
47.3/29.0
52.3/91.4
61.8/363
00/
-7-
73.6
ICON
76.6
+2.8
87.4
+
50 6
+2.3
95.6/92.0
-2.4
62.2/38.9
-2.1
65
78.6/523
+1.5
+0.0
+1.9
71.976477
-0.5
Table 1: Dataset details and the results of our ICON compared with existing methods. Details on in
Section [.] WILDS 2.0 [47] datasets are highlighted in yellow, where models were evaluated on both the test data in S (first number) and in T (second number). The performance gain in T highlighted in red. "ERM (labeled T)" has full supervision in T. -/- means that the method is not applicable to the dataset. Other dataset details and the standard deviation of the results are in Appendix.
5 Experiment
5.1 Datasets
As shown in Table D we extensively evaluated ICON on 10 datasets with standard protocols 34
including 2 classic ones: OFFICE-HOME (60), VISDA-2017 43). and 8 others from the recent WILDS 2.0 benchmark [47. WILDS 2.0 offers large-scale UDA tasks in modalities (image, text, graph) under real-world settings, e.g.. wildlife conservation, medical imaging and remote sensing. Notably, unlike the classic datasets where the same sample set in T is used in training (unlabelled) and testing, WILDS 2.0 provides unlabelled train set in T disjoint with the test set, which prevents explicit pattern mining of the test data. They also provide validation sets for model selection. For datasets in WILDS 2.0 benchmark, we drop their suffix- -WILDS for simplicity (e.g., denoting AMAZON-WILDS as AMAZON).
Details of * in Table OFFICE-HOME has 4 domains: artistic images (Ar), clip art (CI), product images (Pr), and real-world images (Rw), which forms 12 UDA tasks by permutation, and we only show Cl->Rw as an example in the table. FMoW considers the worst-case performance on different geographic regions, where the land use in Africa is significantly different. POVERTYMAP considers the lower of the Pearson correlations (%) on the urban rural arcas. CIVILCOMENT correspond to a semi-supervised setting with no domain gap between and T. and is evaluated on the worst-case performance among demographic identities.
On some WILDS 2.0 datasets, the test data may come from a separate domain different from the training target domain T, e.g., IWILDCAM, AM test data is captured by 48 camera traps disjoint from the 3215 camera traps the training T. We highlight that ICON is still applicable on these datasets. This is because ICON works by learning the causal feature and discarding the environmental feature and the theory holds as long as the definition of and e is consistent across the training and test data, e.g., IWILDCAM is about animal classification across the labeled, unlabeled, and test data.

---

Method
Ar-+CI
Cl-AAr
CI-Pt
Cl-Rw
Pt-CI
Pr-Rw
Avg
DANN
13 (206))
45.6
59.3
70.1
47.0
58.5
60.9
46.1
43.7
68.5
63.2
51B8
76.8
57.6
CDAN
X (011))
50.7
70.6
76.0
70.0
70.0
57.4
50,9
77.3
70,9
56.7
SymNet
69
(2019)
47.7
72.9
78.5
64.2
71.3
74.2
64.2
48.8
79.5
74.5
52.6
82.7
67.6
MDD
70
(2019)
54.9
73.7
77.8
60.0
71.4
71.8
61.2
53.6
78.1
72.5
60.2
82.3
68.1
SHOT
(2020)
57.1
78.1
81.5
68.0
78.2
78.1
67.4
54.9
82.2
73.3
58.8
84.3
71.8
ALDA
(2020)
53.7
70.1
76.4
60.2
72.6
71.5
56.8
51.9
77.1
70.2
56.3
82.1
66.6
GVB 8
(2020)
57.0
74.7
79.8
64.6
74.1
74.6
65.2
55.1
81.0
74,6
59.7
84.3
70.4
TCM 67
(2021)
58.6
74.4
79.6
64.5
74,0
75.1
64.6
56.2
80.9
74.6
60.7
84.7
70.7
SENTRY
(2021)
61.8
77.4
80.1
66.3
71.6
74.7
66.8
63.0
80.9
74,0
66.3
84.1
72.2
CST 22
(2021)
59.0
79.6
83.4
68.4
77.1
76.7
68.9
56.4
83.0
75.3
62.2
85.1
73.0
ToAlign
(2021)
57,9
76.9
80.8
75.6
77.0
67.8
57.0
82.5
75.1
60.0
84.9
72.0
Fixhi 4M (2021)
58.1
77.3
80.4
67.7
79.5
78.1
65.8
57.9
81.7
76.4
62.9
86.7
72.7
ATDOC
(2021)
77.8
78.6
77.9
58.4
83.1
74,8
SDAT 4S (2022)
58.2
77.1
82.2
66.3
77.6
76.8
63.3
57.0
82.2
74.9
64.7
86.0
72.2
MCC+NWD (2022)
79.6
77.9
78.7
66.8
56.0
81.9
73,9
86.1
72.6
ASHOT [55 (2022)
58.2
80.0
82.9
61.1
80.3
80.7
71.3
56.8
83.2
75.5
60.3
86.6
73.9
ICON (Ours)
81.3
82.1
81.0
70.3
61.8
83.7
75.6
Table 2: Break-down of the accuracies in each domain OFFICE-HOME dataset (60]
kSHOT (55)
additionally uses the prior knowledge on the percentage of samples in each class in the testing data. Published years are the brackets after the method names.
5.2 Implementation Details
Feature Backbone. We used the followings pre-trained on ImageNet 46 ResNet-50 [9
on ORMEEE-HOmE, MISDA-2017 and IWILDCAM, DenseNet- 121 [21] on FMOW and Faster-
RCNN 14 on FLOBALWHEAD We used DistilBERT 48 with pre-trained weights from the Trans- formers library CIVI.COMMENT and AMAZON. On CAMELYON17, we used DenseNet-121 [2] pre-trained by the self-supervised SWAV (5] with the training data in S and T. On POVERTYMAP and OGB-MoI.PCH, with no pre-training available, we used multi-spectral ResNet-18 19] and graph isomorphism network [66] trained with the labelled samples in the source domain S, respectively.
Self-training, A general form of the self-training loss Let in Eq. 3 is given below:
(4) where is confident threshold that discards low-confidence pscudo-labels, and CE(.,-) is the cross-entropy loss. We detail the different implementations of Eq (4) in Appendix. ICON is agmostic to the choice of Ltt, and hence choose the best performing self-training baseline for each dataset: FixMatch [53] on OFFICE-HOMe and VIsDA-200 1- NoisyStuddnt [63] on IWILDCAM and FMoW, and AMAZON. ICON also does rely Lst. We did not apply Lst (i.e., 3 = 0) on CAMELYON17, POVERTYMAP, GELOBAL.HEAA and OGB-MOIPPCBA where self-training does not work well.
L Ex-r [1 (maxf(x) > +)CE (f(x), argmax((x))],
Pseudo-label 26
Regression and Detection. For the regression task in POVERTYMAP, f is a regressor that outputs a real number. As the number of clusters in T' is not well-defined, we directly used rank statistics to determine two samples T have the same label. For the BCE loss, we computed the similarity by replacing (x;) f(x,) ) in Eq. T as - (f(x;) f(x;))) For detection in GLOBAL WHEAT, we used the ICON objective in Eq.
train the classification head without other modifications.
Other Details. We aim to provide a simple yet effective UDA method that works across data modalities, hence we did not use tricks tailored for a specific modality (e.g., image), such as mixup (68] or extensive augmentations (16 are used in [LL), On classic UDA datasets, we used entropy minimization and SAM optimizer following CST (32). On VISDA-2017, IWILDCAM, CAMELYON17 1 and FMOW, we pre-processed the features with UMAP 377 (output dimension as 50) and Eqinv [61], which helps fulfill Assumption 1 (i.e., highlight causal feature to improve clustering). ICON is also resource efficient- all our experiments can run on a single NVIDIA 2080Ti GPU.
5.3 Main Results
Classic UDA Benchmarks. In Table [ (first 2 columns), ICON significantly outperforms the existing
state-of-the-ards on OFFICE-HOme (60) and VIsDA-2017 43 by 2.6% and 3.7%, respectively. We
include the breaks down of the performances of OFFICE Home in Table where ICON wins in 10 out
of 12 tasks, and significantly improves the hard tasks (e.g.. Ar- Rw Rw-CI), For ViSDA-2017, we include more comparisons in Table ICON beats the original competition winner (MT+ (6ags) [I] that averages predictions on 16 curated augmented views. ICON even outperforms all methods that
use the much deeper ResNet-101
[19]. Notably, CST 32 and ATDOC (29 also aim to remove
7

---

Method
Backbone
Acc.
Acc. %
Loss
%.aair. fX.gv
MT+16angs
11 (2018)
Re
K2.8
ICON
MDD 70 (2019)
ResNet-50
77.8
FixMatch
TCM
GVB
6
(2020)
(2021)
ResNet-50
ResNet-50
75.8
75.3
8
CST
0.175
0.05
SENTRY
44]
(2021)
ResNet-50
76.7
CST 22
(2021)
ResNet-50
80.6
0.125
0.35
CAN 24
(2019)
ResNet-101
87.2
FixMatch
SHOT 28
FixBi
(2021)
(2020)
ResNet-101
ResNet- 101
87.2
82.9
8
0.075
0.25
ICON
MCC+NWD
6 (2022)
ResNet-100 101
83.7
SDAC 45
(2022)
ResNet-
84.3
Epochs
Epochs
KSHOT 55 (2022)
ResNet-101
86.1
ICON (Ours)
Nes
87.4
Figure (a) Solid lines: test accuracy against train- ing epochs for FixMatch, CST and ICON, Dotted black lines: average loss in each epoch with match- ing markers to denote the three methods. (b) Among the feature pairs where f fails, the percentage of pairs where g succeeds for the three methods.
Table 3: Mean-class accuracy (Acc.) on VisDA-201 Synthetic-RRel task with the choice of feature backbone. ": details in Ta- caption Published years are in the brack- ets after the method names.
the spurious correlation, and both attain competitive performance, validating the importance of this research objective. Yet their training scheme allows the correlation to persist (Section 4 hence ICON is more effective.
WILDS 2.0. In Table I (last 8 columns), ICON dominates the WILDS 2.0 leaderboard despite the challenging real-world data and stringent evaluation. For example, the animal images in IWILDCAM exhibit long-tail distribution that is domain-speciffr locations. Hence under macro-F1 evaluation (sensitive to tail performance), the 2.3% improvement noteworthy. On COMMENTS with text data, ICON improves the worst performance across demographic groups by 2.2%. which promotes AI fairness and demonstrates the strength of ICON even under the SSL setting. On AMAZON, although ICON only improves 0.5% to reach 54.7%, the upper bound performance by training with labelled T (56.4%) is also low due to the strict evaluation metric. We also highlight that ICON increases the performance in T without sacrificing that in (the first number). This empirically validates that the invariance objective of ICON resolves the tug-of-war scenario in Figure 2c In particular, we tried implementing recent methods on this benchmark. Yet they are outperformed by ERM, even applicable (e.g., on text/graph data). This is in line with WILDS 2.0 organizers' observation that ERM as a naive baseline usually has the best performance, which is exactly the point of WILDS 2.0 as diagnosis of the ever-overlooked issues in UDA. Hence we believe that dominating this challenging leaderboard is a strong justification for ICON, and encourage more researchers try this benchmark.
because the rarity of species varies across
Failure Cases. ICON performance is relatively weak on POVERTY MAP and OGB-MOL MOLP with two possible reasons: 1) no pre-training available, the backbone is trained only on S and hence biased in the beginning, and 2) the ground-truth number of classes in is not well-defined in both datasets, hence Assumption 1 in Section 3 may fail.
Learning Curve. In Figure we compare our ICON with two telfniraining baselines: FixMatch [53 and CST (3). The test accuracy and training loss w.E.. training epochs on VISDA-2017 is drawn in colored solid lines and black dotted lines, respectively. We observe that the training losses for all methods converge quickly. However, while the two baselines achieve higher accuracy in the beginning, their performances soon start to drop. contrast, ICON's accuracy grows almost steadily.
Effeect of Supervision in T. To find the cause of the performance drop, we used the optimized g in Section 2.2 to probe the learned f in each epoch. Specifically, we sampled feature pairs X2 in 1" and tested if the predictions of f.g are consistent with their labels, e.g., f succeeds if f = argmaxf(s (x2) when yı1 = y/ (vice versa for A). Then among the pairs where failed, we computed the percentage of them where succeeded, which is shown in Figure 5p. A large percentage implies that f disrespects the inherent distribution in T correctly identified by g. We observe that the percentage has a rising trend for the two baselines, i.e., they become more biased to the spurious correlations S that do not hold T. This also explains their dropping performance in Figure 5p. In particular, CST has lower percentage compared to FixMatch, which suggests that it has some effects in removing the bias. In contrast, the percentage steadily drops for ICON, as we train f to respect the inherent distribution identified by g.
8

---

Method
OFFICE- HOME
VISDA-2017
OFFICE-HOME
VIsDA-2017
FixMarch
69.1
76.6
B
0.25
0.5
1.0
2.0
B
0.25
0.5
1.0
2.0
FixMach-CON
74.1
0.05
85.2
86.2
86.3
81.5
FixMatch+CONni
75.8
87.4
Cluster with 2 #classes
69.7
78.6
Cluster with 0.5 Welasses
67.5
73.9
75.4
75.8
74.4
85.6
86.4
86.7
82.6
Cluster with K-NN
72.1
85.6
0.26
70.2
72.8
75.2
74.5
0.25
76.8
81.2
87.4
80.3
Table 4: Ablations on each ICON component. CON denotes the consistency loss in and T. INV denotes the invariance constraint.
Figure 6: Ablations of a, on the classic UDA datasets.
5.4 Ablations
Components. Recall that the main components of ICON include: 1) Consistency loss (CON) that leverages the supervision from the labels in S and clusters in T; 2) the use of invariance constraint (INV) in Eq. (3): 2) clustering in T with #clusters=#llasses. Their ablations are included in Table I From the first three lines, observe that adding each component leads to consistent performance increase on top of the self-training baseline (FixMatch), which validates their effectiveness. From the next two lines, when #clusters//Hclasses, the model performs poorly, e.g., using one cluster in Figure 26 (i.e., 0.5 #classes,,,,,, the invariant classifier needs to predict the two red circles in S as dissimilar ar (two classes) and the two blue ones in T as similar (one cluster), which in fact corresponds to the erroneous red line (i.e., predictions in T are similarly ambiguous). In the last line, we tried k-means for clustering instead of rank-statistice leading decreased performance. We postulate that rank-statistics is better because its online clustering provides up-to-date cluster assignments. In the future, we will try other more recent online clustering methods, e.g. an optimal transport-basee based method [5]. Overall, the empirical findings validate the importance of the three components, which are consistent with the theoretical analysis in Section
Hyperparameters. We used two hyperparameters in our experiments: 1) the weight of the self- training loss denoted as ot; 2) the invariance penalty strength in the REx (25) implementation of Eq. (3. Their ablations are in Figure on on the widely used classic UDA datasets. We observe that E 0.5, 1.0] and 3 [0.1, 0.25] works well on both datasets. In particular, is standard in the IRM community to choose small 3 (e.g.. 0.1). Then, tuning a follows the same process as other self-training UDA methods. Hence the hyperparameter search is easy.
Confusion Matrix. We use the matrix to concretely show how existing methods are biased to the correlations in the source do- main S. We compare the state-of- the-art self- training method CST with ICON in Figure 7 The matrix on the left reveals that CST predicted most "knife"s as "skateboard''s ss ("board" for short), and some "truck's 's as "car"s. On the right, we see that ICON predicts "knife" accurately and also improves the accuracy on "truck".
CST
ICON
aeroplane
aaroplane bicycle bus car horse
bicydl
horee
knife
knile
motorcydle
motorcycle
-04
person
plant
plant
skateboard
skateboard
train
tran
truck
tnuck
bus
car
harse
egui
untted
yena
Figure 7: CST and ICON confusion matrix on VIsDA-2017.
Grad-CAM Visualization. Why does CST predict "knife" in T' as "board"? To answer this, in Figure 8 we visualized the GradCAM (50) of CST and ICON trained on VisDA-2017. In the first row, we notice that the self-training based CST predicts the two categories perfectly by leveraging their apparent differences: the curved blade versus the flat board ending with semi-circles, However, this leads to poor generalization on "knife" in T. where the background (image 4), the knife component (2) or even the blade (3) can have board-like appearances. Moreover, biased to the standard "board" look in S, CST can leverage environmental feature for predicting "board" in T, egg., under side-view (5) or slight obstruction (6), which is prune to errors (e.g., 8). As the board-like shape alone is non-discriminative for the two categories in T, the second row, the cluster head y additionally leverages knife handles and wheels to distinguish them. In the third row, our ICON combines the

---

preds:
Araffe
CST
preds:
cluader 10
clusder
Cluster
invarlant
Araffe
board
knjfo
board
ICON
43 using CST [32.. the cluster head and the proposed ICON. Ground-truth knives and boards are in solid and dotted boxes, respectively. Predictions from f.g are on top of each row. Red denotes wrong predictions. Pale blue and red background denotes training and testing, respectively. Note that is trained without supervision in T.
Figure 8: GradCAM on VisDA-2017
supervision in S (first row) and 7' (second row) to learn invariant classifier f. In S, ICON focuses on the overall knife shape instead of just blade, and incorporates wheels to predict "board". In T' by comparing images with the same number in the first and third row, we see that ICON focuses more on the discriminative parts of the objects (e.g., handles and wheels), hence generalizes better T.
6 Conclusion
We presented novel UDA method called Invariant CONsistency learning (ICON) that removes the spurious correlation specific to the source domain to generalize in the target domain T. ICON pursues invariant classifier that is simultaneously consistent with the labels in S and the clusters in T,, hence removing the spurious correlation inconsistent in T. We show that ICON achieves generalization in T with theoretical guarantec. Our theory is verified by extensive evaluations on both the classic UDA benchmarks and the challenging WILDS 2.0 benchmark, where ICON outperforms all the conventional methods. We will seek more principled pre-training paradigms that disentangle the causal features and relax the assumption of knowing the class numbers in T to improve regression tasks.
Limitation. Our approach based on the assumptions in Section 4, i.e., the classes in T are separated by clusters, and there enough diversity in T such that causal feature is the only invariance. Assumption is essentially the clustering assumption, which a necessary assumption for learning with unlabeled data [59]. In fact, UDA methods (and semi-supervised learning in general) commonly leverage feature pre-training and data augmentations to help fulfill the assumptions. One can also use additional pre-trained knowledge from foundation models, or deploy improved clustering algorithm (e.g., an optimal transport-besd bondee method (5)). Assumption 2 is necessary to rule out corner cases, e,g., Figure 4b. without additional prior, there is no way for any UDA method to tell if each one of the blue cluster (unlabeled) should be pseudo-labeled as "0" or "1". In practice, these corner cases can be avoided by collecting diverse unlabeled data, which generally abundant, e.g., collecting unlabeled street images from a camera-equipped vehicle cruise.
Broader Impact. Our work aims capture the causal feature in the unsupervised domain adaptation setting. In particular, we significantly improve the state-of-the-art on WILDS 2.0 benchmark, which includes practical tasks that benefit wildlife preservation, tumour classification in healtheare, remote sensing, vision in agriculture, etc. The pursue of causal feature can produce more robust, transparent and explainable models, broadening the applicability of ML models and promoting fairness in AI.
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: 1002-RPP2022-022- MOE AcRF Tier (MOE2201-TT---0662) Alibaba-NTU Singapore Joint Research Institute, the S under its AME YIRG Grant (Project A20F6c0101) and the Lee Kong Chian (L.KC) Fellowship fund awarded by Singapore Manage- ment University.
10

---

References
[1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXhv:190..02893, 2019.
[2] Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distribu- tions with applications adversarial learning and domain adaptation. ICCV, 2019.
[3] David Berthelot, Nicholas Carlini, Ian Goodfellow, Avital Oliver, Nicolas Papernot, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, 2019.
[4] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NeurIPS, 2016.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[6] Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. Reusing the task-specific classifier as a discriminator: Discriminator-free adversarial domain adaptation. In CVPR, 2022.
[7] Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-lernndl loss for domain adaptation. In AAAI, 2020.
[8] Shuhao Cui, Shuhui Wang. Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In CVPR, 2020.
[9] Jacob Devlin, Ming-Was Meig-Wei&o Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[10] Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu, Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In CVPR, 2021.
[11] Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In ICLR, 2018.
[12] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
[13] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarti training of neural networks. The Journal of Machine Learning Research, 2016.
[14] Ross Girshick. Fast r-cnn. In
[15] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. CVPR, 2012.
[16] Chen Gong, Tongliang Liu, Dacheng Tao, Keren Fu, Enmei Tu, and Jie Yang. Deformed graph laplacian for semisupervised learning. IEEE transactions on neural networks and learning systems, 2015.
[17] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurlPS, 2004.
[18] Kai Han, Sylvestre-Alvis Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning visual categories with ranking statistics. In ICLR, 2020.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[20] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matheey Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230. 2018.

---

[21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[22] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In CVPR, 2019.
[23] Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, 2019.
[24] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019.
[25] David Krueger, Ethan Caballero, Joemn-eerik Jacobsen, Amy Zhang. Jonathan Binas, Dinghuai Zhang. Remi Le Priol, and Aaron Courville. Out-of-distribttion generalization via risk extrapo- lation (rex). In ICML, 2021.
[26] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
[27] Shuang Li, Chi Liu, Qiuxia Lin, Binhui Xie, Zhengming Ding, Gao Huang, and Jian Tang. Domain conditioned adaptation network. In AAAI, 2020.
[28] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML., 2020.
[29] Jian Liang, Dapeng Hu, and Jiashi Feng. Domain adaptation with auxiliary target domain- oriented classifier. In CVPR, 2021.
[30] Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Dine: Domain adaptation from single and multiple black-box predictors. In CVPR, 2022.
[31] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: general approach adapting deep classifiers. In ICML, 2019.
[32] Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In NeurlPS, 2021.
[33] Xiaofeng Liu, Zhenhua Guo, Site Li, Fangxu Xing, Jane You, C-C Jay Kuo, Georges Fakhri, and Jonghye Woo. Adversarial unsupervised domain adaptation with conditional and label shift: Infer, align and iterate. In ICCV, 2021.
[34] Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In NewrIPS, 2018.
[35] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip Yu. Transfer feature learning with joint distribution adaptation. In ICCV, 2013.
[36] Mingsheng Long. Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In ICML, 2017.
[37] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018.
[38] Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self-training for unsupervised domain adaptation. In ECCV, 2020.
[39] Kevin P. Murphy. Machine learning a probabilistic perspective. 2013.
[40] Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In CVPR, 2021.
[41] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2010.
12

---

[42] Mohammad Peikari, Sherine Salama, Sharon Nofech-Mozes and Anne L Martel. A cluster- then-label supervvie learning approach for pathology image classification. Scientific reports, 2018.
[43] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.0660 06924, 2017.
[44] Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In ICCV, 2021.
[45] Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and Venkatesh Babu Radhakrishnan. A closer look at smoothness in domain adversarial training. In ICML, 2022.
[46] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
[47] Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, al. Extending the wilds benchmark for unsupervised adaptation. In ICLR, 2022.
[48] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910001108, 2019.
[49] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 2021.
[50] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[51] Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable representations for unsupervised domain adaptation. In NeurlPS, 2016.
[52] Inkyu Shin, Sanghyun Woo, Fei Pan, and In So Kweon. Two-phase pseudo label densification for self-training based domain adaptation. In ECCV, 2020.
[53] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In NeurlPS, 2020.
[54] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
[55] Tao Sun, Cheng Lu, and Haibin Ling. Prior knowledge guided unsupervised domain adaptation. In ECCV, 2022.
[56] Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In CVPR, 2020.
[57] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell, Adversarial discriminative domain adaptation. In CVPR, 2017.
[58] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint 4xXiv:141..3474, 2014.
[59] Jesper Van Engelen and Holger H Hoos. A survey on sumi-supervised learning. Machine learning, 2020.
[60] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[61] Tan Wad, Qianru Sun, Sugiri Pranata, Karlekar Jayashree, and Hanwang Zhang. Equivariance and invariance inductive bias for learning from insufficient data. In ECCV, 2022.
13

---

[62] Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation feature. In NeurIPS, 2021.
[63] Guoqiang Wei, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In CVPR, 2021.
[64] Guoqiang Wei, Cuiling Lan, Wenjun Zeng, Zhizheng Zhang, and Zhibo Chen. Toalign: Task- oriented alignment for unsupervised domain adaptation. In NeurlPS, 2021.
[65] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
[66] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2018
[67] Zhongqi Yue, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Transporting causal mecha- nisms for unsupervised domain adaptation. In ICCV, 2021.
[68] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[69] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In CVPR, 2019.
[70] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML. 2019.
[71] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In /CML, 2019.
[72] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. ICCV, 2019.
14


Reviews:
[{"review_id": "S93iPHCGkq", "reviewer": "Reviewer_cknk", "summary": "The authors propose an unsupervised domain adaptation method, ICON. The algorithm is similar to self-training on the unlabeled target data, but at the start of each epoch, the unlabeled data are first projected from feature space to a reduced-dimension space and clustered. An auxiliary loss enforces consistent pseudo-labels within clusters. The method attained state-of-the-art on the WILDS unlabeled benchmark [1].\n\n[1] Sagawa, S., Koh, P. W., Lee, T., Gao, I., Xie, S. M., Shen, K., ... & Liang, P. (2021). Extending the WILDS benchmark for unsupervised adaptation. *arXiv preprint arXiv:2112.05090*.", "questions": "- The authors present ICON as orthogonal to existing UDA approaches, e.g. consistency regularization using views from data augmentations. Could the authors share how ICON performs when combined with strong data augmentations, e.g. RandAugment?\n- On page 4, the authors state that any clustering algorithm can be used, so long as the number of clusters should be equal to the number of classes. This is a useful ablation; in general, understanding which representation space and how to cluster seems important. Did the authors have k-means experiments to support this?\n- I could not find the UMAP hyperparameters in the appendix as the main paper suggested; in particular, what was the output dimension used?\n- The results of iWildCam2020-WILDS and CivilComments-WILDS are a bit unaligned with the theory, as in these cases, the unlabeled data is *not* drawn from the test distribution. It would be great if the main paper could comment briefly on this discrepancy.", "limitations": "The authors note that the assumptions made in Section 4 are restrictive.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- ICON’s strong empirical performance is striking, especially on WILDS; the authors outperform similar self-training-style methods, e.g.. vanilla self-training, FixMatch, and Noisy Student. Notably, ICON attains this performance without using aggressive data augmentations.\n- The paper evaluates on ten datasets across 3 modalities, and evaluations are conducted over multiple seeds.\n- The presentation is good, and the experiments are notably well-documented.", "weaknesses": "- Given its similarity to other self-training methods, it seems important to analyze, whether through experimental ablations or theory, why ICON outperforms vanilla self-training so strikingly. One unique aspect of ICON is that the unlabeled consistency loss uses clusters computed in a reduced dimension space; this seems important in ablations (Table 2). The authors motivate this by stating that dimensionality reduction acts to suppress environmental features and highlight the causal feature (L138), but this statement seems unsupported to me. Could the authors clarify theoretically why this is?\n- The theory in Section 4 makes two strong assumptions, to the effect of (1) the classes in T are cleanly separated by clusters, and (2) if the model separates classes correctly in S and clusters in T, then the classes in T are predicted correctly, as prediction uses the invariant feature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the in-depth comments and suggestions. We will address all concerns below.\n\n**W1 - Why ICON outperforms self-training.** We clarify that the key to ICON's success is invariant consistency instead of dimensionality reduction with UMAP.\n- For empirical evidence, we perform additional experiments and reformat the results in Table 2 to form Table r1 below. Note that we used FixMatch as a self-training baseline on Office-Home and VisDA-2017. It shows that consistency loss brings the most improvement, followed by invariance constraint, and lastly UMAP.\n- For theoretical analysis, we analyze the failure of self-training in Figure 2a (lines 52-57) and discuss how clusters in the target domain provide additional supervision to prevent the failure (lines 58-65). Then we propose consistency loss (lines 66-71) and invariance constraint (lines 72-80) to incorporate this supervision. We clarify that UMAP is simply a pre-processing technique to improve clustering, which is shown to benefit classification tasks (*i.e.*, highlights causal feature) supported by rigorous theoretical results [1].\n\nTable r1: Ablations on each ICON component. Supplement to Table 2.\n| Method                                            | Office-Home | VisDA-2017 |\n| ------------------------------------------------- | ----------- | ---------- |\n| FixMatch                                          | 69.1        | 76.6       |\n| FixMatch + Consistency                            | 74.1        | 82.0       |\n| FixMatch + Consistency + Invariance               | 75.2        | 86.5       |\n| FixMatch + Consistency + Invariance + UMAP (ICON) | 75.4        | 87.4       |\n\n\n**W2 - Strong assumptions.** The assumptions are actually fundamental for UDA.\n- Assumption 1 corresponds to the classic clustering assumption (or low-density assumption), a necessary assumption for learning with unlabeled data [2]. In fact, UDA methods (and semi-supervised learning in general) commonly leverage feature pre-training and data augmentations to help fulfill the assumptions.\n- Assumption 2 is necessary to rule out corner cases, *e.g.*, in Figure 4b, without additional prior, there is no way for *any* UDA method to tell if each one of the blue cluster (unlabeled) should be pseudo-labeled as \"0\" or \"1\". In practice, these corner cases can be avoided by collecting diverse unlabeled data (lines 213-215), which is generally abundant, *e.g.*, collecting unlabeled street images from a camera-equipped vehicle cruise.\n\n**Q1 - ICON + strong data augmentations.** Actually, self-training baselines already leverage strong data augmentations. For example, FixMatch in Table r1 enforces the predictions on strongly augmented samples (using RandAugment) to be similar to those on weakly augmented samples.\nWe also highlight that ICON's success does not rely on strong augmentations, *e.g.*, ICON still outperforms on text modality (CivilComments and Amazon), where strong augmentations are not available.\n\n**Q2 - k-means experiment.** We tried k-means and got 85.6% on VisDA-2017. We postulate that rank-statistics is better because its online clustering provides up-to-date cluster assignments. In the future, we will try other more recent online clustering methods, *e.g.*, an optimal transport-based method [3].\n\n**Q3 - UMAP hyperparameters.** Sorry for omitting it. We used 50 as the output dimension and used the default values in the official UMAP repo for all other hyperparameters.\n\n**Q4 - Datasets unaligned with theory.** Sorry for the confusion. The two datasets are indeed aligned with the theory. Note that ICON works by learning the causal feature $\\mathbf{c}$ and discarding the environmental feature $\\mathbf{e}$ (lines 66-80). Hence the theory holds as long as the definition of $\\mathbf{c}$ and $\\mathbf{e}$ is consistent across the training and test data, *e.g.*, when the task is digit classification as in Figure 2, $\\mathbf{c},\\mathbf{e}$ corresponds to digit shape and digit color, respectively. In iWildCam-WILDS and CivilComments-WILDS, while the unlabeled data is not drawn from the test distribution, the task is the same across labeled, unlabeled, and test data (*e.g.*, animal classification), *i.e.*, the definition of $\\mathbf{c},\\mathbf{e}$ remains consistent. We will revise the main paper to clarify this.\n\n[1] Leland McInnes, et al. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.\n\n[2] Van Engelen JE, Hoos HH. A survey on semi-supervised learning. Machine learning. 2020 Feb.\n\n[3] Mathilde Caron, et al. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS 2020.\n"}, {"review_id": "eZHd0RXXer", "reviewer": "Reviewer_Q3G5", "summary": "This paper proposes a new UDA method which strives to produce a consistent classifier for labels in source domain and clusters in target domain. Specifically, this paper introduces an auxiliary task for distinguishing whether the input image pair share the same class/cluster or not. This binary classification task would encourage the features from the same class to be similar and features from different classes to be dissimilar. The groundtruth for this binary classification task on target pair is determined by performing clustering on target features. By combining this binary classification task with supervised learning on source domain and self training on target domain, this method obtains state-of-the-art performance on multiple challenging UDA benchmark. ", "questions": "1. In line41 there are double commas.\n2. The BCE loss is only applied on source-source pair or target-target pair. What if applying it on source-target pair? Does it bring better performance?\n3. How the pseudo labels for $L_{st}$ is obtained? A short introduction should be included in the implementation details.\n4. Now that the main idea is to utilize the cluster label to group target domain features, is it necessary to add a BCE task for UDA classification? Is it possible to utilize the cluster label as the pseudo label for self-training?", "limitations": "This paper adequately discussed the limitation of the proposed method and provide an analysis about the possible reasons. No potential negative societal impact is discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper utilizes clustering on target features to determine whether the target image pair share the same class. Compared to previous self-training methods, this design better reduces the noise in the pseudo labels and alleviate overfitting on incorrect pseudo labels.\n2. The proposed method obtains state-of-the-art performance on many challenging UDA tasks. A sufficient ablation study and analysis clearly depicts the advantage of the proposed method, which would provide great insight for furtehr works.", "weaknesses": "Basically I believe the method proposed method greatly meets its motivation and sufficient analysis has proven its effectiveness. The introduction part could be further polished to emphasize the core idea better. For figure 1, the lack of comparison on t-SNE of baselines might make the reader difficult to understand the advantage of ICON.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback. We will fix the typos in Q1 and address all concerns below.\n\n**W - Lack of baselines t-SNE in Figure 1.** Sorry for the confusion. Actually, the goal of Figure 1 is not to compare our ICON with baselines, but to depict the condition where a model generalizes. In Figure 2, we further discuss how ICON achieves this condition and hence improves existing baselines.\n\n**Q2 - BCE between source-target pair?** Thanks for the suggestion. We tried and observed comparable performance (75.3% on Office-Home, 87.0% on VisDA-2017). Specifically, for a source-target pair, we compared the ground-truth label (source sample) and pseudo-label (target sample) to get the binary label for BCE loss (*i.e.*, $b$ in Eq. 1). This is because we cannot directly compare the ground-truth label and cluster label (see Q4). We postulate that the self-training loss ($\\mathcal{L}_{st}$ in Eq. 2) achieves similar effect with the source-target BCE loss, *i.e.*, bringing each target sample closer to its pseudo-label class center.\n\n**Q3 - How to obtain $\\mathcal{L}_{st}$.** Sorry for the confusion. We discussed this in Appendix Section B.3 and E (under self-training details). We mainly used FixMatch and Noisy Student loss as $\\mathcal{L}_{st}$, where pseudo-labels are generated on weakly-augmented target domain samples, and the model is trained to predict the pseudo-labels given strongly-augmented samples. We will give a short introduction in the main text in revision.\n\n**Q4 - Cluster label as pseudo label?.**\nUnfortunately, this is not possible, because the cluster labels (*e.g.*, cluster 1 or 2) are not aligned with the class labels (*e.g.*, \"bed'' or \"clock'') as discussed in lines 155-156, and explicit alignment (*e.g.*, assigning cluster 1 to \"bed'') reduces our method to standard self-training. Hence it is necessary to use the BCE loss to train the classifier, such that its decision boundary simultaneously separates the classes in the source domain and the clusters in the target domain (Figure 2d)."}, {"review_id": "GA7oop9li4", "reviewer": "Reviewer_RV7L", "summary": "This paper deals with unsupervised domain adaptation problem. This paper focuses on how to exploit the inherent distribution of target domain to improve the adaptation performance. In detail, it trains two classifier: one is on source domain and the other one is on target domain. Each classifier is trained with BCE loss on a pair of source samples or target samples. For source domain, the same ground-truth label of a pair of samples indicates a positive pair; for target domain, the same cluster label (obtained through clustering) of a pair of samples indicates a positive pair. Finally, the intersection of two classifier is exploited to form the optimal classifier. Additional cross-entropy loss and self-training loss are added to the objective to supervise the adaptation process. Experiment results on various benchmarks validate the effectiveness of proposed method.", "questions": "See the weakness part.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea is interesting and reasonable to an extent.\n- The paper is generally easy to understand and follow.\n- The method has been tested on various UDA benchmarks.", "weaknesses": "- In one of the claimed contributions, the authors state that their method ICON is orthogonal to previous UDA methods. But I don't see any evidence showing that previous UDA methods combined with ICON should achieve a better adaptation performance.\n\n- In my opinion, it is not suitable to claim outperforming all the conventional methods on WILDS 2.0 UDA benchmark without fully evaluating existing previous works.\n\n- It is interesting to use REx to realize the goal of the second line of Eq. (2). I just wonder what if we simply share the $f$\n between two domains. Will it achieve similar performance? I would like to see such ablations.\n\n- For the experiments, the authors should compare with more recent SOTA UDA methods. The current status of comparisons listed in Table 1 is unsatisfactory and may not justify the performance superiority of proposed method.\n\n- As shown in Fig. 5, the model seems to be sensitive to the hyper-parameter selection. As we know, it is hard to perform hyper-parameter selection in UDA as we don't have labels of target domain data. I feel concerned about whether this work can be really useful in practice.\n\n- The details about how to utilize the self-training loss are not clear. The ablations for self-training terms are missing. Without providing such ablations, we cannot know if the improvement really comes from what the authors claimed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the in-depth review. We will address all weaknesses.\n\n**W1 - Orthogonality of ICON.**\nSorry for the misleading term. We intend to mean that our ICON loss can be plugged into different self-training baselines ($\\mathcal{L}_{st}$ in Eq. 2).\nWe discussed the choice of the self-training baselines in Appendix Section E under self-training details. For example, on Office-Home and VisDA-2017, we use FixMatch, and greatly improve the accuracy from 69.1% and 76.6% (FixMatch) to 75.4% and 87.4% (FixMatch + ICON) on the two datasets, respectively. We will clarify this in the revision.\n\n**W2 - Claim on WILDS 2.0.**\nThanks for the suggestion. We will revise accordingly.\n\n**W3 - Ablation on sharing $f$.**\nWe have tried sharing $f$ without REx, and it’s worse in Table 3 (ICON-INV).\n\n**W4 - Table 1 may not justify ICON.** We gracefully disagree.\n- For Office-Home and VisDA-2017, we've already included the best-performing UDA methods on ResNet-50 in Table 1. We supplement it with a more comprehensive list and results on ResNet-101 in Appendix Table 3 and 4. Notably, ICON even outperforms the previous SOTA on the much deeper ResNet-101.\n- For WILDS 2.0 benchmark, we included all previous results in its official leaderboard. We also tried implementing recent SOTAs on WILDS2.0. However, without official implementation, it is indeed a nontrivial task to fairly reproduce them as what they claimed to be: they are outperformed by ERM, or even not applicable (*e.g.*, on text/graph data). Hence we feel that it may be disrespectful to include results by us, *e.g.*, the result of recent CST on applicable datasets are shown below. This is in line with WILDS 2.0 organizers' observation that ERM as a naive baseline usually has the best performance, which is exactly the point of WILDS 2.0 as a diagnosis of the ever-overlooked issues in UDA. Hence we believe that dominating this challenging leaderboard strongly justifies the superiority of our ICON.\n\n| Method | iWildCAM | Camelyon | FMoW |\n| ------ | -------- | -------- | ---- |\n| ERM    | 32.2     | 82.0     | 34.8 |\n| CST    | 31.3     | 74.5     | 32.6 |\n\n**W5 - Hyper-parameter ablations.**\nOur work is as practical as other UDA methods: We apologize that our presentation on invariant weight $\\beta$ may mislead you. It is standard in the IRM community to choose a small $\\beta$ (*e.g.*, 0.1). Then, tuning $\\alpha$ follows the same process as other self-training UDA methods.\n\n**W6 - Missing details and ablation for self-training loss.**\nSorry for the confusion. We include self-training loss details in Appendix Section B.3 and E. The ablation for self-training loss weight $\\alpha$ is in Figure 5. Please also refer to Reviewer cknk-W1 for a discussion on ICON's improvement over self-training."}, {"review_id": "U61GH4FK9t", "reviewer": "Reviewer_gs8L", "summary": "This paper proposes ICON (Invariant CONsistency learning), a method to utilize the distribution of unlabeled target data in the UDA task.\nAnd it obtains stable performance improvements over 8 UDA tasks.", "questions": "1. Could you provide more analysis about \"why low dimensional pseudo labels are more accurate?\"\n2. If the low-dimensional pseudo labeling works better, why do you still use high dimensions in the final validation?", "limitations": "The author has provided limitations in their appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of the article is simple, but it makes sense that the distribution of unlabeled target data does have an impact on UDA tasks.\n2. In terms of learning the distribution of unlabeled data, this paper has some novelty in using pseudo labels from low-dimension clustering.\n3. This article provides sufficient experiments, and the effectiveness of ICON has been verified under various UDA settings", "weaknesses": "Essentially, I think that the core of the ICON method is a pseudo label-based contrastive learning technology, where the pseudo label is from low-dimension clustering. And the authors provide some experiments about the rationality of the ICON, like Fig. 5,6,8. But none of this fundamentally or theoretically explains why low dimensional pseudo labels are more accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback. We address all questions below.\n\n**Q1 - Why low-dimensional pseudo-labels are more accurate?**\nWe first clarify that our cluster labels are not conventional pseudo-labels, because they are not aligned with the classes in the source domain (details in Reviewer Q3G5-Q4).\nWe don't intend to claim that our low-dimensional cluster labels are more accurate (than conventional pseudo-labels). Instead, cluster labels in the target domain provide *additional* supervision *on top of* pseudo-labels to address their limitation, *e.g.*, we empirically show that cluster labels correct pseudo-labeling error in Figure 6b. We also have illustrative and theoretical explanations:\n- In lines 52-57, we illustrate the limitation of conventional pseudo-labels due to spurious correlation in the source domain.\n- In lines 58-65, we explain how cluster labels provide *additional* supervision, and we show how they remove the spurious correlation in lines 66-80.\n- In Section 4, we present a theoretical analysis.\n- We also highlight that \"low-dimensional'' is cherry-on-top instead of bread-and-butter for ICON. Please refer to Reviewer cknk-W1 for results and explanation.\n\n\n**Q2 - Why use high dimensions in final validation?**\nThis is for practicality and fair comparison: The best-performing dimensionality reduction methods (*e.g.*, t-SNE or UMAP) need to process all data at once. If we train a classifier with low-dimensional features, the final validation must be performed in an offline manner, *i.e.*, having access to all test data to compute low-dimensional features. This is extremely restrictive and impractical, as such a model cannot be deployed to predict online/streaming data. It is also unfair to compare the performance of it against previous methods, which do not have the restriction. Hence we only use dimensionality reduction for clustering the unlabeled training data."}, {"review_id": "JKTzMw0pVF", "reviewer": "Reviewer_CYwX", "summary": "This paper presents a novel unsupervised domain adaptation method called invariant CONsistency learning (ICON). ICON is very simple; assuming that labeled source samples and clustered target samples are available, ICON uses BCE losses to make the inner product of the features (softmax-normalized) of any two samples in a mini-batch in each domain close to 1 if they are of the same class/cluster, and 0 otherwise. Evaluation experiments on 10 different benchmark datasets (Office-Home, VisDA-2017, WILDS 2.0) show that ICON achieves superior performance to several representative methods on most of them.", "questions": "I have no specific questions that I would like the authors to answer, but it would be great if I could read some discussions on the weaknesses I mentioned above, especially on 1. the comparisons with SoTA.", "limitations": "I found a discussion of that limitation in the supplemental material, and it is very reasonable.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method, ICON, is surprisingly simple and easy to use.\n\nA theoretical property of ICON is discussed; ICON gives the optimal classifier under certain assumptions. (this is a fairly high-level characteristic, and the realism of the assumptions are somewhat questionable, though.)\n\nVarious experiments are conducted. Despite its simplicity, ICON achieves sufficiently high accuracy on many datasets.\n\nThe paper is generally well-written and easy to follow.\n", "weaknesses": "1. The performance of ICON on Office-Home and VisDA-2017 is inferior to that of SoTA. For example, CDTrans [Xu+, ICLR2022] achieves 88.4% on VisDA-2017 and 80.5% on OfficeHome, both higher than ICON.\n\n\n2. Since the assumptions underlying the theorem appear to be quite strong, it is questionable to what extent they are valid in practice. (this is discussed more or less in the limitation part in the supplementary material, though.)\n\n\n3. Intuitively, the principle of ICON (i.e., bringing features within the same class/cluster closer together) seems highly similar to that of contrastive learning, which is also a major approach to unsupervised domain adaptation (e.g. [Shen+, ICML2022 ], [Wang+, TMM2023]). Discussing the differences will highlight the property and uniqueness of ICON.\n\n\n4. While this may be outside the scope of this paper, it would be interesting to discuss the possibility of extending to more advanced domain adaptation problems, such as universal domain adaptation and source-free domain adaptation. Since ICON (perhaps implicitly) assumes that the number of classes in the source and target are the same, and the impact of the number of clusters on accuracy is significant (see Table 2). So the performance of ICON on universal domain adaptation, where the number of classes cannot be assumed to be the same, may not be as promising. Application to source-free domain adaptation is also non-trivial.\n\n\n[Xu+, ICLR2022] CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation\n\n[Shen+, ICML2022] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation\n\n[Wang+, TMM2023] Cross-Domain Contrastive Learning for Unsupervised Domain Adaptation\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the in-depth review. We address all weaknesses below.\n\n**W1 - SoTA > ICON?** ICON is SoTA on the ResNet-50 backbone. UDA performance is sensitive to backbone choice. We choose the most classic and widely used ResNet-50 to demonstrate the superiority of ICON. In contrast, CDTrans uses the much larger DeiT-base (more than tripling the number of parameters of ResNet-50). Hence the results are not comparable. We also highlight that ICON is agnostic to the backbone choice, and we will try transformer-based backbones in future work.\n \n**W2 - Strong assumption.** The assumptions are actually fundamental for UDA.\n- Assumption 1 corresponds to the classic clustering assumption (or low-density assumption), a necessary assumption for learning with unlabeled data [3]. In fact, UDA methods (and semi-supervised learning in general) commonly leverage feature pre-training and data augmentations to help fulfill the assumptions.\n- Assumption 2 is necessary to rule out corner cases, *e.g.*, in Figure 4b, without additional prior, there is no way for any UDA method to tell if each one of the blue cluster (unlabeled) should be pseudo-labeled as \"0\" or \"1\". In practice, these corner cases can be avoided by collecting diverse unlabeled data (lines 213-215), which is generally abundant, *e.g.*, collecting unlabeled street images from a camera-equipped vehicle cruise.\n \n**W3 - Differences with methods based on contrastive learning.** Yes, our method can be viewed as contrastive learning. The differences with previous methods lie in ***what to contrast***. For example, [Shen+, ICML2022] contrasts augmented samples, *i.e.*, a sample under different augmented views shares similar features, and different samples have dissimilar features. [Wang+, TMM2023] contrasts ***cross-domain*** sample pairs (one from source domain $S$ and the other from target domain $T$), *i.e.*, pairs from the same class share similar features (and vice versa). Unfortunately, they still generate $T$ pseudo-labels based on $S$ supervision like self-training methods, and hence are prone to spurious correlations (lines 52-57). Our ICON contrasts ***in-domain*** sample pairs (both samples from $S$ or $T$), *i.e.*, pairs from the same class in $S$ or cluster in $T$ share similar features (and vice versa). In this way, our cluster labels in $T$ only capture the inherent distribution of $T$, which helps remove spurious correlations (lines 66-80).\n \n**W4 - Extension to more advanced DA problems**. Interesting question.\n1. In Universal DA, there exist classes that only appear in $S$ and $T$. Still, all classes in $S$ and $T$ are typically under a common task, *e.g.*, [1] uses animal classification in its motivating example. Hence this setting meets the condition of ICON (lines 27-32): any sample is generated from a causal feature determining the class identity (*e.g.*, animal appearance) and an environmental feature (*e.g.*, background). From a technical perspective, the extension is not difficult: instead of setting the cluster number in $T$ as the class number in $S$, one can estimate the cluster number (*e.g.*, with semi-supervised k-means in [2]), or use density-based clustering (*e.g.*, DBSCAN) that does not require cluster number.\n2. Source-Free DA (SFDA) loses access to the data in $S$, *i.e.*, the goal is to adapt a model trained in $S$ with unlabeled data in $T$. Part of ICON can be extended to this setting: In Eq. 2, one can use any loss function in SFDA as $\\mathcal{L}_{st}$ and add our consistency loss $\\mathrm{BCE}(T,f)$ to it. However, the invariance constraint is no longer applicable without data in $S$.\n\nWe will try these settings for future work.\n\n[1] Kaichao You, et al. Universal domain adaptation. CVPR 2019.\n\n[2] Kai Han, et al. Learning to discover novel visual categories via deep transfer clustering. ICCV 2019.\n\n[3] Van Engelen JE, Hoos HH. A survey on semi-supervised learning. Machine learning. 2020 Feb."}]

Author rebuttal:

Meta-review: The final views on the paper ranged from borderline accept to accept. 

The main concerns of one of the reviewers  (borderline accept) were that the claims for causality needs to be carefully made. The point about the same class features being closer and the different class features being further does not imply causality and as raised by the reviewer, this needs to be clarified. The point by the other reviewer with borderline views are also valid, in that the claims of orthogonality are not generally proven. The specific empirical method can be provided as indications of complementarity (rather than orthogonality). 

On balance, it is felt that the paper does provide thorough empirical analysis and the reviewers are convinced by the rebuttal. It is therefore recommended that the paper be accepted. It is strongly recommended to revise the paper by taking all the points raised by reviewers into account while preparing the final version of the paper.
Decision: Accept (poster)