Title: Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: unsupervised domain adaptation

transfer learning
Abstract: Domain Adaptation (DA) is always challenged by the spurious correlation between the domain-invariant features (e.g., class identity) and the domain-specific ones (e.g., environment) that does not generalize to the target domain. Unfortunately, even enriched with additional unsupervised target domains, existing Unsupervised DA (UDA) methods still suffer from it. This is because the source domain supervision only considers the target domain samples as auxiliary data (e.g., by pseudo-labeling), yet the inherent distribution in the target domain---where the valuable de-correlation clues hide---is disregarded. We propose to make the U in UDA matter by giving equal status to the two domains. Specifically, we learn an invariant classifier whose prediction is simultaneously consistent with the labels in the source domain and clusters in the target domain, hence the spurious correlation inconsistent in the target domain is removed. We dub our approach "Invariant CONsistency learning" (ICON). Extensive experiments show that ICON achieves the state-of-the-art performance on the classic UDA benchmarks: Office-Home and VisDA-2017, and outperforms all the conventional methods on the challenging WILDS 2.0 benchmark. Codes are in https://github.com/yue-zhongqi/ICON.

Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation

Abstract

Domain Adaptation (DA) is always challenged by the spurious correlation between domain-invariant features ( e.g. , class identity) and domain-specific features ( e.g. , environment) that does not generalize to the target domain. Unfortunately, even enriched with additional unsupervised target domains, existing Unsupervised DA (UDA) methods still suffer from it. This is because the source domain supervision only considers the target domain samples as auxiliary data ( e.g. , by pseudo-labeling), yet the inherent distribution in the target domain-where the valuable de-correlation clues hide-is disregarded. We propose to make the U in UDA matter by giving equal status to the two domains. Specifically, we learn an invariant classifier whose prediction is simultaneously consistent with the labels in the source domain and clusters in the target domain, hence the spurious correlation inconsistent in the target domain is removed. We dub our approach 'Invariant CONsistency learning' (ICON). Extensive experiments show that ICON achieves the state-of-the-art performance on the classic UDA benchmarks: OFFICE-HOME and VISDA-2017, and outperforms all the conventional methods on the challenging WILDS 2.0 benchmark. Codes are in Appendix.

1 Introduction

̸

Source domain

𝑆𝑆

Target domain

𝑇𝑇

Samples

t-SNE features

Train Accuracy: 90.6%

Test Accuracy: 90.2%

0

Domain Adaptation (DA) is all about training a model in a labelled source domain S ( e.g. , an autopilot trained in daytime), and the model is expected to generalize in a target domain T ( e.g. , the autopilot deployed in nighttime), where T and S has a significant domain shift ( e.g. , day vs. night) [41]. To illustrate the shift and generalization, we introduce the classic notion of causal representation learning [20, 49]: any sample is generated by x = Φ( c , e ) , where Φ is the generator, c is the causal feature determining the domain-invariant class identity ( e.g. , road lane), and e is the environmental feature ( e.g. , lighting conditions). As the environment is domain-specific, the domain shift e S = e T results in x S = x T , challenging the pursuit of a domain-invariant

̸

class model. However, if a model disentangles the causal feature by transforming each x = c , it can generalize under arbitrary shift in e . In Figure 1, even though e T = 'colorful' is significantly

● ●

● ●

●

● ●

●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

● ●

● ● ●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

✘

✘

● ●

● ●

●

● ●

●

●

● ●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

● ●

●

●

●

●●

●

●

● ●

●

(a)

𝐜𝐜

(digit shape)

𝐞𝐞

(digit color)

(b)

(c)

▲ ▲

▲ ▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲ ▲

▲ ▲▲

▲

▲

▲

▲

▲

▲

●

▲ ▲

▲ ▲

▲

▲ ▲

▲

▲ ▲

▲

▲

▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲

▲

▲

▲ ▲

●

▲ ▲

▲ ▲

▲

▲ ▲

▲

▲ ▲

▲

▲

▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲

▲

▲

▲ ▲

▲ ▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲ ▲

▲ ▲▲

▲

▲

▲

▲

▲

▲

▲

●

●

● ●

● ● ●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

▲ ▲

▲

▲

▲

▲

▲

▲

▲ ▲

▲▲

▲

▲

▲

▲▲

▲

▲

▲ ▲

▲

▲ ▲

▲ ▲

▲

▲ ▲

▲

▲ ▲

▲

▲

▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲

▲

▲

●

●

● ●

● ● ●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

●

●

● ●

●

●

●

●●

●

●

● ●

●

(d)

▲ ▲

▲

▲

▲

▲

▲

▲

▲ ▲

▲▲

▲

▲

▲

▲▲

▲

▲

▲ ▲

▲

▲ ▲

▲ ▲

▲

▲ ▲

▲

▲ ▲

▲

▲

▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲

▲

▲

●

●

● ●

● ● ●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

ICON

⋂

=

different from e S = 'red', thanks to the non-discriminative 'red' color in S , the model still easily disentangles c from Φ( c , e S ) and thus generalizes to digits of any color.

Unfortunately, in general, disentangling c by using only S is impossible as e S can be also discriminative, i.e. , spuriously correlated with c . In Figure 2a, the color e S is correlated with the shape c , i.e. , c 1 = '0' and c 2 = '1' in S tend to have e 1 = dark and e 2 = light digit colors, respectively. As both c and e S are discriminative, a classifier trained in S will inevitably capture both (see the red line in Figure 2a). This leads to poor generalization when the correlation between e T and c is different, i.e. , the colors of c 1 = '0' and c 2 = '1' in T tend to be e 2 = light and e 1 = dark instead.

To avoid spurious correlations between e S and c , a practical setting called Unsupervised DA (UDA) introduces additional information about T through a set of unlabeled samples [15, 35, 12]. There are two main lineups: 1) Domain-alignment methods [58, 34, 10] learn the common feature space of S and T that minimizes the classification error in S . In Figure 2a, their goal is to unify '0's and '1's in S and T . However, they suffer from the well-known misalignment issue [31, 71], e.g. , collapsing '0's and '1's in T to '1's and '0's in S respectively also satisfies the training objective, but it generalizes poorly to T . 2) Self-training methods [72, 38, 52] use a classifier trained in S to pseudo-label samples in T , and jointly train the model with the labels in S and confident pseudo-labels in T . Yet, the pseudo-labels can be unreliable even with expensive threshold tweaking [32, 29], e.g. , the ✗ areas in Figure 2a have confident but wrong pseudo-labels. In fact, the recent WILDS 2.0 benchmark [47] on real-world UDA tasks shows that both lineups even under-perform an embarrassingly naive baseline: directly train in S and test in T .

Notably, although both methods respect the idiosyncrasies in S (by minimizing the classification error), they fail to account for the inherent distribution in T , e.g. , in Figure 2a, even though T is unlabelled, we can still identify the two sample clusters in T (enclosed by the blue circles), which provide additional supervision for the classifier trained in S . In particular, the classifier (red line) in Figure 2a breaks up the two clusters in T , showing that the correlation between color e S and c in S is inconsistent with the clusters affected by e T in T , which implies that color e is the environmental feature. Hence, to make the U in UDA matter, we aim to learn a classifier that is consistent with classification in S and clustering in T :

Consistency . We use the Binary Cross-Entropy (BCE) loss to encourage the prediction similarity of each sample pair from the same class of S or cluster of in T , and penalize the pairs from different ones. In Figure 2b, the red and blue lines denote the classifiers that minimize the BCE loss in S and T , respectively. By minimizing the combined loss, we learn the classifier consistent with the sample distribution in both domains (the upright black line), which predicts solely based on the causal feature c ( i.e. , disentangling c from Φ( c , e ) ).

Invariance . However, enforcing consistency is a tug-of-war between S and T , and the BCE loss in one domain can dominate the other to cause failures. Without loss of generality, we consider the case where S dominates in Figure 2c. Due to the strong correlation between e S and c , the classifier that minimizes the combined losses of BCE (black line) deviates from the desired upright position as in Figure 2b. To tackle this, we aim to learn an invariant classifier [1] that is simultaneously optimal w.r.t. the BCE loss in S and T . In Figure 2d, for each of the two losses, we color the candidate regions where the classifier is optimal ( i.e. , gradients ≈ 0 ) with red and blue, respectively. In particular, by

B

B

B

'clock'

'bed'

BCE(

)

BCE(

)

cluster 2

cluster 2

dot-product

loss

model

similarity in

UMAP

0

1

bed

clock

bed

clock

clock

bed

clock

bed

Invariance

?

̸

learning an invariant classifier lying in the intersection (purple region), we successfully recover the desired one as in Figure 2b, even when one domain dominates.

Overall, we term our approach as Invariant CONsistency learning (ICON) . Our contributions:

ICON is a novel UDA method that can be plugged into different self-training baselines. It is simple to implement (Section 2), and disentangles the causal feature c from Φ( c , e ) that generalizes to T with a theoretical guarantee (Section 3).

ICON significantly improves the current state-of-the-art on classic UDA benchmarks [29, 6] with ResNet-50 [19] as the feature backbone: 75.8% (+2.6%) accuracy averaging over the 12 tasks in OFFICE-HOME [60], and 87.4% (+3.7%) mean-class accuracy in VISDA-2017 [43] (Section 5).

Notably, as of NeurIPS 2023, ICON dominates the leaderboard of the challenging WILDS 2.0 UDA benchmark [47] ( https://wilds.stanford.edu/leaderboard/ ), which includes 8 large-scale classification, regression and detection tasks on image, text and graph data (Section 5).

2 Algorithm

UDA aims to learn a model with the labelled training data { x i , y i } N i =1 in the source domain S and unlabelled { x i } i M =1 in the target domain T , where x i denotes the feature of the i -th sample extracted by a backbone parameterized by θ ( e.g. , ResNet-50 [19]), and y i is its ground-truth label. We drop the subscript i for simplicity when the context is clear. As shown in Figure 3, our model includes the backbone θ , a classification head f , and a cluster head g , where f and g output the softmax-normalized probability for each class and cluster, respectively. Note that f and g have the same output dimension as the class numbers are the same in S and T . As discussed in Section 1, the generalization objective of ICON is to learn an invariant f that is simultaneously consistent with the classification in S , and the clustering in T , identified by g . ICON is illustrated in Figure 3 and explained in details below.

2.1 Consistency with S

We use Binary Cross-Entropy (BCE) loss to enforce the consistency of f with S , denoted as BCE ( S, f ) . Specifically, BCE ( S, f ) is a pair-wise loss given by:

BCE ( S, f ) = ❊ ( x i , x j ) ∼ S [ b log(ˆ y ) + (1 -b )log(1 -ˆ y )] , (1) where b = ✶ ( y i = y j ) is a binary label indicating if the pair ( x i , x j ) is from the same class, and ˆ y = f ( x i ) ⊺ f ( x j ) is the predictive similarity. Intuitively, BCE is a form of contrastive loss that clusters sample pairs with the same label (by increasing the dot-product similarity of their predictions) and pushes away pairs with different labels (by decreasing the similarity). Note that in practice, BCE loss is computed within each mini-batch, hence there is no N 2 prohibitive calculation.

2.2 Consistency with T

Unfortunately, BCE ( T, f ) cannot be evaluated directly, as the binary label b in Eq. (1) is intractable given unlabeled T . To tackle this, we cluster T to capture its inherent distribution and use cluster labels to compute BCE ( T, f ) .

Clustering T . We adopt the rank-statistics algorithm [18], as it can be easily implemented in a mini-batch sampling manner for online clustering. The algorithm trains the cluster head g to group features whose top values have the same indices (details in Appendix). Ablation on clustering algorithm is in Section 5.4. Note that our g is trained from scratch, rather than initialized from f trained in S [56, 29, 32]. Hence g captures the distribution of T without being affected by that of S .

Computing BCE ( T, f ) . After training g , we replace b = ✶ ( y i = y j ) for BCE ( S, f ) in Eq. (1) with b = ✶ (argmax g ( x i ) = argmax g ( x j )) for BCE ( T, f ) , which compares the cluster labels of each sample pair. We emphasize that it is necessary to use pairwise BCE loss to enforce the consistency of f with T . This is because cross-entropy loss is not applicable when the cluster labels in T are not aligned with the class indices in S .

2.3 Invariant Consistency (ICON)

ICON simultaneously enforces the consistency of f with S ( i.e. , the decision boundary of f separates the classes) and T ( i.e. , it also separates the clusters). The objective is given by:

Line 1 . The BCE losses train f to be consistent with the pair-wise label similarity in S and T . CE loss in S trains f to assign each sample its label ( e.g. , predicting a clock feature as 'clock'). L st is the self-training loss with weight α , which leverages the learned invariant classifier to generate accurate pseudo-labels in T for additional supervision (details in Section 5.2).

Line 2 . The constraint below requires the backbone θ to elicit an invariant classifier f that simultaneously minimizes the BCE loss in S and T . This constraint prevents the case where one BCE loss is minimized at the cost of increasing the other to pursue a lower sum of the two (Figure 2c).

To avoid the challenging bi-level optimization, we use the following practical implementation:

where the variance term Var( · ) is known as the REx loss [25] implementing the invariance constraint. The self-training weight α and invariance weight β are later studied in ablation (Section 5.4).

Training and Testing . Overall, at the start of training, the backbone θ is initialized from a pre-trained weight ( e.g. , on ImageNet [46]) and f, g are randomly initialized. Then ICON is trained by Eq. (3) until convergence. Only the backbone θ and the classifier f are used in testing.

3 Theory

Preliminaries . We adopt the group-theoretic definition of disentanglement [20, 62] to justify ICON. We start by introducing some basic concepts in group theory. G is a group of semantics that generate data by group action, e.g. , a 'turn darker' element g ∈ G transforms x from white '0' to g ◦ x as grey '0' (bottom-left to top-left in Figure 4b). A sketch of theoretical analysis is given below and interested readers are encouraged to read the full proof in Appendix.

Definition (Generalization). Feature space X generalizes under the direct product decomposition G = G / H×H , if X has a non-trivial G / H -representation, i.e., the action of each h ∈ H corresponds to a trivial linear map in X (i.e., identity map), and the action of each g ∈ G / H corresponds to a non-trivial one.

This definition formalizes the notion of transforming each sample Φ( c , e ) to the causal feature c . Subgroup H ⊂ G and quotient group G / H transform the environmental feature e and the causal c , respectively. If X has a non-trivial G / H -representation, samples in a class are transformed to the same feature c ∈ X regardless of e ( i.e. , representation of G / H ), and different classes have different c ( i.e. , non-trivial). This definition is indeed in accordance with the common view of good features [62], i.e. , they enable zero-shot generalization as c is class-unique, and they are robust to the

domain shift by discarding the environmental features e from Φ( c , e ) . To achieve generalization, we need the following assumptions in ICON.

Assumption (Identificability of G / H ).

1) ∀ x , x ∈ T , y = y iff x ∈ { h ◦ x | h ∈ H} ;

i j i j i j 2) There exists no linear map l : X → R such that l ( x ) > l ( h ◦ x ) , ∀ x ∈ S, h ◦ x ∈ T and l ( g ◦ x ) < l ( gh ◦ x ) , ∀ g ◦ x ∈ S, gh ◦ x ∈ T , where h = e ∈ H , g = e ∈ G / H .

Assumption 1 states that classes in T are separated by clusters. This corresponds to the classic clustering assumption, a necessary assumption for learning with unlabeled data [59]. Figure 4a shows a failure case of Assumption 1, where the intra-class distance ( i.e. , dotted double-arrow) is much larger than the inter-class one ( i.e. , solid double-arrow) in T , i.e. , shape c is less discriminative than color e T . Hence the clusters in T are based on e , causing ICON to capture the wrong invariance. To help fulfill the assumption, UDA methods (and SSL in general) commonly leverage feature pre-training and data augmentations. We also specify the number of clusters as the class numbers in T , which is a necessary condition of this assumption.

̸

̸

●

● ●

●

● ●

●

●

●

● ●

●

●

●

●

●

●

● ●

●

●

●

●

𝐜𝐜

(digit shape)

𝐞𝐞

(digit color)

(a)

▲

▲

▲ ▲

▲

▲ ▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲

▲

▲

▲

▲ ▲

▲

▲ ▲

▲ ▲

▲ ▲

▲

▲

▲ ▲ ▲

▲

▲

▲▲

▲

▲

▲

▲

●

● ●

●

●

●

●

●

●

●

●

●

●●

●

●

●

● ●

●

●

● ● ●

●

● ●

●

●

●

●

●

● ● ●

●

●

●

●

● ●

●

●

●

●

●

●

(b)

▲ ▲

▲ ▲

▲

▲ ▲

▲

▲ ▲

▲

▲

▲ ▲

▲

▲

▲ ▲

▲

▲

▲

▲

▲

▲ ▲

▲ ▲ ▲

▲ ▲

▲

▲

▲ ▲

▲

▲ ▲ ▲

▲ ▲

▲

▲

▲

▲

●

●

● ●

● ● ●

●

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

Assumption 2 states that c is the only invariance between S and T . Figure 4b illustrates a failure case, where there exist two invariant classifiers: both the vertical one based on shape c and the horizontal one based on color e separate the classes in S and clusters in T . Yet as T is unlabelled, there is no way for the model to determine which one is based on c . In practice, this assumption can be met by collecting more diverse unlabelled samples in T ( e.g. , darker '0's and lighter '1's in Figure 4b), or introducing additional priors on what features should correspond to the causal one.

Theorem . When the above assumptions hold, ICON (optimizing Eq. (3) ) learns a backbone θ mapping to a feature space X that generalizes under G / H×H . In particular, the learned f is the optimal classifier in S and T .

By transforming each sample Φ( c , e ) in S and T to c ∈ X , the classifier easily associate each c to its corresponding class label using the labelled S to reach optimal.

4 Related Work

Domain-Alignment methods [13, 4, 57] map samples in S and T to a common feature space where they become indistinguishable. Existing works either minimize the discrepancy between the sample distributions of S and T [36, 2, 27], or learn features that deceive a domain discriminator [69, 63, 33]. However, they suffers from the misalignment issue under large domain shift from S to T [71, 31], such as shift in the support of the sample distribution [23] ( e.g. , Figure 2a).

Self-Training methods [17, 3, 53] are the mainstream in semi-supervised learning (SSL), and recently became a promising alternative in UDA, which focuses on generating accurate T pseudo-labels. This lineup explores different classifier design, such as k-NN [51], a teacher-student network [11] or an exponential moving average model [30]. Yet the pseudo-labels are error-prune due to the spurious correlations in S , hence the model performance is still inconsistent [32].

Alleviating Spurious Correlation has been studied in several existing works of UDA [56, 32, 29]. They train a T -specific classifier to generate pseudo-labels [56, 29]. However, their T -classifiers are still initialized based on a classifier trained in S , hence the spurious correlation in S are introduced to the T -classifiers right in the beginning. On the contrary, our ICON learns a cluster head in T without supervision in S , which is solely for discovering the sample clusters in T . Hence, the spurious correlation in S inconsistent with the clusters in T can be removed by ICON (Section 5).

Graph-based SSL methods [16, 42, 22] share a similar spirit with ICON, where they learn a classifier using the labeled samples that is consistent with the unlabelled sample similarities. However, they lack the invariant constraint in ICON, which can cause the failure cases ( e.g. , Figure 2c).

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

● ●

● ● ●

●

●

● ● ●

●

●

●

ICON

▲

▲

▲

▲

▲ ▲

▲

▲

▲

▲ ▲

▲▲

▲

▲

▲

▲ ▲

▲

▲ ▲

●●

●

●

●

●

●

●

●

●

●

●

●

(d)

Dataset

OFFICE-HOME

VISDA-2017

IWILDCAM

CAMELYON17

FMOW

POVERTYMAP

OGB-MOLPCBA

CIVILCOMMENTS

AMAZON

Sample

𝐱𝐱

object image

object image

camera trap photo

tissue slide

satellite image

satellite image

wheat image

molecular graph

online comment

product review

Label

𝒚𝒚

65 categories

12 categories

182 species

tumor/not

62 land uses

asset wealth

wheat bbox

bioassays

toxic/not

5 review scores

Task

classification

classification

classification

classification

classification

regression

detection

classification

classification

classification

Source

𝑺𝑺

various*

synthetic

images

photos from

243 traps

slides from

hospital 1-3

images from

2002-2013

images in 14

countries

images in

Europe

44,930 scaffold

groups

online articles*

1,252

reviewers

Example

𝑺𝑺

𝑺𝑺

average 3,875

152,397

129,809

302,436

76,863

~10,000

2,943

350,343

269,038

245,502

Target

𝑻𝑻

various*

real photos

photos from

3215 traps

slides from

hospital 5

images from

2016-2018

images in 5

countries

images across

the world

43,793 scaffold

groups

online articles*

1,334

reviewers

Example

𝑻𝑻

𝑻𝑻

average 3,875

55,388

819,120

600,030

173,208

261,396

42,445

517,048

1,551,515

268,761

Evaluation

average acc.

mean-class

accuracy

macro-F1

acc.

worst-region

acc.*

Pearson

correlation* %

acc.

average

precision

worst-group

acc.*

10 th

percentile

acc.

Existing

Methods

GVB

[7]

Empirical Risk Minimization (ERM)

70.4

75.3

47.0 / 32.2

90.6 / 82.0

60.6 / 34.8

65 / 48

77.8 / 51.0

- /

28.3

89.8 / 66.6

72.0

/ 54.2

TCM

[61]

CORAL

[51]

70.7

75.8

40.5 / 27.9

90.4 / 77.9

58.9 / 34.1

54 / 36

- / -

- / 26.6

- / -

71.7 / 53.3

SENTRY

[41]

DANN

[13]

72.0

76.7

48.5 / 31.9

86.9 / 68.4

57.9 / 34.6

50 / 33

- / -

- / 20.4

- / -

71.7 / 53.3

CST

[29]

Pseudo-Label

[24]

72.2

80.6

47.3 / 30.3

91.3 / 67.7

60.9 / 33.7

- / -

73.3 / 42.9

- / 19.7

90.3 / 66.9

71.6 / 52.3

ToAlign

[59]

MDD

[63]

Noisy Student

[60]

72.7

77.8

47.5 / 32.1

93.2 / 86.7

61.3 / 37.8

61 / 42

78.1 / 46.8

- / 27.5

- / -

- / -

FixBi

[37]

MT+16augs

[11]

FixMatch

[50]

Masked LM

[9]

73.0

82.8

46.3 / 31.0

91.3 / 71.0

58.6 / 32.1

54 / 30

- / -

- / -

89.4 / 65.7

71.9 / 53.9

ATDOC

[26]

MCC+NWD

[6]

SwAV

[5]

ERM (labelled

𝑻𝑻

)

73.2

83.7

47.3 / 29.0

92.3 / 91.4

61.8 / 36.3

60 / 45

- / -

- / -

89.9 / 69.4

73.6 / 56.4

75.8

87.4

50.6 / 34.5

95.6 / 93.8

62.2 / 39.9

65 / 49

78.6 / 52.3

- / 28.3

89.7 /

68.8

71.9

/ 54.7

+2.6

+3.7

+2.3

+2.4

+2.1

+1

+1.3

+0.0

+1.9

+0.5

I applaud your

father. He was a

good man! We need

more like him.

As a Christian, I will

not be patronizing

any of those

businesses.

Super easy to

put together.

Very well built.

I am disappointed

in the quality of

these.

ICON

5 Experiment

5.1 Datasets

As shown in Table 1, we extensively evaluated ICON on 10 datasets with standard protocols [34, 47], including 2 classic ones: OFFICE-HOME [60], VISDA-2017 [43], and 8 others from the recent WILDS 2.0 benchmark [47]. WILDS 2.0 offers large-scale UDA tasks in 3 modalities (image, text, graph) under real-world settings, e.g. , wildlife conservation, medical imaging and remote sensing. Notably, unlike the classic datasets where the same sample set in T is used in training (unlabelled) and testing, WILDS 2.0 provides unlabelled train set in T disjoint with the test set, which prevents explicit pattern mining of the test data. They also provide validation sets for model selection. For datasets in WILDS 2.0 benchmark, we drop their suffix -WILDS for simplicity ( e.g. , denoting AMAZON-WILDS as AMAZON).

Details of * in Table 1: OFFICE-HOME has 4 domains: artistic images (Ar), clip art (Cl), product images (Pr), and real-world images (Rw), which forms 12 UDA tasks by permutation, and we only show Cl → Rw as an example in the table. FMOW considers the worst-case performance on different geographic regions, where the land use in Africa is significantly different. POVERTYMAP considers the lower of the Pearson correlations (%) on the urban or rural areas. CIVILCOMENTS correspond to a semi-supervised setting with no domain gap between S and T , and is evaluated on the worst-case performance among demographic identities.

On some WILDS 2.0 datasets, the test data may come from a separate domain different from the training target domain T , e.g. , IWILDCAM test data is captured by 48 camera traps disjoint from the 3215 camera traps in the training T . We highlight that ICON is still applicable on these datasets. This is because ICON works by learning the causal feature c and discarding the environmental feature e , and the theory holds as long as the definition of c and e is consistent across the training and test data, e.g. , IWILDCAM is about animal classification across the labeled, unlabeled, and test data.

5.2 Implementation Details

Feature Backbone . We used the followings pre-trained on ImageNet [46]: ResNet-50 [19] on OFFICE-HOME, VISDA-2017 and IWILDCAM, DenseNet-121 [21] on FMOW and FasterRCNN [14] on GLOBALWHEAT. We used DistilBERT [48] with pre-trained weights from the Transformers library on CIVILCOMMENTS and AMAZON. On CAMELYON17, we used DenseNet-121 [21] pre-trained by the self-supervised SwA V [5] with the training data in S and T . On POVERTYMAP and OGB-MOLPCBA with no pre-training available, we used multi-spectral ResNet-18 [19] and graph isomorphism network [66] trained with the labelled samples in the source domain S , respectively.

Self-training . A general form of the self-training loss L st in Eq. (3) is given below:

L st = ❊ x ∼ T [ ✶ (max f ( x ) > τ ) CE ( f ( x ) , argmax f ( x ))] , (4) where τ is a confident threshold that discards low-confidence pseudo-labels, and CE ( · , · ) is the cross-entropy loss. We detail the different implementations of Eq (4) in Appendix. ICON is agnostic to the choice of L st , and hence we choose the best performing self-training baseline for each dataset: FixMatch [53] on OFFICE-HOME and VISDA-2017, NoisyStudent [65] on IWILDCAM and FMOW, Pseudo-label [26] on CIVILCOMMENTS and AMAZON. ICON also does not rely on L st . We did not apply L st ( i.e. , β = 0 ) on CAMELYON17, POVERTYMAP, GLOBALWHEAT and OGB-MOLPCBA where self-training does not work well.

Regression and Detection . For the regression task in POVERTYMAP, f is a regressor that outputs a real number. As the number of clusters in T is not well-defined, we directly used rank statistics to determine if two samples in T have the same label. For the BCE loss, we computed the similarity by replacing f ( x i ) ⊺ f ( x j ) in Eq. (1) as -( f ( x i ) -f ( x j )) 2 . For detection in GLOBALWHEAT, we used the ICON objective in Eq. (3) to train the classification head without other modifications.

Other Details . We aim to provide a simple yet effective UDA method that works across data modalities, hence we did not use tricks tailored for a specific modality ( e.g. , image), such as mixup [68] or extensive augmentations (16 are used in [11]). On classic UDA datasets, we used entropy minimization and SAM optimizer following CST [32]. On VISDA-2017, IWILDCAM, CAMELYON17 and FMOW, we pre-processed the features with UMAP [37] (output dimension as 50) and EqInv [61], which helps fulfill Assumption 1 ( i.e. , highlight causal feature to improve clustering). ICON is also resource efficient-all our experiments can run on a single NVIDIA 2080Ti GPU.

5.3 Main Results

Classic UDA Benchmarks . In Table 1 (first 2 columns), ICON significantly outperforms the existing state-of-the-arts on OFFICE-HOME [60] and VISDA-2017 [43] by 2.6% and 3.7%, respectively. We include the breaks down of the performances of OFFICE-HOME in Table 2, where ICON wins in 10 out of 12 tasks, and significantly improves the hard tasks ( e.g. , Ar → Cl, Rw → Cl). For VISDA-2017, we include more comparisons in Table 3. ICON beats the original competition winner (MT+16augs) [11] that averages predictions on 16 curated augmented views. ICON even outperforms all methods that use the much deeper ResNet-101 [19]. Notably, CST [32] and ATDOC [29] also aim to remove

Method

Backbone

Acc.

MT+16augs [11] (2018)

ResNet-50

82.8

MDD[70] (2019)

ResNet-50

77.8

GVB [8] (2020)

ResNet-50

75.3

TCM [67] (2021)

ResNet-50

75.8

SENTRY [44] (2021)

ResNet-50

76.7

CST [32] (2021)

ResNet-50

80.6

CAN [24] (2019)

87.2

SHOT [28] (2020)

82.9

FixBi [24] (2021)

87.2

MCC+NWD [6] (2022)

83.7

SDAC [45] (2022)

84.3

kSHOT

∗

[55] (2022)

86.1

ICON (Ours)

ResNet-50

87.4

Acc.%

70

80

Loss

0.125

0.175

%Pairs,

𝒇𝒇

✗

,

𝒈𝒈

✓

0.25

0.35

0.45

Epochs

Epochs

ICON

CST

FixMatch

ICON

CST

FixMatch

(a)

(b)

60

0.075

the spurious correlation, and both attain competitive performance, validating the importance of this research objective. Yet their training scheme allows the correlation to persist (Section 4), hence ICON is more effective.

WILDS 2.0 . In Table 1 (last 8 columns), ICON dominates the WILDS 2.0 leaderboard despite the challenging real-world data and stringent evaluation. For example, the animal images in IWILDCAM exhibit long-tail distribution that is domain-specific, because the rarity of species varies across locations. Hence under macro-F1 evaluation (sensitive to tail performance), the 2.3% improvement is noteworthy. On CIVILCOMMENTS with text data, ICON improves the worst performance across demographic groups by 2.2%, which promotes AI fairness and demonstrates the strength of ICON even under the SSL setting. On AMAZON, although ICON only improves 0.5% to reach 54.7%, the upper bound performance by training with labelled T (56.4%) is also low due to the strict evaluation metric. We also highlight that ICON increases the performance in T without sacrificing that in S (the first number). This empirically validates that the invariance objective of ICON resolves the tug-of-war scenario in Figure 2c. In particular, we tried implementing recent methods on this benchmark. Yet they are outperformed by ERM, or even not applicable ( e.g. , on text/graph data). This is in line with WILDS 2.0 organizers' observation that ERM as a naive baseline usually has the best performance, which is exactly the point of WILDS 2.0 as a diagnosis of the ever-overlooked issues in UDA. Hence we believe that dominating this challenging leaderboard is a strong justification for ICON, and encourage more researchers to try this benchmark.

Failure Cases . ICON performance is relatively weak on POVERTYMAP and OGB-MOLPCBA with two possible reasons: 1) no pre-training available, the backbone is trained only on S and hence biased in the beginning, and 2) the ground-truth number of classes in T is not well-defined in both datasets, hence Assumption 1 in Section 3 may fail.

Learning Curve . In Figure 5a, we compare our ICON with two self-training baselines: FixMatch [53] and CST [32]. The test accuracy and training loss w.r.t. training epochs on VISDA-2017 is drawn in colored solid lines and black dotted lines, respectively. We observe that the training losses for all methods converge quickly. However, while the two baselines achieve higher accuracy in the beginning, their performances soon start to drop. In contrast, ICON's accuracy grows almost steadily.

̸

Effect of Supervision in T . To find the cause of the performance drop, we used the optimized g in Section 2.2 to probe the learned f in each epoch. Specifically, we sampled feature pairs x 1 , x 2 in T and tested if the predictions of f, g are consistent with their labels, e.g. , f succeeds if argmax f ( x 1 ) = argmax f ( x 2 ) when y 1 = y 2 (vice versa for = ). Then among the pairs where f failed, we computed the percentage of them where g succeeded, which is shown in Figure 5b. A large percentage implies that f disrespects the inherent distribution in T correctly identified by g . We observe that the percentage has a rising trend for the two baselines, i.e. , they become more biased to the spurious correlations in S that do not hold in T . This also explains their dropping performance in Figure 5a. In particular, CST has a lower percentage compared to FixMatch, which suggests that it has some effects in removing the bias. In contrast, the percentage steadily drops for ICON, as we train f to respect the inherent distribution identified by g .

5.4 Ablations

73.8

75.0

75.5

73.9

73.9

75.4

75.8

74.4

70.2

72.8

75.2

74.5

0.25

0.5

1.0

2.0

0.05

0.1

0.25

𝜶𝜶

𝜷𝜷

OFFICE-HOME

85.2

86.2

86.3

81.5

85.6

86.4

86.7

82.6

76.8

81.2

87.4

86.3

0.25

0.5

1.0

2.0

0.05

0.1

0.25

𝜶𝜶

𝜷𝜷

VISDA-2017

1.0

73.9

75.4

75.8

74.4

74.6

77.6

70.0

72.2

1.0

Components . Recall that the main components of ICON include: 1) Consistency loss (CON) that leverages the supervision from the labels in S and clusters in T ; 2) the use of invariance constraint (INV) in Eq. (3); 2) clustering in T with #clusters = #classes. Their ablations are included in Table 4. From the first three lines, we observe that adding each component leads to consistent performance increase on top of the self-training baseline (FixMatch), which validates their effectiveness. From the next two lines, when #clusters = #classes, the model performs poorly, e.g. , using one cluster in Figure 2b ( i.e. , 0 . 5 × #classes), the invariant classifier needs to predict the two red circles in S as dissimilar (two classes) and the two blue ones in T as similar (one cluster), which in fact corresponds to the erroneous red line ( i.e. , predictions in T are similarly ambiguous). In the last line, we tried k-means for clustering instead of rank-statistics, leading to decreased performance. We postulate that rank-statistics is better because its online clustering provides up-to-date cluster assignments. In the future, we will try other more recent online clustering methods, e.g. , an optimal transport-based method [5]. Overall, the empirical findings validate the importance of the three components, which are consistent with the theoretical analysis in Section 3.

̸

Hyperparameters . We used two hyperparameters in our experiments: 1) the weight of the selftraining loss denoted as α ; 2) the invariance penalty strength β in the REx [25] implementation of Eq. (3). Their ablations are in Figure 6 on the widely used classic UDA datasets. We observe that α ∈ [0 . 5 , 1 . 0] and β ∈ [0 . 1 , 0 . 25] works well on both datasets. In particular, it is standard in the IRM community to choose a small β ( e.g. , 0.1). Then, tuning α follows the same process as other self-training UDA methods. Hence the hyperparameter search is easy.

Confusion Matrix . We use the matrix to concretely show how existing methods are biased to the correlations in the source domain S . Wecompare the state-ofthe-art self-training method CST with ICON in Figure 7. The matrix on the left reveals that CST predicted most 'knife's as 'skateboard's ('board' for short), and some 'truck's as 'car's. On the right, we see that ICON predicts 'knife' accurately and also improves the accuracy on 'truck'.

aeroplane

bicycle

bus

car

horse

knife

motorcycle

person

plant

skateboard

train

truck

aeroplane

bicycle

bus

car

horse

knife

motorcycle

person

plant

skateboard

train

truck

aeroplane

bicycle

bus

car

horse

knife

motorcycle

person

plant

skateboard

train

truck

aeroplane

bicycle

bus

car

horse

knife

motorcycle

person

plant

skateboard

train

truck

CST

ICON

- 0.8

- 0.6

- 0.4

- 0.2

- 0.0

Grad-CAM Visualization . Why does CST predict 'knife' in T as 'board'? To answer this, in Figure 8, we visualized the GradCAM [50] of CST and ICON trained on VisDA-2017. In the first row, we notice that the self-training based CST predicts the two categories perfectly by leveraging their apparent differences: the curved blade versus the flat board ending with semi-circles. However, this leads to poor generalization on 'knife' in T , where the background (image 1, 4), the knife component (2) or even the blade (3) can have board-like appearances. Moreover, biased to the standard 'board' look in S , CST can leverage environmental feature for predicting 'board' in T , e.g. , under side-view (5) or slight obstruction (6), which is prune to errors ( e.g. , 8). As the board-like shape alone is non-discriminative for the two categories in T , in the second row, the cluster head g additionally leverages knife handles and wheels to distinguish them. In the third row, our ICON combines the

knife

board

𝒇𝒇

preds:

cluster 10

knife

board

board

board

person

board

knife

cluster 5

ICON

CST

Cluster

in

𝑻𝑻

𝒈𝒈

preds:

invariant

𝒇𝒇

:

𝑺𝑺

𝑻𝑻

𝑺𝑺

𝑻𝑻

𝑻𝑻

1

2

3

4

5

6

7

8

1

2

3

4

5

6

7

8

supervision in S (first row) and T (second row) to learn an invariant classifier f . In S , ICON focuses on the overall knife shape instead of just blade, and incorporates wheels to predict 'board'. In T , by comparing images with the same number in the first and third row, we see that ICON focuses more on the discriminative parts of the objects ( e.g. , handles and wheels), hence generalizes better in T .

6 Conclusion

We presented a novel UDA method called Invariant CONsistency learning (ICON) that removes the spurious correlation specific to the source domain S to generalize in the target domain T . ICON pursues an invariant classifier that is simultaneously consistent with the labels in S and the clusters in T , hence removing the spurious correlation inconsistent in T . We show that ICON achieves generalization in T with a theoretical guarantee. Our theory is verified by extensive evaluations on both the classic UDA benchmarks and the challenging WILDS 2.0 benchmark, where ICON outperforms all the conventional methods. We will seek more principled pre-training paradigms that disentangle the causal features and relax the assumption of knowing the class numbers in T to improve regression tasks.

Limitation . Our approach is based on the assumptions in Section 4, i.e. , the classes in T are separated by clusters, and there is enough diversity in T such that causal feature c is the only invariance. Assumption 1 is essentially the clustering assumption, which is a necessary assumption for learning with unlabeled data [59]. In fact, UDA methods (and semi-supervised learning in general) commonly leverage feature pre-training and data augmentations to help fulfill the assumptions. One can also use additional pre-trained knowledge from foundation models, or deploy improved clustering algorithm ( e.g. , an optimal transport-based method [5]). Assumption 2 is necessary to rule out corner cases, e.g. , in Figure 4b, without additional prior, there is no way for any UDA method to tell if each one of the blue cluster (unlabeled) should be pseudo-labeled as "0" or "1". In practice, these corner cases can be avoided by collecting diverse unlabeled data, which is generally abundant, e.g. , collecting unlabeled street images from a camera-equipped vehicle cruise.

Broader Impact . Our work aims to capture the causal feature in the unsupervised domain adaptation setting. In particular, we significantly improve the state-of-the-art on WILDS 2.0 benchmark, which includes practical tasks that benefit wildlife preservation, tumour classification in healthcare, remote sensing, vision in agriculture, etc . The pursue of causal feature can produce more robust, transparent and explainable models, broadening the applicability of ML models and promoting fairness in AI.

References

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation. In ICCV , 2019.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Avital Oliver, Nicolas Papernot, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS , 2019.

Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NeurIPS , 2016.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. Reusing the task-specific classifier as a discriminator: Discriminator-free adversarial domain adaptation. In CVPR , 2022.

Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-learned loss for domain adaptation. In AAAI , 2020.

Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In CVPR , 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu. Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In CVPR , 2021.

Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In ICLR , 2018.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML , 2015.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research , 2016.

Ross Girshick. Fast r-cnn. In ICCV , 2015.

Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR , 2012.

Chen Gong, Tongliang Liu, Dacheng Tao, Keren Fu, Enmei Tu, and Jie Yang. Deformed graph laplacian for semisupervised learning. IEEE transactions on neural networks and learning systems , 2015.

Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurIPS , 2004.

Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In ICLR , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230 , 2018.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , 2017.

Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In CVPR , 2019.

Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domaininvariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics , 2019.

Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR , 2019.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In ICML , 2021.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , 2013.

Shuang Li, Chi Liu, Qiuxia Lin, Binhui Xie, Zhengming Ding, Gao Huang, and Jian Tang. Domain conditioned adaptation network. In AAAI , 2020.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML , 2020.

Jian Liang, Dapeng Hu, and Jiashi Feng. Domain adaptation with auxiliary target domainoriented classifier. In CVPR , 2021.

Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Dine: Domain adaptation from single and multiple black-box predictors. In CVPR , 2022.

Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In ICML , 2019.

Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In NeurIPS , 2021.

Xiaofeng Liu, Zhenhua Guo, Site Li, Fangxu Xing, Jane You, C-C Jay Kuo, Georges El Fakhri, and Jonghye Woo. Adversarial unsupervised domain adaptation with conditional and label shift: Infer, align and iterate. In ICCV , 2021.

Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In NeurIPS , 2018.

Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In ICCV , 2013.

Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In ICML , 2017.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self-training for unsupervised domain adaptation. In ECCV , 2020.

Kevin P. Murphy. Machine learning : a probabilistic perspective . 2013.

Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In CVPR , 2021.

Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks , 2010.

Mohammad Peikari, Sherine Salama, Sharon Nofech-Mozes, and Anne L Martel. A clusterthen-label semi-supervised learning approach for pathology image classification. Scientific reports , 2018.

Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimization via committee consistency for unsupervised domain adaptation. In ICCV , 2021.

Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and Venkatesh Babu Radhakrishnan. A closer look at smoothness in domain adversarial training. In ICML , 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 2015.

Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. In ICLR , 2022.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 2021.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV , 2017.

Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable representations for unsupervised domain adaptation. In NeurIPS , 2016.

Inkyu Shin, Sanghyun Woo, Fei Pan, and In So Kweon. Two-phase pseudo label densification for self-training based domain adaptation. In ECCV , 2020.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. In NeurIPS , 2020.

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV , 2016.

Tao Sun, Cheng Lu, and Haibin Ling. Prior knowledge guided unsupervised domain adaptation. In ECCV , 2022.

Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In CVPR , 2020.

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR , 2017.

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 , 2014.

Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine learning , 2020.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , 2017.

Tan Wad, Qianru Sun, Sugiri Pranata, Karlekar Jayashree, and Hanwang Zhang. Equivariance and invariance inductive bias for learning from insufficient data. In ECCV , 2022.

Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. In NeurIPS , 2021.

Guoqiang Wei, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation. In CVPR , 2021.

Guoqiang Wei, Cuiling Lan, Wenjun Zeng, Zhizheng Zhang, and Zhibo Chen. Toalign: Taskoriented alignment for unsupervised domain adaptation. In NeurIPS , 2021.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In CVPR , 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2018.

Zhongqi Yue, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Transporting causal mechanisms for unsupervised domain adaptation. In ICCV , 2021.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018.

Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In CVPR , 2019.

Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML , 2019.

Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In ICML , 2019.

Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In ICCV , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel unsupervised domain adaptation method called invariant CONsistency learning (ICON). ICON is very simple; assuming that labeled source samples and clustered target samples are available, ICON uses BCE losses to make the inner product of the features (softmax-normalized) of any two samples in a mini-batch in each domain close to 1 if they are of the same class/cluster, and 0 otherwise. Evaluation experiments on 10 different benchmark datasets (Office-Home, VisDA-2017, WILDS 2.0) show that ICON achieves superior performance to several representative methods on most of them.\n\nSTRENGTHS:\nThe proposed method, ICON, is surprisingly simple and easy to use.\n\nA theoretical property of ICON is discussed; ICON gives the optimal classifier under certain assumptions. (this is a fairly high-level characteristic, and the realism of the assumptions are somewhat questionable, though.)\n\nVarious experiments are conducted. Despite its simplicity, ICON achieves sufficiently high accuracy on many datasets.\n\nThe paper is generally well-written and easy to follow.\n\nWEAKNESSES:\n1. The performance of ICON on Office-Home and VisDA-2017 is inferior to that of SoTA. For example, CDTrans [Xu+, ICLR2022] achieves 88.4% on VisDA-2017 and 80.5% on OfficeHome, both higher than ICON.\n\n\n2. Since the assumptions underlying the theorem appear to be quite strong, it is questionable to what extent they are valid in practice. (this is discussed more or less in the limitation part in the supplementary material, though.)\n\n\n3. Intuitively, the principle of ICON (i.e., bringing features within the same class/cluster closer together) seems highly similar to that of contrastive learning, which is also a major approach to unsupervised domain adaptation (e.g. [Shen+, ICML2022 ], [Wang+, TMM2023]). Discussing the differences will highlight the property and uniqueness of ICON.\n\n\n4. While this may be outside the scope of this paper, it would be interesting to discuss the possibility of extending to more advanced domain adaptation problems, such as universal domain adaptation and source-free domain adaptation. Since ICON (perhaps implicitly) assumes that the number of classes in the source and target are the same, and the impact of the number of clusters on accuracy is significant (see Table 2). So the performance of ICON on universal domain adaptation, where the number of classes cannot be assumed to be the same, may not be as promising. Application to source-free domain adaptation is also non-trivial.\n\n\n[Xu+, ICLR2022] CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation\n\n[Shen+, ICML2022] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation\n\n[Wang+, TMM2023] Cross-Domain Contrastive Learning for Unsupervised Domain Adaptation\n\nQUESTIONS:\nI have no specific questions that I would like the authors to answer, but it would be great if I could read some discussions on the weaknesses I mentioned above, especially on 1. the comparisons with SoTA.\n\nLIMITATIONS:\nI found a discussion of that limitation in the supplemental material, and it is very reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposes ICON (Invariant CONsistency learning), a method to utilize the distribution of unlabeled target data in the UDA task.
And it obtains stable performance improvements over 8 UDA tasks.

STRENGTHS:
1. The idea of the article is simple, but it makes sense that the distribution of unlabeled target data does have an impact on UDA tasks.
2. In terms of learning the distribution of unlabeled data, this paper has some novelty in using pseudo labels from low-dimension clustering.
3. This article provides sufficient experiments, and the effectiveness of ICON has been verified under various UDA settings

WEAKNESSES:
Essentially, I think that the core of the ICON method is a pseudo label-based contrastive learning technology, where the pseudo label is from low-dimension clustering. And the authors provide some experiments about the rationality of the ICON, like Fig. 5,6,8. But none of this fundamentally or theoretically explains why low dimensional pseudo labels are more accurate.

QUESTIONS:
1. Could you provide more analysis about "why low dimensional pseudo labels are more accurate?"
2. If the low-dimensional pseudo labeling works better, why do you still use high dimensions in the final validation?

LIMITATIONS:
The author has provided limitations in their appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with unsupervised domain adaptation problem. This paper focuses on how to exploit the inherent distribution of target domain to improve the adaptation performance. In detail, it trains two classifier: one is on source domain and the other one is on target domain. Each classifier is trained with BCE loss on a pair of source samples or target samples. For source domain, the same ground-truth label of a pair of samples indicates a positive pair; for target domain, the same cluster label (obtained through clustering) of a pair of samples indicates a positive pair. Finally, the intersection of two classifier is exploited to form the optimal classifier. Additional cross-entropy loss and self-training loss are added to the objective to supervise the adaptation process. Experiment results on various benchmarks validate the effectiveness of proposed method.\n\nSTRENGTHS:\n- The idea is interesting and reasonable to an extent.\n- The paper is generally easy to understand and follow.\n- The method has been tested on various UDA benchmarks.\n\nWEAKNESSES:\n- In one of the claimed contributions, the authors state that their method ICON is orthogonal to previous UDA methods. But I don't see any evidence showing that previous UDA methods combined with ICON should achieve a better adaptation performance.\n\n- In my opinion, it is not suitable to claim outperforming all the conventional methods on WILDS 2.0 UDA benchmark without fully evaluating existing previous works.\n\n- It is interesting to use REx to realize the goal of the second line of Eq. (2). I just wonder what if we simply share the $f$\n between two domains. Will it achieve similar performance? I would like to see such ablations.\n\n- For the experiments, the authors should compare with more recent SOTA UDA methods. The current status of comparisons listed in Table 1 is unsatisfactory and may not justify the performance superiority of proposed method.\n\n- As shown in Fig. 5, the model seems to be sensitive to the hyper-parameter selection. As we know, it is hard to perform hyper-parameter selection in UDA as we don't have labels of target domain data. I feel concerned about whether this work can be really useful in practice.\n\n- The details about how to utilize the self-training loss are not clear. The ablations for self-training terms are missing. Without providing such ablations, we cannot know if the improvement really comes from what the authors claimed.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new UDA method which strives to produce a consistent classifier for labels in source domain and clusters in target domain. Specifically, this paper introduces an auxiliary task for distinguishing whether the input image pair share the same class/cluster or not. This binary classification task would encourage the features from the same class to be similar and features from different classes to be dissimilar. The groundtruth for this binary classification task on target pair is determined by performing clustering on target features. By combining this binary classification task with supervised learning on source domain and self training on target domain, this method obtains state-of-the-art performance on multiple challenging UDA benchmark.\n\nSTRENGTHS:\n1. This paper utilizes clustering on target features to determine whether the target image pair share the same class. Compared to previous self-training methods, this design better reduces the noise in the pseudo labels and alleviate overfitting on incorrect pseudo labels.\n2. The proposed method obtains state-of-the-art performance on many challenging UDA tasks. A sufficient ablation study and analysis clearly depicts the advantage of the proposed method, which would provide great insight for furtehr works.\n\nWEAKNESSES:\nBasically I believe the method proposed method greatly meets its motivation and sufficient analysis has proven its effectiveness. The introduction part could be further polished to emphasize the core idea better. For figure 1, the lack of comparison on t-SNE of baselines might make the reader difficult to understand the advantage of ICON.\n\nQUESTIONS:\n1. In line41 there are double commas.\n2. The BCE loss is only applied on source-source pair or target-target pair. What if applying it on source-target pair? Does it bring better performance?\n3. How the pseudo labels for $L_{st}$ is obtained? A short introduction should be included in the implementation details.\n4. Now that the main idea is to utilize the cluster label to group target domain features, is it necessary to add a BCE task for UDA classification? Is it possible to utilize the cluster label as the pseudo label for self-training?\n\nLIMITATIONS:\nThis paper adequately discussed the limitation of the proposed method and provide an analysis about the possible reasons. No potential negative societal impact is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an unsupervised domain adaptation method, ICON. The algorithm is similar to self-training on the unlabeled target data, but at the start of each epoch, the unlabeled data are first projected from feature space to a reduced-dimension space and clustered. An auxiliary loss enforces consistent pseudo-labels within clusters. The method attained state-of-the-art on the WILDS unlabeled benchmark [1].\n\n[1] Sagawa, S., Koh, P. W., Lee, T., Gao, I., Xie, S. M., Shen, K., ... & Liang, P. (2021). Extending the WILDS benchmark for unsupervised adaptation. *arXiv preprint arXiv:2112.05090*.\n\nSTRENGTHS:\n- ICON’s strong empirical performance is striking, especially on WILDS; the authors outperform similar self-training-style methods, e.g.. vanilla self-training, FixMatch, and Noisy Student. Notably, ICON attains this performance without using aggressive data augmentations.\n- The paper evaluates on ten datasets across 3 modalities, and evaluations are conducted over multiple seeds.\n- The presentation is good, and the experiments are notably well-documented.\n\nWEAKNESSES:\n- Given its similarity to other self-training methods, it seems important to analyze, whether through experimental ablations or theory, why ICON outperforms vanilla self-training so strikingly. One unique aspect of ICON is that the unlabeled consistency loss uses clusters computed in a reduced dimension space; this seems important in ablations (Table 2). The authors motivate this by stating that dimensionality reduction acts to suppress environmental features and highlight the causal feature (L138), but this statement seems unsupported to me. Could the authors clarify theoretically why this is?\n- The theory in Section 4 makes two strong assumptions, to the effect of (1) the classes in T are cleanly separated by clusters, and (2) if the model separates classes correctly in S and clusters in T, then the classes in T are predicted correctly, as prediction uses the invariant feature.\n\nQUESTIONS:\n- The authors present ICON as orthogonal to existing UDA approaches, e.g. consistency regularization using views from data augmentations. Could the authors share how ICON performs when combined with strong data augmentations, e.g. RandAugment?\n- On page 4, the authors state that any clustering algorithm can be used, so long as the number of clusters should be equal to the number of classes. This is a useful ablation; in general, understanding which representation space and how to cluster seems important. Did the authors have k-means experiments to support this?\n- I could not find the UMAP hyperparameters in the appendix as the main paper suggested; in particular, what was the output dimension used?\n- The results of iWildCam2020-WILDS and CivilComments-WILDS are a bit unaligned with the theory, as in these cases, the unlabeled data is *not* drawn from the test distribution. It would be great if the main paper could comment briefly on this discrepancy.\n\nLIMITATIONS:\nThe authors note that the assumptions made in Section 4 are restrictive.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}