How does Contrastive Pre-training Connect Disparate Domains?
Authors: Kendrick Shen
Robbie Matthew Jones
Ananya Kumar
Sang Michael Xie
Percy Liang
Venue: ICLR 2022 Submitted (2022)
Keywords: pre-training
contrastive learning
robustness
out-of-distribution
domain shift

Abstract: Pre-training on massive unlabeled datasets greatly improves accuracy under distribution shifts. As a first step toward understanding this, we study a popular pre-training method, contrastive learning, in the unsupervised domain adaptation (UDA) setting where we only have labeled data from a source domain and unlabeled data from a target domain. We begin by showing on 4 benchmark datasets that out-of-the-box contrastive pre-training (even without large-scale unlabeled data) is competitive with other UDA methods. Intuitions from classical UDA methods such as domain adversarial training focus on bringing the domains together in feature space to improve generalization from source to target. Surprisingly, we find that contrastive pre-training learns features that are very far apart between the source and target domains. How then does contrastive learning improve robustness to distribution shift? We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart. We propose a new measure of connectivity ---the relative connection strengths between same and different classes across domains---that governs the success of contrastive pre-training for domain adaptation in a simple example and strongly correlates with our results on benchmark datasets.

Under review as conference paper at ICLR 2022
How DOES CONTRASTIVE PRE-TRAINING CONNECT DISPARATE DOMAINS?
Anonymous authors
Paper under double-blind review
ABSTRACT
Pre-training on massive unlabeled datasets greatly improves accuracy under distribution shifts. As first step toward understanding this, we study popular pre-training method, contrastive learning, in the unsupervised domain adaptation (UDA) setting where we only have labeled data from a source domain and unlabeled data from target domain. We begin by showing on 4 benchmark vision datasets that out-of-the-box. contrastive pre-training (even without large-scale unlabeled data) is competitive with other UDA methods. Intuitions from classical UDA methods such as domain adversarial training focus on bringing the domains together feature space to improve generalization from source to target. Surprisingly, we find that contrastive pre-training learns features that are very far apart between the source and target domains. How then does contrastive learning improve robustness to distribution shift? We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart. We propose a new measure of connectivity -hee rel- ative connection strengths between same and different classes across domains that governs the success of contrastive pre-training for domain adaptation in a simple example and strongly correlates with our results on benchmark vision datasets.
INTRODUCTION
In applications such as image recognition for self-driving cars Yuu al. 20200] Sun al. 2020) or medical image diagnosis (AlBadawy et al. [2018] Dai & Gool 2018)) machine learning models often fail on examples drawn from different distribution than training. Pre-training on large-scale unlabeled data (Chen et al., 2020 He et al., 2020 Radford et al. (021) can substantially boost accuracy under
distribution shift Radford et al. 2021 Hendrycks et al. 2019 Fisch al. al. 2019 Yogatama et al. 2019)
Devlin et al.) 2019). For example, CLIP |Radford et al. 2021). which is trained on internet-scale image
and text data, has shown impressive robustness benefits on ImageNet variants such as ImageNet
Similarly, pre-training methods such as BERT (Devlin et al. 2019) have improved robustness in NLP
for example, multilingual pre-training significantly improves performance on unseen languages (Liu 2020). While other robustness methods that unlabeled data such as self-training (Prabhu et al.
et al.
2021 Sohn et al.] 2020 Berthelot et al. (222)) and domain adversarial training (Shu et al. 2018 Ganin al. (0166) jointly optimize a labeled and unlabeled objective, pre-training is particularly
suitable for large unlabeled datasets since we can generically pre-train once on unlabeled data and
then specialize the model downstream labeled data via fine-tuning. How does pre-training improve robustness to distribution shift when the pre- pre-training method is not tailored to any downstream task?
In this paper, we run controlled experiments to understand the robustness benefits of a popular pre-training method, contrastive pre-training (Chen et al. 2020 He al.. 2020 Caron et al. 20200). even without large-scale unlabeled data. We directly apply contrastive pre- training to unsupervised visual domain adaptation (UDA), where unlabeled data from shifted target domain is used to adapt models trained on labeled source dataset. We compare contrastive pre-training (SwAV (Caron et al. 2020h) to UDA methods such as SENTRY (Prabhu et al. 202I) (self-training) and DIRT-T (Shu et al. 2018) (domain adversarial training) where all methods use the same unlabeled data (source and target). We find that SWAV achieves comparable or better results on the DomainNet, BREEDS Living- 17, and BREEDS En0ity-30 benchmarks. Furthermore, we show that pre-training once unlabeled data from many domains further improves the average target accuracy over the consituent source-targe target pairs, showing that extra unlabeled data improves contrastive pre-training for domain adaptation.

---

Under review as conference paper at ICLR 2022
Domain (Real)
Domain 2 (Sketch)
a > and B>
Class (Butterfly)
Real
Sketch
Butterfly
B
B
α
Clock
Class
(Clock)
Simplified augmentation graph
Learned contrastive feature space
(Left) Edges between (class, domain) pairs indicate how "connected" they are by augmentations (e.g., cropping, colorization). Small crop sizes contrastive learning can increase the connectivity between disparate domains. For example, a denotes the probability that augmentations connect examples of the same class across different domains. (Right) When a and B, which measures the connectivity between images of the same class or same domain respectively. are larger than y (different domain and different class connecivity), contrastive pre- pre-training produces a feature space where training on labeled data from the source domain (Real seal-gget green, filled) achieves high accuracy on the target domain (Sketch- gray, hollow).
Figure 1:
Given the strong performance of contrastive pre-training, we seek to understand why contrastive pre-training works for domain adaptation. Domain adversarial training and self-supervision algorithms commonly based on the intuition that domains get merged in feature space (Tzeng al. (0114
Ganin et al. 2016 Tzeng et al. 2017 Sun et al.) 2019 Ben- et al. 2010 Shu et al. 2018 Wang
2021 2002 I'hota& & Leontidis 2021). This is support by theoretical notions such H.Y 7 H-iveergeeeee
where domains that can be discriminated well and results worse generalization bounds for target
accuracy (Ben-David et al. 2010). natural hypothesis is that contrastive pre-training brings domains logether by bringing the representations of strong augmentations of the same image together (Sun)
al. 2019).
Surprisingly, find that the source and target domains still far apart in the feature space learned by contrastive pre-training. A domain classifier trained on the pre-trained representations of strongly augmented images from the sketch and painting domains in DomainNet has 94% accuracy, which is even higher than the accuracy of domain classifier on the original (augmented) input space (75%).
We study this phenomenon from the lens of the recently proposed augmentation graph (HaoChen al. 2021). which forms connections between examples if they have similar augmentations (e.g.. a small crop + grayscale can connect sketch and real photos, Figure IT. These connections are central the contrastive objective, which seeks to bring neighbors in the graph closer together while pushing apart neighbors.... From both simulations and real datasets, we define critical connectivity quantity which seems to govern whether domain adaptation will be successful (Figure TT. In particular, connection strength between the same classes across different domains should be relatively greater than the connection between different classes across different domains. Importantly, the absolute connectivity between domains be small and the domains can be far apart in input space. This suggests that contrastive learning circumvents the separation of domains through the connectivity structure of the (general-Pg )urposs)) augmentations used.
Using our understanding, we ablate contrastive learning by removing subsets of unlabeled data that contribute most to connecting the two domains. We do this by training domain classifier to distinguish between the source and target and removing examples which most uncertain under the classifier. This consistently worsens the target accuracy in comparison to randomly removing the same number of data points (by 2-3% DomainNet), showing the importance of domain connectivity for adaptation.
2

---

Under review as conference paper at ICLR 2022
2 SETUP
We consider classification problem from an input space X XC to label space y [K].
Data. Let Ps and Pr denote the source and target distributions of (J.y) pairs, respectively, where a E X and E y. The labeled source dataset S consists of n.s input-output pairs from Ps. We also have access to an unlabeled target dataset T with wr unlabeled inputs from Pr, and we assume the label sets for the source and target are identical. In some cases, also consider an additional related dataset with n unlabeled inputs from related distribution R.
Model and metrics. The goal is to learn classifier h: :. with low classification error on the target distribution Lo-1 (h) = Es.y urr
Pr [1 (z) * yll The pre-training algorithms we consider consist of two steps: first, we train an encoder function f - Z into representation space Z C R (where d' denotes the representation dimension), then we train a classification head g 2 y. The final classifier h = f composes the encoder with the classification head.
2.1 CONTRASTIVE LEARNING
Contrastive learning (He et al. 2020 Chen et al. 2020 Caron et al. 2020) trains representations to be similar between augmented views of the same example e and dissimila lar between augmented views of random examples. Example image augmentations include random cropping, color jitter, and Gaussian blur (Chen et al. (020). The basic assumption is that the semantic content of the data is invariant under the augmentations.
Though there several variants of contrastive learning, focus the following objective from Hho et al. (2021) for our conceptual understanding. Let X' denote the "augmented" space containing all data-augmented views of natural examples (e.g., all random crops of natural ImageNet images). Data augmentation is defined as a random function A: >', where PT[A(TT= = denotes the probability that ग E X augments to x E X. A pair of augmented views is called a positive pair and denoted x.xx if natural point F first drawn and then x,r are the results of calls to A(T). On the other hand, a negative pair æ,æ arises from two natural data points F,F' when x = A( x = A(x'))
Minimizing the following loss function simultaneously brings together the representations positive pairs x,z and repels the representations of negative pairs 2
[()) = - Bz,z [f(x) fxx +E (((())((<>))
Here the first expectations are taken over the marginal distribution over X and the randomness of A.
2.2 CONSTRASTIVE PRE-TRAINING FOR DOMAIN ADAPTATION
Let D denote the unlabeled input data used for contrastive pre-training (e.g., some combination of the source unlabeled data S, target unlabeled data T, and/or additional unlabeled data Rt). We use the following simple algorithm to apply contrastive pre-training to domain adaptation:
1. (Pre-train) Learn an encoder function f:X' + Z via contrastive learning on D.
2. (Fine-tune) On labeled source data S, train a classifier h go :X'+>> + as the composition of the pre-trained encoder and classifier head g. In this paper, we take "fine-tuning" to mean updating both the parameters of and
CONTRASTIVE PRE-TRAINING IS A STRONG DOMAIN ADAPTATION METHOD
We evaluate contrastive learning and relevant baselines benchmark vision datasets and observe that contrastive learning achieves comparable or better performance in all cases.
3.1 DATASETS
BREEDS (Santurkar et al. 2020). BREEDS subpopulation shift benchmark derived from ImageNet by constructing a hierarchical tree structure of classes from WordNet. Nodes at specified depth of the tree become the labels for the classification task, and descendant nodes are treated as subpopulations that can be randomly partitioned into source and target domains.
DomainNet (Peng et al. 2019). DomainNet is large unsupervised domain adaptation task, consisting of approximately 600,000 images and 345 classes in domains. For our experiments utilize the same filtered version DomainNet from Prabhu et al. (202T), which uses 40 of the 345 classes and the sketch, painting, photograph, and clipart domains.
3

---

Under review as conference paper at ICLR 2022
Source
Real
Sketch
Painting
Clipart
Avg.
Target
Skeich
Painting
Clipart
Real
Painting
Clipari
Real
Sketch
Clipurt
Real
Sketch
Painting
ERM
25.67
37.09
44,80
25.72
18.59
31.87
29.13
14.25
20.48
19.34
12.71
10.79
24.20
(RA(
42.76
47.26
48.26
26.74
18.56
37.56
40.32
32.01
29.82
26.89
22.13
14.30
32.22
SENTRY
41.18
51.66
$9.65
56.73
42.55
55.01
30.13
13.33
20.97
10.52
5.33
6.60
32.81
DANN
44.97
48.88
57.36
26
39.60
52.28
47.77
32.22
35.64
32.98
25.46
15.67
39.92
DANN(SA) (SA)
56.74
23.58
43.99
SwAV
43.76
54.55
53.27
53.48
34.99
40.59
67.08
39.64
32.9%
57.13
34,30
25.09
44,91
SwAV+
57.27
46.75
53.46
48
41.33
51.73
ERM
SENTRY
DANN
SwAV (S)
SwAV (T)
SwAV (S+T)
SwAV+
Living-17
59.00
68.76
66.88
61.89
68.53
73.59
81.82
Eutity-30
50.97
62.80
54.62
52.33
62.03
ERM
Dir-T
SimCLR (T)
63.6
75,3
76.1
Table 1: (Top) Test accuracy (%) of ERM with standard and Sw augmentations. SENTRY, DANN with standard and SwAV augmentations, SwAV, and SwAV extra (abbreviated to SwAV+) all domain pairs of DemainNet. Here, SwAV is run on source + target, and SwAV+ is separated from the other algorithms it uses additional data and is therefore not directly comparable. (Middle) Test accuracy (%) of ERM, SENTRY, DANN, SWAV (with various splits), and SwAV+ on BREEDS. (Bottom) Test accuracy (%) of ERM, Dirt-T, and SimCLR on STL-10 CIFAR-10. ERM and Dirt-T numbers are as reported from Shu et al. (2018).
and
STL-10 CIFAR-10. STL-10 and CIFAR- are two classic image classification datasets, each consisting of 10 classes (Coates et al. 2011 Krizhevsky 2009). Each dataset has class label that not present in the other, and hence we follow the procedure of French et al. (2018) and filter out the examples in the non-overlapping classes, resulting in 9-class classification problem.
3.2 METHODS
Baselines. We empirical risk minimization (ERM) the source dataset for each task, i.e., "source-only" baseline. In addition, for BREEDS and DomainNet we use these ERM models as weight initialization for SENTRY, a self-training algorithm for domain adaptation that achieved state of the art results on several domains of DomainNet when initialized with ImageNet-pretree prettraied models (Prabhu al. 202T). For STL - CIFAR, we use Dirt-T (Shu al.) 2018), the current state of the art algorithm for this task.
Contrastive pre-training. For BREEDS and DomainNet we use SwAV (Caron et al. 2020), a contrastive learning algorithm that achieved -ate--of-the-art linear evaluation accuracy on ImageNet. SwAV contrasts examples in feature space rather than input space and uses a novel multi-crop data augmentation strategy using several crops of different sizes. For the lower-resolution task STL + CIFAR we use SimCLR (Chen et al. 2020), which uses only 2 crops and therefore scales straightforwardly smaller input dimensions.
3.3 RESULTS
Main comparison. For all tasks, contrastive pre-training competitive when compared ERM and SOTA unsupervised domain adaptation basclines. On DomainNet (top of Table contrastive pre-training on source and target unlabeled data achieves higher target accuracy than jiitt-nnniig baselines both 5 of 12 domain pairs and average over all pairs (44.91 % the second best, 43.99%). On Living-17 17 (middle of Table contrastive learning on source and target unlabeled data improves over SENTRY over 4%, while on Entity-30, contrastive learning (source target) within 0.3% of the target accuracy of SENTRY. On STL-CIFAR, SimCLR on the target unlabeled data achieves 0.8% higher target accuracy than Dirt-T.
Extra unlabeled data from related domains. We also consider adding extra unlabeled data from other (related) domains in DomainNet, Living-17, and Entity-30. In particular, we can pre-train once on unlabeled dataset that consists of a superset of the domains want adapt to, and fine-ture on the available source data. While this is not a fair comparison to other domain adaptation methods, the ability to scale to large unlabeled datasets is natural advantage of pre-training.
4

---

Under review as conference paper at ICLR 2022
Feature space learned
Different class, domsin, source
Different class, same domain, target
Same class, different domain
Different class, different domain
Living-17 17
Taput space SwAV
20.94
21.91 2.26
36.58
20.00
1.85
12.67
1.26
DomainNet
Inpat space
32.88
27,36
25.12
ERM
16.85
18.34
10.69
DANN
10.59
11.96
16,59
7.30
SwAV
7.03
7.54
2.46
Table Average separation error of Living- 17 and DomainNet class-domain pairs. On both datasets, the classes become very distinguishable in the feature space learned by contrastive learning and on DomainNet, the domains additionally remain far apart in the pre-trained feature space. Classifiers were trained on augmented images to distinguish class-domain pairs proxy for connectivity (higher separation errors suggest greater connectivity).
On NomainNet, we consider contrastive pre-training with SWAV unlabeled data from all the domains at once, which we denote as SwAV + extra. The top T that adding extra unlabeled data improves the target accuracy of contrastive pre-training on all pairs and improves the average by almost 6% (44.91% % to 50.75%). On Living- 17 and Entity-30 (middle of Table we additionally consider contrastive pre-training with SWAV with 1) only source, 2) only target, and 3) with all of ImageNet as the unlabeled data. We find that increasing the amount of unlabeled data steadily increases the target accuracy, and vre-training on all of ImageNet improves over source target by 8% and 4% respectively for Living- 17 and Entity-30.
improves
Ease of hyperparmeter tuning. The SENTRY self-training baseline requires extensive hyper- parameter tuning (e.g., finding optimal balancing coefficients for the joint losses, learning rate schedules) for every transfer task and is difficult with access only to source labels. In contrast to the baseline methods, for contrastive pre-training we did not need sweep over hyperparameters or use target labels for model selection. We used nearly identical hyperparmeters for both BREEDS datasets (Living- 17 and Entity-30) and all DomainNet pairs (details in Appendix A).
No explicit knowledge of distribution shift. We note that SENTRY uses explicit knowledge of label imbalance within its algorithm, on top of being designed for the domain adaptation task of adapting from source target. In contrast, used off-the-shelf contrastive pre-training, which was designed for general representation learning, with no additional information about distribution shift.
4 CONNECTIVITY MODEL FOR DOMAIN ADAPTATION
How does contrastive pre-training improve target accuracy with unlabeled data? Conventional intuitions and theory for domain adaptation are based reducing the distance between the source and target domains (Ben-David et al., 2010 Ganin et al. 2016).Howert we see empirically that contrastive pre-traiie pre-training keep the domains very separated representation space. Towards building an understanding for how contrastive pre- pre- training connects domains without merging them, we extend a conceptual framework based on connectivity via data augmentations (HaoChen et al. 202T) for distribution shifts. We define crucial measure of domain connectivity that relates properties of the data distribution and contrastive augmentations to the success contrastive learning for domain shifts on a simulated data model, DomainNet, Living- 17, and Entity-30.
4.1 CONTRASTIVE PRE-TRAINING KEEPS DOMAINS APART
Contrary to common intuitions for domain adaptation methods, we find that contrastive learning does not bring the domains together in feature space. Table showss the average test error of classifiers trained to discriminate between examples from the same class but different domains on NetainNet. We train and test the classifiers on images using the same augmentations as SwAV and find that contrastive pre-training learns features that are actually more discriminable between domains than in the original input space (7.5% vs. 27.3% error on average over all pairs). These estimates for the baselines and for individual pairs are provided in table
Next, we measure the separability in feature space and input space between classes within a single domain (c.g., 3% 24% error in the real domain, t)) We note that the separabiliry between domains in feature space is comparable to the separability benween classes, suggesting that contrastive learning may cluster both domains and classes in separate axes.
5

---

Under review as conference paper at ICLR 2022
4.2 USING CONNECTIVITY TO CHARACTERIZE DOMAIN SHIFT
In this section, we set up conceptual model of connectivity used to study contrastive learning.
Connectivity via augmentations in domain adaptation. HaoChen et al. (2021) proposed viewing contrastive learning as operating on an augmentation graph. The augmented inputs from A' constitute the vertices and the edge weight between x,r' represents how likely they are to be generated from the same natural example (i.e., how likely they are to be chosen as positive example during contrastive pre-training). HaoChen et al. (2021 show that when examples from different classes r,æ' E X are sparsely connected via the augmentation graph (i.e., rarely augment into the same augmented example), and the connectivity within a class is strong, the contrastive pre-training leads to features that obtain good accuracy on the downstream task.
Extension domain adaptation. Motivated by the augmentation graph framework, we define a notion of connectivity for domain adaptation on a graph where class-domain pairs are vertices. We denote the set of examples belonging to class and domain (which we refer to as the class-domain pair (c,d)) as P P(c.d)- We formalize this notion in the following definition:
Definition (Connectivity). The connectivity between two class-domain pairs (i,d) and (j.d") is
where x ((t.d) and P(gad))
Intuitively, the connectivity between two class-domain pairs is the probability that two randomly sampled points from each class-domain pair are augmented to the same point.
Definition is defined for two arbitrary class-domain pairs, which then allows us to build further definitions of connectivity involving multiple class-domain pairs. Such connectivity definitions would allow us to more rigorously describe (for instance) our observation in the DomainNet dataset that points of the same class but different domains are more discriminable than points in different classes within the same domain. We define these instances of connectivity using Definition I
Definition 2 ((a,8.n) connectivity). A source-target pair of distributions S and T over X (with domain labels ds, dr, respectively) satisfies (a,,3.)) cooneetivity if
2. [BLsetc](comm((44444444)4 =3. (differeett-aas--ssme--oooe Sater de (ds,dr})
3. (Egsen/(oonm(i444)....44 oatp://oeeetiiooooo////////////r
4.3 ILLUSTRATIVE TOY EXAMPLE: BINARY CLASSIFICATION
As an illustrative example, consider the simple case of binary classification where each class-domain pair consists of single data point. This toy setting pinpoints simple connectivity condition such that linear classification on the pre-trained features results in good target accuracy.
Setting (Figure 2
Assume uniform marginal distribution over the natural inputs, and suppose data augmentation transforms each data point directly into another with a certain probability, so that the entire augmentation graph consists of the 4 points (2 classes, 2 domains) each identified by their class and domain. Formally, the example (i,d) (of class and domain d) has augmentation given by
(i,d)
with probability
(i,d")
with probability (a/2C)
A((i,))=
=
(j.d)
with probability (8/2()
(i,d))
with probability (7/2C)
where j Hi, d' dd d. We also assume <> max(a,,8r)) (i.e., data augmentation is most likely to map each data point to itself) and appropriately constrain + + + = 2Ç(1- so that the probabilities sum to 1. Note that the specific transition probabilites are selected so that the distribution shift satisfies (a.,B,)) conneetivivy We also number the points 1 -4 in the following order: class (source), class 1 (target), class 2 (source), class 2 (target).

---

Under review as conference paper at ICLR 2022
Source
Target
Good case: a'
Bad case: a <
1.0
Class
B
Class
a
35
Figure 3: Scatter plot of random graph simulations, where Ç <= 16,3 7 Shown on the x-axis is and nontrivial target accuracy is not possible until a > 7. Target accuracy also increases with the ratio a/3.
Figure 2: (Left) Illustrative toy example with 2 domains and 2 classes, where each class-domain pair is a single node in the graph. Edge weights denote connectivities (probability of two class-domain pairs augmenting into the same point). (Middle) When connectivity) is greater than y (connectivity across different domains and classes), the domain representation sification on the source classifies the target accurately. (Right) When <, linear classifier can label both the source and target accurately.
and
oriented so that linear clas-
Pre-training. HaaoChen et al. (2021) show that minimizing the spectral contrastive loss (given in D is equivalent to computing the spectral decomposition of the adjacency matrix of the population augmentation graph. The leading eigenvectors of the resulting decomposition are then used as the learned representations, so that the representation of example is the concatenation of the 7th components of those eigenvectors. Because the population here finite, we compute the spectral decomposition algebraically and express the learned representations as a function of connectivity. Agnostic of the ordering of the relevant parameters, the unordered eigenvectors of this toy example are
V1 = [111]]
U2 =2= - -1]", 03 =3= -1 11--11111 1-1"",
84 1 -1
with associated eigenvalues
\ - - - )=
We discard the eigenvector U1, it assigns the same feature every point, and the top 2 remaining cigenvectors as the learned features, For example, if U2 and U3 are the top remaining cigenvectors, then the features for data point 2 would be the 2nd component of U2 and 03 (namely, (-1,1)).
Fine-tuning, We consider learning a linear classifier upon the pre-trained features. The middle panel of FiguureBillustrates jilusttrate the fortunate case, which a >> and 8> (but possibly a S 9). In this case, A2 > Aa and Aa > Aa and therefore the learned features use 122 and and are (1,1),(1,-1) for class 1 and (-1,1),-11,-1) for class Then, linear classifier trained on the source uses only the class information, which is contained in the second index, and labels the target accurately.
Note that the source classifier labels the target correctly if and only if it uses solely the features given by U3 for prediction. Two possible failure modes occur when is smallest (and therefore va is discarded) and when 3 is smallest (and therefore the source classifier can use features from either U3 or v4).
Importantly in this toy setting, whether not the features of the target examples oriented correctly depends only on whether a and /3 are both > y. Consequently, the classes within each domain can be very connected-mont more than across domains-and target accuracy importance of this connectivity ratio on more fine-gr -ne-grained setting, we run simulations and observe sharp thresholding behavior when y surpasses (Figure 3 simulation details in Appendix C).
be high. To illustrate the
4.4 CONNECTIVITY GOVERNS THE SUCCESS OF PRE-TRAINING FOR DOMAIN ADAPTATION
Connectivity on benchmark datasets. We estimate the input space domain connectivity on DomainNet, Living- 17, and Entity-30 by training classifiers between various pairs of class-domain pairs (test errors provided in Table full details provided in Section B). We find that all the datasets satisfy a > the necessary condition for the good case in the toy example-but a < B for DomainNet while 3 for BREEDS. Intuitively, natural distribution shifts is likely to be higher than 7 since the data augmentation should be more likely to connect doas-n pairs where only the domain differs than to bridge differences across both the domain and the class.
We leave the eigenvectors unnormalized for ease of exposition; the normalized variants would multiply the entire vector by 1/2.
7

---

Under review as conference paper at ICLR 2022
3.84
real-cipart
66
30-
skeach-clpart
60-
Targel accuraey gain
painting-clpart resl painting painting elpart-painting
20
.88
Tarqet
56-
sketch-real
Confidence
realtingreal
54-
Random
cspart-real
real-sketch
2k.
Subsompped Dataset Size
4k
6k
8k
10k
12k
1.05
1.10
1.15
1.20
painting-scettch
Connectivity measure
cipart-sketch
Figure 5: Ablation of connectivity by removing unlabeled data points near the margin of a domain classifier. Removing these points from the pre-training dataset reduces target accuracy more than removing random subset of the same size.
Figure 4: Plot our measure of connectivity ratios against target accuracy of contrastive pre-training on DomainNet. Our quantity highly correlates = 0.84 with target accuracy.
Domain and class classifiers are nearly orthogonal. We check the extent to which the domain and class information are separated by training 3 linear classifiers the pre-trained representation space: 1) Source classifier (which discriminates classes source) 2) Target classifier (which discriminates classes in target P(3 Domain classifier (discriminates between source target domains). We compare the linear weights of these classifiers by computing the cosine similarity. We find that
1. The source and target classifiers very similar; average the classes, the cosine similarity of linear weights for the same class from the source and target classifiers is high (around 0.20 for DomainNet, 0.34 for Liv.ng-17).
2. The domain classifier is nearly orthogonal to the source and target classifiers. The average cosine similarity between the weights of any class from the source or target classifiers and the weights of the domain classifier is 0.01.
This suggests that the class information and domain information are contained in different directions of the feature space, and gives an explanation for how both classes and domains can be easily separated in feature space while maintaining the ability to generalize from source to target. Learning to discriminate classes results in classifier with weights that are nearly orthogonal to the weights for domain classification, suggesting a natural amount of 'domain agnosticity" despite not removing domain information from the features. Detailed results on individual pairs are provided in Appendix
Connectivity ratios correlate with target accuracy. We construct scalar connectivity quantity and find that it correlates with target accuracy of contrastive learning on DomainNet. In the simple example, contrastive pre-training for domain adaptation works when a > and 8 >> yy The first condition (a ?) is relatively more important since if (connectivity between different classes in the same domain) is very high, then the classes within a domain are hard to discriminate and the classification task itself will be hard. Thus, we define the quantity
(a/1)-(3/1))
where we take = 0.1 to "discount" the importance of the second ratio. We multiply the two ratios to express the logical "and". The larger this quantity is, the better the target accuracy should be.
Figure&show: a plot of the quantity with target accuracy of contrastive learning on DomainNer pairs. Because different target domains can have different intrinsic errors, we "debias" the target accuracy of each source-target pair by subtracting the worst accuracy over all pairs with the same target domain. We find a strong correlation between the connectivity and target accuracy (Pearson r of 0.87 for S+T pre-training) (Figure[4).
Ablating connectivity.
We find that the connectivity between examples of the same class but different domain is also very important, as those examples "bridge" the domains. We verify this intuition by systematically ablating the connectivity on subset of Living- 17 by removing those examples that contribute most to domain connectivity. Specifically, for each class we trained discriminators between the source and target examples of that class and removed the pre-training data points from each
A Athhough this cannot he done in practice, we only use the target labels here for exploratory analysis;
8

---

Under review as conference paper at ICLR 2022
class-domain pair which the discriminator was least confident; these examples be seen as the ones contributing most to domain connectivity. For subsample sizes of consistently reduce target accuracy compared random subset removal (Figure confidence subsampling increases error relatively by 6% on average compared to random).
5 RELATED WORK
Domain adaptation. Devid et al. (2010) prove generalization bounds the distribution shift setting. Their bounds rely on notion of distance between two data distributions called HAH divergence being small, which many methods such as domain adversarial training are inspired by (Ganin et al. 2016 Tzeng al. 2014 2017). Our conceptual model based around connectivity provides way to explain how contrastive pre-training can work for domain adaptation even when the HAH divergence between domains is large.
Domain adaptation with self-supervision Prior works have explored the application of self- supervision to domain adaptation. (Sun el. (2019) propose optimizing on hand-crafted supervuee tasks such as predicting the angle of rotation of rotated image. These tasks are jointly optimized on the source and target data along with the labeled source loss. As part of their comparisons, they found that first pre- pre-training on target (or source and target) data and then fine-tuning separately (instead of jointly) was highly non non-competitive with their other methods. In contrast, we find that pre-trane pre-training with general, off-the-shelf contrastive learning objective is competitive with other state-of of-the-art domain adaptation techniques.
source data
Contrastive objective for domain adaptation. Contrastive objectives have also been applied to do- main adaptation problems. Wang et al. (2021) jointly train the contrastive loss on source and target data, These methods rely on explicit anchor pairs that encourage domain alignment (Wang et al, 2021), or still employ some form of explicit domain alignment (Thota & Leontidis (02T). We consider pre- pre-tnnining instead of joint training and do not use any domain knowledge about the distribution shift problem.
self-training. Berthelot et al. (2021) use self-training and consistency regularization with a distri- bution alignment method, but require estimating the target label distribution and tuning a confidence thresholding parameter (tuned using target test labels). Cai et al. (202T) give a label propagation analysis for self-tran eefftrrining in domain adaptation. They have only one parameter which governs both the cefferent-class-samee same-domain and small. In our framework, we allow the fffffrrent-clas-ssm--doo connections, requiring both to be connectivity to not be small, in accordance with our observation that domains can be well separated contrastive feature space. They give experiments on self-training when operating but their focus is on self self-training while the pre- pre-traine trained representations are an empirical detail. One inflexible characteristic of standard self-training is that it relies on the unlabeled data having the same classes as the labeled data (in order to pseudolabel unlabeled points), which may not allow them to utilize diverse sources of extra unlabled data like many pre-training methods can.
contrastive pre-trate ree-triine representation space,
CONCLUSION
In this work we focused evaluating and understanding contrastive pre-training under domain adaptation. Pre-training has the advantage of amortizing the cost of processing a large unlabeled dataset into pre-training step that is done only once. While pre-training loses the advantage of seeing labeled data (and thus the downstream task) to adapt its usage of unlabeled data, we find that contrastive pre-training is still competitive with other domain adaptation algorithms. We hope that our connectivity model can give insights to improve pre-training data selection, develop better augmentations for pre-training with distribution shift in mind, and improve fine-tuning methods that exploit the properties of the pre-trained feature space.
ETHICS
Our experiments use standard benchmark vision datasets in domain adaptation, which is publicly available data. In general, however, large-scale unsupervised learning can involve data scraped from the internet which may lead to privacy concerns. Care should be taken when curating datasets for large scale contrastive pre-training in practice.
9

---

Under review as conference paper at ICLR 2022
REPRODUCIBILITY
We detail the hyperparameters and experimental details in the paper and appendix. We have uploaded a zipped code archive and will publish the code on Github upon acceptance.
REFERENCES
EA AlBadawy, A Saha, and MA Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med Phys., 45, 2018.
Shai Ben-David John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151-175, 2010.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint 1rXiv:2106.04732. 2021.
Tianle Cai, Ruiqi Gao, Jason Lee, and Qi Lei. A theory of label propagation for subpopulation shift. In Marina Meila and Tong Zhang (eds.), Proceedings the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1170-1182 1822. PMLR, 18 24 Jul 2021. URL https: pttEps://proceedingee www.angg........ssss......eese.. press/pesssee.......... 39/ caicaalaaa...... html
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 9111999129999 92249922,222 2020.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607, 2020.
Adam Coates, Andrew Ng. and Honlak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15, pp. 215-223, 2011.
Dengxin Dai and Luc Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In 2018 2 International Conference on Intelligent Transportation Systems (ITSC), pp. 011938812222 9-2244 2018. doi: 10.1109/TTSC.2018..555 8.5699877
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL). pp. 4171-411 4186, 2019.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library). lhtpps: / hub. com http//MddryLLLbbb
SaladdryLaab/bbusneessss http://obuusssssseeeeee 2019.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Workshop on Machine Reading for Question Answering (MRQA), 2019.
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference on Learning Representations, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario March, and Victor Lempitsky. Ddwainnadversarial training of neural networks. Journal of Machine Learning Research (JMLR) 17, 2016.
Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. arXiv preprint
100.020000.4451 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In Computer Vision and Pattern Recognition (CVPR), 2020.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre- mre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML.), 2019.
10

---

Under review as conference paper at ICLR 2022
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toroonto, 2009.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and
Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020. doi: 10.1 162/ttallaa0004453333131 ort/ 2ttp://220.aac11..... tac1-1.47
URL
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-soures domain adaptation. In International Conference on Computer Vision (ICCV), 2019.
Viraj Prabhu, Shivam Khare, Deeksha Karthik, and Judy Hoffman. Selective entropy optimization via committee consistency for unsupervised domain adaptation. In International Conference on Computer Vision (ICCV), 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), volume 139, pp. 8748-8763, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 1 115(3)211-2522 2015.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. arXiv, 2020.
Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-T approach to unsupervised domain adaptation. In International Conference on Learning Representations (ICLR), 2018.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying eupi-supeeeieed learning with consistency and confidence. arXiv, 2020.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset, 2020.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A. Efros. Unsupervised domain adaptation through self-supervision. arXiv, 2019.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey. arXiv preprint arXiv: 10320, 2020.
Mamatha Thota and Georgios Leontidis. Contrastive domain adaptation. arXiv, 2021.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412.3444 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2017.
Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. Cross-domain contrastive learning for unsupervised domain adaptation. arXiv, 2021.
Dani Yogatama, Cyprien de Masson d' Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et. Learning and evaluating general linguistic intelligence. arXiv preprint 19X.i:1901..13330 2019.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

---

Under review as conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 DATASETS
BREEDS. We use the dataset creation functions defined in the rcbustness Python library to generate the Living- and Entity-30 tasks from the original ImageNet dataset (Engstrom et al. 2019
Russakovsky et al. 2013). The Living-17 17 dataset is an animal classification task which consists of nodes in the subtree rooted at the "living thing" node the WordNet hierarchy. An example of a label is "ape" with subpopulations of gibbons, orangutans, gorillas, and chimpanzees. The Entity-30 dataset a much more general task, incorporating nodes in the "entity" subtree. Labels include "building" and "home appliance". The trailing number in the dataset name is the total number of classes in that dataset.
We use the official SENTRY repo at hhtpp:: / http//gttttttttte huub com/ ht///viraaajapppp rajprabhu SENTRY, which filters the original DomainNet dataset automatically as described in Section 3 This refinement is done eliminate much of the noise present in the original DomainNet dataset (Tan et al. 2020).
NomainNet.
STL -> CIFAR. CIFAR-10 consists of 32 x 32 images from the former TinyImages dataset, and STL-10 is derived from ImageNet and contains images with resolution 96 96. We resize the STL-10 images to 32 x 32 to match the resolution of CIFAR1O. 10. The two non-overlape con-ovellap classes ("monkey" in CIFAR-10 and "frog" in STL10) are removed from both datasets before training.
A.2 BASELINES AND HYPERPARAMETER TUNING
ERM (source-only).
BREEDS: We the same hyperparameters as Santurkar et al. (2020). For Entity30, we train for 300 epochs and divided the learning rate by 10 every 100 epochs. On Living 17, 7, we train for 450 epochs and divided the learning rate by 10 every 150 epochs. For both datasets we use a ResNet50 with learning rate of 0.1, a weight decay of 10-4, and a batch size of 128.
The SENTRY algorithm runs ERM with class balancing (starting with ImageNet- initialization) prior to beginning entropy minimization, and therefore the SENTRY repository contains an ERM implementation and hyperparameters for DomainNet. We ERM for 1550 epochs and multiply the initial learning rate by 10x, keeping all other hyperparameters constant.
DomainNet:
STL10 > CIFARIO: We report the exact results from Shu et al. (2018).
DIRT-T. DIRT-T (Shu et al. 2018) is domain adaptation method that addresses two flaws of domain adversarial neural networks (Ganin et al. (016)): 1) distribution matching is weak constraint, and 2) in some domain adaptation settings there does not exist a good joint classifier on both source and target. The authors address the first shortcoming by adding conditional entropy regularization term so that the model's 's decision boundaries do not overlap high-density regions of data. This is inspired by the cluster assumption, which states that the input space is divided into well well-separated clusters, one for each class in the label space. The lack of a good classifier on both source and target is then addressed via self training on the unlabeled target data. We report the STL.10 CIFARIO results from Shu et al. (2018).
SENTRY. For each pair of domains on DomainNet we conduct hyperparameter search through Aure E (0.5,1.0,1.5) (the weight on the supervised classification loss) and learning rates E (0.01,0.ck 3.001 and report the model results with the highest source evaluation accuracy.
On BREEDS we keep the same default hyperparameters from the SENTRY repo and search over 3 learn- ing rates 00.000,,0001,000 choosing the model with the highest target evaluation accuracy to report.
SENTRY underperforms the ERM models when the source domain is clipart images (from Net) which could be the result of not using mmageNet-pretraing classifiers for initialization as Prabhu et al. (202T) do. Therefore, as the joint-training bascline for DomainNet we take the maximum of ERM and SENTRY.
A.3 SIMCLR
We the official SimCLR repository from Google for our experiments (https //qithub. com/ google-ooeeeoooooooooooooo www.rreeeeeeeeeee.eee. of 256, learning rate of 0.2, and weight decay 10-1, The projection head has two layers and an output
We use a ResNet18 with a batch size
12

---

Under review as conference paper at ICLR 2022
dimension of 64. We pre-train the model for 400 epochs with square-root learning rate scaling and we train the linear probe for 100 epochs on batches of size 512 and a learning rate of 0.1.
A.4 SWAV
We use the public SwAV repository available at httalabbeehtttppss//ittttt com hww.//aaabbooeeeerr..caaaaa swav and kept almost all of the hyperparameters provided by the original paper for 400 epoch, 256-batch-size training on ImageNet. However, we used batch size of 512 and additionally made the following changes based on the Github issues answered by the original authors:
We avoid using a queue on DomainNet and the subsampled variant of Living- 17 in order to stabilize training. For pre-training the full Living- 17 and Entity-30 datasets, introduce the queue at epoch 60.
2. We set the number of prototypes to be 10 times the number of classes (170, 300, and 400 for Living-17, 17, Entity-30 and DomainNet, respectively).
3. We set c
4. We set the base learning rate to 0.6, following a linear scaling rule based on batch size.
We always fine-tuned from the final iterate of SwAV pre-training (400 epochs).
On DemainNet, we fine-tuned SwAV models for 50 epochs using the ERM implementation the SEN- TRY repository (without rumning any the number of epochs constant. We report the target test accuracy of the final model (after 50 epochs).
algorithm), keeping all hyperparameters other than
On BREEDS Living- 17, fine-tuned SwAV models for 100 epochs with cosine learning rate schedule without restarts. We use SGD with initial learning rate 0. for the classifier head and 0.01 for the encoder, momentum 0.9, and weight decay 0.0001. We use batch size of 96, and once again report the target test accuracy of the final model (after 100 epochs).
On BREEDS Entity-30, which contains 300K examples, instead linear probe.
B RESULTS AND PROTOCOL FOR ESTIMATING (a,,,7) CONNECTIVITY
Table 3 reports the connectivity estimates for the input space and the feature spaces learned by ERM, DANN, and SwAV for all pairs of DomainNet domains.
To estimate the connectivity between 2 class-domain pairs (i,d) and (j.d), we use the following algorithm:
1. Label all training examples of class and domain (i,d) as 0 and all training examples of (j,d') as 1. Discard the remaining examples.
2. Train ResNet50 for 100 epochs using SwAV augmentations and cosine learning rate. We did not exhaustively tune this training step, but we kept the hyperparameters constant for all the pairs.
3. Collect the test set analogously to step 1, and evaluate the classifier on augmented data.
Each domain in DomainNet-S has a unique label distribution (all of which are far from uniform), and therefore in computing the average connectivity we compute the weighted mean, where each pair of (class, domain) pairs is weighted by the ratio the less to more frequent label (0 or 1).
C SIMULATION DETAILS
To obtain an even more fine-grained picture, we run simulations on toy data settings, where we can precisely vary each type of connectivity, and observe:
1. Sharp thresholding behavior when a and y cross over.
2. must be nontrivial.
The simulation is on finite population so that we can study the spectral decomposition directly: how- ever, later show empirically that these ideas still hold in real-world, continuous continuous-population settings. For each domain-clan pair dc, we generate N points in augmentation space. These augmented points inherit the domain-class assignments of their corresponding natural points (we have ensure that all
13

---

Under review conference paper at ICLR 2022
Input space
Different class, domain,
Different class, domain, target
Same class. different domain
Different class, different domain
Real <5 Sketch
24.49
12
21.47
20.51
Real ++ Painting
24.49
33.71
33.62
28.07
Real 4+ Clipart
24.49
35.20
23.12
21.18
Sketch + Painting
38.12
33.71
24.34
23.16
Sketch Clipart
38.12
35,20
30.23
27.72
Painting + Clipart
33.71
35,20 31,36
30.09
Avg.
35.19
27.36
25.12
SwAV pre-trained
Different class, same domain, source
Different class. same domain, target
Same class, different domain
Different class, different domain
space
1.74
Real +> Painting
3.07
7.43
11.79
2.23
Clipart
1.73
Sketch +- Painting
8.50
10.65
5.55
2.66
Sketch Clipart
8.54
8.15
8.10
3.29
Painting +s Cliprt
10.22
8.40
6.13
3.12
Avg.
6.07
7.98
7.54
2.46
ERM feature
Different class, domain,
Different class, domain, larget
Same class, different domain
Different class, different domain
space
Real + Sketch
8:40
13.32
9.99
6.43
Real Painting
16.24
22.23
8.87
Real - Clipart
8.40
13.15
14.16
6.63
Sketch +
11.75
16.36
8.45
Sketch Painting
12.46
18.57
16.51
9.38
Sketch + Clipari
12.46
14.45
22.30
10.90
Painting Real
22.59
13.45
26.50
14.10
Painting + Sketch
22.59
20.43
14.03
11.28
Painting Clipart
22.59
18.73
19.55
14.20
Clipart Real
17.17
15.35
21.18
11.48
Clipart Sketch
17.17
21.45
17.47
12.03
Clipart Painting
17.17
25.30
19.80
14.52
Avg.
15.16
16.85
18.34
10.69
DANN feature
Different class, same domain, source
Different class, same domain, target
Same class, different domain
Different class, different demain
space
Real + Sketch
9.98
9.87
4.53
Real + Painting
6.27
13.07
21.55
6.72
Real + Clipart
6.36
9.82
14.37
4.98
Sketch Real
8.50
7.53
15.38
5.71
Sketch + Painting
8.75
13.24
16.61
6.91
Sketch Clipart
10.43
21.15
7.76
Painting - Rcal
14.93
7.70
22.14
7.46
Painting Sketch
14.57
15.49
9.76
Painting Clipart
14.78
11.31
13.24
7.35
Clipart
Clipart Sketch
12.40
16.18
16.09
8.90
Cliputt Painting
11.51
19.06
15.90
10.44
Avg.
10.59
11.96
16.59
7.30
Table 3: Empirical estimates of the different parameters of connectivity in the input space (top) and feature spaces computed by SwAV (second), ERM (third), and DANN (bottom). The numbers provided are the separation error.
the augmented points are distinct). Then for any new natural point 7 X that we want to augment, we simply choose one of our existing augmented points random and designate as the augmentation of F. Since we have assigned domain-class memberships to this base set of augmentations, we can define connectivity of two domain-class pairs (d C1, d2c2) to be the probability that natural point from dici is "augmented into" an augmented point in d2c2 (and vice-versa). This probability something we can easily change in the simulations to vary this notion of connectivity.
D ADDITIONAL DETAILS FOR SECTION 4.4
As another method for evaluating the connectivity measures (in addition to Figure 4 inspect the weights learned from a linear regression model from features [a,33yy] to the target accuracy (de-biased). The weights are (6..4,55661.0.8) which gives large negative weight a (different-
14

---

Under review as conference paper at ICLR 2022
class cuus-ddiferent-doone domain connecitivity), large positive weight wwwss-diffeeeett--eeeeee domain)) and a small positive weight to 3 (different-class wlae-cllasss----eeeeee same-domain) as expected from our intuitions.
The following table contains more detailed results the cosine similarity between domain and class classifiers.
Class (domain 1)
Class (domain 1)
Class (domain 2)
Class (domain 2)
vs. Domain
Domain
Living-17
Source Target
0.3971
0.0125
0.0158
DomainNet
Real ++ Sketch
0.1999
0.0143
0.0165
Real ++ Painting
0,2280
0.0157
0.0166
Real 6+ Clipart
0.2222
0.0148
0.0149
Sketch Painting
0.1758
0,0146
0.0200
Sketch Clipart
0.1590
0.0155
0.0178
Painting Clipart
0.1383
0.0178
0.0221
Table Cosine similarity of class and domain classifiers trained SwAV representations (average over all classes). Class classifiers trained on the source and target individually learn similar linear weights, evidenced by the cosine similarity in the range 0.13 0.33. However, domain classifiers learn linear weights that are nearly orthogonal to the class classifier weights, suggesting that SwAV pre-training learns features containing domain and class information in somewhat separate directions.
15


Reviews:
[{"review_id": "E4wjGxrRq_m", "reviewer": "Reviewer_W3eA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points of the paper: \n1.\tThe goal of applying contrastive pre-training on unsupervised domain adaptation is a critical topic to study. \n2.\tThe paper proposes to use the connectivity model to study contrastive learning. \n3.\tExperiments are comprehensive with a variety of benchmark tasks.\n\nWeak points of the paper: \n1.\tThis paper does not propose a new method, but proposes a connectivity measure based on the analysis of existing methods. Despite it maybe practical to improve fine-tuning with connectivity measure, the technical contribution is rather limited.\n2.\tThe main experiments in this article are based on existing methods. It would be good to show the results of using domain adaptation methods based on the new connectivity measure. Even a simple solution on the toy task can verify the practicality of the proposed measure to a certain extent.\n", "summary_of_the_paper": "This paper proposes to study the contrastive pre-training on domain adaptation tasks. The key contribution is to design a connectivity model for better data selection and data augmentations of pre-training. To verify the benefit of contrastive pre-training to domain adaptation, experiments are performed on a variety of benchmark tasks including BREEDS, DomainNet and CIFAR-10.", "summary_of_the_review": "Although this paper gives an interesting new idea to solve the problem of domain adaptation from connectivity, it does not extend the idea to a specific method, nor does it conduct sufficient experimental verification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6M-NrbOpO3U", "reviewer": "Reviewer_pgHb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n+ The topic of the study is of great interest, and to me is pretty novel and not explored in the literature.\n+ The observations are intriguing: contrastive learning based methods connect disparate domains in a way that is different from traditional adversarial DA perspective. The findings could have implications on when / how contrastive learning works when data are from different domains.\n+ The evaluation and presentation is clear and thorough. The empirical results are well aligned with the proposed connectivity model, which could be promising for understanding the behavior of CL.\n+ The writing is clear and easy to follow; the paper is well structured.\n\n## Main Weaknesses\n1. __Ambiguous shift assumptions throughout paper__ \n- Throughout the paper, the authors never explicitly define what is the __shift assumption__ across domains. Is the study conducted under the covariate shift assumption (i.e., $p_i(x) \\neq p_j(x)$ and $p_i(y|x) = p_j(y|x)$)? If so, what will be the results under other assumptions (e.g., label shift)? Will this affect the main conclusion?\n\n2. __Investigations limited to visual recognition tasks__\n- First, the title and body texts seem to hold without specifying the data modality. However, all experiments are limited to image data for visual recognition tasks. Nevertheless, CL has been successful beyond image data, for example, texts [1], audios [2] and time series [3]. The authors only experimented on image data, which might not be enough for a rigorous conclusion.\n- The limited experiments only on image data also lead to doubts on whether some claims are true or not. For example, the paper claims \"Contrastive pre-training is a strong domain adaptation method\". This might be true for image data, as it is interpretable by human, and people can design effective augmentation methods (e.g., those in SimCLR). However, does this hold also for other data modalities, especially those without rich priors on how to do augmentation in the first place?\n\n3. __Minor: only single source & single target domain considered__\n- One minor drawback is that the paper considered only single source & single target domain. For self-supervised pre-training, uncurated data is more likely to come from different domains [4]. The analysis under such case however is missing. It would be helpful to discuss the differences / similarities when there are multiple source domains exist.\n\n4. __Minor: no actual algorithms / application scenarios provided__\n- I understand that as a starting point, the paper only analyzed the behavior for CL across disparate domains. It would be nice to also give practical algorithms on how should one select (sub)sets of unlabeled data for better transfer. The analysis in the paper is good; however, I find it a bit disconnected with practical application scenarios.\n\n## Other issues / questions\n- Since the overall objective is agnostic to the data modality, is the proposed scheme also be extended to other data domains beyond images, like time series / texts? What would be the most challenging parts when extending to other modalities?\n- How much is the performance dependent on the quality/effectiveness of the augmentation set $\\mathcal{A}$?\n\n## References\n[1] SimCSE: Simple Contrastive Learning of Sentence Embeddings.\n\n[2] Representation Learning with Contrastive Predictive Coding.\n\n[3] Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding.\n\n[4] Divide and Contrast: Self-supervised Learning from Uncurated Data.", "summary_of_the_paper": "The paper studies how contrastive learning (CL) behaves in the unsupervised domain adaptation (UDA) setting. In particular, it finds that CL brings a different mechanism for UDA compared to traditional adversarial DA methods, where the features learned are far apart between the source and target domains. It further develops a conceptual model for explaining the success of contrastive learning under domain shifts, and empirically demonstrates through experiments on several benchmark datasets.", "summary_of_the_review": "Overall, the idea is interesting and intriguing, however there are still limitations and ambiguities in the current draft. The detailed comments/questions are listed in the weaknesses / questions part.\n\nThe paper has a great potential to the CL field, and could be beneficial and inspiring for broader audience; but issues need to be addressed / made clear. I'm happy to change my score if the feedback addresses my concerns. Please refer to the points in the weaknesses / questions part. I would like to see feedbacks on these comments/questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5khknPQU4Dr", "reviewer": "Reviewer_6UNN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is mainly about a finding that pretraining with SwAV ( a recently proposed self-supervised method) can benefit unsupervised domain adaptation tasks. This is done by pretraining on both source and target models.\n- Overall, the findings are interesting. But it would be more convincing if the authors report the results of initializing the backbone used in standard UDA tasks with pretrained self-supervised model (SwAV) in this case. So this would make comparisons with other approaches more fair. \n-  It would be more convincing if the authors experiment with other self-supervised models like MOCO, SIMCLR. I am curious whether the same conclusion would hold.\n- The presentation could be further improved---there are lots of typos...", "summary_of_the_paper": "The paper investigates why contrastive learning method benefits unsupervied domain adaptation.  In particular, the authors use SwAV, a recently proposed self-supervised learning method to pretrain a model on both targeted and source datasets. The authors found that by using such a model for UDA tasks surprisingly nice results are achieved. The authors analyze the reasons behind using a connectivity model.", "summary_of_the_review": "Overall, I think the observations in this paper are interesting. But I think more experiments would make the paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ornXvEk7F0", "reviewer": "Reviewer_tPmk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength and contribution of the paper is the analysis of the source-domain connectivity, which was interesting to read. It was well illustrated through a toy example, and then applied to some real-life domain adaptation datasets. The whole analysis seems sound and well-though.\n However the main two weakness are **(i)** there is a lack of clarity on the data augmentation used for contrastive learning; these seem to be key to define the whole notion of connectivity so it is a bit surprising they're not discussed in more details; And **(ii)** it's not clear how significant the reported work is and what it could lead to; In particular, one of the claims of the paper is that contrastive pretraining can reach similar accuracies as other UDA techniques while not aiming to align the domains as adversarial UDA techniques do. However there is not much comparison to these adversarial techniques in the second part of the paper; For instance, Figure 5 suggest that, while domain embeddings can be easily separated with contrastive pretraining, there is still some kind of \"domain connectivity\" which is really important for final accuracy, which seems to actually match the \"domain alignment\" goal from adversarial UDA techniques.\n\n\n**Some more precisions on parts of the paper I found a bit unclear:**\n  * (i) **Unclear definition of the positive and negative sets for contrastive learning.** It is briefly mentioned in Section 3.2 that the positive sets in contrastive learned are defined from multi-crop data augmentations, however it's not really well detailed, and in particular it's not clear how this impacts final accuracy (e.g. how much does the features quality degrade with simpler data augmentations ? how much tuning does it require ?)\n  * (ii) **Choices of datasets and baselines.** The experiments of Table 1 only seem to have one \"fair\" baseline (DirT/Sentry) because ERM is trained on the source data online. It would have been interesting to experiment on datasets such as Visda2017 or Office-31 that seem to be more commonly used in general and might have more benchmark results available. \n  * (iii) **Impact of data augmentation**: This links to points (i) and (ii): It's not clear to me how much the data augmentation strategy impacts the model, and whether it could also improve the baselines: Are Sentry and ERM baselines also trained with the same data augmentation as the contrastive learning techniques ?\n* (iv) **Readability of table 2** Table 2 and Section 4.1 would be stronger with more details on the classifiers used. For instance, if I understood correctly, the authors train a classifier to separate the source and target domain on either (i) the input images or (ii) the pretrained contrastive features. Because classifier (ii) as higher accuracy than (i), they conclude that the contrastive features \n\n**Small typos/unclear notations:**\n  * **Table 1, middle, second row**: the highest accuracy (Sentry) is not bold\n  * **Definition 1** Shouldn't the definition of connectivity include some kind of expectation over drawn samples $x$ and $x'$ ?", "summary_of_the_paper": "The authors tackle the problem of unsupervised domain adaptation (UDA), where most state-of-the-art methods aim to learn a joint embedding space for samples from the (labeled) source domain and (unlabeled) samples from the target domain. \n\nIn contrast to this, the authors propose to use contrastive learning pre-training techniques to first learn an encoder using target/source/additional data: Intuitively, this objective pushes positive samples together, and further away from other samples. The notion of \"positive set\" is defined as all samples which are obtained from data augmentations of a same sample natural image.\nA classifier is then learned on top of these features to solve the classification task on the labeled source dataset.\n\nUsing standard contrastive learning techniques, the authors show that they can achieve similar accuracies as some SoTA baselines on three UDA datasets. The remaining of the paper analyses the features learned by this techniques, and in particular, how they do not learn to align the two domains as is the common intuition in UDA. This analysis relies on a \"connectivity graph\" defined for contrastive learning techniques, which, in the current setting, can be used to connect samples from the source and target domain, to determine how likely they are to be in the same positive set.", "summary_of_the_review": "The main strength and contribution of the paper is the analysis of the source-domain connectivity, which was interesting to read. It was well illustrated through a toy example, and then applied to some real-life domain adaptation datasets. The whole analysis seems sound and well-though.\n However the main two weakness are **(i)** there is a lack of clarity on the data augmentation used for contrastive learning; these seem to be key to define the whole notion of connectivity so it is a bit surprising they're not discussed in more details; And **(ii)** it's not clear how significant the reported work is and what it could lead to; In particular, one of the claims of the paper is that contrastive pretraining can reach similar accuracies as other UDA techniques while not aiming to align the domains as adversarial UDA techniques do. However there is not much comparison to these adversarial techniques in the second part of the paper; For instance, Figure 5 suggest that, while domain embeddings can be easily separated with contrastive pretraining, there is still some kind of \"domain connectivity\" which is really important for final accuracy, which seems to actually match the \"domain alignment\" goal from adversarial UDA techniques.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper. While reviewers believe the findings from this paper may be of potential interest, they are fully convinced. For instance, if the authors want to claim the proposed mechanism is general for UDA, then they should demonstrate its effectiveness to other application domain(s), such as the NLP domain, where the pretrain-finetuning strategy is widely adopted for transfer learning. However, the authors did not provide correspondingly additional experiments as requested by a reviewer but claimed they only focused on the CV domain. If the focus is on the CV domain, then the authors need to explain in detail why in the CV domain, the proposed mechanism works well (while in other domains, it may not). There are many other concerns about the assumptions, experimental settings, etc. 

In summary, this is a borderline paper below the acceptance bar of ICLR.
Decision: Reject