HOW DOES CONTRASTIVE PRE-TRAINING CONNECT DISPARATE DOMAINS?
Year: 2021

Abstract:
Pre-training on massive unlabeled datasets greatly improves accuracy under distribution shifts. As a first step toward understanding this, we study a popular pre-training method, contrastive learning, in the unsupervised domain adaptation (UDA) setting where we only have labeled data from a source domain and unlabeled data from a target domain. We begin by showing on 4 benchmark vision datasets that out-of-the-box contrastive pre-training (even without large-scale unlabeled data) is competitive with other UDA methods. Intuitions from classical UDA methods such as domain adversarial training focus on bringing the domains together in feature space to improve generalization from source to target. Surprisingly, we find that contrastive pre-training learns features that are very far apart between the source and target domains. How then does contrastive learning improve robustness to distribution shift? We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart. We propose a new measure of connectivity —the relative connection strengths between same and different classes across domains—that governs the success of contrastive pre-training for domain adaptation in a simple example and strongly correlates with our results on benchmark vision datasets.

heading: 1 INTRODUCTION
text: In applications such as image recognition for self-driving cars (Yu et al., 2020; Sun et al., 2020) or medical image diagnosis (AlBadawy et al., 2018; Dai & Gool, 2018), machine learning models often fail on examples drawn from a different distribution than training. Pre-training on large-scale unlabeled data (Chen et al., 2020; He et al., 2020; Radford et al., 2021) can substantially boost accuracy under distribution shift (Radford et al., 2021; Hendrycks et al., 2019; Fisch et al., 2019; Yogatama et al., 2019; Devlin et al., 2019). For example, CLIP (Radford et al., 2021), which is trained on internet-scale image and text data, has shown impressive robustness benefits on ImageNet variants such as ImageNetV2. Similarly, pre-training methods such as BERT (Devlin et al., 2019) have improved robustness in NLP — for example, multilingual pre-training significantly improves performance on unseen languages (Liu et al., 2020). While other robustness methods that use unlabeled data such as self-training (Prabhu et al., 2021; Sohn et al., 2020; Berthelot et al., 2021) and domain adversarial training (Shu et al., 2018; Ganin et al., 2016) jointly optimize a labeled and unlabeled objective, pre-training is particularly suitable for large unlabeled datasets since we can generically pre-train once on unlabeled data and then specialize the model to downstream labeled data via fine-tuning. How does pre-training improve robustness to distribution shift when the pre-training method is not tailored to any downstream task?
In this paper, we run controlled experiments to understand the robustness benefits of a popular pre-training method, contrastive pre-training (Chen et al., 2020; He et al., 2020; Caron et al., 2020), even without large-scale unlabeled data. We directly apply contrastive pre-training to unsupervised visual domain adaptation (UDA), where unlabeled data from a shifted target domain is used to adapt models trained on labeled source dataset. We compare contrastive pre-training (SwAV (Caron et al., 2020)) to UDA methods such as SENTRY (Prabhu et al., 2021) (self-training) and DIRT-T (Shu et al., 2018) (domain adversarial training) where all methods use the same unlabeled data (source and target). We find that SwAV achieves comparable or better results on the DomainNet, BREEDS Living-17, and BREEDS Entity-30 benchmarks. Furthermore, we show that pre-training once on unlabeled data from many domains further improves the average target accuracy over the consituent source-target pairs, showing that extra unlabeled data improves contrastive pre-training for domain adaptation.
Given the strong performance of contrastive pre-training, we seek to understand why contrastive pre-training works for domain adaptation. Domain adversarial training and self-supervision algorithms are commonly based on the intuition that domains get merged in feature space (Tzeng et al., 2014; Ganin et al., 2016; Tzeng et al., 2017; Sun et al., 2019; Ben-David et al., 2010; Shu et al., 2018; Wang et al., 2021; Thota & Leontidis, 2021). This is support by theoretical notions such asH∆H-divergence, where domains that can be discriminated well and results in worse generalization bounds for target accuracy (Ben-David et al., 2010). A natural hypothesis is that contrastive pre-training brings domains together by bringing the representations of strong augmentations of the same image together (Sun et al., 2019).
Surprisingly, we find that the source and target domains are still far apart in the feature space learned by contrastive pre-training. A domain classifier trained on the pre-trained representations of strongly augmented images from the sketch and painting domains in DomainNet has 94% accuracy, which is even higher than the accuracy of a domain classifier on the original (augmented) input space (75%).
We study this phenomenon from the lens of the recently proposed augmentation graph (HaoChen et al., 2021), which forms connections between examples if they have similar augmentations (e.g., a small crop + grayscale can connect sketch and real photos, Figure 1). These connections are central to the contrastive objective, which seeks to bring neighbors in the graph closer together while pushing apart non-neighbors. From both simulations and real datasets, we define a critical connectivity quantity which seems to govern whether domain adaptation will be successful (Figure 1). In particular, the connection strength between the same classes across different domains should be relatively greater than the connection between different classes across different domains. Importantly, the absolute connectivity between domains can be small and the domains can be far apart in input space. This suggests that contrastive learning circumvents the separation of domains through the connectivity structure of the (general-purpose) augmentations used.
Using our understanding, we ablate contrastive learning by removing subsets of unlabeled data that contribute most to connecting the two domains. We do this by training a domain classifier to distinguish between the source and target and removing examples which are most uncertain under the classifier. This consistently worsens the target accuracy in comparison to randomly removing the same number of data points (by 2–3% on DomainNet), showing the importance of domain connectivity for adaptation.

heading: 2 SETUP
text: We consider a classification problem from an input spaceX ⊆Rd to a label spaceY=[K]. Data. Let PS and PT denote the source and target distributions of (x̄,y) pairs, respectively, where x̄∈X and y ∈Y . The labeled source dataset S consists of nS input-output pairs from PS . We also have access to an unlabeled target dataset T with nT unlabeled inputs from PT , and we assume the label sets for the source and target are identical. In some cases, we also consider an additional related datasetRwith nR unlabeled inputs from a related distributionR.
Model and metrics. The goal is to learn a classifier h :X →Y with low classification error on the target distributionL0−1(h)=Ex̄,y∼PT [1[h(x̄) 6=y]]. The pre-training algorithms we consider consist of two steps: first, we train an encoder function f :X →Z into a representation spaceZ⊆Rd′ (where d′ denotes the representation dimension), then we train a classification head g : Z →Y . The final classifier h=g◦f composes the encoder with the classification head.

heading: 2.1 CONTRASTIVE LEARNING
text: Contrastive learning (He et al., 2020; Chen et al., 2020; Caron et al., 2020) trains representations to be similar between augmented views of the same example and dissimilar between augmented views of random examples. Example image augmentations include random cropping, color jitter, and Gaussian blur (Chen et al., 2020). The basic assumption is that the semantic content of the data is invariant under the augmentations.
Though there are several variants of contrastive learning, we focus on the following objective from HaoChen et al. (2021) for our conceptual understanding. LetX denote the “augmented” space containing all data-augmented views of natural examples (e.g., all random crops of natural ImageNet images). Data augmentation is defined as a random functionA :X →X , where P[A(x)=x] denotes the probability that x∈X augments to x∈X . A pair of augmented views is called a positive pair and denotedx,x+ if a natural pointx is first drawn and thenx,x+ are the results of two calls toA(x). On the other hand, a negative pair x,x− arises from two natural data points x,x′ when x=A(x), x−=A(x′). Minimizing the following loss function simultaneously brings together the representations of positive pairs x,x+ and repels the representations of negative pairs x,x−:
L(f)=−2·Ex,x+ [ f(x)>f(x+) ] +Ex,x− [( f(x)>f(x−) )2] . (1)
Here the first expectations are taken over the marginal distribution overX and the randomness ofA.

heading: 2.2 CONSTRASTIVE PRE-TRAINING FOR DOMAIN ADAPTATION
text: Let D denote the unlabeled input data used for contrastive pre-training (e.g., some combination of the source unlabeled data S, target unlabeled data T , and/or additional unlabeled data R). We use the following simple algorithm to apply contrastive pre-training to domain adaptation:
1. (Pre-train) Learn an encoder function f :X →Z via contrastive learning onD.
2. (Fine-tune) On labeled source data S, train a classifier h,g◦f :X →Y as the composition of the pre-trained encoder f and a classifier head g. In this paper, we take “fine-tuning” to mean updating both the parameters of f and g.

heading: 3 CONTRASTIVE PRE-TRAINING IS A STRONG DOMAIN ADAPTATION METHOD
text: We evaluate contrastive learning and relevant baselines on 4 benchmark vision datasets and observe that contrastive learning achieves comparable or better performance in all cases.

heading: 3.1 DATASETS
text: BREEDS (Santurkar et al., 2020). BREEDS is a subpopulation shift benchmark derived from ImageNet by constructing a hierarchical tree structure of classes from WordNet. Nodes at a specified depth of the tree become the labels for the classification task, and descendant nodes are treated as subpopulations that can be randomly partitioned into source and target domains.
DomainNet (Peng et al., 2019). DomainNet is a large unsupervised domain adaptation task, consisting of approximately 600,000 images and 345 classes in 6 domains. For our experiments we utilize the same filtered version of DomainNet from Prabhu et al. (2021), which uses 40 of the 345 classes and the sketch, painting, photograph, and clipart domains.
STL-10→CIFAR-10. STL-10 and CIFAR-10 are two classic image classification datasets, each consisting of 10 classes (Coates et al., 2011; Krizhevsky, 2009). Each dataset has a class label that is not present in the other, and hence we follow the procedure of French et al. (2018) and filter out the examples in the non-overlapping classes, resulting in a 9-class classification problem.

heading: 3.2 METHODS
text: Baselines. We run empirical risk minimization (ERM) on the source dataset for each task, i.e., a “source-only” baseline. In addition, for BREEDS and DomainNet we use these ERM models as weight initialization for SENTRY, a self-training algorithm for domain adaptation that achieved state of the art results on several domains of DomainNet when initialized with ImageNet-pretrained models (Prabhu et al., 2021). For STL→ CIFAR, we use Dirt-T (Shu et al., 2018), the current state of the art algorithm for this task.
Contrastive pre-training. For BREEDS and DomainNet we use SwAV (Caron et al., 2020), a contrastive learning algorithm that achieved state-of-the-art linear evaluation accuracy on ImageNet. SwAV contrasts examples in feature space rather than input space and uses a novel multi-crop data augmentation strategy using several crops of different sizes. For the lower-resolution task STL→ CIFAR we use SimCLR (Chen et al., 2020), which uses only 2 crops and therefore scales straightforwardly to smaller input dimensions.

heading: 3.3 RESULTS
text: Main comparison. For all 4 tasks, contrastive pre-training is competitive when compared to ERM and SOTA unsupervised domain adaptation baselines. On DomainNet (top of Table 1), contrastive pre-training on source and target unlabeled data achieves higher target accuracy than all joint-training baselines both on 5 of 12 domain pairs and on average over all pairs (44.91% vs. the second best, 43.99%). On Living-17 (middle of Table 1), contrastive learning on source and target unlabeled data improves over SENTRY over 4%, while on Entity-30, contrastive learning (source + target) is within 0.3% of the target accuracy of SENTRY. On STL→CIFAR, SimCLR on the target unlabeled data achieves 0.8% higher target accuracy than Dirt-T.
Extra unlabeled data from related domains. We also consider adding extra unlabeled data from other (related) domains in DomainNet, Living-17, and Entity-30. In particular, we can pre-train once on an unlabeled dataset that consists of a superset of the domains we want to adapt to, and fine-tune on the available source data. While this is not a fair comparison to other domain adaptation methods, the ability to scale to large unlabeled datasets is a natural advantage of pre-training.
On DomainNet, we consider contrastive pre-training with SwAV on unlabeled data from all the domains at once, which we denote as SwAV + extra. The top Table 1 shows that adding extra unlabeled data improves the target accuracy of contrastive pre-training on all pairs and improves the average by almost 6% (44.91% to 50.75%). On Living-17 and Entity-30 (middle of Table 1), we additionally consider contrastive pre-training with SwAV with 1) only source, 2) only target, and 3) with all of ImageNet as the unlabeled data. We find that increasing the amount of unlabeled data steadily increases the target accuracy, and pre-training on all of ImageNet improves over source + target by 8% and 4% respectively for Living-17 and Entity-30.
Ease of hyperparmeter tuning. The SENTRY self-training baseline requires extensive hyperparameter tuning (e.g., finding optimal balancing coefficients for the joint losses, or learning rate schedules) for every transfer task and is difficult with access only to source labels. In contrast to the baseline methods, for contrastive pre-training we did not need to sweep over hyperparameters or use target labels for model selection. We used nearly identical hyperparmeters for both BREEDS datasets (Living-17 and Entity-30) and all DomainNet pairs (details in Appendix A).
No explicit knowledge of distribution shift. We note that SENTRY uses explicit knowledge of label imbalance within its algorithm, on top of being designed for the domain adaptation task of adapting from source to target. In contrast, we used off-the-shelf contrastive pre-training, which was designed for general representation learning, with no additional information about distribution shift.

heading: 4 CONNECTIVITY MODEL FOR DOMAIN ADAPTATION
text: How does contrastive pre-training improve target accuracy with unlabeled data? Conventional intuitions and theory for domain adaptation are based on reducing the distance between the source and target domains (Ben-David et al., 2010; Ganin et al., 2016).However, we see empirically that contrastive pre-training keep the domains very separated in representation space. Towards building an understanding for how contrastive pre-training connects domains without merging them, we extend a conceptual framework based on connectivity via data augmentations (HaoChen et al., 2021) for distribution shifts. We define a crucial measure of domain connectivity that relates properties of the data distribution and contrastive augmentations to the success of contrastive learning for domain shifts on a simulated data model, DomainNet, Living-17, and Entity-30.

heading: 4.1 CONTRASTIVE PRE-TRAINING KEEPS DOMAINS APART
text: Contrary to common intuitions for domain adaptation methods, we find that contrastive learning does not bring the domains together in feature space. Table 2 shows the average test error of classifiers trained to discriminate between examples from the same class but different domains on DomainNet. We train and test the classifiers on images using the same augmentations as SwAV and find that contrastive pre-training learns features that are actually more discriminable between domains than in the original input space (7.5% vs. 27.3% error on average over all pairs). These estimates for the baselines and for individual pairs are provided in table 3.
Next, we measure the separability in feature space and input space between classes within a single domain (e.g., 3% vs. 24% error in the real domain, table 3). We note that the separability between domains in feature space is comparable to the separability between classes, suggesting that contrastive learning may cluster both domains and classes in separate axes.

heading: 4.2 USING CONNECTIVITY TO CHARACTERIZE DOMAIN SHIFT
text: In this section, we set up our conceptual model of connectivity used to study contrastive learning.
Connectivity via augmentations in domain adaptation. HaoChen et al. (2021) proposed viewing contrastive learning as operating on an augmentation graph. The augmented inputs fromX constitute the vertices and the edge weight between x,x′ represents how likely they are to be generated from the same natural example (i.e., how likely they are to be chosen as a positive example during contrastive pre-training). HaoChen et al. (2021) show that when examples from different classes x,x′ ∈X are sparsely connected via the augmentation graph (i.e., rarely augment into the same augmented example), and the connectivity within a class is strong, the contrastive pre-training leads to features that obtain good accuracy on the downstream task.
Extension to domain adaptation. Motivated by the augmentation graph framework, we define a notion of connectivity for domain adaptation on a graph where class-domain pairs are vertices. We denote the set of examples belonging to class c and domain d (which we refer to as the class-domain pair (c,d)) as P(c,d). We formalize this notion in the following definition:
Definition 1 (Connectivity). The connectivity between two class-domain pairs (i,d) and (j,d′) is
conn((i,d),(j,d′)),P[A(x)=A(x′)],
where x∼P(i,d) and x′∼P(j,d′).
Intuitively, the connectivity between two class-domain pairs is the probability that two randomly sampled points from each class-domain pair are augmented to the same point.
Definition 1 is defined for two arbitrary class-domain pairs, which then allows us to build further definitions of connectivity involving multiple class-domain pairs. Such connectivity definitions would allow us to more rigorously describe (for instance) our observation in the DomainNet dataset that points of the same class but different domains are more discriminable than points in different classes within the same domain. We define these instances of connectivity using Definition 1 below:
Definition 2 ((α,β,γ)9 connectivity). A source-target pair of distributions S and T over X (with domain labels dS , dT , respectively) satisfies (α,β,γ)9connectivity if
1. Ei∈[K][conn((i,dS),(i,dT ))]=α. (same-class-different-domain)
2. Ei,j∈[K][conn((i,d),(j,d))]=β. (different-class-same-domain, d∈{dS ,dT } )
3. Ei,j∈[K][conn((i,dS),(j,dT ))]=γ. (different-class-different-domain)

heading: 4.3 ILLUSTRATIVE TOY EXAMPLE: BINARY CLASSIFICATION
text: As an illustrative example, consider the simple case of binary classification where each class-domain pair consists of a single data point. This toy setting pinpoints a simple connectivity condition such that linear classification on the pre-trained features results in good target accuracy.
Setting (Figure 2). Assume a uniform marginal distribution over the natural inputs, and suppose data augmentation transforms each data point directly into another with a certain probability, so that the entire augmentation graph consists of the 4 points (2 classes, 2 domains) each identified by their class and domain. Formally, the example (i,d) (of class i and domain d) has augmentation given by
A((i,d))=  (i,d) with probability ζ (i,d′) with probability (α/2ζ) (j,d) with probability (β/2ζ) (j,d′) with probability (γ/2ζ)
where j 6= i, d′ 6=d. We also assume ζ >max{α,β,γ} (i.e., data augmentation is most likely to map each data point to itself) and appropriately constrain α+β+γ= 2ζ(1−ζ) so that the probabilities sum to 1. Note that the specific transition probabilites are selected so that the distribution shift satisfies (α,β,γ)9connectivity. We also number the points 1−4 in the following order: class 1 (source), class 1 (target), class 2 (source), class 2 (target).
0 5 10 15 Same Class, Different Domain Connectivity
0.0
0.2
0.4
0.6
0.8
1.0
Ta rg
et T
es t A
cc ur
ac y
Figure 3: Scatter plot of random graph simulations, where ζ = 16,β = 4, and γ=2. Shown on the x-axis is α and nontrivial target accuracy is not possible until α > γ. Target accuracy also increases with the ratio α/β.
Pre-training. HaoChen et al. (2021) show that minimizing the spectral contrastive loss (given in Eq. 1) is equivalent to computing the spectral decomposition of the adjacency matrix of the population augmentation graph.The leading eigenvectors of the resulting decomposition are then used as the learned representations, so that the representation of example i is the concatenation of the ith components of those eigenvectors. Because the population here is finite, we compute the spectral decomposition algebraically and express the learned representations as a function of connectivity. Agnostic of the ordering of the relevant parameters, the unordered eigenvectors of this toy example are:1
v1 =[ 1 1 1 1] > , v2 =[ 1 −1 1 −1] > , v3 =[ 1 1 −1 −1] > , v4 =[ 1 −1 −1 1] >
with associated eigenvalues
λ1 =ζ+β+α+γ, λ2 =ζ+β−α−γ, λ3 =ζ−β+α−γ, λ4 =ζ−β−α+γ We discard the eigenvector v1, as it assigns the same feature to every point, and use the top 2 remaining eigenvectors as the learned features. For example, if v2 and v3 are the top remaining eigenvectors, then the features for data point 2 would be the 2nd component of v2 and v3 (namely, (−1,1)).
Fine-tuning. We consider learning a linear classifier upon the pre-trained features. The middle panel of Figure 2 illustrates the fortunate case, in which α>γ and β>γ (but possibly α≤β). In this case, λ2>λ4 and λ3>λ4 and therefore the learned features use v2 and v3 and are (1,1),(1,−1) for class 1 and (−1,1),(−1,−1) for class 2. Then, a linear classifier trained on the source uses only the class information, which is contained in the second index, and labels the target accurately.
Note that the source classifier labels the target correctly if and only if it uses solely the features given by v3 for prediction. Two possible failure modes occur when α is smallest (and therefore v3 is discarded) and when β is smallest (and therefore the source classifier can use features from either v3 or v4).
Importantly in this toy setting, whether or not the features of the target examples are oriented correctly depends only on whether α and β are both >γ. Consequently, the classes within each domain can be very connected–more so than across domains–and target accuracy can be high. To illustrate the importance of this connectivity ratio on a more fine-grained setting, we run simulations and observe sharp thresholding behavior when γ surpasses α (Figure 3, simulation details in Appendix C).

heading: 4.4 CONNECTIVITY GOVERNS THE SUCCESS OF PRE-TRAINING FOR DOMAIN ADAPTATION
text: Connectivity on benchmark datasets. We estimate the input space domain connectivity on DomainNet, Living-17, and Entity-30 by training classifiers between various pairs of class-domain pairs (test errors provided in Table 2, full details provided in Section B). We find that all the datasets satisfy α>γ–the necessary condition for the good case in the toy example–but α<β for DomainNet while α>β for BREEDS. Intuitively, on natural distribution shifts α is likely to be higher than γ, since the data augmentation should be more likely to connect class-domain pairs where only the domain differs than to bridge differences across both the domain and the class.
1We leave the eigenvectors unnormalized for ease of exposition; the normalized variants would multiply the entire vector by 1/2.
Domain and class classifiers are nearly orthogonal. We check the extent to which the domain and class information are separated by training 3 linear classifiers on the pre-trained representation space: 1) Source classifier (which discriminates classes in source) 2) Target classifier (which discriminates classes in target)2 3) Domain classifier (discriminates between source or target domains). We compare the linear weights of these classifiers by computing the cosine similarity. We find that
1. The source and target classifiers are very similar; on average over the classes, the cosine similarity of linear weights for the same class from the source and target classifiers is high (around 0.20 for DomainNet, 0.34 for Living-17).
2. The domain classifier is nearly orthogonal to the source and target classifiers. The average cosine similarity between the weights of any class from the source or target classifiers and the weights of the domain classifier is 0.01.
This suggests that the class information and domain information are contained in different directions of the feature space, and gives an explanation for how both classes and domains can be easily separated in feature space while maintaining the ability to generalize from source to target. Learning to discriminate classes results in a classifier with weights that are nearly orthogonal to the weights for domain classification, suggesting a natural amount of ‘domain agnosticity” despite not removing domain information from the features. Detailed results on individual pairs are provided in Appendix D.
Connectivity ratios correlate with target accuracy. We construct a scalar connectivity quantity and find that it correlates with target accuracy of contrastive learning on DomainNet. In the simple example, contrastive pre-training for domain adaptation works when α > γ and β > γ. The first condition (α>γ) is relatively more important since if β (connectivity between different classes in the same domain) is very high, then the classes within a domain are hard to discriminate and the classification task itself will be hard. Thus, we define the quantity
(α/γ)·(β/γ) (2)
where we take = 0.1 to “discount” the importance of the second ratio. We multiply the two ratios to express the logical “and”. The larger this quantity is, the better the target accuracy should be.
Figure 4 shows a plot of the quantity with target accuracy of contrastive learning on DomainNet pairs. Because different target domains can have different intrinsic errors, we “debias” the target accuracy of each source-target pair by subtracting the worst accuracy over all pairs with the same target domain. We find a strong correlation between the connectivity and target accuracy (Pearson r of 0.87 for S+T pre-training) (Figure 4).
Ablating connectivity. We find that the connectivity between examples of the same class but different domain is also very important, as those examples “bridge” the domains. We verify this intuition by systematically ablating the connectivity on a subset of Living-17 by removing those examples that contribute most to domain connectivity. Specifically, for each class we trained discriminators between the source and target examples of that class and removed the n pre-training data points from each
2Although this cannot be done in practice, we only use the target labels here for exploratory analysis;
class-domain pair on which the discriminator was least confident; these examples can be seen as the ones contributing most to domain connectivity. For subsample sizes of {1176,2352,5880,11760} we consistently reduce target accuracy as compared to random subset removal (Figure 5; confidence subsampling increases error relatively by 6% on average compared to random).

heading: 5 RELATED WORK
text: Domain adaptation. Ben-David et al. (2010) prove generalization bounds in the distribution shift setting. Their bounds rely on a notion of distance between two data distributions called H∆H divergence being small, which many methods such as domain adversarial training are inspired by (Ganin et al., 2016; Tzeng et al., 2014; 2017). Our conceptual model based around connectivity provides a way to explain how contrastive pre-training can work for domain adaptation even when theH∆H divergence between domains is large.
Domain adaptation with self-supervision. Prior works have explored the application of selfsupervision to domain adaptation. Sun et al. (2019) propose optimizing on hand-crafted self-supervised tasks such as predicting the angle of rotation of a rotated image. These tasks are jointly optimized on the source and target data along with the labeled source loss. As part of their comparisons, they found that first pre-training on target (or source and target) data and then fine-tuning on source data separately (instead of jointly) was highly non-competitive with their other methods. In contrast, we find that pre-training with a general, off-the-shelf contrastive learning objective is competitive with other state-of-the-art domain adaptation techniques.
Contrastive objective for domain adaptation. Contrastive objectives have also been applied to domain adaptation problems. Wang et al. (2021) jointly train the contrastive loss on source and target data. These methods rely on explicit anchor pairs that encourage domain alignment (Wang et al., 2021), or still employ some form of explicit domain alignment (Thota & Leontidis, 2021). We consider pre-training instead of joint training and do not use any domain knowledge about the distribution shift problem.
Self-training. Berthelot et al. (2021) use self-training and consistency regularization with a distribution alignment method, but require estimating the target label distribution and tuning a confidence thresholding parameter (tuned using target test labels). Cai et al. (2021) give a label propagation analysis for self-training in domain adaptation.They have only one parameter which governs both the different-class-same-domain and different-class-different-domain connections, requiring both to be small. In our framework, we allow the different-class-same-domain connectivity to not be small, in accordance with our observation that domains can be well separated in contrastive feature space. They give experiments on self-training when operating on a contrastive pre-trained representation space, but their focus is on self-training while the pre-trained representations are an empirical detail. One inflexible characteristic of standard self-training is that it relies on the unlabeled data having the same classes as the labeled data (in order to pseudolabel unlabeled points), which may not allow them to utilize diverse sources of extra unlabled data like many pre-training methods can.

heading: 6 CONCLUSION
text: In this work we focused on evaluating and understanding contrastive pre-training under domain adaptation. Pre-training has the advantage of amortizing the cost of processing a large unlabeled dataset into a pre-training step that is done only once. While pre-training loses the advantage of seeing labeled data (and thus the downstream task) to adapt its usage of unlabeled data, we find that contrastive pre-training is still competitive with other domain adaptation algorithms. We hope that our connectivity model can give insights to improve pre-training data selection, develop better augmentations for pre-training with distribution shift in mind, and improve fine-tuning methods that exploit the properties of the pre-trained feature space.
ETHICS
Our experiments use standard benchmark vision datasets in domain adaptation, which is publicly available data. In general, however, large-scale unsupervised learning can involve data scraped from the internet which may lead to privacy concerns. Care should be taken when curating datasets for large scale contrastive pre-training in practice.
REPRODUCIBILITY We detail the hyperparameters and experimental details in the paper and appendix. We have uploaded a zipped code archive and will publish the code on Github upon acceptance.

heading: A ADDITIONAL EXPERIMENTAL DETAILS
text: A.1 DATASETS BREEDS. We use the dataset creation functions defined in the robustness Python library to generate the Living-17 and Entity-30 tasks from the original ImageNet dataset (Engstrom et al., 2019; Russakovsky et al., 2015). The Living-17 dataset is an animal classification task which consists of nodes in the subtree rooted at the “living thing” node in the WordNet hierarchy. An example of a label is “ape” with subpopulations of gibbons, orangutans, gorillas, and chimpanzees. The Entity-30 dataset is a much more general task, incorporating nodes in the “entity” subtree. Labels include “building” and “home appliance”. The trailing number in the dataset name is the total number of classes in that dataset.
DomainNet. We use the official SENTRY repo at https://github.com/virajprabhu/ SENTRY, which filters the original DomainNet dataset automatically as described in Section 3. This refinement is done to eliminate much of the noise present in the original DomainNet dataset (Tan et al., 2020).
STL→ CIFAR. CIFAR-10 consists of 32×32 images from the former TinyImages dataset, and STL-10 is derived from ImageNet and contains images with resolution 96×96. We resize the STL-10 images to 32×32 to match the resolution of CIFAR10. The two non-overlapping classes (“monkey” in CIFAR-10 and “frog” in STL10) are removed from both datasets before training.
A.2 BASELINES AND HYPERPARAMETER TUNING ERM (source-only).
• BREEDS: We use the same hyperparameters as Santurkar et al. (2020). For Entity30, we train for 300 epochs and divided the learning rate by 10 every 100 epochs. On Living17, we train for 450 epochs and divided the learning rate by 10 every 150 epochs. For both datasets we use a ResNet50 with a learning rate of 0.1, a weight decay of 10−4, and a batch size of 128.
• DomainNet: The SENTRY algorithm runs ERM with class balancing (starting with ImageNet-pretrained initialization) prior to beginning entropy minimization, and therefore the SENTRY repository contains an ERM implementation and hyperparameters for DomainNet. We run ERM for 150 epochs and multiply the initial learning rate by 10x, keeping all other hyperparameters constant.
• STL10→CIFAR10: We report the exact results from Shu et al. (2018).
DIRT-T. DIRT-T (Shu et al., 2018) is a domain adaptation method that addresses two flaws of domain adversarial neural networks (Ganin et al., 2016): 1) distribution matching is a weak constraint, and 2) in some domain adaptation settings there does not exist a good joint classifier on both source and target. The authors address the first shortcoming by adding a conditional entropy regularization term so that the model’s decision boundaries do not overlap high-density regions of data. This is inspired by the cluster assumption, which states that the input space is divided into well-separated clusters, one for each class in the label space. The lack of a good classifier on both source and target is then addressed via self-training on the unlabeled target data. We report the STL10→CIFAR10 results from Shu et al. (2018).
SENTRY. For each pair of domains on DomainNet we conduct a hyperparameter search through λsrc∈{0.5,1.0,1.5} (the weight on the supervised classification loss) and learning rates∈{0.01,0.001} and report the model results with the highest source evaluation accuracy.
On BREEDS we keep the same default hyperparameters from the SENTRY repo and search over 3 learning rates {0.001,0.01,0.1}, choosing the model with the highest target evaluation accuracy to report.
SENTRY underperforms the ERM models when the source domain is clipart images (from DomainNet), which could be the result of not using ImageNet-pretrained classifiers for initialization as Prabhu et al. (2021) do. Therefore, as the joint-training baseline for DomainNet we take the maximum of ERM and SENTRY.
A.3 SIMCLR We use the official SimCLR repository from Google for our experiments (https: //github.com/google-research/simclr). We use a ResNet18 with a batch size of 256, a learning rate of 0.2, and weight decay 10−4. The projection head has two layers and an output
dimension of 64. We pre-train the model for 400 epochs with square-root learning rate scaling and we train the linear probe for 100 epochs on batches of size 512 and a learning rate of 0.1.
A.4 SWAV We use the public SwAV repository available athttps://github.com/facebookresearch/ swav and kept almost all of the hyperparameters provided by the original paper for 400 epoch, 256-batch-size training on ImageNet. However, we used a batch size of 512 and additionally made the following changes based on the Github issues answered by the original authors:
1. We avoid using a queue on DomainNet and the subsampled variant of Living-17 in order to stabilize training. For pre-training on the full Living-17 and Entity-30 datasets, we introduce the queue at epoch 60.
2. We set the number of prototypes to be 10 times the number of classes (170, 300, and 400 for Living-17, Entity-30 and DomainNet, respectively).
3. We set =0.03.
4. We set the base learning rate to 0.6, following a linear scaling rule based on batch size.
We always fine-tuned from the final iterate of SwAV pre-training (400 epochs).
On DomainNet, we fine-tuned SwAV models for 50 epochs using the ERM implementation in the SENTRY repository (without running any joint-training algorithm), keeping all hyperparameters other than the number of epochs constant. We report the target test accuracy of the final model (after 50 epochs).
On BREEDS Living-17, we fine-tuned SwAV models for 100 epochs with a cosine learning rate schedule without restarts. We use SGD with initial learning rate 0.1 for the classifier head and 0.01 for the encoder, momentum 0.9, and weight decay 0.0001. We use a batch size of 96, and once again report the target test accuracy of the final model (after 100 epochs).
On BREEDS Entity-30, which contains 300K examples, we instead linear probe.
B RESULTS AND PROTOCOL FOR ESTIMATING (α,β,γ)9CONNECTIVITY Table 3 reports the connectivity estimates for the input space and the feature spaces learned by ERM, DANN, and SwAV for all pairs of DomainNet domains.
To estimate the connectivity between 2 class-domain pairs (i,d) and (j,d′), we use the following algorithm:
1. Label all training examples of class and domain (i,d) as 0 and all training examples of (j,d′) as 1. Discard the remaining examples.
2. Train a ResNet50 for 100 epochs using SwAV augmentations and cosine learning rate. We did not exhaustively tune this training step, but we kept the hyperparameters constant for all the pairs.
3. Collect the test set analogously to step 1, and evaluate the classifier on augmented data.
Each domain in DomainNet-S has a unique label distribution (all of which are far from uniform), and therefore in computing the average connectivity we compute the weighted mean, where each pair of (class, domain) pairs is weighted by the ratio of the less to more frequent label (0 or 1).

heading: C SIMULATION DETAILS
text: To obtain an even more fine-grained picture, we run simulations on toy data settings, where we can precisely vary each type of connectivity, and observe:
1. Sharp thresholding behavior when α and γ cross over.
2. αmust be nontrivial.
The simulation is on a finite population so that we can study the spectral decomposition directly; however, later we show empirically that these ideas still hold in real-world, continuous-population settings. For each domain-class pair dc, we generateN points in augmentation space. These augmented points inherit the domain-class assignments of their corresponding natural points (we have to ensure that all
the augmented points are distinct). Then for any new natural point x∈X that we want to augment, we simply choose one of our existing augmented points x at random and designate x as the augmentation of x. Since we have assigned domain-class memberships to this base set of augmentations, we can define the connectivity of two domain-class pairs (d1c1,d2c2) to be the probability that a natural point from d1c1 is “augmented into” an augmented point in d2c2 (and vice-versa). This probability is something we can easily change in the simulations to vary this notion of connectivity.

heading: D ADDITIONAL DETAILS FOR SECTION 4.4
text: As another method for evaluating the connectivity measures (in addition to Figure 4), we inspect the weights learned from a linear regression model from 3 features [α,β,γ] to the target accuracy (de-biased). The weights are [−6.94,5.61,0.87], which gives a large negative weight to α (different-
class-different-domain connecitivity), large positive weight to γ (same-class-different-domain), and a small positive weight to β (different-class-same-domain) as expected from our intuitions.
The following table contains more detailed results on the cosine similarity between domain and class classifiers.

Review summaries:
The authors tackle the problem of unsupervised domain adaptation (UDA), where most state-of-the-art methods aim to learn a joint embedding space for samples from the (labeled) source domain and (unlabeled) samples from the target domain.   In contrast to this, the authors propose to use contrastive learning pre-training techniques to first learn an encoder using target/source/additional data: Intuitively, this objective pushes positive samples together, and further away from other samples. The notion of "positive set" is defined as all samples which are obtained from data augmentations of a same sample natural image. A classifier is then learned on top of these features to solve the classification task on the labeled source dataset.  Using standard contrastive learning techniques, the authors show that they can achieve similar accuracies as some SoTA baselines on three UDA datasets. The remaining of the paper analyses the features learned by this techniques, and in particular, how they do not learn to align the two domains as is the common intuition in UDA. This analysis relies on a "connectivity graph" defined for contrastive learning techniques, which, in the current setting, can be used to connect samples from the source and target domain, to determine how likely they are to be in the same positive set.

The paper investigates why contrastive learning method benefits unsupervied domain adaptation.  In particular, the authors use SwAV, a recently proposed self-supervised learning method to pretrain a model on both targeted and source datasets. The authors found that by using such a model for UDA tasks surprisingly nice results are achieved. The authors analyze the reasons behind using a connectivity model.

This paper proposes to study the contrastive pre-training on domain adaptation tasks. The key contribution is to design a connectivity model for better data selection and data augmentations of pre-training. To verify the benefit of contrastive pre-training to domain adaptation, experiments are performed on a variety of benchmark tasks including BREEDS, DomainNet and CIFAR-10.

The paper studies how contrastive learning (CL) behaves in the unsupervised domain adaptation (UDA) setting. In particular, it finds that CL brings a different mechanism for UDA compared to traditional adversarial DA methods, where the features learned are far apart between the source and target domains. It further develops a conceptual model for explaining the success of contrastive learning under domain shifts, and empirically demonstrates through experiments on several benchmark datasets.