ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations
Authors: Xuyang Zhao
Tianqi Du
Yisen Wang
Jun Yao
Weiran Huang
Venue: ICLR 2023 poster (2023)
Keywords: Contrastive learning
out-of-distribution generalization

Abstract: Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shift scenarios, where the downstream and training distributions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transferability of self-supervised contrastive learning, by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of contrastive learning depends largely on the choice of data augmentation.  Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn domain-invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments on several datasets and show that ArCL significantly improves the transferability of contrastive learning.

Published as a conference paper at ICLR 2023
ARCL: ENHANCING CONTRASTIVE LEARNING WITH JUGMENTATION-ROBUST REPRESENTATIONS
Xuyang Zhao 1.2 Tianqi Du 1.3+ Yisen Wang 3.4
Jun Yaod Weiran Huang?!
School of Mathematical Sciences, Peking University
2 Qing Yuan Research Institute. Shanghai Jiao Tong University
3 National Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University Institute for Artificial Intelligence, Peking University
Huawei Noah's Ark Lab
ABSTRACT
Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising perfor- mance distribution shift scenarios, where the downstream and training distribu- tions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transfer- ability of self-suuer contrastive learning. by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of con- trastive learning depends largely on the choice of data augmentation. Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called sugmentation-robut Contrastive Learning (ArCL), which guaran- tees to learn domain-invarian invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments several datasets and show that ArCL significantly improves the transferability of contrastive learning.
INTRODUCTION
common assumption in designing machine learning algorithms is that training and test samples are drawn from the same distribution. However, this assumption may not hold in real-world applications, and algorithms may suffer from distribution shifts, where the training and test distributions differ. This issue has motivated plethora of research in various settings, such as transfer learning, domain adaptation and domain generalization (Blanchard et al., 2011; I; Muandet et al., 2013; Wang et al., 2021a la; Shen et al., 2021). Different ways of characterizing the relationship between test and training distributions lead to different algorithms. Most literature studies this in the supervised learning scenario. It aims find features that capture some invariance across different distributions, and assume that such invariance also applies to test distributions (Peters et al., 2016; Rojas-Carulla Carulla et al., 2018; Arjovsky et al., 2019; Mahajan al., 2021; Jin et al., 2020; Ye et al., 2021).
Self-Supervised Learning (SSL) has attracted great attention in many fields (He et al., 2020; Chen al., 2020; Grill et al., 2020; Chen & He, 2021; Zbontar et al., 2021). It first learns a representation from large amount of unlabeled training data, and then fine-tunes the learned encoder to obtain a final model on the downstream task. Due to its two-step nature, SSL more likely to encounter the distribution shift issue. Exploring its transferability under distribution shifts has become an important topic. Some recent works study this issue empirically (Liu et al., 2021; Goyal et al., 2021; von Kügelgen al., 2021; Wang et al., 202 Shi et al., 2022). However, the theoretical understanding is still limited, which also hinders the development of algorithms.
In this paper, we study the transferability of self-supervised contrastive learning in distribution shift scenarios from a theoretical perspective. In particular, we investigate which downstream distribu-
"Equal Contribution. This work was partially done when Xuyang was visiting Qing Yuan Research Institute. Cooresponneece Weiran Huang (weiran haarrannhuangge @ outlaok.com).

---

Published as a conference paper at ICLR 2023
tions will result in good performance for the representation obtained by contrastive learning. We study this problem by deriving a connection between the contrastive loss and the downstream risk. Our main finding is that data augmentation is essential: contrastive learning provably performs well on downstream tasks whose distributions are close to the augmented training distribution.
Moreover, the idea behind contrastive learning is to find representations that invariant under data augmentation. This is similar to the comain-invariaan since applying each kind of augmentation to the training data can be viewed as inducing a specific domain. Unfortunately, from this perspective, we discover that contrastive learning fails to produce Nomain-invariann representation, limiting its transferability. To address this issue, we propose new method called Augmentation-robee various widely used contrastive learning algorithms, such as SimCLR (Chen et al., 2020) and MoCo (He et al., 2020). In contrast to the standard contrastive learning, ArCL. forces the representation to align the two farthest positive samples, and thus provably learns domain-invarian representations. We conducted experiments various downstream tasks to test the transferability of representations learned by ArCL on CIFAR 10 and ImageNet. Our experiments demonstrate that ArCL significantly improves the standard contrastive learning algorithms.
based supervised learning methods,
Contrastive Learning (ArCL), which can be integrated with
RELATED WORK
Distribution shift in supervised learning. Distribution shift problem has been studied in many literature (Blanchard et al., 2011; Muandet et al., 2013; Wang et al., 2021a; Shen et al., 2021). ). Most works aim to learn representation that performs well on different source domains simultaneously (Rojas-Carulle et al., 2018; Mahajan et al., 2021; Jin et al., 2020), following the idea of causal in- variance (Peters et al., 2016; Arjovsky al., 2019). Structural equation models are often assumed for theoretical analysis (von Kügelgen et al., 2021; Liu et al., 2020; Mahajan et al., 2021). Distribu- tionally robust optimization optimizes a model's worst-case performance over some uncertainty set directly (Krueger et al., 2021; Sagawa et al., 2019; Duchi & Namkoong, 2021; Duchi et al., 2021). Stable learning (Shen et al., 2020; Kuang et al., 2020) learns a set of global sample weights that could remove the confounding bias for all the potential treatments from data distribution. Disentan- gled representation learning (Bengio et al., 2013; Träuble et al., 2021; Kim & Mnih, 2018) aims to learn representations where distinct and informative factors of variations in data are separated.
Theoretical understanding of contrastive learning. A number of recent works also aim to theo- retically explain the success of contrastive learning in IID settings. One way to explain it is through the mutual information between positive samples (Tian et al., 2020; Hjelm al., 2018 8; Tschannen et al., 2019). Arora et al. (2019) directly analyze the generalization of InfoNCE loss based on the assumption that positive samples are drawn from the same latent classes. In the same setting, Bao et al. (2022) establish equivalence between InfoNCE and supervised loss and give sharper upper and lower bounds. Huang al. (2021) take data augmentation into account and provide generalization bounds based on the nearest neighbor classifier.
Contrastive learning in distribution shift. Shen et al. (2022) and HaoChen et al. (2022) study contrastive learning in unsupervised domain adaptation, where unlabeled target data are obtained. Shi et al. (2022) show that SSL is the most robust on distribution shift datasets compared to auto- encoders and supervised learning. Hu et al. (2022) improve the SSL from an SNE perspective. Other robust contrastive learning methods (Kim et al., 2020; Jiang et al., 2020) focus on adversarial robustness while this paper focuses on distributional robustness.
performance of
2 PROBLEM FORMULATION
Given a set of unlabeled data where each sample X is i.i.d. sampled from training data distribution D on X C Rd, the goal of Self-Supervised Learning (SSL) is to learn an encoder f: X + RT for different downstream tasks. Contrastive learning is a popular approach of SSL, which augments each sample X twice to obtain a positive pair X2), and then learns the encoder f by pulling them close and pushing random samples (also called negative samples) away in the embedding space. The data augmentation is done by applying a transformation to the original data, where A is randomly selected from transformation set A according to some distribution x. We use
2

---

Published as a conference paper at ICLR 2023
DA to denote the distribution of augmented data under specific transformation A, and use D, to denote the distribution of augmented data, i.e., D = Ia DA di((A)) The loss of contrastive learning typically consists of two terms:
Leon (J: (on((, T)
Calign (J:D, m) Aorog ();
|| (AI [(A2(X))1 measures the alignment of positive samples', and Feg (f:D, σ) is the regularization term to avoid feature collapse. For example, the regularization term in InfoNCE loss is
where Lalign (J: = (AI
log (A))( ( ((AA)) (((XX)
E () (x,X')='² (A (A1.Ag,A'))r*
E
+
In this paper, we consider multi-class classification problems X' -R- (1........ K} as down- stream tasks, and focus on the covariate shift setting, i.c., target distribution D is different from D but [X) is fixed. We use the best linear classifier on top of f for downstream classification, and evaluate the performance of f on Diar by its risk (also used by Shi et al. (2022)):
R(f: ((f: =
min E l(h f))),
(2)
NERRXm X-DI
where é is the loss function. We focus on the case where E the square loss. With a slight abuse of notation, for the classifier h o f I - RR, we also use R(h f: DDr) to denote its risk. Our goal is to study the transferability of representation learned by contrastive learning on different downstream distributions. For simplicity, we assume that [x] = for every z X and f is
L-Lipschitz continuous throughout this paper.
At the end of this section, we remark the difference between transformations and augmentations. Transformations are deterministic mapping from X' to X, and augmentations are random application of them. We give examples to demonstrate that many augmentation methods used in practice indeed satisfy this model.
Example 2.1. The augmentations used in SimCLR (Chen et al.. 2020) are compositions of random transformations. such as RandomCrop, HorizonalFlip and Color distortion. Each A denotes a spe- cific composition here. Users will determine probabilities that each transformation will be applied. If used, the parameters of the random transformation such as crop range are also selected ran- domly. Until the parameters of each used transformations are set, the composited transformation is deterministic. Therefore, we can view these augmentation processes as randomly selecting some deterministic transformations, following the distribution π determined by users.
Example 2.2. Some recent works (Jahanian et al., 2021) propose to augment data by transfor- mations in the latent space. Suppose we have a pre-trained generative model with an encoder T : X + Z and generator G - X, where Z is a latent space. Since Z is usually simple, one can augment data in X by randomly shifting T(X). parameterized by Ao(X)= G(T(X) + 0) for 0 E Z. The distribution π of 0 is usually chosen to be a normal distribution.
this setting, the augmentation can be
TRANSFERABILITY ANALYSIS: THE CRUCIAL ROLE OF DATA AUGMENTATION
In this section, we theoretically demonstrate that data augmentation plays a crucial role in the trans- ferability of contrastive learning.
The first step is to establish the connection between contrastive loss 2ron (f: D, σ) and downstream risk R(ʃ: Dtar). ). For this purpose, adopt the (a, (a,8)-an -uaummmennnio notion proposed by Huang al. (2021), which was used there to study the KNN error of contrastive learning.
Definition 3.1 ((o, 8) -ttransformaiin)... Let CR C X be the set of all points in class k. A data transformation sel A is called (a, o)-transformation on D for some a E {0,1] and 8 > O, if for every l E [K], there exists C2 C& such that
PXND(X E CΦ) 2 a PXxD(XX E Ck), and sup a S 8,
where A (2) == inf A,,AsEA (z1), Ag(x2)) for some distance d(.,-)-
In this paper, notation|
stands for nnomm or Frobenius norm for vectors and matrices, respectively.
3

---

Published as a conference paper at ICLR 2023
This definition measures the concentration of augmented data. A transformation set with smaller d and larger o clusters original data more, i.c., samples from the same class are closer after augmen- tation. Therefore, one expect the learned representation f to have better cluster performance, which was measured by KNN error in (Huang et al., 2021). We remark that most of contrastive learning theories (Arora et al., 2019; Ash al., 2021; Bao et al., 2022) assume that given class k positive pairs are obtained by ii.d. sampling from Px ( E Ck). Compared with this data generation model, our (o, 6) notion is more practical.
By taking into account the linear classifier on top of f. we extend the KNN result in (Huang et al., 2021) to the downstream risk, and obtain the following lemma.
Lemma 3.1. For any distribution D and encoder J, define pt (R;[DD) Ex-D (X) for k [K]. Suppose that A is a (a, o)-transformation. For any representation f X' + lRd and linear layer h RK xds, we have
R:
Dz) cllbil Ko 2
| (1l 8)+ Σ D.lCCallea-hoou( (3)
where e an absolute constant and T 8) is a constant which only depends on (a, 8). The detail form of is shown in the appendix.
The first term in equation is the aligmment of f, which is optimized during the contrastive learning pre-training on D. The second term is determined solely by the (a, 6) quantity of the data aug- mentation. Larger a and smaller 5 induce smaller T(o, 5). These two terms together quantify the cluster level of the representation f learned by contrastive learning. The third term related the linear layer h top of f, and will be minimized in downstream training, where each class center converted to the corresponding ground-truth label. If the regularization term Lrog is appropriately chosen, the class centers can be distinguished from each other, and the third term can be reduced to 0 by h.
Taking the linear classifier into account, obtain the following main result.
Theorem 3.2. Suppose that A is (o, trrassfsofoonnnaannnn on D, and is the augmentation distribu- tion on A. For an encoder f. define
Vive (f:D,z) = (Calllinfff (.D, T max (f: pa (f:Dz)
max kyk
for some positive constants 01 and cz. Then for any f such that (uo(f: [z)] are linearly inde- pendent and 'reg (f: TT > 0, we have
R(f:Dt) (Tre) (Calign (f; + D.
(4)
where e is an absolute constant.
Remark 1. (a) The linear independence of (ux condition is weak, since its complement event is a null set in Euclidean space under Lebesgue measure, and have probability under any distribution that is absolutely continuous w.r.t. Lebesgue measure.
(b) The quantity Trr is essentially the bound of L2 norm of the linear classifier h which reduces the third term Σk= Dr, (Cx)) h [s<[]; Dz) in equation 3 to O, i.e., mapping the class centers to their corresponding ground-truth labels. The more orthogonal the class centers, the larger the Tee and the better the bounds. The theory developed in (Huang al., 2021) shows that popular algorithms such as SimCLR and Barlow Twins can indeed push away the class centers, and the constant "reg be ensured have a constant gap away from O. Therefore, for simplicity directly impose this requirement on f.
This theorem shows that contrastive learning on distribution D with augmentation æ is essentially optimizing the supervised risk on the augmented distribution D.. instead of the original distribution D. Therefore, the augmentation π is crucial to the transferability of contrastive learning. the downstream distribution Dr is close Dr, the encoder obtained by contrastive learning performs well on it. The more diverse π is, we can expect D, to approximate downstream distributions more accurately, and contrastive learning has better transferability.
4

---

Published as a conference paper at ICLR 2023
Besides, if we have some prior knowledge of downstream distribution, augmentation such that D, and Di'"r are close in order to improve downstream performance. For example, consider the case where original data D are gray digits, and the downstream distribution Diar is obtained by dying them red. Then if we use random coloring as data augmentation and conduct contrastive learning D, the learned feature will generalize Diar well, even though DDar is far away from D.
specify appropriate
4 TRANSFERABILITY LIMITATION: A DOMAIN-INVARIANCE VIEW
Our theoretical results in the last section suggests that data augmentation is crucial to the trans- ferability of contrastive learning. However, there exists fundamental limitation of contrastive learning: the learned representation is not generalize to more downstream datasets (Peters al., 2016; Arjovsky al., 2019).
which is important for a model to
Roughly speaking, tures, which are invariant across different domains. In supervised domain generalization, common setting is to have a training domain set consisting of multiple source domains. In contrastive learn- ing, correspondingly, we also consider D transformation A, and consider (D A AEA as the training domain set. Since the goal of contrastive learning is also align different DA. we naturally expect that the features it learn invariant. However, since the alignment loss Laligu in contrastive learning is obtained by averaging over different transformations (i.e., taking expectation), the learned features are not invariant even across (DA) In other words, encoders learned through contrastive learning could behave ex- tremely differently different AA
means that the learned model can extract some intrinsic fea-
transformation-induee domain for each data
domain-
The following toy model gives an illustrative example.
Proposition 4.1. Consider two-dimensional classification problem with data (X1;X2) (2) The label Y satisfies Y 1(X1 2 0), and the data augmentation is to multiply X2 by standard normal noise, i.e.,
Ax(X) (x1,0xXx
~~ ((0,1..
The corresponding transformation-imuuce domain set :D.:D. (X1,c- X2) e E R). We consider the 0-1 loss in equation 2. Then for every > 0. there exists representation f and two domains De and De such that
Catign (f; <E
but
RRf: 1
This example shows that there exists representation with arbitrary small contrastive loss, but with very different performance over different transformationiinduse domains. The intuition behind this example is that in order to make Caligo small, it is sufficient to align different domains (induced by different transformations) in an average rather than uniform sense. Therefore, the representation may still suffer large alignment loss on some rarely selected augmented domains.
ARCL:
SIIIENNTATIO-ROBUSSS CONTRASTIVE LEARNING
To further improve the transferability performance of contrastive learning, we consider how to learn representation that is domain-invariant invariant across (DA) AEA. AAAA First of all, we need formal definition that mathematically characterize domain-invariance, so that we can design algorithms based on it. Here, we borrow the notion proposed by (Arjovsky et al., 2019).
Definition 5.1. We say that representation f elicits an invariant predictor ho o I across domain set P, if there is classifier ho simultaneously optimal for all domains in P. that is, hg E arg min, R(h f; f:D) D) for all D P.
This definition is equivalent to learning features whose correlations with the target variable stable, and has been shown to improve distribution shift transferability both practically and theo- retically in supervised learning. Interestingly, by setting P to be (DA)AEA this definition is well
5

---

Published as a conference paper at ICLR 2023
suited to contrastive learning. Specifically, define the angmmentatiion-rouust loss LAR(::))== :- IE sup [I((A())) - ((A((
XED A,ME
which is uniform version of the original alignment loss Lalign. Then we have the following results.
Theorem 5.1. For any two transformations A and A', linear predictor h and representation f, we have
sup R(h f; - RR(h f; (: Sc- <-IIILLARRR [[]]LAAR(/D))
(5)
A,A'EAA
Moreover, fix f and let A arg min, R(h f, DA). Then have
RR(hA o f:Dx) - R(h o f: D:)) < 2c (lhalll + [IAA|))AAR(f:))
(6)
Compared with Calign. CAR replaces the expectation over A by the supremum operator, hence Calign (f; x) < (fR(f: for all and т. If (fR(f) is small, the above proposition shows that R(h o DA) varies slightly with A, so that the optimal h for DA is close to that for DA' other words, representation with smaller CaR tends elicit the same linear optimal predictors across different domains, a property that does not hold for original alignment loss. Small LaR not only requires / to push the positive pairs closer, but also forces this alignment be uniform across all transformations in A.
Therefore, propose train the encoder using the augmentation-rooust loss for better distribution shift transfer performance, i.c., replace Lalign by CAR in contrastive loss equation 1. In practical ff f(A2(X)) can not be computed exactly, unless |A]]
implementation, sup AAEEA Il.f (AA finite and small. Therefore, use the following approach approximate it. For each sample X, we first randomly select m augmentations to obtain set of augmented samples denoted by An, (X). Then, rather than taking average as in standard contrastive learning, we consider use
sup
(X)() - (X)2
A (X)
as the alignment term, which is an approximation of A. (I] f(22 The integer is called the number of views, which controls the approximation accuracy. Then, given n unlabeled samples (,....,,),,,... the empirical augmentation--rouus loss is
ExR(f) = Σ sup 1=1 (X,
- -
We remark that our proposed method can applied to any contrastive learning methods whose objective can be formulated by equation I, such as SimCLR. Moco, Barlow Twins (Zbontar et al., 2021) and so on. We give the detailed form of SimCLR + ArCL as an example in Algorithm 1, and provide MoCo ArCL in the Appendix. The difference between Algorithm and SimCLR is only the construction positive pairs. In each epoch t, for every sample X, we first randomly select m transformations denoted by Á(x)= = {AI...., Am Then based on current encoder f and projector g. we select two transformations that have worst alignment (minimal inner product) among all pairs in A(X), and use them to construct the finally used positive pairs of X. The construction of negative pairs and update rule are the same SimCLR.
5.1 THEORETICAL JUSTIFICATION
We now give thcoretical guarantees of our proposed approximation, i.c., bound the gap between [AR(S) and LAR(S). For each representation / E F. where F is some hypothesis space, de- fine (x)) sup (A)r) S( [A22r)ll2 and 2, {2) E F}. Then we the Rademacher complexity of Z to define the alignment complexity of F as
R (F) == Rr (Z,)) = E sup ZEZ
I Σ
(xx(x.)) im1
7
This quantity is a characteristic of the complexity of in terms of its alignment. If most of the functions in F have good alignment under A, then RR, is small even though R, (F) could be large. Using this definition, have the following result.

---

Published as a conference paper at ICLR 2023
Algorithm 1: SimCLR + ArCL.
input Batch size temperature т, augmentation т, number of views m, epoch encoder f, projector
for t = 1,...., do sample minibatch (x;) for i 1,...,,. N do
draw m augmentations Ā (Ai...., Am ~ Zi.j = =[[[( ((J((A for [m]; select the worst positive samples;
5 = minj,ke(m]] 21./2,l) =
8
select the negative samples;
for j ,,..,, N do
B.j 21, 23,11 (1l 2,, 1, i.j+N = -2,1120 2 2 (1=
compute L = - Σ log expla /r) 12s explo7,/r)
13
update f and g to minimize L;
14 return
Theorem 5.2. Suppose that θ Ba (1), i.e. the d--diennsoonal unit ball. Suppose that all A in A L Lipichhhit continuous. Let π be distribution on θ with density pn such that infece Pr (0) Cy. Then with probability at least 1 2e, for every f F,
(
(log(2n/E) C+
2
CAR(F) S CAR(S) (F)
LLA
n
where n. is the training sample size and m is the number of views.
The bound in Theorem 5.2 consists of two terms Onn-1/2 and O(m-1/da). The first one is due finite sample issue, and the second caused by the finite maximization approximation of the supremum. 1/cz acts like the "volume" of the transformation set and is of constant order. As m increases, the second term decreases and the bound becomes tighter. Note that we need assume that the augmentation distribution æ behaves uniformly, i.e., inføce (0) 2 c for some positive constant e, otherwise we can not ensure that the finite maximization is an acceptable approximation.
EXPERIMENTS
We conduct experiments on several distribution shift settings to show the performance our pro- posed methods. To show the adaptability of our approach, apply ArCL on SimCLR (Chen al., 2020) and MoCo V2 (He et al., 2020) respectively. Following the setting of SimCLR and MoCo V2, we add projection head after the backbone in the pre-training stage and remove it in the down- stream task. For the linear evaluation, we follow the protocol that the encoder is fixed and a linear classifier the top of the encoder is trained using test set. For the full fine-tuning, also add linear classifier on the top of the encoder but we do not fix the encoder.
6.1 EXPERIMENTS ON MODIFIED CIFARIO AND CIFARION
Setup. Our representations are trained on CIFAR-I0 (Krizhevsky, 2009) using SimCLR modified with our proposed ArCL. Different batch sizes (256, 512) and number of views (m 4, 6, 8) are considered. We use ResNet-18 18 as the encoder and a 2-layer MLP as the projection head. We train the representation with 500 epochs. The temperature is 0.5. Warm-up is not used. We conduct linear evaluation with 100 epochs on different downstrcam datasets to test the OOD performance of the learned representations.

---

different Published as a conference paper at ICLR 2023
Table 1: 5 different augmentations.
Grayscale
RandomCrop
HorizontalFlip
ColorJitter
Aug
Aug
Aug 3
Aug 4
Aug
Table 2: Linear evaluation results (%) of pretrained CIFARIO models CIFARIO, CIFARIOS and
their modified versions.
Method
Batch Size
Aug
Aug 2
Aug
Aug
Aug
Original
SimCLR
256
86,36
83.21
86.93
86,45
86.13
86.76
SimCLR + ATCI. (viewss))
256
88.68
86.77
89.01
88.70
88.31
88.95
CIFAR10
SimCLR + ArCL (views=6)
256
88,95
87.18
89.54
88.92
88.61
89.11
SimCLR
512
88.62
86.27
88.96
88.56
88.37
88.81
SimCLR ArCL (views=4)
512
89.97
88.06
90.48
89.91
89.59
90.20
SimCLR ArCL (views=6)
512
90.24
89.54
90.69
90.43
90.07
90.69
SimCLR + ArCL (views=8)
512
90.44
88.96
90.98
90.63
90.31
90.84
SimCLR
256
51.65
47.55
53.17
52,05
51.36
52.75
SinCCLR=ACLLVveww==))
256
53.76
49.80
$5.68
54,19
52.96
54.83
CIFAR100
SinCCLR+ArLL(viewss
256
54.13
50.74
55.74
54.75
53.46
55.29
SimCLR
512
52.28
48.09
53.45
52.58
51.53
53.12
SinnCLR=ArCL(veewss
512
53.40
50.16
54.92
53.77
52.61
54.20
SinCILR=ArCLLviewwss
512
54.00
50.57
56.24
55.04
53.77
55.60
SinCLRR+ArCLvveeee
512
54.59
50.85
55.74
54.62
53.21
55.96
Datasets. We CIFARIO to train the representation. The data augmentation used follows the setting in SimCLR, ie., a composition of RandomCrop. RundomHorizontalPiip.. ColorJitter and Random Grayscale. To evaluate the transferability of the learned representation, downstream datasets to train the linear classifier and test its accuracy. Downstream datasets are generated by modifying the original CIFARIO and CIFARI00 datasets, i.e., applying different data augmentations to them. We choose 5 different augmentations as shown in Table 1. The results are shown in Table 2.
Results. In all settings, our proposed approach improves the transferability of SimCLR significantly. As the number of views m increases, the accuracy also increases, which matches our theoretical results. Besides, the performance improvement brought by increasing m tends to saturate, which suggest that m does not need to very large.
6.2 EXPERIMENTS ON IMAGENET TRANSFERRING TO SMALL DATASETS
Setup. In this part, will first train encoder ImageNet (Deng al., 2009) using MoCo v2 modified with our proposed ArCL. We select the number of ArCL views from (2, 3, 4]. Notice that since there exists asymmetric network architecture MoCo, there difference between MoCo and MoCo ArCL(views=2). We use ResNet50 as the backbone and follow all the other architecture settings of MoCo. The memory bank size K is set 65536 and the batch size the training epochs, we propose three schemes: 1) 200 epochs training from scratch, 2) 50 epochs training from the model which is pretrained by MoCo for 800 epochs, 3) 100 epochs training from the model which is pretrained by MoCo for 800 epochs. For each setting, we search the best start learning rate {0.03, 0.0075, 0.0015) and the best temperature in {0.20, 0.15, 0.12, 0.10, 0.05). We use the SGD optimizer and apply the cosine learning rate decay.
256. For
After the model has been pre-trained by contrastive learning, we test its transferability with linear evaluation and full fine-tuning various downstream small datasets, following the settings in (Eric- sson et al., 2021). For the target datasets, we adopt FGVC Aircraft (Maji et al., 2013), Caltech-101 (Fei-Fei et al., 2004), Stanford Cars (Krause et al., 2013), CIFARIO, CIFAR100 100, DTD (Cimpoi et al., 2014), Oxford 102 Flowers (Nilsback & Zisserman, 2008), Food- 101 (Bossard et al., 2014) and Oxford-IIIT Pets (Parkhi et al., 2012). For linear evaluation, multinomial logistic regression is
8

---

Published as a conference paper at ICLR 2023
Table 3: Results comparison on línear evaluation (up) and finetuning (down) of pretrained ImageNet models on popular recognition datasets. Additionally, we provide a supervised bascline for compar- ison, standard pretrained ResNet50 available from the PyTorch library. The supervised baseline results are from (Ericsson et al., 2021). Results style: best in the same epoch setting. Avg takes the average of the results on all the nine small datasets.
Epochs
ImugeNet
Aircran
Caluchh01
CIFARIO
CIFAR100
Fvod
Pets
Avg
800
41.79
92.28
74.90
800+50
41.00
75.14
68.55
McCo (vCC-vv
808+50
69.70
44.29
89.79
42.15
03.07
76. 76.70
74.20
90.40
70.94
83.68
73.91
MoCo AcCLiviesss))
900+50
69.90
$4.57
39.48
42.11
93.29
77.33
74.63
91.13
71.15
84.23
74.21
McCo (rC[(vice==))
800+50
69.80
44.62
89.66
42.88
93:22
76.83
75,00
91.59
71.35
833.9
74.35
Linear
MoCo
800+100
70.54
41.65
87.64
39.31
92.12
75.03
73.94
89.53
68.31
83.55
72.34
McCo ArCL(viees=)
800+100
70.26
41.86
89.52
60.21
92.64
75.75
74.04
88.97
70.06
84.31
73.04
MoCo ArCLivicsss))
800-100
45,87
93.30
72.95
84,01
(viewss))))))
800-100
45.55
200
40.02
90.90
72.43
13.88
66.97
90.00
ArCL/vices=e)
200
71.21
67.87
McCo AvCl-vvo
200
43.21
88.26
38.27
75.02
74.68
89.61
68.31
N1.64
77.30
MoCo (viievi
200
68.92
41.65
88.42
38.77
92.78
75.73
75.43
88.05
68.63
81.60
72.43
Superiised
77.20
43.59
90.18
44.02
91.42
73.90
72.23
89.03
69.49
91.45
74.12
800
83.56
82.54
95.89
71.81
69.95
76.81
95.83
800-50
83.15
72.58
79.
36..2
MoCo + A
8001500
86.05
87.58
87.28
96.53
79.
72.18
95.20
81.36
89.01
86.10
McCo+ ArCL(vinn=))
800+50
84.03
87.64
65.34
96.88
80.98
72.87
96.14
81.90
89.20
86.22
MoCo (viewss)))
800-50
84.19
88.42
86.67
95.68
81.17
73.09
95.00
81.70
89.52
86.37
Fineunee
MoCo
800-100
83.18
84,50
84.27
96.01
72.14
70.27
95.53
78.23
88.75
83.65
MoCo ArCL/viees=)
300-100
84.45
86.84
87.20
96.40
78.40
71.91
95.93
80.54
88.55
85.58
ArCL/viccsss))
800+100
85.65
88.50
96.91
$1.29
73.35
81.82
39.30
200
83.18
82.66
95.51
72.54
70:43
77.39
86.12
McCo ArC((vies==))
200
81.09
83.93
86.54
95.83
76.18
70.69
91.14
76.76
86.95
63.61
MoCo ACCL/viess)))
200
84.79
85.61
85.39
96.56
78.81
70.59
95.84
80.71
87.91
85.13
MoCo e 4)
200
$4.88
86.19
85,90
96.35
78.62
70.69
95.77
80.46
88.00
85.21
Supervised
83.50
91.01
82.61
96.39
82,91
73.30
95.50
84.60
92.42
86.92
fit on the extracted features. For full fine-tuning, the full models are trained for 5,000 steps using SGD with Nesterov momentum. We calculate the average results of the nine downstream datasets.
Results. We compare our methods with the MoCo baseline. There are about an average rise of 2% in the linear evaluation setting and about average rise of 3% in the finetuning setting. In particular, we gain a huge improvement on CIFAR 100 finetuning, where our proposed methods can outperform the baseline for up to 10%. It is also worthy to mention that methods reach the supervised results on nearly all the datasets. These facts can all indicate the effectiveness of our proposed methods.
Besides, the results also show that, under the same epoch setting, the average performance grows as the number of the views increase, which fits our theoretical analysis. One also notice that there is only a small improvement in the 800 + 100 epoch setting compared to the 800 + 50 epoch setting. This implies that our methods also converge in rather fast speed, which can also be effective within a few epochs of training. Compared to CL methods, ArCL gets to see more augmentations in total since for each original sample, we construct its m augmented versions.
6.3 COMPARISON WITH AVERAGE ALIGNMENT LOss
Compared to CL methods, ArCL gets to see more augmentations in total since for each original sample, we construct its m augmented versions. To make the comparison between ArCL and CL fairer, for original CL, also construct m views while the average of the similarity between all positive pairs among the m views, namely average aligmment loss (AAL), as the learning objective. Results for SimCLR CIFAR10 and CIFAR100 100 and MoCo on ImageNet and 6 in the appendix. Detailed experimental settings are also deferred there. As the experiments show, contrastive learning with AAL has similar performance with vanilla methods, which still much worse than ArCL on distribution shift downstream tasks.
be seen in Table 5

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
Yisen Wang is partially supported by the National Key R&D Program of China 20222D00603044 the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang Lab (No. 2022RC0AB0S), and Huawei Technologies Inc.
We would like express our sincere gratitude the reviewers of ICLR 2023 for their insightful and constructive feedback. Their valuable comments have greatly contributed to improving the quality of our work.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: (007.0290000299 2019.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 00t::/990.0000 092.09229, 2019.
Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. arXiv preprint arXiv:2106. 09043, 2021.
Han Bao, Yoshihiro Nagano, and Kento Nozawa. On the surrogate gap between contrastive and
supervised losses. International Conference on Machine Learning, pp. 1585-1606. PMLR, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 2013.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Proc. of NeurIPS, 2011.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative com- ponents with random forests. In ECCV, 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. of ICML, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proc. of CVPR, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-2442244 205, 2009. doi: 10. 111.1///R..00099556666
John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distribu- tionally robust optimization. The Annals of Statistics, 2021.
John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A
generalized empirical likelihood approach. Mathematics of Operations Research, 2021.
Linus Eriesson, Henry Gouk, and Timothy M. Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion (CVPR), pp. 5414 4443,,,, June 2021.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training ex- amples: An incremental bayesian approach tested 101 object categories. In 2004 Con- ference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004. doi: (009//VPP22004..00
10

---

Published as a conference paper at ICLR 2023
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXIv:2103.01988 2021.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bermardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et Bootstrap your own latent-a new approach to self-supervised learning. Proc. of NeurlPS, 2020.
Jeff Z HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations. arXiv preprint arXiv:2204. 0288082 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. of CVPR, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchiln, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprini arXiv: 0600000000067 2018.
Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. arXiv preprint arXiv:2205. 14814, 2022.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self- supervised learning. arXiv preprint arXiv:2111.0001 2021.
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. arXiv preprint wwx.v:2106.05220 2021.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. Proc. of NeurlPS, 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Domain extrapolation via regret minimization. arXiv preprint 2xXx::0006.3390 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In Proc. ICML, 2018.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. Proc. of NeurIPS, 2020.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine- grained cars. 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
David Krueger, Ethan Caballero, foern-Henri Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distributioo generalization via risk extrapo- lation (rex). In Proc. ICMI, 2021.
Kun Kuang, Ruoxuan Xiong, Peng Cui, Susan Athey, and Bo Li. Stable prediction with model misspecification and agnostic distribution shift. In Proc. of AAAI, 2020.
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie- Yan Liu. Learning causal semantic representation for out-of-distribution prediction. arXiv preprint arXiv:2011.0168) 2020.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXiv preprint arXiv:2110.0500 2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching, In Proc. of ICML, 2021.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- fine-grained visual classification of aircraft, 2013. URL http:ss//arxiiv.oo//aaasss//////

---

Published as a conference paper at ICLR 2023
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In Proc. of ICML, 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, doi: 10. 10110//CGIP...00.......
pp. 722-729, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1498-3505 2012. doi: 201.0///VVRR...2.222222222
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre-
diction: identification and confidence intervals. Journal of the Royal Statistical Society. Series (Statistical Methodology), 2016.
Mateo Rojas-Carulla Bernhard Schölkopf. Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. Proc. of ICIR, 2019.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NeurlPS.
Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie. Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. arXiv preprint arXiv:2204.003: 00570, 2022.
Zheyan Shen, Peng Cui, Tong Zhang, and Kun Kuang. Stable learning via sample reweighting. In Proc. of AAAI, 2020.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: survey. arXiv preprint arXiv:2 108. /3624, 2021.
Yuge Shi, Imant Daunhawer, Julia E Vogt, Philip HS Torr, and Amartya Sanyal. How robust are pre-trained models to distribution shift? arXiv preprint arXiv:2206.0887 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Proc. of ECCV, 2020.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf. and Stefan Bauer. On disentangled representations learned from correlated data. In Proc. of ICML, 2021.
Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. arXiv preprint 1907. 2019.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably 2021.
isolates content from style. arXiv preprint arXiv:2106.044
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. arXiv preprint arXiv:2 20Xiv22103.03097 2021a.
Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Proc. of NeurlPS, 2021b.
Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of out-of-distribution generalization. arXiv preprint arXiv:2106.04496. 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Proc. of ICML, 2021.
12

---

Published as a conference paper at ICLR 2023
Appendix
A MoCo + ARCL
Algorithm 2: MoCo + ArCL.
input Batch size temperature т, memory bank M, augmentation 9, number views epoch T, encoders fgr fR, projector
for t = -,...,, do sample minibatch {x,) for = l...= N do
draw m augmentations Ā = J ((g((Axx))) for j [m]:
Am) ~
5
i.j = (A,.) for j E [m]:
select the worst positive samples; IT s.¹ miinjJe(m]eii (orr22, Rig <3/((1=)]lll=3.])
# select the negative samples; for m E .M do f. 2 m/ 26,1 mll):
10
compute - ビ log exple /r)
Emem explei, /T))
13
update f andd g to minimize update M;
15 return f
B PROOFS IN SECTION
Proof of Lemma 3.1. For simplicity we omit the notations D and 9 in this proof, and use 2 to denote the positive sample, i.e. the augmented data, use pk denote (Cx))) and use pa denote (.)) From Theorem 2 and Lemma B.1 in (Huang et al., 2021), we have
E xEC₁
Ilf (x1) pall s <eV Pk
Lpos (f) + T(
(8)
for some constant c, where
T(, 110
L8
Note that r is decreasing with and increasing with ö. Then we have
Pk
(foo(f)++10) +(
Pk
E nECk
lff(1))
pxll
K
Pk
E
E
l1 f(x1) - h mill
| zEC, zierr
Pk
E
E
|| f(x1) - -eHll
pallex h ukl
|
|
K
R(h f)
Pk lel - h 1u
|
13

---

Published as a conference paper at ICLR 2023
for all linear layer h RRxdd Therefore, obtain
K
K
R(h o f) v c|ll||
Vpx + [|llT((, + Σ
Phllen - 11 Mkll
k=1
(10)
SclillVK + [1,1 0) + Σ pkllex h m
Proof of Theorem 3.2. By the triangle inequality and equation 8, have
| Liou (f) T(, ph
||x| 2 1 - E zEC,
[1f(x1) uel - CV.
Therefore,
I Lipos (f) T(o,0).
min /1lll22 2 - CV
Pn
Define U (Mi..... px) E 12 x and let hg haU 1x. Then we have
U be the Moore-Peenoon inverse U such that
| |l
| sup =ER
|Ux]
For every (x) ( Tai) E Rdi with | 1, have
1121
2 || 12
Σ au Pi
min || 2
max | My 4.
Σ |
|z:i [[1zd)]
min ||p5112 2 - max 1l! Mll
-1
(,)
min ||m.12 2 - (K - max 1p, pqyl
6,j
1
1 V
[P) T(0,8) K max 1n 1sy
Po
W
Theg (J;
Therefore, we have
||ool| S Treg (f; 0). Then, by Lemma 3.1, we complete the proof by R(f) min R(h f) < R(ho f) cYreg (f:D, Looo DD)
C PROOF IN SECTION
Proof of Proposition 4.1. For any ε 0, let alignment loss of f satisfies
Ve/2 and f(11, =11 + tzz. Then, the
Latign (f: D. TT = f E
E (0, (1..0))~~(001)))
(0, -02)² 2t²
Let c = 0 and c' = =1t. Then obviously
R(f: 0,
but
R(f:De) P(X, < 0, X 0) P(X; 2 0, + =
14

---

Published as a conference paper at ICLR 2023
D PROOFS IN SECTION 5
Proof Theorem 5.1. For any f and h with ||h]| S ch. we have
R(h J;DA) R(h (v) (x,Y)
E (1h f(AX) Y 11 ((AX)) -
E
(ho (0hof(A)) [Axx - ho ((A'x))) s(Ax) ho [(A'x)) +2)
(x.Y)DD
E
|1 f(Ax) ho f(A'x)
(x,
S c||b||
E [1f(Ax) - f(A'x))
(X.X)DD
S <cllh|LAR(())
for some constant e which depends on Ch.
proof of Theorem 5.2. We first fix the sample "f1,...,,nll and consider the randomness of
(z)). Define (z,) and (2,),) as (z))), 42(x,) arg sup
(I) (((1)))(((A22)))) f(A2z.)
ALEA
Note that their order doesn't 't matter. For every > have
(log(2n/e)
[[{(A);; - f(A22))
S Σ
[If(A)); - f(A25;)12 I.2I'3
sup ALEA
sup
n i=1 A1. (mm
Cym
[lo((2n/E)
>P
max
inf (AEAm (z.)
||A - -- inf (w.)
||A - A3(z))l2
iE(n|
Cym
| 11-A40 Ai;)lll (log(2n/E)
(log(2n/c)
21
P
inf AEAm (x:)
inf AEAm(a))
I|A - Ab(xs)lle 5
com
com
im I
1og((n/E3)
21
P
3E E Am(x))
AA-AÇ(z))le 5
Cam
(log(2n/E)
3
P Am:)) A| Ab(x:)l12 s
cam
i=1
(i0)
co log 2n/E)
21 2n
com
(1)
21 E.
(i) comes from the Lipschitz continuity of and A; (ii) is simple application of set operations:
is derived from the fact that the volume of a d-dimensional ball of radius r is proportional rd:
iiii) comes from the inequality (1 1/a)a s < for a 1. Therefore, with probability at least
E, we have
sup 11/(A1æ;) -[Ara:))((Aaa)))))) [(A22))l12 S [AR(s)+ + 1.713, (log(2n/s) 1=1 AL,AE
Besides, for every we have
LAR(f)
1
Σ sup [1/(A)>;) f(A22:) 22 (F) E=
(log(1/s)
ALAAEA
m
with probability at least - E. Then we finally obtain that with probability at least 2s, for every
fe F.
LAR(f) S DAR(f)+ + + (log(1/E)
llog(2n/E)
+ 133L3
Cøm
15

---

Published as a conference paper at ICLR 2023
Table 4: Linear evaluation results of pretrained CIFARIO models using MoCo v2 CIFAR1O, CIFARI00 and their modified versions.
Method
Aug
Aug 2
Aug 3
Aug
Aug 5
Original
MoCo
88.34
87.44
89.14
88.76
88.
89.29
MoCo ArCL (views=2)
89.12
88.13
89.62
89.25
88.91
90.44
CIFARIO
MoCo ArCL (views=3)
90.22
89.11
90.93
90.13
89.75
91.22
MoCo + ArCL (views=l)
90.77
89.64
91.22
90.86
90.18
91.45
SimCLR
52.98
48.77
54.47
53,18
52.93
55,88
53.37
49.14
55.09
$3.46
53. 14
56.62
CO
SinCCR--ArCL.vvoos=:)
54.49
50.29
55.33
53.78
53.62
57.13
SinCLRR-ArCL((viows=))
55.42
51.42
55.96
53.76
54.41
57.98
E ADDITIONAL EXPERIMENTS
In order to further verify the generality of our method, we conduct more experiments on different settings and compare objective with different loss.
E. MoCo ON CIFAR10
Setup. In this part, we conduct experiments with MoCo v2 pretrained on CIFARIO. We follow the setup MoCo v2. We use ResNet- 18 as the encoder and train the representation with 400 epochs. The temperature is set to 0.2 and the initial learning rate is set to 0.15. The memory bank size is set to 4096. The SGD optimizer We conduct linear evaluation with 100 epochs modified CIFARIO and CIFAR100 100 which are used in Section 6.1.
cosine schedule for learning rate are used. Warm-up is not used.
Results. Results can be seen in Table 4. We can see that the original results still maintain in the MoCo method. Our proposed approach improves the transferability of MoCo. As the number of views grows, the accuracy also increases.
E.2 COMPARISON WITH AVERAGE ALIGNMENT LOSS
To make the comparison between ArCL and CL fairer, we add new experiments for original CL. For each sample, we also construct m views and use the expectation of the similarity between positive pairs, namely average alignment loss(AAL), as the learning objective. The settings and results are shown in the following part.
Loss Function. For image 24. the normalized features of its views from the online branch are 2/1, 242,... Zim and the normalized features from the target branch are 2/1 2/2...., .., 2/m0- Our ArCL alignment loss is
L
align ArCL
-
min 2n / jæk
And the average alignment loss should be
I, align
Σ avg a -
-m).
Average
jifk
The uniformity loss keeps the same and the total loss is the sum the alignment loss and uniformity loss.
Hyperparameters and Datasets. For SimCLR, we train the models on CIFARTO and evaluate them modified CIFARIO 10 and CIFARI00 just views to be and the batch size to be 512. For MoCo, We conduct experiments on ImageNet, with the 800-epochs preerrind model as the initializing model. We train the model for 50 epochs and use the same setting as in Section 6.2.
original paper does. We choose the augmentation
Results. Results for SimCLR experiments can be in Table 5. We compare the linear evaluation results on modified CIFARIO and CIFARIO0 Results for MoCo experiments can be seen 6. We compare both the linear evaluation and finetuning results small datasets. As the experiments
16

---

Published as a conference paper at ICLR 2023
Table 5: Linear evaluation results of pretrained CIFARIO models using SimCLR with two different alignment losses on CIFARIO, CIFARI00 and their modified versions.
Method
Aug
Aug
Aug 3
Aug
Aug
Original
SimCLR
88.62
86,27
88.96
88.56
88.37
88.81
CIFARIO
SimCLR AAL (views=4)
88.90
86.34
88.72
88.65
88.44
89.97
SimCLR ArCL (viows=4)
89,97
88,06
90.48
89,91
89,59
90.20
SimCLR
52.28
48.09
53.45
52.58
51.53
53.12
CIFARI0O
SinCLR-AAL(viewer
52.34
48.77
53.12
52.44
51.82
53.21
S
53,40
50.16
54,92
53.77
52.61
54.20
Table 6: Results comparison on linear evaluation (up) and finetuning (down) of pretrained ImageNett models with ArCL alignment loss and average alignment loss on popular recognition datasets. Re- sults style: best in the same augmentation view setting. Avg takes the average of the results all the nine small datasets.
Epoctss
ImageNet
Aircraft
Caltech101
Cars
CIFARIO
CIPARI00
DTD
Flowers
Food
Pets
Arg
MoCo
92.28
74,90
83.30
Linear
MoCo AAL(views-2)
71.12
40.53
87.80
38.64
92.23
75.14
74.95
$8.64
69.24
83.17
72:26
MoCo ArC((vie)))))
69:70
44.29
89.79
42.15
93.07
76.70
74,20
90.40
70.94
83.68
73.91
MaCo AAL/viows)))
71.37
83.79
72.84
MoCo + ACL(viess)))
69.80
44.57
89.48
42.11
93.29
77.33
74.63
91.13
71.16
84.23
74.21
MoCo
83.56
82.54
85.09
95.89
71.81
69.95
95.26
76.81
88.83
83.30
Finetume
MoCo e AAL(viens-2)
83.87
82.76
85.90
96.38
71.43
72.71
95.50
76.95
$9.05
83.84
MoCo (vVeeve))))))
87.28
79.39
89.03
86.10
MoCo (AI/views)))
83.07
83.21
85.19
96.37
72.02
72.55
95.74
79.62
84.07
MoCo 3
84.03
87.64
86.34
96.88
80.98
72.87
96.14
81.90
89.20
86.22
show, when contrastive learning methods with average alignment loss use the same amount of data
as ArCL, although they can have higher accuracy compared to vanilla methods (those which only two views) ID set, they still have similar performance with vanilla methods OOD set, which is much worse than ArCL. This further verifies the superiority of our method.
E.3 EXPERIMENTS ON MNIST-CIFAR
MNIST-CIFAR is synthetic dataset proposed by (Shah et al.). It consists of 3 64 32 synthetic images, each of which is vertical concatenation of a 3 x 32 x 32 MNIST image from class 0 or class and 3 x 32 32 CIFARIO image from class class We follow the distribution settings proposed by (Shi et al., 2022):
ID train: 1.0 correlation between MNIST and CIFARIO labels. Contains two classes: class 0 with MNIST "0" and CIFAR10 "automobile". and class 1 with MNIST "1" and CIFARIO "plane".
OOD train: no correlation between MNIST and CIFARIO labels, images from the two classes of MNIST and CIFAR10 10 are randomly paired. We choose the label of CIFAR10 to be the label of the concatenated image.
OOD test: generated similarly to the OOD train set using the test set of MNIST and CI- FAR1O.
In this experiment, we train the model on the train set, conduct the linear evaluation on the OOD train set, and calculate the accuracy on the OOD test set. We adopt SimCLR as the CL framework and use the 4-layer CNN as the backbone just as Shi et al. (2022) does. We fix the base feature size of CNN C 32 and the latent dimension L 128. Batch size is set to 128, the optimizer is set to SGD and the Ir. scheduler is set to warmup cosine. The learning rate and weight decay are searched in the range given in Table of Shi et al. (2022). Since the training process easy converge, set the training epoch and the linear evaluation epoch to 5. We also compare the average alignment loss (AAL for short) mentioned in the Appendix 6.3.
17

---

Published as a conference paper at ICLR 2023
Table 7: Linear evaluation results of pretrained models using SimCLR with different alignment losses on MNIST-CIFAR dataset. The average results under three diffrent random seeds are given.
Methods
Accuracy(%)
Methods
Accuracy(%6)
SimCLR
85.6
SimCLR + ArCI_vviews=3)
86.0
SimCLR+AAM SAmCC.RAAA..viewwsssss (views=3)
85.4
SimCI. R + ArCI (viewss))
87.2
SimCI.R+AA RR+AAL (views=l)
86.1
SimCLR + ArCL(views==)
87.3
SimCLR+AAL (
86.3
SimCLR + ArCL(views=-)
88.4
SimCLR+AAL(vicwws))
85.8
Results. We can see that by using ArCL, SimCLR enjoys a rising performance. The accuracy also grows as the number of views grows, which fits our theory. We also notice that the accuracy of SimCLR with AAL does not vary too much as the number of views grows, which indicates that key point is the usage of multi-view (adopting the minimum instead of the average).
18


Reviews:
[{"review_id": "HYr3mhffnih", "reviewer": "Reviewer_y4QB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work established a theoretical framework to analyze the OOD performance of contrastive learning based self-supervised learning and suggest that the effective of contrastive learning mainly comes from data augmentation. Further, they proposed augmentation-robust contrastive learning to show the better OOD performance.", "summary_of_the_review": "Since I do not have too much background in theoretical analysis, I could hardly measure the novelty and contribution of this work to the community. The proposed augmentation-robust contrastive learning (ArCL) seems to work well on two framework SimCLR and Moco although I am not sure why the authors do not provide the two frameworks on all the three datasets.\n\n", "strength_and_weaknesses": "+ This work starts from a theoretical perspective to analyze the OOD performance of contrastive learning in self-supervised learning. The analysis reveals that the better performance of contrastive learning mainly comes from data augmentation.\n+ This work proposes a simple augmentation-robust contrastive learning to improve OOD generalization of SSL.\n- The empirical results seem not to be very extensive. Can authors provide the results of ArCL applied to both SimCLR and MoCo on all the three datasets rather than picking one model for each of these datasets?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9iuXOVeOM_", "reviewer": "Reviewer_QCLS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the benefit/role of learning representations with contrastive learning for out-of-distribution (OOD) generalization.\nIt argues that one way contrastive learning could help with covariate shift is through the diversity in the augmentation distribution, even if the input distributions shift a lot.\nThis is justified with some theory to upper bound the supervised risk of a representation by the contrastive loss and some other “representation diversity” terms, under some clusterability assumptions on augmentations from prior work.\nHowever the paper finds that vanilla contrastive learning is not sufficient to learn *domain invariant representations* and thus might do poorly in OOD settings.\n\nViewing different augmentation transformations (like random cropping, gray scaling, brightness) as different “domains”, the paper notes that contrastive learning only tries to make representations invariant to these on average over the augmentation distribution.\nInstead it proposes an augmentation-robust contrastive learning (ArCL) that encourages representation invariance for the worst case sets of augmentations, i.e. minimizing $\\sup\\_{A,A’} ||f(A(x) - A’(x)||^2$ instead of $\\mathbb{E}\\_{A,A’} ||f(A(x)) - f(A’(x))||^2$.\nThis ArCL alignment term (combined with MoCo) seems to do much better on linear probing and fine-tuning evaluation while transferring from ImageNet -> one of many downstream image classification tasks.", "summary_of_the_review": "Overall I think the paper makes interesting contributions to study the benefit of contrastive learning in the transfer setting. While the findings and results paper are quite promising, the issues with the incorrect usage of OOD generalization and some other issues about unjustified claims, I believe that the paper could use benefit from round of reviews. Thus for now I would assign a score of reject.", "strength_and_weaknesses": "**Strengths**\n\n- The new augmentation-robust contrastive learning algorithm is an interesting variant of contrastive learning that goes beyond “averaging over augmentations”. To my knowledge this is a novel idea and it also clearly learns better representations than standard contrastive learning, based on the experiments on ImageNet.\n- The paper is clearly written for the most part and easy to follow\n- Theoretical results are provided to justify the need for going beyond the average over augmentations setting, and also to upper bound the transfer risk for ArCL\n\n**Weakness**\n\n- One of the main concerns about this paper is the use of the term *OOD generalization* for the settings discussed in the paper. In the entire discourse, including experimental evaluations, a representation is learned using unlabeled data from a source domain (ImageNet) and this is evaluated with linear probe or fine-tuning by using **labeled data** from the target domain (CIFAR, Caltech101, etc.). While I am not an expert in OOD generalization, I believe that the standard OOD setting assumes no targets (and sometimes no inputs) from the target domain. I verified this with researchers who constantly publish OOD generalization papers and also checked some surveys (see Section 2.1.2 in [1]). In fact the setting discussed in this paper is much closer to something like “self-supervised transfer learning”. Even the original SimCLR paper [2] evaluate in this setting; see Table 8. OOD generalization is typically evaluated using benchmarks such as WILDS [3]. In light of this, a lot of the claims in the paper will need to be changed, including the title, or actual OOD evaluations need to be included.\n\n- The abstract makes the following claim: **recent work claims that contrastive learning learns more robust representations than supervised learning, our results suggest that this superiority mainly comes from the data augmentation used, i.e., more data are fed to the model.** However I do not think this is adequately justified in the paper. I could not find any experimental evidence to claim that the superiority of contrastive learning mainly comes from “more data fed to the model”. One way to test this is the following: how well does *supervised learning* with the same augmentations do on OOD? If it does well, then one could believe that augmentations played a huge role. Otherwise it could be that contrastive learning somehow learns better features even with the same augmentations. Either the claim needs to be changed to *augmentations can play a role* (based on Section 3), or some experiments are required to justify *superiority mainly comes from …*.\n\n- Section 3 is used to justify that augmentations play a huge role in the good OOD (actually transfer) performance, but the section seemed a little out of place after reading the full paper because it is not really related to the subsequent ideas. It just seems like an “improved upper bound” for the performance of contrastive learning representations on a downstream task. In fact even the analysis from [4] shows an upper bound on supervised learning risk for the augmentation distribution (which they later convert to guarantees for the input distribution). I think the first equation in Theorem B.3 from the arxiv version of [4] shows this. So I think it is important to both, mention the bound in [4] and justify the section better.\n\n\nOther comments/questions:\n- Could it be that the superior performance of ArCL compared to CL is due to the fact that it gets to see more augmentations in total? If so, some ablation study to isolate this effect could be useful\n- Treating augmentations as domains for the results in Section 5  is a little weird, because they are not really domains of interest for any downstream evaluation. How does this idea (and Theorem 5.2) connect to the ImageNet experimental settings?\n- Is there any connection to ViewMaker network [5] that learn augmentations in an adversarial manner? Those augmentations are more robust to common corruptions in CIFAR. It would be a useful experiment to see if ArCL helps over those augmentations as well.\n- Proposition 5.1 $h_A \\in \\arg\\min$ might be more appropriate\n- In Theorem 5.2, it seems like the bound if non-vacuous only if $m$ (number of views used) is of the order of $1/c\\_{\\pi}$. However $c\\_{\\pi}$ will be at least $|\\mathcal{A}|^{-1}$ and so one would need $m$ to be as large as the total number of distinct transformations, which is unreasonably high. Of course this is just an upper bound, but it might be useful to discuss this a bit more. Maybe some covering number argument on $\\mathcal{A}$ might lead to a tighter bound since many $A$ might be close to each other.\n\n\n[1] Shen et al. Towards Out-Of-Distribution Generalization: A Survey\n\n[2] Chen et al. A Simple Framework for Contrastive Learning of Visual Representations\n\n[3] Koh et al. WILDS: A Benchmark of in-the-Wild Distribution Shifts\n\n[4] Haochen et al. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss\n\n[5] Tamkin et al. Viewmaker Networks: Learning Views for Unsupervised Representation Learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1t56XCYomXJ", "reviewer": "Reviewer_jXHS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the topic of contrastive SSL. In particular, the paper studies whether using multiple data augmentations and choosing the one the model performs worst at is a more effective approach than Contrastive SSL. The paper calls this ArCL and also provides more experiments.", "summary_of_the_review": "* In Equation (2), the linear classifier is evaluated as the best linear classifier on the target domain. While this is usually not done in practice, some recent works (e.g. [1]) used this to empirically observe the robustness of SSL methods. The authors should discuss this and other similar works as they provide empirical  use cases of their setting. Perhaps, Lemma 3.1 indeed applies to the experimental setting in these papers.\n\n* Theorem 3.2 shows a kind of equivalence between SSL and Supervised for appropriately chosen regularisation term, This is similar to the recent work of bao et. al, which is also not cited in this paper. In addition Bao et. al. quantifies the effect of the number of negative samples in the regularisation term and presents a more realistic results as increasing number of negative examples work better in practice.\n\n* The example in Proposition 4.1 presents a multiplicative noise model. This can indeed lead to interesting theoretical examples showing that minimising expectation across domains is arbitrarily worse than  minimising supremum across domains. However, I am not sure if this is a realistic problem to address.\n\n* Theorem 5.2 has a dependence on m and n, but I don't see the dependence on N, the number of negative examples. I am also not sure what $n$ is here. It seems to be an application of Rademacher uniform convergence result, but $n$ is not defined here. Could the author discuss what this is ?\n\n* In the experiments, doing an ArCL with m views requires m forward passes with m augmentations and then selecting the worst one. A normal SimCLR uses just one random augmentation instead if I understand correctly. That is an unfair comparison. Could the authors do a fairer comparison either by multiplying the number of epochs with the number of views or doing an expectation over m views for SimCLR.\n\n* Could the authors also more extreme distribution shift experiments on more realistic/synthetic distribution shifts using the framework provided in [1] ?\n\n* How do the results change with varying $k$ as in [2] ?\n\n\n=========== Post Rebuttal =================\n\nAs the discussion below shows, we had an interesting discussion with the authors and due to the reasons stated below, I am of the position that I keep the score but do object acceptance if the other reviewers want to fight for it.\n\n\n[1] How robust are pre-trained models to distribution shift? https://arxiv.org/abs/2206.08871\n[2] Bao, Han, Yoshihiro Nagano, and Kento Nozawa. \"Sharp learning bounds for contrastive unsupervised representation learning.\" arXiv preprint arXiv:2110.02501 (2021).\n[3] Balcan, Maria-Florina, Avrim Blum, and Ke Yang. \"Co-training and expansion: Towards bridging theory and practice.\" Advances in neural information processing systems 17 (2004). ", "strength_and_weaknesses": "# Strength \n* The paper studies the role of data augmentations in contrastive SSL in a rigorous manner. I wonder if it can be linked to the concept of expansion [3]. This can serve as a basis for more theoretical studies on the role of data-augmentations in various fields within ML.\n\n* The concept of ArCL is quite natural but showing how it might be necessary is an interesting take. \n\n# Cons\n\n* As I explain below, the experimental comparisons between ArCL and SImCLR might be unfair. Further, if the topic is distribution shift, I would recommend using more distribution shifts for example _Controllable Shift_s in [1].\n* The significance of the theoretical results are not immediately clear. Ofcourse, the accuracy increases with more augmentations , but other than that there does not seem to be any takeaway from the theoretical results. Also some terms, like $n$  and the absence of $N$ in Thm 5.2 is unclear.\n* Discussion of similar work like [2] is also missing. This particular work also discusses the same question but highlights the importance of the number of negative samples, which is completely ignored in this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster