The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning
Authors: Zhenmei Shi
Jiefeng Chen
Kunyang Li
Jayaram Raghuram
Xi Wu
Yingyu Liang
Somesh Jha
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Contrastive Learning
Self-Supervised Learning
Foundation Model
Complexity

Abstract: Pre-training representations (a.k.a. foundation models) has recently become a prevalent learning paradigm, where one first pre-trains a representation using large-scale unlabeled data, and then learns simple predictors on top of the representation using small labeled data from the downstream tasks. There are two key desiderata for the representation: label efficiency (the ability to learn an accurate classifier on top of the representation with a small amount of labeled data) and universality (usefulness across a wide range of downstream tasks). In this paper, we focus on one of the most popular instantiations of this paradigm: contrastive learning with linear probing, i.e., learning a linear predictor on the representation pre-trained by contrastive learning. We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously. 
Specifically, we provide analysis using a theoretical data model and show that,  while more diverse pre-training data result in more diverse features for different tasks (improving universality), it puts less emphasis on task-specific features, giving rise to larger sample complexity for down-stream supervised tasks, and thus worse prediction performance. Guided by this analysis, we propose a contrastive regularization method to improve the trade-off. We validate our analysis and method empirically with systematic experiments using real-world datasets and foundation models.

Published as a conference paper at ICLR 2023
THE TRADE-OFF BETWEEN UNIVERSALITY AND LA- BEL EFFICIENCY OF REPRESENTATIONS FROM CON- TRASTIVE LEARNING
Zhenmei Shi Jiefeng Chen Kunyang Li Jayaram Raghuram'
Xi Wu Yingyu Liang' Somesh Jha
University of Wisconsin-Madden hhmee shi, j efeng, k1 kli253, faliis,,ayaaalee..eaeeeee jayaramr, ylinr,,yiinn... jha .ec...e wi sc edu, wux @goog 1 com
Google LLC "XaiPient
Equal contribution
ABSTRACT
Pre-training representations (a.k.a. foundation models) has recently become a prevalent learning paradigm, where one first pre-trains a representation using large-scale unlabeled data, and then learns simple predictors top of the rep- resentation using small labeled data from the downstream tasks. There are two key desiderata for the representation: label efficiency (the ability to learn accu- rate classifier on top of the representation with a small amount of labeled data) and universality (usefulness across wide range of downstream tasks). In this paper, we focus on one of the most popular instantiations of this paradigm: contrastive learning with linear probing. i.e., learning linear predictor on the representa- tion tre-trained by contrastive learning. We show that there exists trade-off between the two desiderata so that one may not be able achieve both simul- taneously. Specifically, we provide analysis using a theoretical data model and show that, while more diverse pre- pre-training data result more diverse features for different tasks (improving universality), it puts less emphasis task-specific fea- tures, giving rise to larger sample complexity for down-stream supervised tasks, and thus worse prediction performance. Guided by this analysis, we propose contrastive regularization method to improve the trade-off. We validate our analy- sis and method empirically with systematic experiments using real-world datasets and foundation models.
INTRODUCTION
Representation pre-training is a recent successful approach that utilizes large-scale unlabeled data to address the challenges of scarcity of labeled data and distribution shift. Different from the traditional supervised learning approach using large labeled dataset, representation learning first pre-trains a representation function using large-scale diverse unlabeled datasets by self-supervised learning (e.g., contrastive learning), and then learns predictors on the representation using small labeled datasets for downstream target tasks. The pre-trained model is commonly referred to as a fooundation model (Bommasani et al. 2021). and has achieved remarkable performance in many applications, e.g., BERT (Devlin et al. 2019). GPT-3 (Brown et al.] 2020), CLIP (Radford et al., 202I). and Flamingo (Alayrac al.] 2022). To this end, we note that there are two properties that are key to their success: (1) label efficiency: with the pre-trained representation, only a small amount of labeled data is needed to learn accurate predictors for downstream target tasks; (2) universality: the pre-trained representation can be used across various downstream tasks.
In this work, we focus on contrustive learning with linear probing that learns a linear predictor on the representation trainrd by contrastive learning, which is exemplary pre-training ap- proach (c.g., (Arora et al.) 2019 Chen et al. (2020)) We highlight and study a fundamental trade-off between label efficiency and universality, though ideally, one would like have these two key prop- erties simultaneously. Since pre-training with large-scale diverse unlabeled data is widely used in practice, such trade-off merits deeper investigation.
Theoretically, we provide an analysis of the features learned by contrastive learning, and how the learned features determine the downstream prediction performance and lead to the trade-off. We

---

Published as a conference paper at ICLR 2023
propose hidden representation data model, which first generates hidden representation contain- ing various features, and then uses it to generate the label and the input. We first show that con- trastive learning is essentially generalized nonlinear PCA that can learn hidden features invariant to the transformations used to generate positíve pairs. We also point out that additional assumptions on the data and representations are needed to obtain non-vacuous guarantees for prediction perfor- mance. We thus consider setting where the data are generated by linear functions of the hidden representation, and formally prove that the difference in the learned features leads the trade-off. In particular, pre-training on more diverse data learns more diverse features and is thus useful for prediction on more tasks. But it also down-weight task-specific features, implying larger sample complexity for predictors and thus worse prediction performance on specific task. This analysis inspires us to propose a general method- contrastive regularization - that adds a contrastive loss to the training of predictors to improve the accuracy on downstream tasks.
Empirically, we first perform controlled experiments to reveal the trade-off. Specifically, we first pre-train on a specific dataset similar to that of the target task, and then incrementally add more datasets into pre- training. In the end, the pre-training data includes both datasets similar to the target task and those not so similar, which mimics the practical scenario that foundation models are pre-trained on diverse data to be widely applicable for various downstream tasks.
0,210
average test accuracy on tasks
Aueraged Thct Arcuracy
0.205
test accuracy on the target task
1095
0.200
0.90
0.195
0,390
Trgeet Tack
0.385
Fig. gives an example of this experiment: As we
0.180
increase task diversity for contrastive learning, it in- creases the average accuracy on all tasks from 18.3% to 20.1%, while it harms the label efficiency of an individual task, on CIFAR-10 the accuracy drops from 88.5% to 76.4%. We also perform experi- ments on contrastive regularization, and demonstrate that it can consistently improve over the typical fine- tuning method across multiple datasets. In several cases, the improvement is significant: 1.3% test curacy improvement for CLIP on ImageNet, 4.8% for MoCo v3 on GTSRB (see Table [ and / for de- tails). With these results, we believe that it is of im- this
0.175
C
CS
CSG
CSCI
00.75
10P
Number of unlabeled data
Figure 1: Illustration of the trade-off between universality and label efficiency, æraxis: from left to right, incrementally add CINIC-10 (C), SVHN (S), GTSRB (G), and ImageNet32 (I) for pre- training MoCo v2. For example, "CS" means CINIC- 10+SVHN. The average test accuracy of prediction on 4 datasets (red line) increases with more diverse pre-training data, while that on the target task CIFAR-10 (blue line) decreases. (The variance of the blue line too small be seen.) Please refer to Section 3.1 details.
portance to bring the community's 's attention trade-off and the forward path of foundation models.
Our main contributions are summarized as follows:
We propose a hidden representation data model and prove that contrastive learning is es- sentially generalized nonlinear PCA, and can encode hidden features invariant the trans- formations used in positive pairs (Section 2.T).
We formally prove the trade-off in a simplified setting with linear data (Section 2.2
We empirically demonstrate the trade-off across different methods and datasets for con- trastive learning with linear probing (Section 3.1 and 3.2)
We propose contrastive regularization method for training the predictor on target task (Section 2.2 which achieves consistent improvement our experiments (Section 3.3)
Related Work on Representation Pre-training, This paradigm pre-trains a representation func- tion on large dataset and then uses for prediction on various downstream tasks (Devlin et al, 2019 Kolesnikov et al.] 2020 Brown et al., 20200, Newell & Deng 2020). The representations are also called foundation models (Bommasani et al., 2021). There are mainly two kinds of ap- proaches: (1) supervised approaches (e.g., (Kolesnikov et al. (0200) that pre-train on large labeled datasets; (2) self-supervised approaches (e.g., (Newell Deng (020)) that pre-train on large and diverse unlabeled datasets. Recent self-supervised pre- pre-training
compete with or outperform pervised pre-traine training on the downstream prediction performance (Ericsson et al., 202T). Practical examples like BERT (Devlin al., 2019). GPT-3 (Brown et al. 2020), CL.P (Radford al. 2021)
2

---

Published as a conference paper at ICLR 2023
DALL-E (Ramesh et al.
2022). PaLM (Chowdhery et al. (022)) and Flamingo (Alayrac et al. 2022)
have obtained effective representations universally useful for a wide range of downstream tasks.
A popular method is contrastive learning, i.c., to distinguish matching and non -on-matching pairs of
augmented inputs (c.g.. (van den Oord et al. 2018 Chen et al. 2020 He et al.] 2020 Grill et al.
2020 Chen & He 2021 Zbontar al. 2021
Gao et al. 202
Some others solve "pretext tasks"
like predicting masked parts of the inputs (e.g.. DDeersch et al. 2015 Devlin et al. 2019)).
Related Work on Analysis of Self-supervised Pre-training. There exist abundant studies analyz- ing self-supervised pre-training (Arora et al., |019 Tsai et al., 2020 Yang et al. 2020; Wang &
Isola 2020 Garg & Liang 2020; Zimmermann et al. 2021]Tosh et al.
202 HaoChen
al.
2021
Wen & Li 2021] Liu et al. [2021] Kotar et al.) 2021 Van Gansbeke et al., [021,] Lee et al. 2021
Saunshi et al., 2022a] Shen al. 2022 Kalibhat al. (022). They typically focus on pre-training
or assume the same data distribution in pre-training and prediction. Since different distributions are the critical reason for the trade-off we focus on, we provide a new analysis. Some studies have
connected contrastive learning to component analysis (Balestriero & LeCun| 2022, Tian 2022 Ko (et al. 2022). Our analysis focuses on the trade-off, while also showing a connection to PCA based on our notion of invariant features and is thus fundamentally different. Recently, Cole et al. have attempted to identify successful conditions for contrastive learning and pointed out that diverse pre- training data can decrease prediction performance compared to pre-training on the specific task data. However, they do not consider universality and provide no systematic study. Similarly, [Bommasan] et al. call for more research specialization vs. diversity in pre-training data but provide study. We aim to provide a better understanding of the trade-off between universality and label efficiency.
THEORETICAL ANALYSIS
Our experiments in Section 3.1 demonstrate a trade-off between the universality and label efficiency of contrastively pre-trainet representations when used for prediction distribution different from the pre-training data distribution. See Fig. T for an example. Intuitively, from the unlabeled data, pre-training can learn semantie features useful for prediction even different data distributions. To analyze this, we need to formalize the notion of useful semantic features. So we introduce a hidden representation data model where a hidden representation (i.e., set of semantic features) is sampled and then used for generating the data. Similar models have been used in some studies (HaoChen al., [02T]] Zimmermann et al.] 2021), while we introduce the notion of spurious and invariant features and obtain novel analysis for contrastive learning.
Using this theoretical model of data, Section 2.1 investigates what features are learned by contrastive learning. We show that contrastive learning can be viewed as a generalization of Principal Compo- nents Analysis, and it encodes the invariant features not affected by the transformations but removes the others. We also show that further assumptions on the data and the representations are needed nec- essary for any non-vacuous bounds for downstream prediction. So Section [2.2] considers a simplified setting with linear data. We show that when pre- pre-trained diverse datasets (modeled as mixture of unlabeled data from different tasks), it encodes all invariant features from the different tasks and thus is useful for all tasks. On the other hand, it essentially emphasizes those that are shared among the tasks, but down-weights those that are specific to a single task. Compared to pre-training only on unlabeled data from the target task, this then leads to a larger sample complexity and thus worse generalization for prediction on the target task. Therefore, we show that the trade-off between uni- versality and label efficiency occurs due to the fact that when many usefull features from diverse data are packed into the representation, those for a specific target task can be down-weighted and thus worsen the prediction performance on it. Based on this insight, we propose a contrastive regular- ization method for using representations in downstream prediction tasks, which achieves consistent improvement over the typical fine-tuning method in our experiments Section 3.3
Contrastive Learning. Let X' C Rd denote the input space, ) the label space, and Z C R the out- put vector space of the learned representation function. Let & denote the hypothesis class of repre- sentations Φ X' + Z, and F& the hypothesis class of predictors on Φ. A task is simply data distri- bution over X' x Y. In pre-training using transformations on unlabeled data from the tasks, we have some pre- pre-train distribution Drre over positive pairs x+) and negative examples x where x* are obtained by applying random transformations on the same input (c.g., cropping or color jitter for images), and x: is an independent example. The contrastive loss is e (0(x) (0(x+) ((x-)))
3

---

Published as a conference paper at ICLR 2023
where ((t) is suitable loss function. Typically, the logistic loss {(t) log(1 exp(-t)) is used, while our analysis also holds for other loss functions. A representation Φ is learned by:
min
[xxx DDDgr pree [ (o(x) - 6(x )))] MEO
(1)
(We simply consider the population loss since pre-training data are large-scale.) Then predictor learned on top of & using m labeled points {(ri, M= from a specific target task
1 (c((((()))
min JEFo
where le is a prediction loss (e.g. cross-entropy). Usually, f is a linear classifier (Linear Probing) with bounded norm: Fa [f(2) u 2: E R* ||u||| S B), where || denotes the Eg norm.
Hidden Representation Data Model. We now consider the pre-train distribution D Dpre over (r, " To capture that pre-training can learn useful features, we assume hidden representa- tion for generating the data: first sample a hidden representation = E 2 from a distribution D2 over Rd, and then generate the input x and the label y from 2. (The space Z models semantic features, and can be different from the learned representation space Z.) The dimensions of z are partitioned into two disjoint subsets of [d] := {L. (1. .d: spurions features U that are affected by the transformations, and invariant features R that are not. Specifi- cally, let Dy. DR denote the distributions of zy and 2R. respectively, and let = g(2 denote the generative function for Then the positive pairs (x, xt) are generated as follows:
some hidden representation space
== [zn; zu]
Dz, = ~ Du, z = [=n: =t], == ===),,,x*+ = g(=+).
That is, x are from the same but two random copies of 2u that model the random transfor- mations. Finally, x is an i.i.d. sample from the same distribution as z: 2 Dz. = g(z
2.1 WHAT FEATURES ARE LEARNED BY CONTRASTIVE LEARNING?
To analyze prediction performance, we first need to analyze what features are learned in pre-training.
Contrastive Learning Generalized Nonlinear PCA. Recall that given data æ from a distribution D, Principal Components Analysis (PCA) (Pearson [901 Hotelling 1933) aims find linear pro- jection function é on some subspace such that the variance of the projected data ((x) is maximized, i.e. it is minimizing the following PCA objective:
E [1)(())))))))))))())) - E (0)(=)=']== (Eg=lllo(2) - 201
(4)
where on :- ((((('))) is the mean of the projected data. Nonlinear PCA replaces linear representa- tion functions 6 with nonlinear ones. We next show that contrastive learning is a generalization of
nonlinear PCA on the smoothed representation after smoothing out the transformations.
Theorem 2.1. If E(t)
-t, then the contrastive loss is equivalent to the PCA objective Ozn'
E le (d(x) (0(x)) - 60x ))) E [||Ø=R -bl1
where Dax = E(Ø(x) =n] = B[p(g(=)) =n) !f additionally ((x) is linear in then it is equiva- lent to the linear PCA objective -E [[ - <oll on data ह := [[ = E[g(2)|<R]
So contrastive learning is essentially nonlinear PCA when E(t) -t, and further specializes to linear PCA when the representation is linear. As PCA finds directions with large variances, the analogue is that contrastive learning encodes important invariant features but not spurious ones.
Contrastive Learning Encodes Invariant Features and Removes Spurious Features. For a for- mal statement we need some weak assumptions on the data, the representations, and the loss:
(A1) zR can be recovered from z, i.., the inputs x = g(2) from different 2R 's are disjoint.
(A2) The representation functions are the regular functions with [| B, (Vz) for some B, > 0. Being regular means there are finite L and partition of Z into finite number of subsets, such that in each subset all @ o g have Lipschitz constants bounded by L.
(A3) The loss {() is convex, decreasing, and lower-hounded.
4

---

Published as a conference paper at ICLR 2023
The first condition means the invariant features 2R can be extracted from (note that g need not be invertible). The regular condition on the representation is to exclude some pathological cases like the Dirichlet function; essentially reasonable functions relevant for practice satisfy this condition, e.g., when is Lipschitz and Φ are neural networks with the ReLUU activation. Also, note that the logistic loss typically used in practice satisfies the last condition.
We say a function f(z) is independent of a subset of input dimensions 2g. if there exists a function f' such that (() (===s) s) with probability I, where 2-s denotes the set of all 2j with j & S. We say the representation & encodes a feature 24, if Ø o g : Z + Z is not independent of 2; as long as the generative function (z)) is not independent of 2;.
Theorem 2.2. Under Assumptions (A13)) the optimal representation o* satisfies:
(1) 6* does not encode the spurious features zu: 6- g(2) is independent of zu.
(2) For any invariant feature E R. there exists B, > such that as long as the repre- sentations' norm B, 2 B;, then d² encodes 2;. Furthermore, if 2 is finite, then B; is monotonically decreasing in Pr(zR(() = = + 24 the probability that in zg and zn. the i-th feature varies while the others remain the same.
So contrastive learning aims to remove the spurious features and preserve the invariant features. Then the transformations should be chosen such that they will not affect the useful semantic features, but change those irrelevant the label. Interestingly. the theorem further suggests that contrastive learning tends to favor the more "spread-out" invariant features 2,, as measured by Pr(zn(()) = (n((1)) + 2; ]. As we increase the representation capacity B,, B, passes the threshold B; for more features 2i 6 first encodes the more spread-out invariant features and then the others.
This further suggests the following intuition for the trade-off. When pre-trained on diverse data modeled as mixture from multiple tasks with different invariant features, the representation en- codes all the invariant features and thus useful for prediction on all the tasks. When pre-trained on only specific task, features specific to this task are favored over those that only show up in other tasks, which leads to smaller sample complexity for learning the predictor and thus better prediction. However, to formalize this, some inductive bias assumptions about the data and the representation are necessary to get any non-vacuous guarantee for the prediction (see discussion in Appendix[A.] [..I) Therefore, Section 2.2 introduces additional assumptions and formalizes the trade-off.
2.2 ANALYZING THE TRADE-OFF: LINEAR DATA
To analyze the prediction performance, we first need to model the relation between the pre- pre-trainig data and the target task. We model the diverse pre- vre-training data as a mixture of data from T
D
P2
different tasks D.'s, while the target task is one of the tasks. All tasks share public feature set S of size and each task D, additionally owns a private disjoint feature set Pt of size r 8, i.e., P. E S 0 and Pc, Pt2 0 for invariant features for DD are then R2 = S USUI P.. All invariant features are -T=RRe Re, and spurious features are U [d],R.
la (Fig. 2 The
Figure 2: Illustration of the features in our data distributions.
In task D,, the (z, x²) are generated as follows:
ZR. ~ (), ZRAR = O, zu ~ 1(0,1), = [ my]. [ +
x = ==)); x = g(2
(6)
= ~
(7)
and x is simply an i.idd copy from the same distribution as x. In practice, multiple in- dependent negative examples are used, and thus we consider the following contrastive loss minge B( [ (d(x) (Ø(x+)= (((=))( E,, <(x])) for a convex and decreasing E(t) to treitrain a representation ø. Then, when using ø for prediction in the target task D., the predictor class should contain a predictor matching the ground-truth label:
=o.= = [f(2) = u 2::EER RK ||uu|| S Bo.t)
(8)
where Ba. is the minimum value such that there exists ue E Fe,t with y = u1 ((x) on Dr.
5

---

Published as a conference paper at ICLR 2023
Now, given the necessity of inductive biases for non vacuous guarantees (see Appendix A.I]. and inspired by classic dictionary learning and recent analysis on such data (c.g., Olshausen & Field 1997]
Wen & Li (202I); Shi et al. 2022)), we assume linear data and linear representations:
x: is linear 2: x = g(2) = 2 where M E gdd is an orthonormal dictionary. Since linear probing has strong performance pre-trained representations, thus assume that the label in each task is linear in its invariant features (ut) zR, for some uj E RR.
The representations are linear functions with weights bounded spectral/Frobeeius norms:
4 = {)(z)) = Wr: : WeR**d fxd || |SVT).
Here the norm bounds are chosen to be the minimum values to allow recovering the invari- ant features in the target task, i.c., there exists
such that ((x) [=R, 0j.
We compare two representations: a specific one pre-trained on unlabeled data from the target task Dr, and a universal one pre-trained on an even mixture of data from tasks. (Appendix B provides analysis for more general cases like uneven mixtures.) This captures the situation that the pre- training data contains some data similar the target task and also other less similar data. Let be the weights the shared and task-specific invariant features, respectively. Also, assume the prediction loss &e is I--Lipschitz.
Ur, Ejes(u;) (u;); )3 and vx2 = Ejep, (u)) )
Proposition 2.3. The representation o* obtained on an even mixture of data from all the tasks : T) satisfies 6² og(2)= = (Ejes Vazzez e EyEm, VBzjes) for some E [0, 1]. min (1, T1-))))) ). where ej are the basis vectors and Q is any orthonormal matrix.
The Empirical Risk Minimizer à E For on o² using m labeled data points from Dr has risk
B(x,g)_,,1 (E (x(x),y]
Ut,1 a
V1.
r ( sa (r () + +8
2 2l(60/)
44
3
m
Proposition 2.4. The representation of obtained on data from 'D. satisfies of o g(2) =
Q (EjER, z4e, where ej 's are the basis vectors and Q is any orthonormal matrix.
The Empirical Risk Minimizer à E For on of using m labeled data points from Dr has risk
B(a.)) le (f))), S 4L / 1|Gll +1 8V 2l1(4/0)
While on task (i t), any linear predictor on of has error at least min,, ED, [eo(" 2ss w)].
Difference in Learned Features Leads to the Trade-off. The key of the analysis (in Appendix B about what features are learned in the representations. Pre-trained on all T tasks, o² is a rotation of the weighted features, where the shared features are weighted by va and task-specific ones are weighted by vB. Pre-trained on one task D,, ot is rotation of the task-specific features RG So compared to of. ø² encodes all invariant features but down-weights the task-specific features Pt.
The difference in the learned features then determines the prediction performance and results in trade-off between universality and label efficiency: compared to of o² is useful for more tasks but has worse performance on the specific task D, For illustration, suppose task-specific features equally important for the labels on the target task: v4,1 V2 ||u;] I| /2.
2s, and the shared and
In Appendix B.3 we show that d' has a 1, # and the error is O (L Tr m (l while the error using of is (L me II) Therefore, the error when using representations pre-trained on data from T tasks is O(vT) worse than that when just pre-training on data from the target task. On the other hand, the former can be used all T' tasks and the prediction error diminishes with the While the latter only encodes R, and the only useful features the other
labeled data number tasks are zs. then even with infinite labeled data the error can be large (2 min, El 2g, y)}, the approximation error using only the common features 2ş for prediction).
Improving the Trade-off via Contrastive Regularization. The above analysis provides some guidance improving the trade-off, in particular, improving the target prediction accuracy when

---

Published as a conference paper at ICLR 2023
given pre-trained representation ø² It suggests that when 6' is pre- pre-trained on diverse data, one can update it by contrastive learning on some unlabeled data from the target task, which can get better features and better predictions. This is indeed the case for the illustrative example above. We can show that updating o² by contrastive learning on Dr can increase the weights & on the task-specific features 2 and thus improve the generalization error (formal analysis in Appendix [B44). (B.4).
In practice, typically one will learn the classifier and also fine-tune the representation with a la- beled dataset {224, yi)J from the target task. We thus propose contrastive regularization for fine-tuning: for each data point (a, y). generate contrastive pairs R = {[z Ø 3 _-]}} by applying transformations, and add the contrastive loss on these pairs as a regularization term to the classifica- tion loss:
((f()(z)).)) +
E
(((TT) (o(Et) - o(t))))
((6())(((Et)))
R
1E,++,2-EER
This method is simple and generally applicable to different models and algorithms. Similar ideas have been used in graph learning (Ma al.] 202T]. domain generalization (Kim et al.) 202T) and semi-supervised learning (Lee et al. [022]. while we use it in fine-tuning for learning predictors. Our experiments Section [33 show that it can consistently improve the prediction performance compared to the typical fine-tuning approach.
3 EXPERIMENTS
We conduct experiments to answer the following questions. (Q1) Does the trade-off between uni- versality and label efficiency exist when training on real datasets? (Q2) What factors lead to the trade-off? (Q3) How can we alleviate the trade-off, particularly in large foundation models? Our experiments provide the following answers: (A1) The trade-off widely exists in different models and datasets when pre-training on large-scale unlabeled data and adapting with small labeled data (see Section 3.1. This justifies our study and aligns with analysis.
(A2) Different datasets own many private invariant features leading to the trade-off, e.g., FaceScrub and CIFAR-10 do not share many invariant features (see Section 3.2). It supports our analysis in Section 2.2 (A3) Our proposed method, Finetune with Contrastive Regularization, can improve the trade-off consistently (see Section 3.3 Please refer to our released code for more details.
3.1
VERIFYING THE EXISTENCE OF THE TRADE-OFF
0.231
overage test accuracy on tadks best the target task
test accurccy on the torget task
average test accuracy on tasks
0:3
0.3
test tacts
0.47
Trrpee Trask Textt Arr
6.35
0.3
1.45
0.30
0,58
0.32
0.30
Task
Averaged
3.2
0.43
Gars
0.32
0.02
ca0e
6:0
0:30
10
55
C5G
C5c
FSS 3.40
Namber of unlaabeled data
Number of unlabeled data
Number unlabeled data
(a) CIFAR-10
(b) MNIST
(c) Fer2013
Figure 3: Trade-off between universality and label efficiency for MoCo v2. Appendix C.5 shows similar results for more methods and datasets. r-axis: incrementally add datasets for pre-training MoCo v2. (a) Pre- training data: CINIC-10 (C), SVHN (S), GTSRB (G), and ImageNet32 (I). E.g., "CS" the z-axis means CINIC-I0+SVHN Target task: CIFAR-10. 10. Red line: average test accuracy of Linear Probing on all 4 datasets.
Blue line: test accuracy on the target task. (b) EMNIST- EMMNSST-Digit&&LL Digiititti&ttteee (E), Fashion-MNIST (F), GTSRB (G), ImageNet32 (I). Target: MNIST. (c) FaceScrub (F), CIFAR-I0 (C), SVHN (S), ImageNet33 Nt33 (I). Target: Fer2013. Note that training does not follow the online learning fashion. e.g., the model will pre-train from scratch (random initialization) the CSG datasets, rather than using the model pre-trained on the CS datasets.
Evaluation & Methods. We first pre-train a ResNet18 backbone (He et al. 2016) with different con- trastive learning methods and then do Linear Probing (LP, i.e., train a linear classifier on the feature
https::/ whw. com zhmei shi http///reeeoooo oot.ddd--a-ccootraatttaa vee learning

---

Published as a conference paper at ICLR 2023
0.8
average test accuracy on all tasis the
average test accuracy tacks
Reciaracyy
task
0.0
Aeseagee Teet
035
6.23
Task
05-
Number of unlabaicd data
Number of otuntabeles sata
(a) MoCo v3 (backbone ViT-S)
(b) SimSiam (backbone ResNet50)
Figure 4: Trade-off between universality and label efficiency on ImageNet. æ-axis: from left to right, crementally add ImageNet-Bing Bird (B), ImageNet- ImageNet-Vehicle (V),
(+), and ImageNet (AIL)) for pre-training (a) MoCo v3 with hackhone ViT-S (h) SimSiam with backbone ResNet50. For example, "BV" means ImageNct-Bal Bird + ImageNet- Vehicle. Target: ImageNet-Bird.
extractor) with the labeled data from the target task. We report the test accuracy a specific target task and the average test accuracy on all pre-training datasets (i.e., using them as the downstream tasks). Appendix C.2 presents full details and additional results, while Fig. shows the results for the method MoCo v2. The size and diversity of pre-training data are increased on the x-axis by incrementally adding unlabeled training data from: (a) CINIC-10, SVHN, GTSRB, ImageNet32 (using only a 500k subset); (b) EMNISS-DDigis&LLettee
Fashion-MNIST. GTSRB, ImageNet32; (c) FaceSerub, CIFAR-10, SVHN, ImageNet32. We further perform larger-scale experiments: (1) on ImageNet (see Fig. 4 (2) on ImageNer22k and GCC-15M (see Appendix CC2.1).
Results. The results show that when the pre-training data becomes more diverse, the average curacy on all pre-training datasets increases (i.e., universality improves), while the test accuracy on the specific target task decreases (i.c., label efficiency drops). This shows a clear trade-off between universality and label efficiency. It supports our claim that diverse pre-training data allow learning diverse features for better universality, but can deig-weight the features for specific task result- ing in worse prediction. Additional results in the appendix show similar trends (e.g., for methods NNCLR and SimSiam). This validates our theoretical analysis of the trade-off.
3.2 INSPECTING THE TRADE-OFF: FRATURE SIMILARITY
Here we compute the similarity of the features learned from dif- ferent pre-training datasets for a target task. For each pre-trained model, we extract set of features for the target task Fer2013 using the pre-trained representation func- tion. Then we compute the similar- ities between the extracted features based on different pre-training dataset pairs using linear Centered Kernel Alignment (CKA) (Korn- blith et al. 2019), widely used tool for high-dimensional feature comparison. Figure reports the results (rows/columns are pre- training data; numbers/colors show the similarity). The left figure shows that the features from different pre-training datasets have low similarities. This is consistent with our setup in Section 2.2 that different tasks only share some fea- tures and each owns many private ones. The right figure shows decreasing trend of similarity along each row. This indicates that when gradually adding more diverse pre- pre-training data, the learned rep- resentation will encode more downstream-tar iirrelevant features, and become less similar to that prior to adding more pre- vre-training data. Additional results with similar observations, finer- finer-grained investigation into the trade-off, and some ablation studies are provided in Appendix C3
LIAST SHIN crew dee
0.058
0.07
0.15
0.39
PANH SCBATT 4011
0.11
0.25
0:31
0.64
0.56
0.19
0.24
MONE
0.3
0.83
SOND
Ne132
W432
Figure 5: Lincar CKA similarity among Fer2013 features from MoCo v2 pre-trained different datascts. Left: each representa- tion the first four columns/rows is pre-trained "Union" indicates model pre-trained joint datasets. Right: from left column to right, from top row to bottom, we incrementally add datasets for pre-training.
single dataset. union the four dis-
8

---

Published as a conference paper at ICLR 2023
3.3 IMPROVING THE OFAD FINETUNE WITH CONTRASTIVE REGULARIZATION
Evaluation & Methods. We pre-train ResNer18 by MoCo v2 as in Section 3.] and report the test accuracy on CIFAR- 10 when the predictor is learned by: Linear Probing (LP), Finetune (FT), and Finetune with Contrastive Regulariza- tion (Ours). LP follows the training pro- tocol in 3.. FT and Ours learn a linear predictor and update the represen- tation, and use the same data augmenta- tion for fair comparison. FT follows MAE (He et al. 2022 while Ours uses MoCo v2 contrastive loss and regularization coefficient À = 0.1. More details and results are given in Appendix C.4
Pre-training dataset
Method
CINIC-10 10
+SVHN
+GTSRB
#ImageNet32
LP
88.41
85.18:0011
82.07:00011
75.64
FT
93.58:0014
93.35::0.00
93,42:1:0.15
92.92
Ours
94.51=0002
94.26=00011
94.32:t0.13
93.66=2012
Table 1: Test accuracy on CIFAR-I0 with different evalua- tion methods on MoCo v2 by using all CIFAR- 10 training data, From left to right: incrementally add datasets for pre-training.
Results. Table I shows that our method consistently outperform the other baselines. In partic- ular, it outperforms the typical fine-tuning method by about 0.7% - 1%, even when the latter also uses the same amount of data augmentation. This confirms the benefit of contrastive regularization. To further support our claim, Fig. 13 Appendix visualiisl the features of different methods by t-SNE, showing that contrastive regularization can highlight the task-specific features and provide cleaner clustering, and thus improve the generalization, as discussed in our theoretical analysis.
CLIP
MoCo v3
SimCSE
Method
NmtgeNet
SVHN
GTSRB
CIFAR-10
SVHN
GTSRB
IMDB
AGNews
LP
77.84+0..02
63 63.44:00.11
86.56:20.01
95.82:t0.01
61.92::0.01
75.37::0.01
86.49:0..16
87.76::0.66
FT
83.65:t0.01
18.22210.18
90.74::0006
96. 17::0.11
65 35.336:::33
76.45:::0.99
92.31
93.57:10.23
Ours
84.94
78.72
92.01:10.28
96.71
66.29:10.20
81.28
20.03
93.94:0002
Table 2: Test accuracy for different evaluation methods on different datasets using training data and using foundation models from CLIP, MoCo v3, and SimCSE. Data augmentation is not used for LP (Linear Probing). For FT (Finetune) and Ours (our method), 10 augmentations to each training images are used for CLIP, MoCo v3, and unique augmentation in each training step is used for SimCSE. More results are in Appendix C.4.1
Larger Foundation Models. We further evaluate our method on several popular real-world large representation models (foundation models). On some of these models, the user may be able fine- tune the representation when learning predictors. On very large foundation models, the user typically extracts feature embeddings of their data from the models and then trains a small predictor, called adapter (Hu et al. 2021] (ung et al., 2022). these embeddings. We evaluate CLIP (ViT-L (Doso- vitskiy et al. 2020 as the representation backbone), MoCo v3 (ViT-B backbone), and SimCSE (Gao et al. 2021) (BERT backbone). They are trained on (image, text). (image, image). and (text, text) pairs, respectively, so cover a good spectrum of methods. For CLIP and MoCo v3, the backbone is fixed. LP uses linear classifier, while FT and Ours insert two-layer ReLU network as adapter between the backbone and the linear classification layer. Ours uses the SimCLR contrastive loss on the output of the adapter. For SimCSE, all methods use linear classifiers. LP fixes the backbone, while FT and Ours train the classifier and fine-tune the backbone simultancously. Ours uses the SimCSE contrastive loss on the backbone feature. We the regularization coefficient À 1.0.
Table 2 again shows that our method can consistently improve the downstream prediction perfor- mance for all three models by about 0.4% 4.8%, and quite significantly in some cases (c.g.. 1.3% for CLIP on ImageNet, 4.8% for MoCo v3 on GTSRB). This shows that our method is also useful for large foundation models, even when the foundation models cannot be fine-tuned and only the extracted embeddings can be adapted. Full details and more results are provided in Appendix ...11
4 CONCLUSION AND FUTURE WORK
In this work, we have shown and analyzed the trade-off between universality and label efficiency of representations in contrastive learning. There are many interesting open questions for future work. (1) What features does the model learn from specific pre- pre-training and diverse pre-training datasets beyond linear data? (2) Do the other self-supervised learning methods have a similar trade-off? (3) Can address the trade-off better gain both properties at the same time?

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The work is partially supported by Air Force Grant FA9550-18-1-0166 the National Science Foun- dation (NSF) Grants CCF-FMitf--18369)
IIS-2008559, SarCC-Frontirr--18044 CCF-2046710 10 and ECCF-1652144 and ARO grant number W91 INF-17-1-0000.. Jiefeng Chen and Somesh Jha are partially supported by the DARPA-GARD problem under agreement number 885000. Jayaram Raghuram was partially supported through the National Science Foundation's ' grants CNS-21 12562 and CNS-2003129
REFERENCES
lean-Baptist Alayrac. Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 14198, 2022.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In 36th International Conference on Machine Learning, ICML 2019, pp. 9904-9923, International Machine Learning Society (IMLS), 2019.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning cover global and local spectral embedding methods. arXiv preprint arXiv:2205. //508, 2022.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quiney Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing. Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Ku- ditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.07258, 2021. URL /ttpss:
org/ abs/ / 2110/1011.100 07258
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httpt:ttts//ppeeeeeeeeeeeeeeeeee 1457c0d6bfcbbl wwt.effcc4444 496 741 Soffbacc142fe0 hatteeaaaaaattttttaaaaa .tml
neurips httccc//pa. wwtppppr///e.cccppcc..c..ppe.. 2020/hash/
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize tail-tail visual concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Muchine Learning Reseurch, pp. 1597-1607. PMLR, 2020. WtRLhtt:: http:hhttt/p::ceeeeeee http:///ccceeee.g..
mlr. www.ppssssve... v wttp:///999cche2m hhmll
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE
Conference on Computer Vision and Puttern Recognition. CVPR 2021, virtual, June 19. 25, 2021, pp. 15750-15758. Computer Vision Foundation / IEEE, 2021. URL https:
thecvf. wtt.
Chen Exploring.
Si Simple Simmese htt.a/ssseeeeeeeeteeeeooaaaaannn 2021___________1 paper.r........ .html
10

---

Published as a conference paper at ICLR 2023
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN) pp. 2921-2926. IEEE, 2017.
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does contrastive visual representation learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14755-14764 2022.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey, Cinic- 10 is not imagenet 2018.
or cifar-10. arXiv preprint
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Jacob Devlin, Ming-War Wii Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4186 Association for Computational Linguistics. 2019. doi: 10.18653/ v1/n19- 1423. URL http::tttt://tt http://ooo..rr/1////// 1ww53/
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. 20/5 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1422-1430. IEEE Computer Society, 2015. doi:
10.1109//CC.2201..1.11 URL httpt: O0tE::///o..o00/0..1111//111 111.1.111111/11111111100
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman, With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588-9597 2021.
Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How well do self-supervised models transfer? In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 5414-5423. Computer Vision Foundation IEEE, 2021. URL htttps://ppnacceess.oo thecvf. com/ cotm://onneeet///PP//////ttaaaa
How Wwwl Do Sesl1_po_Self-300 Lel___ Maceedddddeeleeeeeeeeeeddeeeeeee _ CVPR 2021 wwwpppppppppra.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence
embeddings. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Siddhant Garg and Yingyu Liang. Functional regularization for representation learning: unified theoretical perspective. Advances in Neural Information Processing Systems, 33:17187-17199 2020.
Ian J Goodfellow, Dumitru Erhan, Pierre Lue Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In International conference on neural information processing, pp. 117-124. Springer, 2013.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent A
11

---

Published as a conference paper at ICLR 2023
new approach self-supervised learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, De- cember 62. 2020, virtual, 2020. URL httpttts///rrcceeinnggg.s. neurips. cc/ paper/ 2020/hash/ 3ada 701 42b1 7b81 aatt
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9726- URL
9735. Computer Vision Foundation IEEE, 2020. doi: 10. 10010//C2226000220000000 2020. 00975.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Harold Hotelling. Analysis of a complex statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Neha Mukund Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Towards better understanding of self-supervised representations. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9619-9628, 2021.
Ching- Yun Ko, Jeet Mohapatra, Sijia Liu, Pin-Yu Chen, Luca Daniel, and Lily Weng. Revisiting contrastive learning through the lens of neighborhood component analysis: integrated frame- work. In International Conference on Machine Learning, pp. 11387-11412. PMLR, 2022.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, volume 12350 of Lecture Notes in Computer Science, pp. 491-507. Springer, 2020. doi: 10. 008-3---00-8558-77777 29. URL https:: /tlpss:/doi.. /wo. org 10 1000 --1777833-3000000 3-030 58558-7 0000//978----000--8----------
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR. 2019.
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting
contrastive self-supervised representation learning pipelines. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9929 993999999.9.01 IEEE, 2021. doi: 10. 10011099CC4489222220000000 100.1109/IcC44B922 20014482222222....... 2021. 00980]
URL htttt:::: /oi. ////
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11) (6(11))2278-2324 1998.
12

---

Published as a conference paper at ICLR 2023
Doyup Lee, Sungwoong Kim, Ildoo Kim, Yeongjae Cheon, Minsu Cho, and Wook-Shin Han. Con- trastive regularization for semi-superrised learning. In Proceedings of the IEEE/CVE Conference on Computer Viston and Pattern Recognition, pp. 3911-3920. 2022.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems, 34:309- 323, 2021.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning more robust to dataset imbalance. In NeurlPS 2021 Workshop Distribution Shifts: Connecting Methods and Applications, 2021.
lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Kaili Ma, Haochen Yang, Han Yang. Tatiana Jin, Pengfei Chen, Yongqiang Chen, Barakeel Fanseu Kamhoua, and James Cheng. Improving graph representation learning by contrastive regulariza- tion. arXiv preprint arXiv:2101. 1155, 2021.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150,
Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/www. acw.aaclio web. org/ www.toooooooyyoooooo
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seatlle WA, USA, June 13-19, 2020, pp. 7343-7352. Computer Vision Foundation / IEEE, 2020. doi: 10.1199CVPR42600.2000000 com/ com//onnentt_vVve 2010RR22200222 Pretraining for L caasks_cvvR__c2___pp_ ptt.ppaperr.mmll
URL
thecvf.
How Usefuleellll.. Is See_SSSff_--e- Supervised
Hong- Hong-Wei Wei Ng and Stefan Winkler. data-driven approach to cleaning large face datasets. In 2014
IEEE international conference image processing (ICIP), pp. 343-347. IEEE, 2014.
B. Olshausen and D. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 11133111-325, 1997.
Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space. The London Edinburgh, and Dublin philosophical magazine and journal of science, 2(1 2(11):559-572, 1901.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763, 206.7777...... PMLR, 2021. URL
mlr www.m.....s......
www.foooddddooo html
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 2022.
Tal Ridnik, Emanuel Ben-Baruce Asaf Noy, and Lihi Zelnik-Manor Imagenet-21k pretraining for the masses. In Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 1), 2021.
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202. 14037, 2022a.
13

---

Published as a conference paper at ICLR 2023
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19250-19286. PMLR, 17-23 Jul 2022b. URL
html
Piyush Sharma, Nan Ding, Sehastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyme, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-2565, 2018.
Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 19847-19878. PMLR, 2022.
Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations, 2022.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323- 332, 2012. doi: www.0o/...euee.2222.....o neunet 2012 02.016
URL https::/dol. org/ 10.6/j
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022.
Yuandong Tian. Deep contrastive learning provably (almost) principal component analysis. arXiv preprint arXiv:2201. /2680, 2022.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory. pp. 1179-1206. PMLR, 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from a multi-view perspective. In International Conference on Learning Representations, 2020.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807 007.03748, 2018. /ttpp//:://arxxx org/ abs/1807 03748
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Gool. Revisiting contrastive methods for unsupervised learning of visual representations. Advances in Neural Information Processing Systems, 14:16238-162511 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9933 PMLR, 2020.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pp- 11112-11122. PMLR, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf.
a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 178.07777 2017.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Uni- fied contrastive learning in maage-text-label space. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 19163-19173, 2022.
14

---

Published as a conference paper at ICLR 2023
Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang, Shanghang Zhang, and Pengtao Xie. Transfer learning or self supervsie learning? A tale of two pretraining paradigms. CoRR, abs/2007.04234, 2020. URL httlt: /ttpp://arxii..... org/ abs 0020000.0233
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Proceedings of the 38th International Con- ference on Machine Learning, ICML 2021, 18-24 July 2021. Virtual Event, volume 139 of Proceedings of Machine Learning Research, Pp. 12310-1233 PMLR, 2021. URL http: //proceadings.. ml wwt.sss..r.rrree/eep.1. //tottoootaaaaa httll
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
15

---

Published as a conference paper at ICLR 2023
Appendix
A PROOFS FOR SECTION 2.1
Theorem A.1 (Restatement of Theorem 2.1 ((t) -t, then the contrastive loss is equivalent
the PCA objective on Øsni
(e(x))
o(x }])] = E [||0=R - 00l
(10)
If additionally Φxx) is linear in then the contrastive loss is equivalent to the linear PCA objective on data from the distribution ps of Ezu [x]]:
E (é(x) [p(x*)
(o(x)))((xx)))>>>>)))))))))))))) - 6(x )I)] = -E [[|Ó(z) - <b 1bol1
(11)
Proof. We first present some preliminaries for the proof. Recall that in our hidden representation data model g(=). The learned representation is ((x) ((g()) - e g(z). For brevity, let us
define Φ(x) = Φ g(=) == l(==). Also, the hidden representations corresponding to (I, xt, xx)) are
given by (z, 2 where
z = [ER; Zu,, x+ = [ =t), = = - =7].
where zR and n are sampled independently from the distribution DRi and wy, z and žn are sam- pled independently from the distribution Dy. The expectation an arbitrary function f(z, =,,² can be simplified follows:
(a,atio-)) [f(e, 2t,
7 ((2,22,27)
= (en. ( [f(z,zt, x-) 2n, 27o
The second step follows the law of iterated expectations.
The negative expected contrastive loss is
(n 1
-(x [ ([(((())TT(((((+t)))))) - x(g(=)))))
(12)
= E ,
(13)
(x,
[M(=) [1(=+) - - []]
(14)
E [h(2) [h(=)T(h>*)) [h(=+) ->>>>>] ) 2n . ]]
(15)
E (ER
IE. (M(2) zn] (TE [A(=+) =M]]
E [h<=-) [
(16)
E (FR==))
E (xx) |
EE [o(x+) [
E [(((x)) | |
(17)
(e
$ +
(18)
The second equality follows from the choice of loss E(1) = -1, and the fourth equality follows from the fact that 2U, 2 and zy are sampled independently from the distribution Dy. Also, we have defined ΦzR := E [6(x) zn).
Denote the centered representation as den Φzn dv. Then we have
- -Ei ---) [e (d(x)) [[((=t) - 6(x }1)]
(19)
T @0en - -0=a]]
= E (2R
(20)
(0En + Φo)T
(ØEn + $0 - क. (b0)
(21)
(5ee + T0)
(22)
(((
2z
T J - 6
+ E
[
(2.n - -0=z)
(23)
2n
16

---

Published as a conference paper at ICLR 2023
Since 6zn and Ozn independent with mean O, have (en 10 Pm 0, E (=x.= 100 p=nl = O, and B (=x,,= n 1 Φ =n
O. Therefore,
Beexot, x (((())]((())
(24)
Ben [22²² pzn]
(25)
Bun [1
(26)
Bug [1 doll")
(27)
which is the PCA objective on the mean representation 6AR- If additionally 6(x) is linear in x, then
-B,0 [ (((()))](<>>))))))))())(()))))))
[d(x o(x
(28)
BEg 11 doll
(29)
Ez [| ((x))
(30)
which is the linear PCA objective on the data from the distribution of ã = E[x[aR]-
Theorem A.2 (Restatement Theorem 2.2 Under Assumptions (A1)(2)AA3):
(1) The optimal representation 6" does not encode zu: o" o g(2) is independent of zu-
(2) For any invariant feature E R, there exists B, > O such that as long the representa- tions' norm B, 2 B,, the optimal representation encodes 2y. Furthermore, if 2R is discrete, then B, is monotonically decreasing in Pr[zn,(1) 2RV(),2 24 # 2, J. the probability that in zg and zg. the i-th varies while the others remain the same.
Proof. (1) Recall that
*=R El s(=) =R), Φ Ez(d s(=)) = EZR [0
(31)
Then the contrastive loss at pre-training is:
Ex _-))
[ [0(x [ })}
(32)
= E (2,2t
(((oo(()) 0o g(z )))
(33)
R (
6o g(= =n, =
(34)
E (x
(0) (00g((z) - [
zn, = 1
(35)
= E (=R.FR)
(E(oo((z)) [{((ooog(2) R (
œg - Eló o g(>-) (2)R)) )
(36)
E (=n!ag)
(oT, OsR
(37)
where the inequality comes from the convexity of {(2) and Jensen's inequality applied to the inner expectation. The inequality becomes equality when the representation function o is invariant to the spurious features zu, i.e., with probability over the distribution, & g(z) Φzn- Therefore, the spurious features zy are not encoded in the optimal representation, proving the first part.
(2) First consider the case when has discrete values from finite set. When the generative function g(z) is not independent of 2;. we assume for contradiction that the optimal representation & is independent of 24. From (1), we know that it independent of zy. there exists an f such that g(2) f(znt1). Without loss of generality, suppose U = 0. then Φ g(2) f(z_.).
Since the generative function g(z) not independent of 2,, there exist and = such that z 2I4-
2; # 2, ((2) # g(= ) and 2, 2 have non-zero probabilities. So Pr[z-: =_, 24 # 2 0.
Now construct new representation function E k n = |Z| such that do g(2) = h(z)) as follows:
h(2)) (11- 101--af(2a0) all
(38)
17

---

Published as a conference paper at ICLR 2023
where I is the one-hot encoding of the value 2. Note that ø still satisfies that norm bound since [|x(z)) = [|((2 = [[ We next show that the contrastive loss of ø can be smaller than that of Φ, leading to contradiction and finishing the proof.
The contrastive loss of Φ (using the fact that a = 2 when U = 0) is
B(z,a-) (h(2) T h(2) - h(2) ((=)T<(==))))) m(=]))]
(39)
= B(),,-)) (M(== M(x) - M==)
+ Re, Pr(==) = Pr(= pr(
(40)
We only need to consider the first term.
[= ) [ [1())) (=)) h(2=))) 2a # 2 Pr(z # 2
(41)
= B(a [ (11((==)))
af) f(zI;)) 12- & 2Ix Pr|z_: # zI,
(42)
+ (x [ ([ |= 24 + 2, Pr[z_; 2I, 24 7 27
(43)
T2
When a = 0, the above reduces to the corresponding terms for Φ so we would like to show that there exists non-zero a that leads to smaller loss values.
Recall that E(.) is decreasing by property (A3). Let V1/2/B,, where B, (II- Then when switching from & to Φ. T2 goes from {()) to E(1/2), constant reduction. For Ti. if f(=_1)" ((x=,) is positive, then Zi decreases; if f(=1) f(zī,) is negative, then Ti in- creases from {BB? f(z_.)
f(z,)) -((==) |a² f(=z)) S. 1 (by the Cauchy-Schwarz inequality); so the increase in Ti diminishes when B, grows, by the property (A3) of f. Then when B, is large enough, the increase in T₁ is smaller than the decrease in T2. So from o to o. the contrastive loss decreases, contradicting that is opti- mal. Finally, since the reduction in (43) is smaller when Pr(z-i = =_ 2i # 2, is smaller, then B; needs to be larger. So B, is monotonically decreasing in Pr(z_; = =_, # 2,
to L(B?
((==))((([)) + a²
Note that
Now consider the general case when 2 may not be from a finite set. For any 6o > O, there exists a & ball B of bounded radius such that the probability of outside the ball is most ep. Since o g's are regular by assumption, there exists a partition Z n B into finitely many subsets such that in each subset and for each Φ o g. the function value varies by at most cp. Construct a new distribution D' for 2: select a representative point in each subset, and put a probability mass to it equal to that of the original distribution D. this subset, and normalize the probabilities over the subsets. The new distribution is over finite set so the above argument holds. Furthermore, the difference in the Ti term for D' and D₂ can be made arbitrarily small by choosing sufficiently small co: similarly for T2. Then the argument also holds for D,, which completes the proof for the general case.
A.1 INDUCTIVE BIASES ARE NEEDED FOR ANALYZING PREDICTION SUCCESS
We have analyzed what features are encoded in the representation. However, encoding the infor- mation does not equate good prediction performance, in particular, with linear predictors. Re- cently, Saunshi et al. (2022b) demonstrated that existing analyses that ignore the inductive biases of the model and algorithm cannot adequately explain the prediction success, and provided examples where such analysis can lead to vacuous bounds. One may wonder if our hidden representation data model can provide inductive biases that avoid such vacuous bounds. Unfortunately. similar issues as in Saunshi et al. (2022b) remain.
To illustrate that inductive biases are still needed in our data model, consider the following simple example. Suppose zn E 12 and can be recovered from æ; the label y is simply the first coordinate in æR. Suppose the representation satisfies Φ(x) E R², [²]](( = 1, and contrastive h(=R), and h((-1,-1)) = (1, (((1, = (0, - -1), ()) = (0, 1). It can be verified that this ó is optimal for the contrastive loss. However,
learning uses the logistic loss {(z). Let d(x) be such that e g(2)
(,,00,,((-1,,,
the representation o, the classification is an XOR- problem (Fig. (). for which there is no non-trivial error bound for linear predictors. This contradicts the success of Tinear probing in practice.
18

---

Published as a conference paper at ICLR 2023
XOR structure in the space of & in 2-d
y-1
y=.1
-1.5
120
-0.5
0-8
0
0.5
1.5
-0.5
-1:0
-1.
Figure 6: A two-dim example of XOR structure in the space of Φb
Furthermore, some restrictions on the data distributions are also needed. Suppose all optimal rep- resentations are linearly separable with certain inductive biases the representation function class. Suppose the label y depends on 2g. Without restrictions on the labeling function, one can consider random y [[11++1} over any 2R. Then for any linear predictor any optimal representation, in expectation the error 1/2, so there is always a labeling function for which no non-trivial error be achieved. Our analysis thus requires restrictions on the dependence of the label zn (in particular, we will assume linear dependence).
B PROOFS AND MORE ANALYSIS FOR SECTION 2.2
B.1 LEMMAS FOR A MORE GENERAL SETTING
We will prove the results in a more general setting, where the mixture can uneven and the vari- ances of different types of features can be different. The results in Section 2.2 then follow from these lemmas.
In the more general setting, the diverse pre-training data is mixture of data from T different tasks D,'s, while the target task one of the tasks. In the mixture, the task Dj has weight w, > 0 and Ei = 1. All tasks share public feature set S of size s, and each task D, additionally owns a private disjoint feature set Pt of size r 8, i.e., P.
for [T] and Pt, n Pt. 0 for t * tz. The invariant features for DD are then R; = SU P.. All invariant features are U 1 R₂ S R, = |/], and spurious features are U = [d] \ R. In task D., the positive pairs (æ, x are generated
as follows:
2s ~ N(0, 2 I), zn. N(O. oh 1), 2RAR, 0,
(44)
20 N/O, o(, = x=
(45)
= ~ N NO, = = x g(=
(46)
and x is simply an i.i.d. copy from the same distribution as In practice, multiple independent negative examples are used, and thus we consider the following contrastive loss
min B(z,zt) [ (6(x) (0(x*) E.- p(x 1))] dE@
(47)
to pre-train representation Φi Then, when using á for prediction in the target task D,, the predictor class should contain predictor matching the ground-truth label, consider the class:
Fo, (5(=) u. ue
R ||u, S Bo.
(48)
where Ba is the minimum value such that there exists E Fa. with y = ue ((x) on D..
Recall that assume linear data model and linear representation functions d:
19

---

Published as a conference paper at ICLR 2023
x: is linear in g(2) M2 where M R is orthonormal dictionary. The label in task Dr is linear in its invariant features = (u.) 2R, for some ut R'.
The representations are linear functions with weights of bounded spectral/Frobenius norms:
= fő(z) = Wr WeR*xd
| [Wll|
Here the norm bounds are chosen to be the minimum values to allow recovering the invari- ant features in the target task, i.e., there exists
such that ((z) = [=R, 0].
Lemma B.1. Consider the above setting. Let a, ault E [T]) be the optimizer for
min
Σ
wyl [e (ão's.Z + ā.o R. 2i)]
(49)
R.01.....G
tm1
subject
âs Σ àr(r 5) sr
(50)
ā ā [0,1].
(51)
where Z ~ X2 and Ze 2=_..
Then the optimal representation o² (I) the loss (47 in contrastive learning satisfies 6" (x) = W* with any W* of the form:
W = [QA", 0
(52)
where Q BRxk is any orthonormal matrices, A is a k k diagonal matrix with
(Va Voi
if S, i E Pt, otherwise,
AJ,
(53)
and the matrix of zeros has size l x (d - k).
Proof. For each D..
E [e [e(x)) E, [[]]
(54)
= ( [e( Mz) (W - E.- MM Mx D)]
(55)
(x,,z)) [e (M W (M)(** 12-122_1)
(56)
2 EER [ [((Erol=)) M W W
(57)
= EeR [ ([ZRi " T W ([>R: -
(58)
E.R [ ([[M 0]][3]]
(59)
where the inequality comes from the convexity of E(t) and Jensen's inequality. Similar to The- orem 2.2] the equality holds if and only if W Mz does not depend on zū and W + does depend on z7. so the optimal solution should satisfy this condition.
Let WM = [AR, Ay where Ar E g Au E (d)(d))), By rotational invariance of 2g, and xp, without loss of generality, we can assume Ar = QA where is diagonal matrix with diagonal entries ajj* 's and Q is any orthonormal matrix. Furthermore, Ag 0 in the optimal solution since does not affect the loss but only decreases the norm bound on AR. So on data from the task D..
ED, [ (I M Zm; 0]13)]
WERR
e
Σ JER,
(60)
20

---

Published as a conference paper at ICLR 2023
Then the mixture,
(x, [e (
[6(xt) 15-0(z II)]
(61)
T
Σ
wy Besxt
Σ
(62)
jER
wy B (E;~
e
Eaj
+ Σ
(63)
jES
jEP.
[{ H)
(64)
where each ž is random variable drawn from standard Gaussian.
Now consider the minimum of the function {((a;)) }) on the right hand side, under the constraints that |كرز| I 1 and Σ, 0i r. Before finishing the proof of Lemma B.1 we have the following claim for this optimization.
Claim 1. There exist a, on satisfying 0 S ax, ar S and as + -),
= -E, Sr.
such that the minimum of the above optimization 64 is achieved when ai = for any j E S, and
ae on for any j Pe and E [T].
Proof. We need to prove that to achieve the minimum,
(1) ai = urrr for any + E S:
(2) az = a for any E + l' E Pt and any t E [T];
2
For (1): By symmetry of 2j 's and the convexity of ((-) for any [T'.
]
E
Σ
(65)
jER,
E
Σ
23 + 020 +
22
2 2 +
Σ
2 ?
2
(66)
2
JEPt
+
Σ
23 + 12122
+ 2
2
+
Σ
23
(67)
jE8.J
jEP:
2
E
Σ
23 +
a² +
27
Σ
2
(68)
JES.J+t.j44*
jEP.
Then
s((an))) 2
Σ E E
Σ JESJHIU#
22
ag + a
28
2
+
ugg
2
28
2
+
Σ
JEP.
(69)
Therefore, the minimum is achieved when ab = az
A similar argument above proves statement (2).
These statements mean that, for any t E [T], the minimum is achieved when a = for j E S, and
aZ 0e for j E Pt, for some values a, 0z 2 0. Let 7 Ejes 23,, 7, Ejer, #7. Then Z x3
21

---

Published as a conference paper at ICLR 2023
and Zz X and we have:
s({aff)) Σ wilk <Ead3
0o 3 +
Σ
a aonny
(70)
(ES
jEP
Σ
wiyE [e (003, 2 a
(71)
Given the constraint as Σ (r Ej azs S 0 S a, S 1, we complete the proof of Lemma B.1
IT
Given this result we can now analyze the generalization error when predicting on the target task Dr. Lemma B.2. Consider any f. [T]. Let Ut Ejes(u?)) and Ut,2 Djep, (u Suppose in 6 (calculated in Lemma B. a, a₁ 0. Suppose the prediction loss l, L-Lipschitz.
Then the Empirical Risk Minimizer Ui E Far i on o² using m labeled data points from Dr has risk
2 (44/5)
((2.))),,1Cc T (xxx), v)1 s 8V
m
(
savos, S,t + (r s)alo Rt +0 (1
max ( 8, n1,) soo + (r - 0) R,
+ A
Ut,1
11,2
m a
a
Proof. For any [T], only need bound the Rademacher complexity Run (For (t) of For the statement then follows from standard generalization bounds,
2ln(4/))
B (ea (x)), S (Fo + 8y
m
Given the representation o² Lemma B.I to ensure there exists a predictor F matching the ground-truth label, (x) = u. x (x) y = (ut) ZR, predictor u, should satisfy
ED, v) VVE.. Ue [QA", 0]M M [ZR, 0; un 2R.
(72)
VVRR, un QAAAA [ZR, : = ut ZR.
(73)
B (Q
(1) 1.r.1 = ui
(74)
<Vv E R", Ut Quik 1:r (Airr, 11 ut + Q 1:
(75)
The (+) from non-zero variance for 2n, d4 = 1u (Air (A
-- is the least-norm optimal
solution, have B (11...177) - should be
Ft 22 So the predictor class
For Fo==(110)
u7 To E R² |lulll S B (1(0)
U1,1
12
(76)
Ot
The empirical Rademacher complexity and Rademacher complexity of For with m samples are
Rm (For.e) = E sup fu, EFatee 2mm
Σ ox (
(77)
Eg
sup
Jw T QA*
(8)
0
(78)
|| < B izl
σ 1 Ai (6)
E,
sup
U1
(79)
||al||
E=1
B .t E
Q1.k A (8)
(80)
m
22

---

Published as a conference paper at ICLR 2023
Rm
=Ben,
Rm (o(,Fr)))
(81)
Bot
m
Eg
Σ
0 1
Am.11 (6)
(82)
m
-Rs
B
mT
Ai Airr,
(83)
m
-R1
R
For any t [T], define X₁ == 1irr1
Ei=t
(c)
Note that for j S, Xt, =
Ei= (1) is
R.
Gaussian of mean zero and variance E[X?,] aE - (6) 2 aB E (1)
is a Gaussian of mean zero and variance E[X? 2 mayah. Since Xt.j sub- sub-gaussian, X? man's for j E and X? -mao for j E Pt are sub-exponential and more precisely
3.0 Similarly, for j E Pt, Xz, = 2i=1
|| - m.l 5 &ill = C 12 s C2mao's, jES, ||x2 mayo S G1Ix2, 11u1 -2 ll C2maroRi Pt,
(84)
(85)
where C₁ C2 are absolute constants and C2 > 1. Let = max(C2mao Oymoro = Cam max max 5,43 n, and m(scros. + (r s) By Bernstein's 's inequality, we have for every 2 that
P
(1|x,112 - 1) 27} S 2exp
-cmin
(86)
K2 K
|
>P/
(87)
72
7
< 2 exp
[ min
m² max (ax (ao 3,0 040 R, 22' m mmax(fof 010 st,t1
f(.1)
r
(88)
where e is an absolute constant. For all numbers 2 2 0, we have = - 11 2 = max (5, 5²). Thus, for any 5 2 O, we have
122
1| >
Xell
P
8
(89)
V/
|
<P
2 max(6, 82)
(90)
2
1 max 82)
I ma 82)
<2exp
min
(91)
C2
mmmaxx (ao 8 a0 R. |r
m max max S.00 o R.
2
<2exp
c
min ((max(6, max (6, 84))
(92)
C²
m mamax
a 2
12
=2 exp
82
(93)
C² m² max max((ar' (0n s 0,00
2p
Rtt
where the last inequality from = m(s00 + (r s).,o & m max (003 a1o Changing variables to f = 5/44 we obtain the desired sub-gaussian tail
P X||||||||
V
00 <2exp
P C3 m² max a 127 82
(94)
23

---

Published as a conference paper at ICLR 2023
By generalization of integral identity, we have
P[ v > 0yd0 - 1 P| - vi 03d0
E ||||
vull
(95)
<2
P - V > 0}d0
(96)
<4
exp
C3 [ m² max fao's R )2r 0² d0
c
1
(97)
max max
S a R.IVi
<C3
(98)
where C3 is an absolute constant. Thus, we have
Rum (Far
V
1 m
vt.1
+
(sao3 (r- t)
(99)
a
at
Bor
[|
|E
(100)
m
$ 0
(
1
Ur.1
12.2
max(ao3
S.t. 015 2 2r
R. 2,
(101)
m
a
02.
sao 2 Stt
+ (r 5) 0),, 2 R,,t
B.2 PROOFS OF PROPOSITION 2.3 AND PROPOSITION 2.4
Given the lemmas for the general case, we now ready to prove the results in Proposition 2.3 and
Proposition 2.4
Proposition B.3 (Restatement of Proposition 2.3 Suppose as,, = ORR = OU. = 1 for any
t E [T]. The representation d' obtained an mixture of data from all the tasks {Di
1 S t S T satisfies o² (g(z) =Q Q(iies Vazze; + EjERAS VBazes) for some E [0, 11.
8 min (I, (ī.))=))))
). where ej 'ss are the basis vectors and Q is any orthonormal matrix.
The Empirical Risk Minimizer i E For on o² using m labeled data points from DD has risk
E(z,g))D,lfar ( (x(),))
Ut
U1,2
(Voa +
(Vo
T + (r 8)3
2 2lm(4/8)
<aby
+
8)3 + 0
+ +8V
B
m
Proof. This follows from Lemma
B.I and considering the optimal a, az for the following:
g(fa, = Σ w [e (ao3,, Z + oro 2:]]
(102)
E (az + anZi)l
(103)
T t=1
T Σ
2 E
0Z Z1
00
(104)
The second equation is from that Z₁'s follow the same distribution by the symmetry of ž The inequality comes from the convexity of E(l) and Jensen's inequality. So the minimum is achieved when aa == for any / [T], leading
o(fa, = E (aZ + (Zz)
(105)
24

---

Published as a conference paper at ICLR 2023
subject to the constraints as + T - sr, Then ge 6" og(2) W>
Q Ejes vazzez + EJERyS v for some E 10, 3 min (1, ((rs))))))))))) where ej are the basis vectors and Q is any orthonormal matrix.
Finally, the generalization bound follows from Lemma B.2 and that
(v
ma sa + (r s3
(V
0 sa (r- s)3
(106)
This completes the proof.
Proposition B.4 (Restatement of Proposition 2.4p. Suppose 0gH ORR OU, = 1. The represen-
tation of obtained on data from Dr satisfies 07 g(2) = =Q =EjER 230,) where ej 's are the basis vectors and Q is any orthonormal matrix. The Empirical Risk Minimizer à E Fez, of using labeled data points from Dr has risk
2 2l((4)))
E B (ec( (T.. S 4L,
+8
m
m
While on task D,(i + t), any linear predictor on of has error at least min,, ED, lla(" 2s. v)]
Proof. Following Lemma B.1 (with = 3), we get of o g(=) = Q (EjER, zjes)
where ej ey's are the basis vectors and any orthonormal matrix. Following the same argument in the proof of Lemma B.2 we get
[
R (For.))
E
FB
(107)
*R
| |l
(108)
where the last inequality comes from the property of chi-squared distribution expectation.
B.3 IMPLICATION FOR THE TRADE-OFF
The propositions then imply the trade-off between universality and label efficiency. Below we for- malize the example discussed in Section 2.2
Proposition B.5 (A specific version of Proposition 2.3 Suppose ds.t = OR. OU. 1 for any
I E [T] and 2s. The representution o² oblained on an even mixture of data from all the tasks
(De S L < T} satisfies o² 9(=)) = Q (Efes 25ez + EjeRyS /fzee;) where ej 's are the
basis vectors and Q is any orthonormal matrix.
The Empirical Risk Minimizer à E For o² using m labeled data points from DD has risk
2 2m((4/)
(( 0² (z)))))
4LV
(v - Tv22)
+ +0(1)
+ +8V
2T
Proof. This follows from Proposition 2.3 and noting that when r = 2s, a = and 8 1/T are the optimal for:
o(fa, 9)) E (oZ [Zz)]
(109)
r as
= E
aZ
T(r - s) Zn
(110)
- 711
E
07
(111)
< r. a, 3 < 1. To see this, note that x3 and Z₁ ~ 2 x²_, x follow the same distribution, so aZ + T
subject to the constraints as a
Zi for 1 will stochastically dominate its
value for other E [0, 1). The optimal is then achieved when
and 2-0 = n
25

---

Published as a conference paper at ICLR 2023
B.4 IMPROVING THE ORA BY CONTRASTIVE REGULARIZATION
The above analysis shows that contrastive learning a representation on unlabeled data from the target task help in prediction on this target task. This suggests that given a representation ø' pre-trained on diverse data, one can fine-tune it by contrastive learning on some unlabeled data from the target task to get representation that can lead better prediction on the target task. In the following, we will formally show that this is indeed the case for the illustrative example in Section 2.2
Recall that in this example, Sst RRR OUt 1, 2s, and 1t1 Vt.2- The representation
6" obtained on an even mixture of data from all the tasks (D; 1 S t s T} satisfies 6 g(2) =
Q (Ejes Vazzez + EjeR, vBzje;) where ej's are the basis vectors and Q is any orthonormal matrix, and a = 1,3 T
Now, suppose we are given unlabeled data from D., and we use them tun-tune o (x) W' by contrastive learning on these unlabeled data. That is, we find W near W to minimize the contrastive loss on the unlabeled data from Dr;
min Wx
(d(xt) E (o(x)) Eg- ((x)))()) 1))
(112)
subject to
IW - w S 1[1 1
(113)
for some small > 0.
Proposition B.6. For 112 CCR, satisfying the following on x from Dr is an optimal representation:
&CR+ о 0(() = Σ
Vazze, +
Σ VBzeg
jES
JEP.
where 1a=1,18 min (1, + }
Proof. Following the argument in Lemma B.1 we still have that (CRI(1) (x) Wr where W for any orthonormal matrix Q2 and some diagonal matrix Ag = (ia with
af} = va for j e and ajj- vB for j E Pt for some a, 3 E [o, 1]. And the contrastive loss is:
E [e (0(x) ((())))(dll (Ø(x+) E.- 6(x [()]=
E
l
Σ
(114)
JER,
E
l
Σ
+ Σ
2/
(115)
jES
jeP.
Recall that of (x) = W x with W = Q[A;0] for any orthonormal matrix Q and some diagonal matrix A, with Aj = 1 for j E S and Aj /1/T for E R, \ S for any i i [T]. Then
I W/ =| [[Q2[A2; 0JMM Q[A:0]MM~111
(116)
1|Q2A2 - QA
(117)
-1|42- - 1 QAllE.
(118)
Since Q7 2 is a rotation and A, Ag are diagonal, we can always set Q2 = Q without increasing
I|W W ||r. Then
I W 12 = 1A - A = =s(Va- 1)2 + v(53 VI/T)² + Σ -1/7)-
(119)
(120)
JEP
To minimize the contrastive loss, we need a, 3 to be as large as possible, subject to I|
W 1| Σ 22 and a, B, (A2)3, E [0, The optimal is then achieved when 1, /B
min (1, / + 3x). and (A2) / I/T for E P.,
26

---

Published as a conference paper at ICLR 2023
Now, recall that by Proposition 2.3
the ERM has risk:
E (, [í T x (x).
1
Ct +
5 (/
v2.2
(
8)8+0
)
(4/5))
$ 4L
+ 8
m
sa + (r - =)8
m
||
|u211
||
(Vaa+ 88 +80 +8V
2 2l1(4/6)
+
20
2/3
r||u@ 1|2
a
2
+
With the fine-tuning using contrastive learning, in the representation learned, a remains to be 1, while 3 increases from 1/T to (VI/T + 7/ /s)" Then the error bound decreases. This shows that fine-tuning with contrastive learning on unlabeled data from the target task can emphasize the specifc features sp,, which then leads to better prediction performance.
C MORE EXPERIMENTAL DETAILS AND RESULTS
C.1 DATASETS
CIFAR-10. CIFAR-10 (Krizhevsky al. 2009) dataset consists of 60,000 32 32 color images in 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. Each class has 6,000 images. There are 50,000 training images and 10,000 test images.
CINIC-10. CINIC-I0 (Darlow et al. 2018) consists of 32 x 32 color images from both CIFAR and ImageNet and has 90,000 training images with ten classes identical to CIFAR-IO.
ImageNet. ImageNet (Deng et al. 2009) is a huge visual dataset which is composed of 1,281,167 training data and 50,000 test data with 1,000 classes. We crop each color image 224 224 as the standard setting.
ImageNet32. ImageNet32 (Deng al.] 2009) huge dataset made up of small color images called the kown-sampled version of ImageNet. ImageNet32 comprises 1,281,161 training data and 50,000 test data with 1,000 classes. Each color image is down-sampled to 32 32.
ImageNet22k. ImageNet22k (Deng et al. 2009); Ridnik et al. 2021) is superset of ImageNet which contains 14.2M training data and 522,500 test data with 21. 1,841 classes. We crop each color image to 224 x 224 as the standard setting.
ImageNet-Bird. The ImageNet- is subset of ImageNet and contains all bird-related images from ImageNet, which have 59 classes and 76k training images.
ImageNett Vehhccle. The ImageNet- imageNet-Vehicle is subset ImageNet and contains all vehicle-related images from ImageNet, which have 43 classes and 55k training images.
The
is a subset images from ImageNet, which have
of ImageNet and contains all 76 classes and 100k training images.
GCC-15M. GCC-15M denotes the merged version of GCC-3M (Sharma et al. 2018) and GCC- 12M (Changpinyo et al. 202T). a diverse dataset of visual concepts with image-text pairs meant to be used for vision- and languag pre-training. GCC-15M contains 15M training data and more than 600k concepts.
SVHN. The Street View House Numbers (Netzer al. 2011) contains 10 digits color images of size 32 x 32 in the natural scene. It has 73,257 digits for training and 26,032 digits for testing.
MNIST. The Modified National Institute of Standards and Technology (LeCun et al. 1998) is database of handwritten gray-scale digits size 28 x 28. It contains 60,000 training images and 10,000 testing images.
27

---

Published as a conference paper at ICLR 2023
EMNIST. Extended MNIST (Cohen al., 2017) includes gray images of handwritten letters and digits. The images in EMNIST were converted into the same size 28 x 28 by the same process as MNIST. EMNIST-Letters has 145,600 lowercase characters with 26 balanced classes, and EMNIST- Digits has 280,000 characters with ten balanced classes.
Fashion-MNIST. Fashion-MNIST (Xiao al., 2017) is dataset of 28 x 28 gray-scale images with ten classes: T-shirt/top. trouser, pullover, dress, coat, sandal, shirt, sneaker, bag, ankle boot. The training set size is 60,000, and the test set size is 10,000.
Fer2013. Fer2013 is a dataset (Goodfellow et al. 2013) of 48 x 48 gray-scale images with 7 face expression classes: angry, disgust, fear, happy, sad, surprise, neutral. The training set size is 28,709, and the test set size is 3,589.
FaceScrub. FaceScrub (Ng & Winkler 2014) is a dataset with 141,130 color face images of 695 public figures.
The German Traffic Sign Recognition Benchmark (Stallkamp et al., 2012 dataset of color images depicting 43 different traffic signs. The images are not of fixed dimensions and have a rich background and varying light conditions as expected of photographed images of traffic signs. The original training set contains 34,799 images, and the original test set contains 12,630 images. We resize each image to 32x32. The dataset has a significant imbalance in the number of sample occurrences across classes. We use data augmentation techniques to enlarge the training data and balance the number of samples in each class. We construct a pressppreerriin data augmentation pipeline consisting of rotation, translation, and projection transforms and apply this pipeline to the training images until each class contains 2,500 examples. So we construct a new training set con- taining 107,500 images in total. We also construct new test set by randomly selecting 10,000 images from the original test set for evaluation.
GTSRB.
IMDB. IMDB (Maas et al. 2011) is large movie review text dataset. The dataset is for binary sentiment classification, positive review or negative review. The dataset contains 25,000 movie reviews for training and 25,000 for testing.
AGNews. AGNews (Zhang et al. 2015) is a sub-dataset of AG's corpus of news articles for text topic classification. It covers the largest classes: world, sports, business, sci/tech. The AG News contains 30,000 training and 1,900 test samples per class.
C.2 VERIFYING THE EXISTENCE OF THE TRADE-OFF
Model. We evaluate three popular contrastive learning frameworks, MoCo v2 (He et al. 2020).
NNCLR (Dwibedi et al. 2021 and SimSiam (Chen & He 2021 MoCo v2 can be regarded as SimCLR (Chen et al. (020) equipped with memory bank, while NNCLR uses neireott
as the positive pairs. SimSiam can be regarded as a modification from BYOL (Grill et al. 2020)
similar to Barlow Twins (Zbontar et al.] [021), which does not need negative pairs. We follow the same data augmentation methods as SimSiam (Chen & He 2021) for all datasets.
Datasets. We consider three sets of data. In the first set, our downstream task is CIFAR-10, and the pre-training datasets may include CINIC-10, SVHN, GTSRB, and ImageNet32. CINIC-10 has classes identical CIFAR-I0 and is the most target-relevant, while the others are less sim- ilar to CIFAR-IO. This then provides more and more diverse pre-training data w.r.t. the target task. In the second set, our downstream task is MNIST, and the pre-training datasets may include SMNISST-igitss&Lettee Fashion-
GTSRB, and ImageNet32. Here, the handwritten dataset the last set, our downstream task is Fer2013,
EMNIST-DDigis&&Lettess is the most target-relevant. face expression classification dataset. The pre-training datasets may include FaceScrub, CIFAR- 10, SVHN, and ImageNet32, where the face dataset Facescrub is the most target-relevant.
feature extractor under different contrastive learning methods using SGD for 800 epochs with a cosine learning-rate scheduler, the base learning rate of 0.06, weight decay 5e-4, momentum 0.9 and batch size 512. Then we fix the pre- praited feature extractor and train a linear classifier (Linear Probing, LP) on 1%, 5%, 10%, 20%, 100% of the labeled data from the downstream task. For LP we use SGD for 200 epochs with a cosine tearning-rate rate scheduler, the base learning rate of 5.0, no weight decay, momentum 0.9, and batch size 256. We finally report the test accuracy on specific target task and
Evaluation & Methods. We pre-train a ResNet18 8 network (He et al. 2016)
28

---

Published as a conference paper at ICLR 2023
Terget Tilk 12, tnass sou clerr)))
Trade off Target Tack vs Ang Jiner eval on dfbr10).
Trrget
0,90
0.2
0.025
6.0
0.492
S-ge Tock Tas Acalroc
Tex "0A Roc
5,425
0.75
0.797
ubbked
lubild data
6.702
25% Milned data
koblld daca
lueded sale
25% Ea--ee cac
(6% sah-ted dic
680
30% ldbeled
20% lubeled cite
100%
E313
0.6 6.11 0.20 5.22
3.25
022
0.73
2
630
2,35
6.40
Averaged Accurecy
Avaraged Test Accurecy
Averaged Text Accuracy
(a) MoCo v2: CIFAR-10
(b) NNCLR; CIFAR- 10
(c) SimSiam; CIFAR-10
Trade-of off Thrget vs. Aug inneir eval meid))
0.325
3.95
22244 T2a Tot AU0M16
2.993
0.82
0.332
bbvind dets
520 15%
ubsked
deta
0.55
1% Dbrk-1 de
LES sabiled dac
3.73
10% abckelldde
lazend
abokeldde
6TC
635
237
52
0:2
2:13
L-2
61
6:5
5:1
5,t
6.20
Accurity
(d) MoCo v2; MNIST
(e) NNCLR; MNIST
SimSiam: MNIST
Tarpet Taakss Inocc (or2013)
Trade-oth Target Tece londr aus Ne220331
3.5 Jode
(smriia
0.44
6.49)
abeld
57% inteediaaa
27% lbbeedaa dita
0.400
Boooe bok Tixt
3,40
0.36
Tuak
0.300
0.20
2.35
lablled dets
Uilisbele data
0.34
lbbeled docs
6.303
12% dasa
Skilsbeld dats.
100% lusoic
2275
daca
201
6.10
3.15
0.26
6.55E Ea05
ETX a158 S1% 1038 0.203 Accuncyy
610
0.70
(g) MoCo v2; Fer2013
(h) NNCLR; Fer2013
(i) SimSiam; Fer2013
Figure 7: Trade-off of universality and label efficiency for MoCo v2, NNCLR, SimSiam on down- stream tasks CIFAR-10, MNIST, Fer2013. "1, 2, 3, 4" means incrementally adding datasets for pre-training. The z-axis the average test accuracy of Linear Probing on all 4 datasets. The y- axis is test accuracy on the target task. Pre-training data: (a)(b)() CINIC-10, SVHN, GTSRB, and Ie33e Target task: CIFAR-10. (f)(e) EMNIST-Digits& Letters, Fashon-MNIST, GT- SRB, ImageNet32. Target: MNIST. (g)(h(ii FaceScrub, CIFAR-I0, SVHN ImageNet32. Target: Fer2013.
the weighted average test accuracy on all pre-training datasets (i.e., using them the downstream tasks). We use the class number of each pre-training dataset as the weight, which is consistent with random guessing Thus, we have nine tasks total. For each task, we have two pre-trained models initialized by different random seeds. We evaluated each model three times and we report the average test accuracy with standard deviation based on multiple runs (six evaluations).
baseline. We have three types of models pre-trained
three sets of datasets.
In Figs. (b)(())). report results for MoCo v2, NNCLR, SimSiam (respectively) CIFAR-10 as the downstream task. The size and diversity of unlabeled data for pre-training are increased on the r-axis by incrementally adding datasets in the following order: CINIC-10 (C), SVHN (S), GTSRB (G), and ImageNet32 (only use a 500k (D) Then, we LP on CIFAR- 10 using different proportions of labeled samples. Similarly, in Figs. (f)) report results for three models
29

---

Published as a conference paper at ICLR 2023
MNIST. We incrementally add pretraining datasets in the following order: MANNIST-Digis&&Lette (E), Fashion-MNIST (F), GTSRB (G), ImageNet32 (I). In Figs. (h)(),, the downstream task is Fer2013 and incrementally add datasets in the following order: FaceScrub (I), CIFAR-10 (C), SVHN (S), ImageNet32 (I).
Results. In Figs. 7 when the pre-training data becomes more diverse, the average test accuracy on all pre-training datasets increases (i.e., universality improves), while the test accuracy on the specific target task decreases (i.e., label efficiency drops). This shows a clear trade-off between universality and label efficiency. Moreover, with fewer labeled data (from the green line to the red line), the trade-off phenomenon will be more significant. It supports our claim that diverse pre-training data allow learning diverse features for better universality, but can down-weight weight the features for a specific task resulting in worse prediction, As more diverse unlabeled data are included, more labeled data from the target task is needed to achieve comparably-gon prediction accuracy. This validates our analysis of the trade-off in Section 2.2
In Figs. (aah,,,))) the average test accuracy (z-axis) decreases in the end because when we add one pre-training dataset, it may hurt the test accuracy of all other pre-training datasets. In Figs. (h), the downstream task test accuracy (y-axis) increases in the beginning because when the pre-training unlabeled data from relevant tasks is not sufficiently large, introducing other pre- training datasets will help the model to learn features relevant for the downstream task. However, the downstream task test accuracy will drop later as in other figures.
Please refer Appendix C.5 for more figures.
C.2.1 LARGER SCALE EXPERIMENTS
The datasets involved are ImageNet (1.2M data points, classes), ImageNet22k (14M, 22k classes), and GCC-15M (15M). We compare two UniCI. representations (Yang et al. 2022): the more specific representation pre-trained on ImageNet, and the one pre-trained on the more diverse dataset ImageNet+GCC-15M We compare their performance in two tasks: classification Ima- geNet (using 2k labeled data) and classification on ImageNe122k (using 44k labeled data).
The results are reported in Table 3 From the specific representation to the diverse one, we observe that the test accuracy on ImageNet decreases (i.e., efficiency drops), while the test accuracy over ImageNet22k increases (i.e., universality improves). This again confirms the trade-off.
LP Accuracy
Pre-training dataset
Target dataset
ImageNett
ImageNat+GGC.... 15M
ImageNet (2k label)
79.05
77.66
ImageNet22k (44k label)
9.02
9.86
LP test accuracy on ImageNet and ImageNer22k with UniCL (Swin-T) pre-trained 500 epochs on ImageNet and ImageNet+GCC-I5M
Table 3:
C.3 INSPECTING THE TRADE-OFF
C.3.1 FEATURE SIMILARITY
For each set of pre-training data, we extract a of features for the target task, CIFAR- 10, MNIST, and Fer2013 respectively, using the pre- pre-trained representation function. In Fig. g (rows/columns are pre-training data; numbers/colors show the similarity) first row, the features from different pre- training datasets have low similarities. This is consistent with our setup in Section 2.2 that different tasks only share some features and each owns many private ones. In the second row, we can see decreasing trend of similarity in each row of each sub-figure. This indicates when gradually adding more diverse pre-training data, the obtained representation will encode more downstream task-irrelevant information and become less similar that before adding more pre-training data. and
will hurt the downstream task performance. This result is consistent with our Proposition 2.3
Finally, we would like to verify in Theorem 2.2 via CKA similarities. The theorem says that when we increase the norm bound, the representation can encode more and more features. To verify this, our
30

---

Published as a conference paper at ICLR 2023
6T3R8 SWHI CRUCO
0.26
0.096
0.62
0.62
0.33
0.56
0.64
Fooz
Semth
0.058
0.07
0.15
0.11
0.24
0.33
Fashen
MUST
0.067
0.29
0.4
0AW CIAR13
0.11
0.23
0.11
0.084
0.11
0.13
0.095
0.19
0.14
-64
Image Ner
image Nert0
UNon
GIC110 SWHN
GTSR8
Inage
Union
EVNSIT Fishin
GTSRB
Image
Union
Face
CHARRO
Image
Unio
(a) CIFAR- 10; Independent
(b) MNIST; Independent
(c) Fer2013; Independent
+39M Ceucio
0.76
0.75
0.62
DMASTT
0.79
0.77
0.64
S4000 000000 1000
Pace
0.49
0.47
0.39
44
644
HOTSRS
0.7
CINC10 +SVI8
+5TTRB
+imaze
EMISST
Farhion
+STTSBB
+Image
Pare Scub
+CIARIO
+SWNN
+Image
(d) CIFAR-10; Incremental
(c) MNIST; Incremental
(f) Fer2013; Incremental
Figure 8: Linear CKA similarity score among downstream task features from MoCo v2 pretrained on three sets of datasets. For "Independent", each representation model in the first four columns/rows is pre-trained on single dataset. "Union" indicates the model pre-trained on the union among four disjoint datasets. "Incremental" means from left column right, from top row bottom, incrementally add datasets for pre-training.
0.00005
0.65
0.64
-1
0.00005
-14
D.205
0.00305
0.0005
0.005
Figure 9: Linear CKA similarity among CIFARIO features from MoCo v2 pre-trained on CINICIO. Each representation the first three columns/rows pre-trained with a different weight decay value.
experiments will vary the weight decay regularization coefficient (larger weight decay corresponds to a smaller norm bound and fewer features learned in the representation). Fig. p shows that the linear CKA similarity decreases with the increase of the weight decay, then it provides support for the theorem.
C.3.2 THE EFFECT OF PRE-TRAINING AND LABELED DATA SIZES
We have also conducted finer mner-grained investigations into the trade-off by varying the pre-training dataset size and the downstream labeled data the specific task. Table 4 shows the results for different pre-training datasets (ImageNet -erd ImageNet, and 10% of ImageNet data) and different labeled datasets (500 8k samples from ImageNet-Bird, and the whole ImageNet). Table shows
31

---

Published as a conference paper at ICLR 2023
the results for similar setting with ImageNet- Vehicle. Table 6 shows the results of UniCL with Swin-T backbone using different pre-training datasets (ImageNet, and ImageNet+GCC-I5M and different labeled datasets (2k samples from ImageNet, 44k samples from ImageNet22k, the whole ImageNet, and the whole ImageNer22k).
First, we find that the trade-off is hidden when small amount of data from the specific task is used for pre-training. The results show that when the specific pre-training data is small, the representation learned is noisy and the downstream prediction performance is poor. This is not surprising: in the extreme case with only pre-training image from the bird task or vehicle task, there is essentially no information for pre-training. this case, the results are well inside the Pareto front of the trade-off curve and thus cannot demonstrate the trade-off.
Second, we find that the trade-off hidden when a large amount of labeled data are available for learning predictors on the representation in the specific task. If large amount of labeled data is available for training the predictor, the prediction performance is similar when using the specific or universal representations. This is consistent with the insights from our analysis: when pre-training on diverse data, the features specific to the target task are down-weighted but can still be in representation, then with large amount of labeled data from the specific task, the sample complexity issue is not significant, and thus the trade-off is hidden.
The above experimental studies show that the trade-off is revealed when we have large-scale pre- training data and limited amount of labeled data from the target task, which is indeed the typical interesting for using pre- me-trrined representations (especially the large foundation models). The trade-off significant in this case and thus crucial for further development of pre-tree pre-training represen- tations.
LP Accuracy
Pre-training dataset
Target dataset
ImageNet-Bint
ImageNet
10% ImageNet
ImageNet-Bird (500 label)
76.00
58.78
30.06
ImageNet-Biie label)
81.42
35.96
ImageNet- Bird (2k, label)
82.88
79.66
39.49
ImageNet-Biin -Brd (4k label)
83.83
83.59
44.13
ImageNet Bird (8& label)
84.71
85.93
48.50
ImageNer (all label)
41.38
73.20
54,08
Table 4: LP test accuracy on ImageNet-Bird and ImageNet with MoCo v3 (ViT-S) pre-trained on ImageNet-Bird and ImageNet.
LP Accuracy
Pre-training dataset
Target dataset
ImageNet Vehicle
ImageNet
10% ImageNet
ImageNet Vehicle lahel)
ImageNett-ehhin Vchicle (Ik label)
70.93
67.90
37.76
ImageNett-Vhhin Vehicle (2k label)
72.09
69.81
39.07
ImageNett Vehicle (4k label)
74.14
72.93
39.62
ImageNett- Vehiicle (8k label)
75.53
74.55
43.53
ImageNet label)
73.20
$4.08
Table 5: LP test accuracy on ImageNet- Vehicle and ImageNet with MoCo v3 (ViT-S) pre-trained on ImageNet- Vehicle and ImageNet.
Vehhcle
LP Accuracy
Pre-training dataset
Target dataset
ImageNet
ImageNer=GCC-I5M
ImageNet (2k label)
79.05
ImageNer22k (44k label)
9.02
9.86
ImageeNet (all lahel)
79.61
81.37
ImageNer224 (all label)
30.90
36.69
Table 6: LP test accuracy on ImageNet and ImageNe122k with UniCL (Swin-T) pre-trained 500 epochs on ImageNet and ImageNet+GCC-IMM.
32

---

Published as a conference paper at ICLR 2023
C.3.3 MORE ABLATION STUDIES
We report results from three ablation studies: (1) varying the class number of ImageNet32. (2) varying the percentage of target-relevant pre-training data, and (3) replacing CINIC-10 with CIFAR- 10 the pre-training dataset. The results consistently show the trade-off.
Tradeeolf on CIABN-1 for lebre for Clexa Divarrdy)
Pradeolf on S3A6-10 5mitem Staiation for Diveraly) 0.00
1% kiddedd sara
1% iiseed ia
5% lubiic data
5% luzs ed sata labalsd
12% lubu cate
3.50
100%
100% istrled dVt
2.80
9.50
0.62
0.06
19.0
100
Number Imagehett2 classes
Number Imagetiet22 denaes
(a) McCo v2
(b) SimSiam
Figure 10: Pre-train MoCo v2 and SimSiam CIFAR-10 ImageNec322(200)) with varying num- ber of classes of ImageNet32 from 50 to 1000 (æ-axis) under a fixed size of pre-training data. The y-axis is LP test accuracy on CIFAR-10.
Tasse off CITAA MoCo pretrained 100% CRIC-2
Thateolll on CPAA MuCerrt ansIEee tn 50% OVC
Tr 21 CIAR fox Stcce aedtoreed ON 20% C.D0
soslic OED
3% JCENNG
5% kooid
0.69
2855 lawile
20% abeic CaN
20% luteed Can
383% data
00.00
6.75
0.2%
coming eesto
0.00
0.20
Edee
-500
107968
20197
Number of es data
Mamber of sceled data
Memzer unsszeled dass
(a) MoCo v2; 100% CINIC-10
(b) MoCo v2; 50% CINIC-10
(c) MoCo v2; 20% CINIC- 10
Toseeuff CHRN sieniam pretraines ORIC-E
Tradl off on CEXD Strdiam pretrs
pndraiied hndd 20% CMEC-0
0.90
NE deta
0.00
0.00
17S Loo-lel calla
285 useed
0.30
19Th lnneeirata
500% soaled pars
0,027
632-
6.75
0.20
9.0.
15940
270957
Aumber uuvoteele dat
Mamber of aniaaaeess dira
Memser unloreed dica
(d) SimSiam; 100% CINIC-10
(c) SimSiam; 50% CINIC-10
(f) SimSiam; 20% CINIC-10
Figure 11: Trade-off on CIFAR-IO test accuracy (y-axis) for MoCo v2 and SimSiam with varying target relevant (CINIC- 10) pre-training data percentage (100%, 50%, 20%).
Varying the Class Number of ImageNet32. To further support A1, we show that the trade-off between universality and label efficiency also exists under a fixed dataset size. In Fig. 10 we pre- train MoCo v2 and SimSiam CIFAR-10 + ImageNe(32(200k) and keep the same setting as Fig except that we vary the class number of ImageNe:32(2000k) In previous experiments, we randomly pick 500,000 images from ImageNet32 without considering labels. Here, we fix the number of classes to 50, 100, 200, 500, 1000. Then we randomly sample 200,000 images from the subset of classes. The downstream task is CIFAR-10. In Fig. 10 we observe that with fixed pre-training
33

---

Published as a conference paper at ICLR 2023
Trade-of on CHAR--1 for Simbiam
5hr (a-laded) data
35 intele dats
10% abiled date
50% luteled cola
landee
0.850
bave
20% labeird cabs
(00%) libe ed 60s3
0.775
rimigeh Hmaggehatstto
230757
120157
Aannd
13987
humber of uniabeled daia
Number of unlaboled duta
Figure 12: Trade-off on CIFAR-IO LP test accuracy (y-axis) for MoCo v2 and SimSiam pre-trianed on datasets including CIFAR-10.
datasets size, e.g., 250,000, when the data is more diverse, the pre-training will learn more irrelevant features, and the performance will drop the downstream task. This supports analysis as well.
Varying Target-Relevant pre-training Data Percentage. In Fig. 11 we use (a)d) 100% (b) 50% (f)(f) 20% CINIC-10 to train MoCo v2 and SimSiam, and keep the same setting Fig. 7 For Fig. I (b) with 50% CINIC-10, test accuracy drops, e.g., the test accuracy of 1% CIFAR-I0 in Fig. 11 aa 80.63% vs. (b) 76.45%. We can also see the decreasing curve in Fig. (b). On the other hand, we also have test accuracy drops in Fig. 11 (c))))tf, However, we can see U-curve rather than strictly decreasing curve Fig. I (T) ImageNet32 more relevant with CIFAR- than SVHN and GTSRB, consistent with human intuition. When we have a small partition of CINIC-10 that does not cover all target-relevant features, the feature extractor can learn these missing features from Im3geNet32. Although there are many irrelevant features in ImageNet32, the positive effect is larger than the negative effect, and so it plots a U-curve. is consistent with our statement that we need a large and target-relevant pre-training dataset rather than diverse irrelevant one.
we keep the same setting as Fig. exxepp we replace CINIC-10 with CIFAR-10. Note that downstream task is still CIFAR-I0. 10. In Fig. 12 can see the same phenomena and similar performance as Fig. 7 Thus, we have good choice of task-relevant pre-training dataset, we can get similar performance pre-training the down- stream task domain directly.
Replacing CINIC-10 With CIFAR-10. Fig. 12
The pre- pre-training unlabeled data from diverse tasks may have positive effect when the pre-training unlabeled data from similar (relevant) tasks is not sufficiently large. Moreover, if we choose good task-relevant pre-training dataset, we can directly get similar performance pre-training on the downstream task. However, when the task-relevant dataset is sufficient, the performance will drop we introduce irsseirteeevant data in the pre-training dataset (Fig. (a)).))
C.4 IMPROVING THE TRADE-OFF: FINETUNE WITH CONTRASTIVE REGULARIZATION
Pretrain data
Method
1% label data
5% label data
10% label data
20% label data
100% label data
LP
80.63=0011
84.42=0.01
08.88:0005
86.75=00001
88.41-20.01
CINICIO
FT
68.74-1046
81.46220 34
85.20 14
88.6910029
93.58:10.14
Ours
83.66:20.43
03.04.20.08
16.18:10.24
89.61-10.12
L.P
77.83
01.43
82.95-10001
83.53-2:0.01
88.18:10.01
+SVHN
FT
66.01:20.52
80.20:10007
83.96:1:.466
88.01
93.35:1:0100
Ours 79,95=2936
81.77=0.11
84.98=2002
88.97-20.14
94.26-2:0001
LP
75.64.10.01
08.18:10.01
79.80.1
79.81:10.01
82.07-10.01
+GTSRB
62.79-20.57
78.90=2041
89:00000000
87. 87.85-00.03
93. 03.42:00.13
Ours 76.20.10.36
80.26.10.05
84.11 0006
88.74.10.01
94.32.10.15
LP
68.26=00.01
71.04=001
73.20=t0.02
73.73=+0.02
75 73.644:00033
+emageNe332
FT
65.40.10.16
1..29
82.96.L0.17
16.75.10.10
92.92.1na.6
Ours
71.70=1120
78.94=0.06
83.48:1000
87.77=10.13
93.66:1:0.12
Table 7: Test accuracy on CIFAR-10 with different evaluation methods on MoCo v2 under different percentages of labeled data. From top to bottom: incrementally add datasets for pre-trale pre-training.
34

---

Published as a conference paper at ICLR 2023
45
(a) Linear Probing
(b) Finetune
(c) Ours
Figure 13: The t-SNE visualization (Van der Maaten & Hinton 2008 for CIFAR-10 training data
normalized features from different evaluation methods, where the model pre-trained on (CSGI)
defined in Fig. 3 FT and Ours are trained the 20% CIFAR-10 training dataset. Different colors correspond to different classes.
Evaluation & Methods. We use the feature extractor (ResNet18) pre-trained as in Section 3.1 by MoCo v2. Then, report three evaluation methods, Linear Probing (LP), Finetune (FT), and Hine- tune with Contrastive Regularization (Ours) on CIFAR-10 under different percentages of labeled data as in Appendix C.2 LP follows the training protocol in Section [3.1] FT and Ours learn linear predictor and update the representation, and use the same data augmentation for fair comparison. FT follows the training in MAE (He et al., 2022) mostly, using SGD for 200 epochs with cosine leamiig-rrae scheduler, the base learning rate of 0.06, weight decay 5e-4, momentum 0.9 and batch size 256. Moreover, Ours uses the contrastive loss from MoCo and regularization coefficient = 0.1.
Results. Table the trade-off phenomenon also exists for FT evaluation, where the FT test aceu- racy drops when the pre-training dataset contains more diverse data points. 7 shows that Ours can achieve better performance than the other baselines. In particular, outperforms
typical fine- tuning method consistently, even when the latter also uses the same amount of data augmentation. This confirms benefit of contrastive regularization. Fig. [3 visualizes features of different methods by t-SNE. It shows that contrastive regularization down weight the downstream task- irrelevant invariant feature, so can improve the model generalization ability, which is consistent with the discussion in Section 2.2
Target data
Method
Aug
1% lahel data
5% label data
10% lahel data
20% label data
100% label data
66.0411o.2
76.28-10.0
78.06:1000
78.731:0003
77.84-1002
FT
71.09
75.32-10.11
76.381001
76.92.10.10
82,971002
73.28.1..1
76.43.10.44
78.69.1001
81.56.1..10
83.65
ImageNet
FT
100
71.83.Lo.05
77.55
80.16.L0.9
82.23.Loor
83,58
Ours
10
74.36:t0.099
78.60=t0003
80.02:b0.e
81.28:t0.07
84.94:0.0
Ours
100
73.03-t0.04
78.41
80.48:2005
82.42+t0.03
85.34:1o07
59.24.10.02
07.25:1:008
56.32.1o.r
58.35.10.08
63.44:1001
FT
55.20:10.16
61.73:t::32
64.91
64,70
65.76:1..10
61.241tn.08
65.89
68.6 68.61110.55
70.89 70.89:10000
78.22:1:..8
SVHN
FT
100
65.23:t0.05
72.34
75.13:10.11
77.20:t0.03
80.13:to.1
Ours
62.87:t00.2
23
70.16:t0.0
72.23-t0.00
78.72-20.37
Ours
100
65.30-t0.44
72.61 +ir0.07
75.34
77.64-b0.05
82,50-t-aa
71.20.Lo.61
83.97-L0.08
85.00
86.34 86.34.1:0066
86.56.L001
FT
17.18.L0.80
87. 17.65.Lo.s
88.56.Lo75
88.76.1001
89.83 39.83.
84.35.Lo.30
88.74.Lo.32
89.11
89,52.1..1
90.74_Lo.6
GTSRB
FT
100
86.28:t0.26
20.50=t0.29
90.83:2008
91.15::0001
91.89:to.12
Ours
84.84+00.35
89.53
89.441001
91.35
92.01=b028
Ours
100
87.24:t0.51
90.86=ta.55
91.29-1026
92.11:b0.15
92.65-1o.10
Table Test accuracy for different evaluation methods on different datasets using foundation model CLIP (backbone ViT-L). We do not data augmentation for LP. We evaluate FT without data augmentation, with 10 augmentation and with 100 augmentation to each training images. For Ours, use 10, 100 augmentation.
35

---

Published as a conference paper at ICLR 2023
Target data
Method
Aug
lahel data
label data
10% lahel data
20% label data
100% label data
LP
91.44_10.01
93.51
94.11
94.22.1001
95.82_oof
89.93.1o.38
94,21 Lo.35
95.15
95.80.Lo.6
96.12.1o.1
FT
10
90.82.10.19
93.24_1008
94.59
95.48.L0.1
96.1710.12
CIFAR-IO
100
90.84:10.03
93.05.1..02
94.23
95.37.1:000
97.01
Ours
10
90.73:t0.11
94,28=0003
95.433-0015
95,83=t0014
96.71
100
91.04:5..06
94.00-t0006
95.29-t:000
95.72:t:005
96.86=to00
I.P
39.40:to.00
50.50:t:0o
54.61
57.66:tnot
61.92:::o1
38.82:t..10
38.15
51.03
52.07:10.99
56. 19:-0333
FT
10
45.00:to.34
22.61=t044
54.45
57.05:to.11
65.36tt0.3
SVHN
100
45.70:t..15
24.51
57.76:t000
61.13:t0..9
19:t0993399
Ours
10
46.53-t0.22
55.59
57.32-t00
59. 59.20=t0.09
66.29-to.0
Ours
100
46.27:t0.09
55.70-t0007
59,45--0004
61.97:::0.1
70.22-t:001
LP
48.52.10.03
Lo.08
71.69
72.27.Lo.02
75.37_o0
$2.54.10.67
72.19.10.3
73.75.1071
73.30.10.46
75.16.1000
FT
10
58.68.10.01
75.21:100
75.8711006
76.75.1..8
76.45:1029
GTSRB
100
61.65:ta.16
75.22-t00m
76.49-t0031
77.67-1-008
78.05=t030
Ours
10
59.28:t0.04
75.35=t0.31
77.38+0009
80.52:t0.12
81,28=t0.00
Ours
100
63.61:1:.922
76.70-20.17
77.85-1023
80.13:::0.01
80.92:t0.19
Table 9: Test accuracy for different evaluation methods different datasets using foundation model MoCo v3 (backbone VīT-B), We do not use data augmentation for LP. We evaluate FT without data augmentation, with 10 augmentations and with 100 augmentations each training image. For Ours, we use 10, 100 augmentation.
Target data
Method
1% lahel data
5% label data
10% lahel data
20% label data
100% label data
LP
79.72-10:06
83.08:1020
81.48:00899
84.87-10.03
16.49.10.16
IMDB
FT
82.54.122.8
87.78
87 87.96_11227
89.49.1
92.31 ::026
Ours
84.48
90.12
90.41
90.79 1. 1.58
92.85 Le00
LP
85.52-t0.31
86.78:b0.2
86.75=mu.6
87.624t0.24
87.76=ta.66
AGNews
FT
88.744t0.54
90.76
91.22=
92.36=t0.44
93.57-t022
Ours
89.20=b0.72
91.22:t-0.6
91.33=1333
92.45
93.94=20.02
Table 10: Test accuracy for different evaluation methods on different datasets using foundation model SimCSE (backbone BERT). We do not use data augmentation for L.P. We evaluate FT and Ours with the same data augmentation as SimCSE.
C.4.1 LARGER FOUNDATION MODELS
We verify our method's effectiveness in real-world scenarios. When users use foundation models, they cannot access the model and can only call the official API, (e.g. GPT-3, DALL-E (Ramesh al. ). The pricing for GPT-3 get feature embedding is $0.20 / 1 tokens. If users would like to use a foundation model on their downstream task. the most efficient way is to get feature embedding of their data from the API and train small model, called adapter (Hu et al. 2021 Sung et al. 2022), on these embedding rather than on raw input data.
We evaluate CLIP (with ViT-L as the representation backbone), MoCo v3 (VTT-B backbone), and SimCSE (Gao et al.] 2021) (BERT backbone). They are trained on (image, text), (image, image), and (text, text) contrastive pairs, respectively, so cover a good spectrum of methods.
For CLIP and MoCo v3, we fix the backbone for all evaluation methods. For LP, we add a linear FC layer on top of the backbone. For FT and Ours, we insert a two-layer ReLU neural network (1024 dimensions for the hidden layer) layer. For Ours, we apply SimCLR contrastive loss on the adapted feature (output of adapter) and set À 1.0. We the same training strategy Section 3.1 for LP and as Table 7 for FT and Ours. In line with the actual situation, we control the number of augmentation number used FT and Ours (more data augmentation leads higher prices in practice).
adapter between the backbone and the linear classification
We conduct experiments on NLP tasks as well. SimCSE proposes a contrastive learning method for sentence emheddings, which uses the dropout feature from BERT data augmentation. The
36

---

Published as a conference paper at ICLR 2023
sequence embedding length is 512. For SimCSE, all three evaluation methods use linear classifier. For all evaluation methods, we use AdamW (Loshchilov & Hutter 2018) with weight decay = 0.01 and train 3 epochs with batch size 16. For LP, we fix the backbone and set the learning rate as 5e-3. For FT' and Ours, we train the backbone and linear classification layer simultaneously using unique data augmentation in each training step and the learning as 5e-5. For Ours, apply SimCSE contrastive loss on the feature (output of the backbone) and set À = 1.0.
As in Appendix C.2 report LP, FT, and Ours on 1%, 5%, 10%, 20%, 100% of the labeled data from the downstream task in Table 8 Table 9 and Table TO for CLIP, MoCo v3, and SimCSE respectively. For CLIP and MoCo v3 Table and Table we consider different data augmentation numbers, c.g. 10, 100. For SimCSE, we use the standard data augmentation, i.c. generating unique augmentation for each gradient step.
These tables again show our method can consistently improve the downstream prediction perfor- mance in all three real-world scenarios, and quite significantly some cases (e.g.. MoCo on GTSRB). This shows that the method is also useful for large foundation models, including the case when the foundation models cannot be fine-tuned and only the extracted embeddings can adapted.
C.4.2 THE EFFECT OF CONTRASTIVE REGULARIZATIN
LINEAR PROBING
We show that contrastive regularization can also improve over linear probing. Note that the con- trastive regularization loss term is only relevant to the backbone; see the definition in Equation (9). While linear probing fixes the backbone. Thus, we cannot do contrastive regularization and linear probing at the same time. To show its effect, we first apply contrastive regularization to update the backbone, and after that use linear probing.
The results in Table T show that contrastive regularization leads to better prediction accuracy. Fur- thermore, the improvement is more significant on diverse pre-training data, consistent with our anal- ysis.
Pre-training dataset
Method
CINIC-10
+SVHN
+GTSKB
+ImageNer32
1.P
88.41
18
82.07
75.64
Contrastive regularization then Linear probing
88.38
86.91
85.95
82.43
Table 11: Test accuracy on CIFAR- 10 with different evaluation methods on MoCo v2 with ResNet18 backbone. From left to right: incrementally add datasets for pre-training.
C.4.3 THE EFFECT OF CONTRASTIVE REGULARIZATIR ON CLOSING THE GAP
We show that contrastive regularization can reduce the target task performance gap between pre- training on the specific dataset (the same or similar as the target task) and that on diverse datasets.
We pre-train with MoCo v3 (backbone ViT-S) on ImageNet-Bin Bird the whole ImageNet and then perform linear probing (LP) on the target task of ImageNet-Bird with Ik labeled samples. The results in Table 12 show that pre-training on the diverse data leads to worse performance. Then pre-train on ImageNet followed by contrastive regularization on ImageNet-BRl linear probing (LP) on the target task. This leads improved performance than without contrastive regularization, closing the gap between pre-training on diverse and specific data. Similar results are observed in Table T3 when ImageNet- Imahhhee- used. This confirms the benefits of contrastive regularization for highlighting task-specific features.
and then perform
I.P Accuracy
Pre-training datnset
Target dataset
ImageNet-Bird Bird
ImageNet
ImageNet then Contrastive Reguarlization ImageNet- Bird
ImageNet-Biin (Ik label)
69.39
Table 12: LP test accuracy on ImageNet-Bird and ImageNet with MoCo v3 (ViT-S) pre-trained on Bird Bird and ImageNet.
ImageNet-Biin
37

---

Published as a conference paper at ICLR 2023
I.P Accuracy
Pre-training dataset
Target dataset
ImageNet- Vehicle
ImageNot
ImageNet then Contrastive Reguarlization on ImageNer- Vehicle
ImageNet- Vehicle (Ik label)
70.93
67.90
71.34
Table 13: LP test accuracy on ImageNet- Vehicle and ImageNet with MoCo v3 (ViT-S) pre-trained on ImageNet- Vehicle and ImageNet.
C.5 ADDITIONAL RESULTS VERIFYING EXISTENCE OF THE TRADE-OFF
averaage lest acturacy un test accurscy on the target task
33
ararage lest accurasy un al test accurscy on the target task
1.9
2:34
arerage lest acturacy aay on lasks test accuraty on the targ tack
Torgee Task Trest Accuney
2.33
0.39
fouraged best
2.r
0.33
0.00
tm
0.75
Task:
fvaraged
600
8.79
8.c
606
6m
CSG
CSG
Number otunbbeed eata
Number of uniaboled data
Number of unlobsles data
(a) MoCo v2: CIFAR-10
(b) NNCLR; CIFAR-10
(c) SimSiam; CIFAR-10
avenge test accuracy on all tasks test accuracy on the barget taxt
0.
tasks
awerage test accuracy on tasks text accurary on the tegel task
3.35
Salger Book Teet Arcusacy
accuracy harget
Aouragas Target bazk Test Accuracy
6.00
Aoeraged twat
8.07
Ercrapes
EEg
EFG
EFGI
EE
Number utunbbeleddooo cata
Number untabeled data
Number unlabeled
(d) MoCo MNIST
(e) NNCLR; MNIST
SimSiam: MNIST
8.22
Accunicy
2377
6.04
average lest acturary on test accuracy on the torget task
6.33
0.33
Serget Taok lent Recuterry
Avaraged Accuracy
lest al tasts the tack
1.0%%
6.32
3.40
6:44
Bed
6.2
RE4
R62-
660
32
6:0
1276
ECS
PCS
EC
ECS
EC5 130
Number untiabeeed
Mumber of unlaboled data
Number unlabblled deta
(g) MoCo v2; Fer2013
(h) NNCLR; Fer2013
(i) SimSiam; Fer2013
Figure 14: Trade-off of universality and label efficiency for MoCo v2, NNCLR, SimSiam on down- stream tasks CIFAR-10, MNIST, Fer2013. r-axis: incrementally add datasets for pre-training. Pre- training data: (a)(b(c) CINIC-10 (C), SVHN (S), GISRB (G), and ImageNet32 (I). For example, "CS" on the r-axis means I0-SHN, Target task: CIFAR-I0. Red line: average test ac- curacy of Linear Probing on all 4 datasets. Blue line: test accuracy on the target task. (f)(e(() EMNiST-Digits&Letters (E), Fashon-MNIST (F), GTSRB (G), ImageNet32 (I), Target: MNIST. (g) FaceScrub (F), CIFAR-10 (C), SVHN (S), ImageNet32 (I). Target: Fer2013. All evalua- tions are trained with 1% labeled data.
38

---

Published as a conference paper at ICLR 2023
Avrrgged Recurecy
averag: test accuracy on tacks test accurayy on the target task
Trypee The Testt Accureyy
8:33
test accuriiyy the targiet task
werag: test accuracy on all tasks
1.33
Avaragee Tanget Teek Thett ACCELL 1
0.38
average test accuracy on tasks test accuracy on the targ tack
K.M
6.00
1.35
0.22
6:44
605
CSG
C5
CSG
CSG
E55
CSG
CSC
unubbbeleddd eata
Number unlabeled
Number uniabales
(a) MoCo v2: CIFAR-10
(b) NNCLR: CIFAR-10
(c) SimSiam; CIFAR-10
average accureey
average accuresy
average accuracy
accuracy Uhe larget Laok
that Teet Accurncy
accuracy the Larget
Tergel Selk Sea recuunecc
Aouraqud Tupt Aucuracy
8.32
test accurasy on largat
3.00
6.50
600
0.3
0.0-
6.7
6:77
0.15
ere
1..0
0.8
600
6x
Number unllbbeed data
Mumber of unlabeled datt
Number of unlobeled datt
(d) MoCo v2; MNIST
(e) NNCLR; MNIST
(f) SimSiam; MNIST
test sccuracy on the target took
avatage accuracy on
Singet teck Armurnc
0.00
averagt test accuracy on tasks the
Targer Taek Twxel Accurney
Avsragid Accurscy
6154
average tost acouracy altsk
the bank
0.2.
2-2
6.00
E:399
Dowrogeed teet
Rownaged
6.04
0.22
6:2
10
ECS
505
6:0
EC
22'
FCS.
FCSt
40003
COE
unlabeled
unlapoled dota
(g) MoCo v2; Fer2013
(h) NNCLR; Fer2013
(i) SimSiam; Fer2013
Figure 15: Trade-off of universality and label efficiency for MoCo v2, NNCLR, SimSiam down- stream tasks CIFAR-10, 10, MNIST, Fer2013. æ-axis: incrementally add datasets for pre-training, Pre- training data: (a)))(c) CINIC-10 10 (C), SVHN (S), GTSRB (G), and ImageNet32 (I). For example, "CS" on the z-axis means CINIC-10+SVHN Target task: CIFAR-I0. Red line: average test ac- curacy Linear Probing on all datasets. Blue line: test accuracy on the target task. (d)(e)(f) EMMIIT-Digits&LLetten (E), Fashon-MNIST (F), GTSRB (G), ImageNet32 (I). Target: MNIST. (h) FaceScrub (F)) CIFAR-10 (C), SVHN (S), ImageNet32 (I). Target: Fer2013. All evalua- tions are trained with 5% labeled data.
39

---

Published as a conference paper at ICLR 2023
Averege tedt Aecurncy
A.17
average test acouracy
accuracy on the larel taxt
035
average test accuracy on accurary the bargt Lonk
1.31
0.22
average accuracy tasks
teit accuresy on lagal
Argee Thak Thstt Arc
Serget Tesk Text Ace
0.2
6,15
6.6-
Averaged
1.30
0:4
648
632
6dn
Number of uniaabedd cata
Number of unlabeled data
Number of unlabeled data
(a) MoCo v2; CIFAR-10
(b) NNCLR; CIFAR-10
(c) SimSiam; CIFAR-10
average acuurayy accuracy on the larget Lank
average test accuracy on accuracy the Largat
average accuracy
Best Accuracy
Brgge Trak Test Accurccy
1.8
Averagul Tast Accuracy
accurasy largat
1.80
6.p-
0.22
Sergen Tesk
8.00
6.5
EFG
Gx Number unlabeled data
Ex
Number of unlabeled datt
Number of unlobeled data
(d) MoCo v2; MNIST
(e) NNCLR; MNIST
(f) SimSiam; MNIST
weeaage test accurary on all tacks tast socurbiy the target tack
424
Target Tenk Tenst Arrrmmyy
2:1
accuracy target
The Herget Tank Teel Accurntyy
accuracy torg
6.0
0:33
Auar bust
1-22
6.0
6:0
FCS
12. Number of unfisbeieed data
Number of unibreled data
Number of unbbeled data
(g) MoCo v2; Fer2013
(h) NNCLR; Fer2013
(i) SimSiam; Fer2013
Figure 16: Trade-off of universality and label efficiency for MoCo v2, NNCLR, SimSiam on down- stream tasks CIFAR-10, MNIST, Fer2013. æ-axis: incrementally add datasets for pre-training, Pre- training data: (a)()(c) CINIC-10 (C), SVHN (S), GTSRB (G), and ImageNet32 (I). For example, "CS" on the --axis means CINIC-10+SVHN. Target task: CIFAR-10. 10. Red line: average test curacy of Linear Probing on all datasets. Blue line: test accuracy on the target task. (d)() EMMIST-Digitt&LLettee (E), Fashon-MNIST (F), GTSRB (G), ImageNet32 (I). Target: MNIST. (h)( FaceScrub (F), CIFAR-10 (C), SVHN (S), ImageNet32 (I). Target: Fer2013. All evalua- tions trained with 10% labeled data.
40

---

Published as a conference paper at ICLR 2023
6.12
averag: test accuracy on tocks test accuracy the target tack
Targes Teak Thitt Accurreyy
0.2
C24
wrergg test accuracy on all tasks test accuriity the targitt task
8:4
024
average test accuracy on tasks test accuracy on the targ tack
6.31
Avaraped Targer Teck fest AC
C22
0.39
4.
Derrgged
0:35
CSG
C5G
C5
CSG
CSG
C5
CSG.
CSC
srunbbeled
Number unlabeled
Number uniabeles
(a) MoCo v2: CIFAR-10
(b) NNCLR: CIFAR-10
(c) SimSiam; CIFAR-10
iverage accureey
accuracy Uhe larget
accuracy the Larget
accuresy
Terget Sankc Sin Accuracy
0.35
test accuracy the tanget tack
average last aracyy on tands
2.90
1.35
twxt
0.35
0.22
32
0:30
6:22
606
BF.. Number of undabbleed deta
6x
Number unlabeled data
Number of unlabeled daste
(d) MoCo v2; MNIST
(e) NNCLR; MNIST
(f) SimSiam; MNIST
--. test sccuragy on the target took
aratage accuracy on
Jururncy
average test accuracy on all tasks the
Aoaraggad Targer Task Teet Accosaary
Accuracc
0.28
average test accuracy tasks
606
6.2
Shngee Teek
6.0
Ere
8.00
ECS
505
EC
FCS
FCSI
FC
FES
FCSI
22t
unlabeled
(g) MoCo v2; Fer2013
(h) NNCLR; Fer2013
(i) SimSiam; Fer2013
Figure 17: Trade-off of universality and label efficiency for MoCo v2, NNCLR, SimSiam down- stream tasks CIFAR-10, 10, MNIST, Fer2013. æ-axis: incrementally add datasets for pre-training, Pre- training data: (a)))(c) CINIC-10 10 (C), SVHN (S), GTSRB (G), and ImageNet32 (I). For example, "CS" on the z-axis means CINIC-I0+SVHN Target task: CIFAR-I0. Red line: average test ac- curacy Linear Probing on all datasets. Blue line: test accuracy on the target task. (d)(e)(f) EMMIIS-Digits&LLettee (E), Fashon-MNIST (F), GTSRB (G), ImageNet32 (I). Target: MNIST. (h)( FaceScrub (F)), CIFAR-I0 (C), SVHN (S), ImageNet32 (I). Target: Fer2013. All evalua- tions are trained with 20% labeled data.
41

---

Published as a conference paper at ICLR 2023
6226
average lest accurssy on al lasks
6:22
average lest accuracy on lasts
bect Accurees
0.re
accuracy target
Argge Toxk: Tent Arrunrry
the Laok
2.32
aourayed target Took Senll Arc uecuunrryyy
6:2
E.11
buii acturacy the largel
0.3
6155
3:
1.90
630
C5
C3G
C5
C5G
C5G
634
E55
CSC
10
Number unlabeled
Number uniabeles
(a) MoCo v2; CIFAR-10
(b) NNCLR: CIFAR-10
(c) SimSiam; CIFAR-10
average test accuracy
0.2
average test acouracy on test accurscy on the torget task
average accuracy
2.02
Barrgged Ted Recunncy
test accursccy on the torget task
00
Meggen Task Text Arruraey
Avaragad 2ccurryy
633
test accuracy on the torget task
0:55
3.89
2:8
6:34
sange Talk
52
0.30
0.
5:08
0.
Gx Number orubbbeldd cata
Ex1P Number of uniobales data
Number of uniabaled data
(d) MoCo v2; MNIST
(e) NNCLR; MNIST
(f) SimSiam; MNIST
avrrage tast accuracy on all tasks test socuracy the target tack
tasks
0.50
Sent Accuricy
Auurgge Taxt Accurarcy
accuracy
6.2
Sorge Tankk Tint
0.1E
Aowaged
6.2
80B.
0.00
10
-0025
5p
FCS
Number of uniobeted data
Number of uniobeled data
Number of unbbeled data
(g) MoCo v2; Fer2013
(h) NNCLR; Fer2013
(i) SimSiam; Fer2013
Figure 18: Trade-off of universality and label efficiency for MoCo v2, NNCLR, SimSiam on down- stream tasks CIFAR-10, MNIST, Fer2013. æ-axis: incrementally add datasets for pre-training, Pre- training data: (a)()(c) CINIC-10 (C), SVHN (S), GTSRB (G), and ImageNet32 (I). For example, "CS" on the z-axis means CINIC-IO+SVHN. Target task: CIFAR-10. Red line: average test ac- curacy of Linear Probing on all datasets. Blue line: test accuracy on the target task. (d)() EMNIST-Digits&LLete
(E), Fashon-MNIST (F), GTSRB (G), ImageNet32 (I). Target: MNIST. (h)( FaceScrub (F), CIFAR-10 (C), SVHN (S), ImageNet32 (I). Target: Fer2013. All evalua- tions trained with 100% labeled data.
42


Reviews:
[{"review_id": "rHIYjjcHFV", "reviewer": "Reviewer_gG3v", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to explore the pre-training of representations with contrastive learning and the impact of training data diversity on the prediction performances of linear predictors. They propose that a trade-off exists between universality of pre-trained representations (diverse data) and efficiency on downstream tasks. They confirm their hypothesis first by a formal demonstration on a simplified model and then by empirical evaluations. Based on resulting insights, they propose a contrastive regularisation loss and proceed to empirical comparisons with other training protocols. \n\nFor their formal demonstration, the authors propose to formalize the concept of semantic features useful for specific tasks. They then introduce a hidden representation data model which allows to sample over the distribution of the hidden representations and to separate the learned features into two subsets: spurious features that are affected by transformations and invariants features that stay constant over different tasks. The authors then use this formal description to show that contrastive learning encodes features from unlabelled data which appear to be shared by a diversity of tasks, while down-weighting features specific to a task. Their analysis also shows that contrastive learning favours the invariant features with high variances, the authors conclude that contrastive learning is a generalized nonlinear PCA. \n\nBased on the theoretical analysis, the authors proposed to test their claim empirically. For this, they evaluated the similarity of features using Centered Kernel Alignment in two ways. First with pre-trained features on different dataset for a target task, and secondly with features pre-trained on gradually increasing datasets. Results showed low similarities between features pre-trained on the different datasets, which confirms that contrastive learning encodes mainly features that are private to these tasks. Results also showed decreasing similarities between features pre-trained on all datasets and those specific to a task, confirming that representations obtained on diverse data encodes more information irrelevant to downstream tasks. \n\nThe authors also evaluated their proposed contrastive regularization method empirically. For that, they compared test results obtained with a predictor trained with contrastive regularization and other training protocols. Results showed that contrastive regularisation outperformed other fine-tuning methods. Finally, authors compared their proposed method on large representation models (foundation models) to the same baseline fine-tuning protocol and showed that their method can improve prediction performance  ", "summary_of_the_review": "Even if the compromise between invariant and spurious representations presented in the paper seems to me to be a reformulation of well-known principles in machine learning, the formal and empirical analysis proposed by the authors brings interesting and important contributions to the community. The experimental results confirm the theoretical analysis and the interest of contrastive regularization for models based on pre-training and contrastive learning. ", "strength_and_weaknesses": "In this paper the authors argue for the existence of a trade-off between the universality and the label efficiency of pre-trained representations. While the formal and empirical proofs for the existence of this trade-off are convincing and strongly supported, it seems to me that it can be reduced to the traditional no-free lunch principle well known in machine learning, and the universality/task specific features to the bias/variance trade-off also well known. It is therefore not surprising that representations that are good for a variety of tasks are not well adapted for a specific one. \n\nHowever, I think the authors are correct when they state that this trade-off required a deep and complete study and the formal and experimental analysis proposed by the authors bring valuable information, results and insights. Although the formal analysis could be clearer and explained more simply, it shows what features are learned by contrastive learning when trained on diverse or specific data. It also shows how the data chosen for pretraining impact the prediction performances of predictor and gives an interesting framework for further investigation. Furthermore, I think that the evidence is numerous enough and that the experiments carried out are rigorous enough to support the assertions of the authors. \n\nThe more interesting contribution of this paper is the contrastive regularisation loss. Here also the evidence is clearly stated and strong enough to demonstrate a significant effect on the prediction performances. Results shows clearly that adding contrastive regularisation improves performances over traditional training protocol, which is a valuable information and contribution for the community. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "n6carQOhbF", "reviewer": "Reviewer_P3pG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper theoretically analyzed the trade-off between **universality** (measured by the average performance of multiple tasks) and **label efficiency** (measured by the amount of labeled data needed for a downstream task) in **constrastive learning**. Both empirical evidence and theoretical guarantee were given.", "summary_of_the_review": "This paper presents solid work on the analysis of contrastive learning with theoretical guarantees and empirical evidence. However, I'm not certain how significant and novel the result is. I may need more time to read it again to give my recommendation. For now, I lean toward acceptance with low confidence.", "strength_and_weaknesses": "(I'm not very familiar with theoretical analysis of contrastive learning and several previous works. I did not check the proof details in the appendix. My review might be biased.)\n\n### Strengths\n\n- The author aimed to analyze an important aspect of contrastive learning.\n- This paper is well-written and contextualized. The introduction section provides a nice overview of this work.\n- The author conducted abundant experiments to support their theory.\n\n### Weaknesses\n\n- How Theorem 2.2 is related to label efficiency is unclear to me. The explanation sounds more like \"pre-training on multiple datasets vs. training for a specific dataset\" instead of \"representation that achieves the optimal performance for many tasks but requires a lot of labeled data (extreme case: identity function) vs. representation that is easy to train for a specific task but can not be used for other tasks (extreme case: the optimal predictor for a task)\".\n- The experimental settings seem detached from the theory. Figures 1 and 3 show the number of unlabeled data instead of labeled data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MFuoML2Tf", "reviewer": "Reviewer_heJa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work claims there is a trade-off between two properties of representation learning 1. When a model is pre-trained, the learned representation will be useful for downstream tasks with a small number of labels. 2. The representation is useful for different downstream tasks. i.e. the average accuracy on multiple downstream tasks increases while single-task test accuracy decreases with more unlabeled data. The work provides some theoretical analysis. Starting from the connection between infoNCE loss and nonlinear PCA, the work conjecture that contrastive learning encodes important invariant features but not spurious ones. ", "summary_of_the_review": "Overall this is a good theoretical paper for understanding contrastive learning.", "strength_and_weaknesses": "This is a well-written paper, the idea is very motivated, and the analysis seems solid and interesting. \n\nOne thing I'd like to see in the experiments is some results that can verify Thm 2.2. Fig. 4 shows some, but not sufficient. Is it possible to show some results related to the increase of $B_r$? For instance, will it be good to show some similarity matrices with different dimensions of features, e.g., 512, 1024, 2048, similar to Fig. 4? Then by comparing the off-diagonal similarity, if the average similarity decreases with the increase of dimensions, then it will verify the theorem (correct me if I am wrong), to a certain degree.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4KknYXf6awa", "reviewer": "Reviewer_mgh6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work shows a trade-off between training on a large dataset and the performance on a target dataset. The authors first show some theoretical analysis of why this tradeoff appears. The authors then show empirical results that this tradeoff happens when training the networks on small-scale and small-resolution datasets. Furthermore, they show that a finetuning trick by including the contrastive loss during the finetuning helps the final performance, even in large-scale and large-resolution datasets with deep architectures.", "summary_of_the_review": "This paper shows interesting results about the tradeoff between the specific target task and the diversity in the training dataset. However, it’s unclear how important this tradeoff is in real-world applications. It is also not clear how novel the things in this paper are.", "strength_and_weaknesses": "Strength:\n\nThe theoretical analysis and empirical results about the tradeoff between target performance and the pretraining dataset are interesting, at least on the small-scale and small-resolution datasets. The improvement by the contrastive regularization trick is also exciting and promising. \n\nWeaknesses:\n\nThe biggest issue, in my opinion, is that it’s unclear how significant this tradeoff is in large-scale and large-resolution datasets. The tested case is mini compared to the actual training datasets people would use to get foundation models in both the resolution and the scale. Isn’t the fact that the CLIP, MoCo v3, and MAE can work already suggesting that this tradeoff is not critical? If the authors can show this tradeoff in high-resolution datasets in the 1M to 10M scales (which is still much smaller than what people may use but is much closer), I would be more convinced.\n\nMany things in this work are not novel, which is another big issue, IMO (see the next section for more details).\n\nAlthough the regularization trick leads to consistent improvement, fine-tuning is much more important in almost all cases. Is this trick useful in the Linear Probing case?\n\n(some of these weaknesses have been addressed by the authors, see their response and my follow-up comment)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%