Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work shows a trade-off between training on a large dataset and the performance on a target dataset. The authors first show some theoretical analysis of why this tradeoff appears. The authors then show empirical results that this tradeoff happens when training the networks on small-scale and small-resolution datasets. Furthermore, they show that a finetuning trick by including the contrastive loss during the finetuning helps the final performance, even in large-scale and large-resolution datasets with deep architectures.
strength_and_weaknesses: Strength:

The theoretical analysis and empirical results about the tradeoff between target performance and the pretraining dataset are interesting, at least on the small-scale and small-resolution datasets. The improvement by the contrastive regularization trick is also exciting and promising. 

Weaknesses:

The biggest issue, in my opinion, is that it’s unclear how significant this tradeoff is in large-scale and large-resolution datasets. The tested case is mini compared to the actual training datasets people would use to get foundation models in both the resolution and the scale. Isn’t the fact that the CLIP, MoCo v3, and MAE can work already suggesting that this tradeoff is not critical? If the authors can show this tradeoff in high-resolution datasets in the 1M to 10M scales (which is still much smaller than what people may use but is much closer), I would be more convinced.

Many things in this work are not novel, which is another big issue, IMO (see the next section for more details).

Although the regularization trick leads to consistent improvement, fine-tuning is much more important in almost all cases. Is this trick useful in the Linear Probing case?

(some of these weaknesses have been addressed by the authors, see their response and my follow-up comment)
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The novelty might be an issue here, as I think the new regularization trick is not first proposed in this work. But the tradeoff analysis might be, though it relies on too strong assumptions. The results should be reproducible. 
summary_of_the_review: This paper shows interesting results about the tradeoff between the specific target task and the diversity in the training dataset. However, it’s unclear how important this tradeoff is in real-world applications. It is also not clear how novel the things in this paper are.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work claims there is a trade-off between two properties of representation learning 1. When a model is pre-trained, the learned representation will be useful for downstream tasks with a small number of labels. 2. The representation is useful for different downstream tasks. i.e. the average accuracy on multiple downstream tasks increases while single-task test accuracy decreases with more unlabeled data. The work provides some theoretical analysis. Starting from the connection between infoNCE loss and nonlinear PCA, the work conjecture that contrastive learning encodes important invariant features but not spurious ones. 
strength_and_weaknesses: This is a well-written paper, the idea is very motivated, and the analysis seems solid and interesting. 

One thing I'd like to see in the experiments is some results that can verify Thm 2.2. Fig. 4 shows some, but not sufficient. Is it possible to show some results related to the increase of $B_r$? For instance, will it be good to show some similarity matrices with different dimensions of features, e.g., 512, 1024, 2048, similar to Fig. 4? Then by comparing the off-diagonal similarity, if the average similarity decreases with the increase of dimensions, then it will verify the theorem (correct me if I am wrong), to a certain degree.
clarity,_quality,_novelty_and_reproducibility: well-written, high quality, the novelty is clear, and reproducibility should be good (easy to implement)
summary_of_the_review: Overall this is a good theoretical paper for understanding contrastive learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose to explore the pre-training of representations with contrastive learning and the impact of training data diversity on the prediction performances of linear predictors. They propose that a trade-off exists between universality of pre-trained representations (diverse data) and efficiency on downstream tasks. They confirm their hypothesis first by a formal demonstration on a simplified model and then by empirical evaluations. Based on resulting insights, they propose a contrastive regularisation loss and proceed to empirical comparisons with other training protocols. 

For their formal demonstration, the authors propose to formalize the concept of semantic features useful for specific tasks. They then introduce a hidden representation data model which allows to sample over the distribution of the hidden representations and to separate the learned features into two subsets: spurious features that are affected by transformations and invariants features that stay constant over different tasks. The authors then use this formal description to show that contrastive learning encodes features from unlabelled data which appear to be shared by a diversity of tasks, while down-weighting features specific to a task. Their analysis also shows that contrastive learning favours the invariant features with high variances, the authors conclude that contrastive learning is a generalized nonlinear PCA. 

Based on the theoretical analysis, the authors proposed to test their claim empirically. For this, they evaluated the similarity of features using Centered Kernel Alignment in two ways. First with pre-trained features on different dataset for a target task, and secondly with features pre-trained on gradually increasing datasets. Results showed low similarities between features pre-trained on the different datasets, which confirms that contrastive learning encodes mainly features that are private to these tasks. Results also showed decreasing similarities between features pre-trained on all datasets and those specific to a task, confirming that representations obtained on diverse data encodes more information irrelevant to downstream tasks. 

The authors also evaluated their proposed contrastive regularization method empirically. For that, they compared test results obtained with a predictor trained with contrastive regularization and other training protocols. Results showed that contrastive regularisation outperformed other fine-tuning methods. Finally, authors compared their proposed method on large representation models (foundation models) to the same baseline fine-tuning protocol and showed that their method can improve prediction performance  
strength_and_weaknesses: In this paper the authors argue for the existence of a trade-off between the universality and the label efficiency of pre-trained representations. While the formal and empirical proofs for the existence of this trade-off are convincing and strongly supported, it seems to me that it can be reduced to the traditional no-free lunch principle well known in machine learning, and the universality/task specific features to the bias/variance trade-off also well known. It is therefore not surprising that representations that are good for a variety of tasks are not well adapted for a specific one. 

However, I think the authors are correct when they state that this trade-off required a deep and complete study and the formal and experimental analysis proposed by the authors bring valuable information, results and insights. Although the formal analysis could be clearer and explained more simply, it shows what features are learned by contrastive learning when trained on diverse or specific data. It also shows how the data chosen for pretraining impact the prediction performances of predictor and gives an interesting framework for further investigation. Furthermore, I think that the evidence is numerous enough and that the experiments carried out are rigorous enough to support the assertions of the authors. 

The more interesting contribution of this paper is the contrastive regularisation loss. Here also the evidence is clearly stated and strong enough to demonstrate a significant effect on the prediction performances. Results shows clearly that adding contrastive regularisation improves performances over traditional training protocol, which is a valuable information and contribution for the community. 
clarity,_quality,_novelty_and_reproducibility: The purpose of the research, the issues and the principles of the contribution are clearly explained and defined. The subject is clearly introduced and explained as well as the previous works. 

The theoretical analysis could be better explained and simplified for readers who are not familiar with the notions presented. 

For the experiments, the methodology is presented correctly, and the goal linked to the hypotheses introduced by the authors. 

However, the results could be better presented and explained. For example, the results shown in Figure 4 could be better explained. At first glance, it is difficult to understand what was evaluated, and the relationship between the observed similarities and the data used for pre-training. 
summary_of_the_review: Even if the compromise between invariant and spurious representations presented in the paper seems to me to be a reformulation of well-known principles in machine learning, the formal and empirical analysis proposed by the authors brings interesting and important contributions to the community. The experimental results confirm the theoretical analysis and the interest of contrastive regularization for models based on pre-training and contrastive learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper theoretically analyzed the trade-off between **universality** (measured by the average performance of multiple tasks) and **label efficiency** (measured by the amount of labeled data needed for a downstream task) in **constrastive learning**. Both empirical evidence and theoretical guarantee were given.
strength_and_weaknesses: (I'm not very familiar with theoretical analysis of contrastive learning and several previous works. I did not check the proof details in the appendix. My review might be biased.)

### Strengths

- The author aimed to analyze an important aspect of contrastive learning.
- This paper is well-written and contextualized. The introduction section provides a nice overview of this work.
- The author conducted abundant experiments to support their theory.

### Weaknesses

- How Theorem 2.2 is related to label efficiency is unclear to me. The explanation sounds more like "pre-training on multiple datasets vs. training for a specific dataset" instead of "representation that achieves the optimal performance for many tasks but requires a lot of labeled data (extreme case: identity function) vs. representation that is easy to train for a specific task but can not be used for other tasks (extreme case: the optimal predictor for a task)".
- The experimental settings seem detached from the theory. Figures 1 and 3 show the number of unlabeled data instead of labeled data.
clarity,_quality,_novelty_and_reproducibility: Although the paper is overall well written, a few unclear parts exist. I may need more information to evaluate the quality and novelty of this work.

- Figure 1:
  - "Specifically, we first pre-train on a specific dataset similar to that of the target task, and then incrementally add more datasets into pre-training": Does this mean the training is done in an online learning fashion? For example, when training on the CSG datasets, was the trained model for the CS datasets used? If so, does the order of datasets matter much?
  - Shouldn't one dimension (e.g., x-axis) be the number of *labeled* target task data if the *label efficiency* is defined with pre-trained representation and downstream prediction task?
  - The two y-axes may be confusing. The difference in average accuracy is around 2%, while the difference in target task accuracy is more than 10%. The absolute value seems suspicious too. Why is the average test accuracy less than 20% while the target task accuracy is more than 75%?
- Notation: $\phi: \mathcal{X} \mapsto \bar{\mathcal{Z}}$ does not follow the convention. A function $f$ with domain $X$ and codomain $Y$ is denoted $f: X \to Y$, and a function $f$ mapping $x$ to $y$ is denoted $f: x \mapsto y$, e.g., $\mathrm{square}: \mathbb{R} \to [0, \infty) := x \mapsto x^2$.
- Maybe assumption A1 can be restated more succinctly as "$g: z \mapsto x$ is injective".
- The term "regular" is used in many fields of mathematics, but I'm unaware of its use in this sense. Is there a more informative name for it? By the way, the implication is not obvious to me. Is it proven somewhere?
- The loss $-t$ was referred to as "unhinged loss" in some papers. It is not negatively bounded, so it does not satisfy A3. Then, what is the purpose of Theorem 2.1? It seems not so related to the trade-off to me.
- The trade-off sounds really like disentanglement. It would be better if the author could discuss their relationship and differences.
summary_of_the_review: This paper presents solid work on the analysis of contrastive learning with theoretical guarantees and empirical evidence. However, I'm not certain how significant and novel the result is. I may need more time to read it again to give my recommendation. For now, I lean toward acceptance with low confidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on models pre-trained by contrastive learning and studies two important properties of such models: (i) the capacity to learn efficient predictors on top of these models when using (small) labelled data, (ii) their universality in the sense of their ability to relevant for diverse tasks.  

The contribution of the paper highlights tradeoff between universality and label efficiency.  
All reviewers agreed that the contribution is interesting and the is novel.   
Some weaknesses about the experiments and other explanations have been raised.  

In the rebuttal, authors have provided additional experimental studies that answer the issues raised in rebuttal. 
The integration of these elements in the final version is important. The paper can then be improved in terms of presentation and clarity.

Overall, there is a clear consensus for accepting this paper.   
I propose then acceptance. 
justification_for_why_not_higher_score: Many additional experimental results have been provided in the rebuttal and the authors did not modify the paper to integrate these elements. In addition, some remarks about the presentation have been raised. So I'm not leaning toward an oral.  


justification_for_why_not_lower_score: The contribution is interesting and authors have provided additional results to complete the paper. The result is of importance for the community. I think that it can be an interesting contribution for a spotlight.
Decision: Accept: notable-top-25%