Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper study a general class of loss function for contrastive learning that contains existing losses as special cases, which is formulated as a coordinate-wise optimization. The authors further proved analyses on the max player on deep linear networks and 2-NN.
strengths_and_weaknesses: Pros:
+ The problem itself is relevant and important for many practitioners in the community.
+ The methodology is novel and general; the perspective of coordinate-wise optimization is interesting and the analyses are sound.
+ The paper is well-written and easy to follow.

Cons:
- The experiments is a bit weak: CIFAR-10 and STL-10 are relatively simple datasets. Can the authors also consider more involved datasets such as CIFAR-100?
- Figure 2 does not have confidence bands (although I noted Table 1 does report standard deviations).
- Assumption 1 seems a bit restrictive. It might be not satisfied by real-world (even simple ones), which makes Theorem 5 and 6 less interesting.

Summary: although the theoretical analyses on the 2-NN case relies on a strong assumption, I think the problem formulation and the perspective of coordinate optimization are useful to the community. I recommend the authors (1) add experiments on larger real datasets, and add confidence bands in the figures; (2) empirically verify whether Assumption 1 holds; or construct synthetic datasets based on Assumption 1 and baselining it over other methods.
questions: Please see the "Strengths and Weaknesses" section.
limitations: As discussed in the "Strengths and Weaknesses" section, the core analyses on 2-NN relies on a strong Assumption 1, which needs remedy.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors provide a different perspective of InfoNCE with a theoretical analysis. A broad family of loss functions could be represented by a coordinate-wise optimization on the network parameter and pairwise importance. The authors connect the relationships between PCA and InfoNCE. Based on this point, the authors demonstrate optimizing the network parameter equals recovering the optimal PCA solution.
strengths_and_weaknesses: Strengths
1. This paper, although very theoretical, is still very easy to follow. I really enjoyed reading this paper. 
2. The novelty is undoubted. Although the relationship between contrastive learning and PCA is not the first time to propose. This paper gives a very interesting point about the min player and the max player. It also addresses the how the temperature parameter influences the hard samples mining.
3. This paper gives a much formal reformulation to demonstrate the max player is equivalent to Principal Component Analysis (PCA) for deep linear network, and almost all local minima are global and rank-1, recovering optimal PCA solutions. 
4. The two-layer MLP based analysis provides a good reason for the dimensional collapse in InfoNCE and other methods. 

Weakness:
The relationship between contrastive learning and PCA is not the first time to propose. In contrastive laplacian eigenmaps (Neurips2021), the random regular graph is used. It is a Laplacian form of PCA. 
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, authors show that contrastive learning (CL) is actually under a broad family of loss functions, and propose a unified formulation called \alphaCL. This formulation unifies not only various existing contrastive losses, but also is able to extrapolate to give novel contrastive losses beyond popular existing ones, providing a guide for designing new CL loss.
strengths_and_weaknesses: Strength:

1.	Although lots of CL loss has been proposed in the past few years motivated by different points, there is no work studying the interactions among them. This work is the first one, to my best knowledge, to reveal the connections or relations among these existing popular CL losses, with efficient proofs and clear descriptions.

2.	Up to now, the research of CL still remains an open problem what features does CL learn. Especially when we use black-box deep neural networks, it is difficult for us to understand the learned features. Based on the analysis of proposed general CL loss in this work, authors presents some special case to help up understand the representation learning by CL.

Weakness:

1.	Current analysis and discussion are still based on some constrained assumptions, such as assumption 1 in the paper. And the understanding for the learned features of CL is still for some easy special case. 

2.	The experimental results are very easy but with lots of theoretical analysis in the main paper. For example, the quantitative results are obtained only on CIFAR-10 and STL-10, which is a very small dataset and easy task. Besides, authors just select ResNet-18 as the backbone. Readers may ask how about applying the new formulation in the big dataset such as Imagenet? How about the performance if we select different backbones? How about applying the new proposed formulations for other CL tasks such as graph learning? Moreover, it is better for authors to provide some visualized results to help readers understand their theories.

questions: Question:

1.	I see authors analyze the representation of CL based on two-layer ReLU network in section 5. We know that Relu is local linear function so that it provides some advantages for analysis. How about other functions such as leaky relu?

2.	In the experiments, authors propose different formulations for the selection of r. Could you tell readers how you design them, or which should they focus on if they want to design new loss. Such discussions may help readers to design their own losses based on their problems with your theorys. 

limitations: Yes
review_title: null

Meta-review: metareview: This paper offers an interesting perspective on the min player and the max player for contrastive learning. It also addresses how the temperature parameter influences hard sample mining. The meta-reviewer recommends acceptance of the paper as a poster.
confidence: Certain
Decision: Accept