Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper set out to investigate the learning dynamics of contrastive learning (e.g.CLIP) in, investigate how contrastive learning learn to align the representations from different views efficiently.
strength_and_weaknesses: Strengths:
(a) The paper provided a fresh view that the alignment and balance of representation play a important role in the training dynamics of contrastive learning, and the training can be decoupled into two stages.
(b) The paper is backed by a series of theory-grounded definitions and proof.
Weaknesses:
(a) The experiment part only covers simulation results, not real data, the model only cover linear models.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and original, the method is theory-grounded and novel.
summary_of_the_review: This paper is well-written and original, it provides an interesting view in investigating the training dynamic of contrastive learning with alignment and balance of feature representations, however, the experiment seems only done on simulated data and linear models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors examine the role of contrastive loss across modalities (such as in CLIP).  They demonstrate that the contrastive loss promotes the learning of “aligned” and “balanced” representations.  The authors investigate the learning dynamics of the optimization process (under specific listed conditions) and show that the dynamics can be interpreted in the form of two stages.
strength_and_weaknesses: The strength of the paper lies in its supposed mathematical rigor and derivations.  A weakness of this paper is that it is limited to a linear data-generating model, and therefore the conclusions are unable to directly translate to understanding multimodal contrastive losses used in practice.

I enjoyed how the authors included both contrastive and non-contrastive loss formulations.  The authors analyze the non-contrastive loss (such as BYOL) in comparison with the contrastive loss to demonstrate why contrastive losses are better (in terms of going beyond alignment only).
clarity,_quality,_novelty_and_reproducibility: In terms of clarity, I believe the paper would benefit from more summarizations and takeaway comments describing each key formula or theorem.  It is virtually impossible to completely check the math thoroughly in this work in the timespan allotted to the reviewers.  As such, it is quite difficult to truly gauge the correctness of this work.

There are also numerous typos and grammatical errors (e.g. Now, we consider the contrastive The main” on page 8, etc.) which should be cleaned up.

I believe that the topic this paper tackles is quite novel, and I believe the experiments listed in the work have no issues with reproducibility.
summary_of_the_review: Given the novelty of the work, as a comprehensive step towards understanding multimodal contrastive losses, I am inclined to support this paper for acceptance.  This is predicated on the correctness of the listed mathematics - I willingly confess that a large caveat of my review is that I was not able to carefully check all the math or the claims in this paper.  Therefore, I will provide a marginal acceptance and look forward to extended discussions with the AC.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper theoretically studied the joint-embedding training dynamics in a linear setting - which may be suitable for a multimodal environment. They studied both contrastive and non-contrastive cases and showed that contrastive negative pairs are essential for preventing representations from becoming a rank-one solution. The analysis largely follows Tian et al. (2021), Wen & Li (2021), and Jing et al. (2022) using gradient flows. Lastly, the authors provide a numerical simulation that verifies their claims. 
strength_and_weaknesses: Strength:
1. The authors considered the normalization step in the contrastive/non-contrastive loss when conducting the analysis of their dynamics. This is a big improvement over previous theoretical works (Tian et al. (2021), Wen & Li (2021), Jing et al. (2022)) on this topic.
2. The phenomenon of the two-stage training dynamics is novel. Most previous theoretical work ignores training dynamics and only focuses on generalization bounds that are unable to explain such phenomena.


Weaknesses:
1. The non-contrastive setting is known to collapse. In fact, all successful non-contrastive methods like BYOL/DINO rely on asymmetric architecture. This is already proven by Tian et al. (2021). The authors derive an obvious conclusion (align->1, balance->0) that does not provide any insight into the problem. 
2. The overall claim on contrastive loss is shallow. The authors show that under perfect conditions, align-> one and balance->same rank as input. For linear settings, this is obvious that the optimal solution exists. However, the analysis seems impossible to scale to nonlinear settings as the definition of `balance` becomes trivial for nonlinear networks.
3. There is no discussion on possible follow-up based on this work. For example, based on theoretical discovery, Tian et al. (2021) and Jing et al. (2022) propose the corresponding solution. It is unclear how this theory can boost empirical advancement.
 
clarity,_quality,_novelty_and_reproducibility: The paper is well written. All theorems/lemmas are provided with solid assumptions and proof.
The simulation does not provide any detail.
summary_of_the_review: This is a well-written theory paper studying contrastive learning dynamics but only limited to the small scope and seems not to provide real insight.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper studies the training dynamics of contrastive learning using a simplified model (linear data model, 1-layer net, etc). The main finding is that there are two phases of learning: first the model aligns positive pairs, second the model spreads out negative pairs. The mathematical analysis is supported by simulations and experiments on MSCOCO.

Strengths:
* Learning dynamics of contrastive learning are relatively unstudied
* Simulation results corroborate the theory

Weaknesses:
* Gap between theory and practice may be large (assumptions of the theory are not realistic and MSCOCO experiments don't show two clear phases)
* Similar results have been shown before (e.g., collapse of noncontrastive methods, relationship of contrastive learning to alignment and uniformity/balance)
* The analysis does not lead to algorithmic improvements

justification_for_why_not_higher_score: Reviewers appreciated the mathematical rigor and found the topic to be relatively unexplored and important. However, concerns were raised about the depth of the results (how generally do they hold) and the gap between theory and practice. The AC and the most critical reviewer both considered the rebuttal and appreciated the clarifications but were unconvinced that the paper is quite at the ICLR level. The AC also appreciates the MSCOCO experiment but was not able to discern the two phases described by the theory.
Decision: Reject