Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper revisits contrastive learning objectives from the feature propagation perspective. Specifically, by casting alignment and uniformity as two message propagation procedures on two respectively graphs, we can establish equivalence between contrastive learning and message passing graph neural networks. In this way, we can inherit existing techniques in graph neural networks to improve contrastive learning performance.
strength_and_weaknesses: Strengths:
+ This paper presents a very interesting perspective on understanding contrastive learning by formulating alignment and uniformity as message passing on graphs
+ This paper is well-written and the idea is easy to follow
+ The message passing formulation is supported by both theoretical analysis and empirical evidence

Weaknesses:
I only have several minor points:
- The second paragraph in the introduction section is a little bit confusing. I would like to suggest the authors to elaborate on "model distribution" and "data distribution".
- I wonder how we can understand the MoCo approach that maintains a memory bank for negatives by constructing similar message passing graphs?
- In Page 1, GAT is not cited correctly.
- In Page 5, reference (Eq. 30) should be (Eq. 11).

clarity,_quality,_novelty_and_reproducibility: This paper is clearly written. I do not spot any obvious errors. I think this paper makes a novel contribution both theoretically and empirically for understanding and improving contrastive learning. The authors do not provide source code so I cannot comment on the reproducibilty.
summary_of_the_review: I like the paper very much. I think the authors make a nontrivial contribution in understanding and improving contrastive learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies contrastive learning (CL) by looking the dynamics of (unconstrained) gradient descent for the contrastive loss and connecting it to message passing schemes on the augmentation graph.

- The "alignment update" for CL is a message passing scheme on the augmentation graph, because each feature is updated as a weighted sum of neighborhood features in the augmentation graph
- The "uniformity update" is equivalent to message passing on a "fake affinity graph" that is itself defined by the similarities induces by the current features.
- Under this view, the contrastive update is a competition between two message passing rules and this helps characterize the "equilibrium solution"

It further elucidates connections/analogies between various techniques in contrastive learning and message passing methods on graph neural networks (MP-GNNs), like layer/batch normalization, feature collapse etc.
Inspired by these connections, the paper designs new contrastive learning variants that borrow techniques from GNNs like graph attention jumpy knowledge etc. that perform reasonably well on standard benchmarks.

- Using graph attention mechanism for adaptive positives with a weighted alignment term $\alpha_{f}(x, x^+) f(x)^{\top} f(x^+)$, to avoid considering far off positives. Leads to ~1% improvement over vanilla evaluation.
strength_and_weaknesses: **Strengths**

- The paper establishes many connections between contrastive learning updates and message passing methods that could potentially help both fields from advancements in the other.
- It is also reasonably easy to follow and well written (comments towards the end).
- The proposed method that utilizes graph attention mechanism perform slightly better than vanilla CL

**Weaknesses**

- While the paper makes many connections, most are derived from pretty straightforward or known calculations (the connections can still be valuable). If not technical, ideally the main contribution could have been to leverage the connections for new contrastive learning algorithms that do well. The only one that gets good performance (and improvements) is the graph attention idea.

- Using multi-stage aggregation, i.e. using features from multiple epochs rather than just the current one, does avoid complete feature collapse without negatives. However the final performance is still far from CL; I think ~45% on CIFAR-10 is achievable with even a randomly initialized network, so I'm not sure how much to read into those results of ~50%.

- Typos/bugs in derivations and proofs (that may be fixable)
  - I fail to see why $LSE(FF^{\top})$ is the same as $\mathbb{E}\_{x} \log(\mathbb{E}\_{x'} \exp(f(x)^{\top}f(x^{+})))$ in Eq. (2b). It seems like it should be something like $Tr(D \log(deg(D \exp(D^{-1/2}FF^{\top} D^{-1/2}))))$. The proof in Eq (20) seems wrong because the $w$ term vanishes from the $\exp$ in the next step. Similar the expression for $A'\_{\text{exp}}$ below Eq (7) seems wrong and other places that use $FF^{\top}$ kind of term by ignoring the $D$ factors. I'm not entirely sure if this affects other results (seems like it won't for uniform distribution), but it would be nice to get some clarification from the authors about this
  - In Eq (8) should the first term be $(1+2\alpha) I$ instead of $(1+\alpha)I$? An so should Eq (9) not have the $(1-\alpha)$ term? If not, then the Equilibrium distribution calculation will be incorrect, since $\bar{A} = A'$ will give $F^{(t+1)} = (1-\alpha) F^{(t)}$.

- It might be worth mentioning in Proposition 3 that the equilibrium distribution is true only if $f$ is allowed to be arbitrarily expressive and that it ignores sample complexity considerations. Also, such a result of distribution recovery has been shown in prior work using different techniques, Theorem 1 in [1].

- Other comments/questions
  - Missing citation for GAT on page 1
  - "Empirically, we show that both techniques leads to clear benefits on benchmark datasets": I would not say that there are clear benefits for both techniques


[1] Zimmerman et al. Contrastive Learning Inverts the Data Generating Process
clarity,_quality,_novelty_and_reproducibility: The connection between contrastive learning and message passing GNN methods is novel and potentially useful, to the best of my knowledge. Most of the paper is clearly written and it was easy to follow even in 1 pass. Some typos and bugs in equations/derivations have been pointed out above.
summary_of_the_review: I think the connection between CL and GNNs could open up new avenues for ideas in contrastive learning. This paper showed one such connection through graph attentions that lead to a small benefit. Overall it would have nicer if more such connections/insights arose. I believe that the overall contribution of the paper is positive. However due to bugs in the derivations and proofs (listed above), that may or may not affect some results, I am hesitant to accept the paper immediately, and have thus assigned a score of weak reject. Happy to raise the score after further clarification from the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper casts a new understanding of contrastive learning, which interprets the alignment update and uniformity update as message-passing steps and bridges contrastive learning and message passing. It interprets some techniques used in contrastive learning from the perspective of message passing neural network and proposes some inspiration for the modification of contrastive learning, some empirical results also justify these changes inspired by MPNN.
strength_and_weaknesses: Strengths:
1. This paper studies a significant problem about how data augmentation affects the learning dynamics of contrastive learning, although previous works have made some trials, this paper has a totally new perspective that is more interpretable. 
2. The analogy from MP-GNN is very interesting, such as the oversmoothing problem and feature collapse, this is inspiring for the future study of contrastive learning.
3. It proposes some new designs which are also justified by the experimental results, the GAT-like attentive score weights is reasonable and promising.

Weaknesses:
1. The analogies and two new designs mainly focus on the alignment updates, while negative sampling for the uniformity updates is a well-known problem. I think the authors should also pay attention to the negative sampling part such as the hard negative sampling [1].


[1] Contrastive Learning with Hard Negative Samples. Robinson et al., ICLR 2021.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and technically sound. Detailed proofs are attached in the appendix. Basically, this paper makes a novel interpretation of contrastive learning which is very interesting. The experimental details are also provided which enhances its reproducibility.
summary_of_the_review: This paper is well-written and presents an innovative understanding of contrastive learning. It connects contrastive learning with MP-GNN in both theory and the analogy of problems/techniques. It also provides some ideas for new designs in contrastive learning inspired by the existing tools in MP-GNN. I think this is a solid paper and vote for accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a novel perspective on contrastive learning by relating it to message passing on graphs. It shows that the alignment and uniformity updates in contrastive learning can be seen as message passing schemes on two different graphs, one induced by data augmentation and the other by feature similarity. It also draws connections and analogies between various techniques and challenges in contrastive learning and message passing graph neural networks, such as layer normalization, feature collapse, and oversmoothing. Based on these insights, the paper introduces two new variants of contrastive learning that leverage graph attention and multi-stage aggregation, and demonstrates their effectiveness on standard benchmarks.

The reviewers generally appreciate the paper's contributions and find it well-written and technically sound. They agree that the paper establishes interesting and potentially useful connections between contrastive learning and message passing, and that the proposed methods are novel and empirically validated. They also point out some minor issues, such as typos, missing citations, and unclear statements. Reviewer 2 also raises some concerns about the correctness of some derivations and proofs, and the significance of some results. The authors are expected to address these issues in the final version of the paper.
justification_for_why_not_higher_score: Overall, the paper makes a positive and significant contribution to the field of contrastive learning and graph neural networks, and deserves to be accepted as a poster. The paper is not strong enough for an spotlight presentation, as it does not provide a major breakthrough or a comprehensive evaluation of the proposed methods. It also does not fully explore the implications of the message passing perspective for the negative sampling and uniformity update, which are important aspects of contrastive learning. The paper could be further improved by clarifying some of the claims and assumptions, fixing the errors in the derivations and proofs, and discussing more related work and limitations.
justification_for_why_not_lower_score: The paper should not be rejected because it presents a novel and insightful view of contrastive learning, which could inspire future research and applications. It also provides some evidence of the benefits of the proposed methods, which are based on solid theoretical foundations. The paper is well-organized and easy to follow, and the authors provide sufficient details and proofs in the appendix. The paper does not have any major flaws or ethical issues that would warrant a rejection.
Decision: Accept: poster