Propagation Alone is Enough for Graph Contrastive Learning

Authors: Xiaojun Guo, Yifei Wang, Mingjie Li, Yisen Wang
Primary area: learning on graphs and other geometries & topologies
Keywords: graph contrastive learning, polynomial GNNs

Abstract:
Graph contrastive learning has recently gained substantial attention, leading to the development of various methodologies. In this work, we reveal that a simple training-free propagation method PROP achieves competitive results over dedicatedly designed GCL methods across a diverse set of node classification benchmarks. We elucidate the underlying rationale for PROP’s effectiveness by drawing connections between the propagation operator and established unsupervised learning algorithms. To investigate the reasons for the suboptimal performance of GCL, we decouple the propagation and transformation phases of graph neural networks. Our findings indicate that existing GCL methods inadequately learns effective transformation weights while exhibiting potential for solid propagation learning. In light of these insights, we enhance PROP with learnable propagation, introducing a novel GCL method termed PROPGCL. The effectiveness of PROPGCL is demonstrated through comprehensive evaluations on node classification tasks.

Decision: reject
Reviewer scores: 6, 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730213921730

thread_type:
official_review

<thread_object>
creator: reviewer_xd6g

<content>
summary:
This paper makes an interesting observation about graph contrastive learning models. While recent GCL methods have become increasingly complex with sophisticated architectures and designs, the authors demonstrate that simple feature propagation alone can achieve competitive performance. They introduce PROP, a training-free propagation method that simply propagates node features along the graph structure. Despite its simplicity, PROP achieves competitive or even superior performance compared to state-of-the-art GCL methods across various benchmarks, particularly on heterophilic graphs. The authors further deepen their investigation by analyzing existing GCL methods from a decoupling perspective, separating the propagation and transformation components. Their analysis reveals that GCL methods struggle to learn effective transformation weights (performing no better than random), while showing potential in learning propagation coefficients. Based on these insights, they propose PROPGCL, a simplified version that removes transformation layers and only learns propagation coefficients. PROPGCL not only maintains competitive performance but also offers computational advantages.

strengths:
1. The paper effectively shows an interesting fact that simple propagation can work as well as complex GNN encoders in graph contrastive learning.  

2. The paper's key strength lies in its systematic experimental design, particularly in Section 5.2. Through well-designed ablation studies comparing fixed propagation coefficients, fixed transformation weights, and an all-one baseline, the authors convincingly show that GCL methods can learn good propagation coefficients but struggle with transformation weights.

3. The experimental evaluation is remarkably comprehensive, comparing against diverse GNN architectures including spectral-based models (ChebNetII), spatial-based methods (GCN, GAT), and polynomial variants (GPRGNN, BernNet). Their simple PROP method consistently matches or outperforms these sophisticated architectures, especially on heterophilic graphs. The authors then show that their PROPGCL variants can further improve upon PROP.

weaknesses:
This paper has a few areas that could be strengthened, primarily in its theoretical analysis.

1. In Section 4.1, where the authors present propagation as a non-parametric learning approach, the theoretical foundation could be more rigorous. The claim that propagation corresponds to gradient descent on Dirichlet energy lacks rigorous mathematical proof, and the cited papers (Zhu et al., 2021a) don't very clearly support this interpretation. The connection between propagation and non-parametric unsupervised learning remains more of an intuitive argument than a mathematically grounded one.

2.  The claim "the propagation operator can also be understood as a special GCL method, where the positive samples are randomly drawn from neighboring nodes" lacks rigorous proof. The authors directly make this connection without formal mathematical justification, and do not explain how propagation's deterministic neighbor aggregation truly corresponds to random sampling of positive pairs in contrastive learning.

3. Theorem 4.1 shows propagation minimizes the alignment loss, this actually corresponds to a mode collapse scenario where all node representations become identical - a situation that contrastive learning typically aims to avoid.

questions:
1. In Section 4.1, the notation \( x^{+} \) appears in the alignment loss equation. Could you clarify what \( x^{+} \) specifically represents in this context? Explicitly defining this and other notations throughout would enhance clarity and help readers follow your ideas more smoothly.

2. Suggestion: The paper currently presumes familiarity with the standard Graph Contrastive Learning (GCL) pipeline. Including a brief overview of the GCL pipeline in the main text would benefit readers who may not have a deep background in GCL.

3. Suggestion: The computational efficiency appears to be a major advantage of PROP/PROPGCL, but this important result is placed in the appendix instead of main context.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730366752392

thread_type:
official_review

<thread_object>
creator: reviewer_fgwu

<content>
summary:
The paper claims that non-parametric propagation is a special form of GCL and outperforms many existing GCN and GCL methods. The authors observe that GCL can effectively learn optimal propagation parameters, leading them to propose PROPGCL, which focuses solely on learning the propagation coefficients.

strengths:
- Adequate experiments.
- The paper is easy to follow.

weaknesses:
- The authors claim the PROP is a special GCL because it has an alignment operation between the node and its neighbors. However, the GCL is aligned node with itself on another view, this is different. Moreover, the commonly used infoNCE also includes the comparison part of negative samples. Therefore, I can't entirely agree that PROP is a special GCL.

- The PROP without a transformation matrix means that can not transform the node feature from the feature space to the class space. How to make PROP to make the prediction?
- The PROP form has been proposed in many previous works, such as SGC, PMLP[1], etc. What is the difference between PROP and SGC?

- The details of the proposed method should be provided, such as PROP-GRACE.

[1] Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730585933902

thread_type:
official_review

<thread_object>
creator: reviewer_h333

<content>
summary:
This paper focuses on graph contrastive learning (GCL). The authors reveal that a simple training-free propagation method PROP can achieve competitive results over dedicatedly designed GCL methods.
To further study the underlying reasons, they decouple the propagation and transformation phases of graph neural networks in GCL and find that GCL inadequately learns effective transformation weights while exhibiting potential for solid propagation learning. Motivated by this, the authors proposed a novel GCL method termed PROPGCL which demonstrates its effectiveness on various datasets.

strengths:
1. This paper is well motivated by detailed observation and conducts a thorough analysis.
2. The proposed method is well supported by theoretical findings and previous works. The findings refresh the customary viewpoints on designing GCL methods.
3. The proposed method is simple but shows outstanding performance compared to previous GCL and GSSL methods.

weaknesses:
1. The proposed method is only verified on node-level tasks with strong feature-connection relationships (i.e., strong homophily and heterophily). No study on the more complicated scenario, for example:
   - complex relationship between node features and edges, e.g., partial multidimensional homophily (Block & Grund, 2014)
   - graph structures introduce extra information instead of hinting homophily/heterophily, e.g., superpixel datasets (CIFAR10, MNIST) in Dwivedi et al. (2022).

Without this, the work can only show that previous GCL methods cannot adequately learn feature transformation. However, it cannot demonstrate that learning feature transformation is unnecessary or harmful if we can manage to learn adequately. A further study on it might lead to a stronger statement on the proposed method.

2. There is no evaluation of the inductive setting, including the single-graph datasets (e.g., datasets in Hamilton et al. (2017))
  and multiple-graph datasets (i.e., a dataset containing multiple graphs, such as those in Dwivedi et al. (2022)).













----

- Block, P., & Grund, T. (2014). Multidimensional Homophily in Friendship Networks*. _Network Science_, _2_(2), 189–212. 
- Dwivedi, V. P., Joshi, C. K., Laurent, T., Bengio, Y., & Bresson, X. (2022). Benchmarking Graph Neural Networks. _Journal of Machine Learning Research_.
- Hamilton, W., Ying, Z., & Leskovec, J. (2017). Inductive Representation Learning on Large Graphs. _Adv. Neural Inf. Process. Syst._, _30_.

questions:
1. Is $\mathbf{A}$ including self-loops? If not, $\mathbf{A}^K$ or $\hat{\mathbf{A}}$ might not include all $K$-hops neighboring nodes as described in the text.

2. Is it possible to extend the PROP-DGI to multiple graph settings, such as the datasets in Dwivedi et al. (2022)? Any specific reasons that PROP-DGI is only evaluated in transductive setting?



- Dwivedi, V. P., Joshi, C. K., Laurent, T., Bengio, Y., & Bresson, X. (2022). Benchmarking Graph Neural Networks. _Journal of Machine Learning Research_.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677564040

thread_type:
official_review

<thread_object>
creator: reviewer_3pxn

<content>
summary:
The paper investigates a training free, non-parametric method called PROP for graph contrastive learning. "PROP" is a method which aggregates node features within K-hop neighbors, without any trainable weights. After such a feature propagation phase, the aggregated node features are fed to a downstream classifier for evaluation on node classification / graph classification tasks. The paper claims that for unsupervised learning, only performing "PROP" is enough and is in fact comparable in performance to other learnable methods from literature that utilize sophisticated contrastive objectives. Given the strong performance of "PROP" and claim that propagation of features is sufficient for GCL, the authors further propose "PROPGCL" paradigm, where encoders of traditional GCL methods are replaced with trainable propagation only polynomial graph neural networks. The authors then experimentally show how "PROPGCL" provides performance gains compared to standard encoder choices in said traditional GCL methods.

strengths:
1) Viewing polynomial GNNs (propagation) as performing graph contrastive learning sheds new light into our understanding of contrastive learning methods for graphs.

2) Interesting insight via experiments that show feature transformations are not necessary in certain conditions for effective graph contrastive learning.

3) The paper provides a comprehensive set of experiments, with detailed analysis.

weaknesses:
1) While the experiments are well done, the main claim is not fully supported / backed up. For e.g., in the graph level tasks (Appendix A), the proposed paradigm of only "PROP" is inferior to other GCL focused methods from literature and performance is not comparable. So why is it effective only for node level tasks?

2) The heterophilic graph datasets chosen in the paper are known to be problematic, choosing recent heterophilic datasets (for e.g.., from https://arxiv.org/abs/2302.11640) will provide a more robust argument. IMO the chosen datasets for node level tasks are quite small and limited to meaningfully test GCL in a realistic setting. Investigating the proposed paradigm of "PROP" and "PROPGCL" on large scale graphs from OGB (Open graph benchmark) will provide a more convincing argument. 

3)  The novelty of the proposed paradigm of "PROP" is quite limited since it is well known to the community that feature aggregation is a powerful inductive bias that GNNs possess. In fact, many GCL methods commonly employ randomly initialized and untrained GNNs as a baseline [c.f. ADGCL (https://arxiv.org/pdf/2106.05819) See for e.g. RU-GIN.]. Moreover, on node level tasks in heterophilic datasets, it is well known that such feature aggregation property of GNNs is very important.

4) While the authors acknowledge that in cases where there are low quality node features (sparse / noisy / absent), propagation alone can be insufficient. This IMO is one of the biggest drawbacks of the proposed approach. Previous methods aim to deal with such cases.

5) There is very little discussions on the results in conclusion section to string together different experiments into a cohesive argument. IMO the readers will appreciate a concise explanation rather than digging through the appendix.

questions:
1) While the authors cite the paper "A critical look at the evaluation of GNNs under heterophily: Are we really making progress?" Platonov e.t. al, ICLR 2023 (https://arxiv.org/abs/2302.11640) why are the newer datasets not chosen for heterophilic datasets ?

2) Why is PROPGCL not evaluated for graph level tasks.?

3) How is PROP setup for graph level tasks?. Are resulting node feature after feature aggregation pooled into a single feature for downstream classification?

------- 
I raise my score from 3-> 5, given that the authors have answered some of my questions and concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695432473

thread_type:
official_review

<thread_object>
creator: reviewer_qb5w

<content>
summary:
In this paper, the authors investigate how the propagation and transformation components affect the Graph Contrastive Learning (GCL) methods. They first illustrate that simple uniform propagation can achieve competitive performance compared with GCL baselines through empirical experiments. Then, they further conduct experiments on how the propagation and transformation layers contribute to the GCL process. The results suggest that the GCL process could not help learn very meaningful transformation parameters but could be helpful in learning propagation hyper-parameters. Based on these observations, they further proposed a GCL framework named PROGCL, which adopts different training-free PROP as backbones. Specifically, they replace the encoder in GCL methods with propagation-only polynomial GNNs. The effectiveness of these methods are verified by empirical experiments.

strengths:
1. Originality: The paper majorly investigates how the components in GCL methods work. The findings in this paper are quite novel. 

2. Quality: The paper seems to be quite sound. Empirical experiments support the conclusions of this paper well. 

3. Clarity: In general, the paper is well written. All the important points and observations are clearly presented. The paper is also well-organized such that different investigations are quite logically presented.

4. Significance: The outcomes from this paper could be significant as they could help provide some guidance on designing effective and efficient GCL methods.

weaknesses:
1. The title and narrative in this paper have been talking about GCL in general. However, this paper is more about GCL for node classification tasks only. Will these conclusions in the paper hold true for other tasks, especially graph classification tasks? It would be better if the authors could conduct further investigations. If not, it might be better if the authors could narrow their scope to GCL for node classification tasks. 

2. It would be more convincing if the authors could conduct experiments on larger datasets in addition to the adopted ogbn-arxiv dataset.

questions:
See the questions in the Weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732203791571

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Updates

comment:
We thank all reviewers for their constructive comments. We have updated the paper accordingly with the following major changes:

-  **Scope Refinement.** Narrow the study's focus to **GCL for node classification tasks**, explicitly stating that future work will explore graph classification scenarios.
- **New Experiments on Larger Datasets.** Add results for the **ogbn-products** dataset (Table 4), demonstrating the scalability and computational efficiency of PROPGCL on large graphs.  
- **Enhanced Theoretical Foundation.** Include a formal theorem (Theorem 4.1) connecting the propagation procedure to the Dirichlet energy objective.
-  **Improved Accessibility of Results.** 1). Move the efficiency analysis to Section 6.3 to highlight PROP/PROPGCL's computational advantages. 2). Relocated the results for newer heterophilic benchmarks from the appendix to the main body (Table 7) for better accessibility.
- **Others.**  1). Add a detailed description of the GCL pipeline to the background section. 2). Provide detailed explanations of PROP-GRACE in Section 6.2. 3).  Revise the conclusion section to provide a cohesive discussion of results.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work provides an analysis of Graph Contrastive Learning (GCL) methods, revealing that simple training-free propagation can achieve competitive results compared to state-of-the-art GCL approaches. The authors' investigation into the decoupled propagation and transformation phases of GCL yields possible insights. Building on their analysis, they propose PROPGCL, a GCL framework that replaces traditional encoders with trainable propagation-only polynomial graph neural networks. The empirical results demonstrate the effectiveness and computational advantages of PROPGCL across small datasets in node classification tasks.

The reviewers find the observations interesting and the proposal of PROPGCL well-motivated. In response to a request for clarification on the connection between propagation and non-parametric unsupervised learning, the authors provide a theorem in the rebuttal that establishes a formal link between the Dirichlet energy objective and the propagation procedure.

However, some concerns and questions remain. Several reviewers express skepticism about the sufficiency of propagation (PROP) for unsupervised learning, citing inferior performance on graph-level tasks and limitations in handling low-quality node features. The authors acknowledge these limitations, which suggests that further research is needed to fully explore the potential of PROPGCL. Additionally, there are questions regarding the inductiveness of the method.

Some reviewers also raise a point about whether non-parametric propagation (PROP) can be considered a special form of Graph Contrastive Learning (GCL), as it could be viewed as a type of negative sampling. The authors provide a response, arguing that GCL can incorporate neighboring nodes as a type of view and that negative samples are not necessary for preventing representation collapse in PROP. This discussion highlights the need for further clarification on the boundaries between different approaches to graph learning.

Overall, this work contributes useful insights into the mechanisms underlying Graph Contrastive Learning and proposes a new framework that warrants further exploration. While some questions and concerns remain, the authors' responses and acknowledgments of limitations demonstrate a clear understanding of the challenges and opportunities. I believe the paper needs a little bit more work before publication.