Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: DGI has shown successful performance as the graph contrastive learning method.
This paper points out the technical defect of DGI, that is, the sigmoid function in the loss makes the embedding uniform.
From that observation, the authors suggest to modify the original loss to much more simplified version (i.e., graph discrimination).
They propose GGD based on the graph discrimination loss, and show the experimental results to prove its efficiency.
strengths_and_weaknesses: **Strengths**
- They improve both the performance and the resource efficiency.
- It is a well-organized paper: easy to catch the idea, and contains detailed but concise explanation.

**Weaknesses**
- It would be better if there are more empirical explanations how GGD achieves the higher performance than DGI, for example, by the ablation study.
- There are minor mistakes: overlapping letters in Table 9, unit error in line 387 (% $\rightarrow$ %p).
questions: - What is the rationale for saying that "GD-based method focuses on the general edge distribution of graphs instead of node-specific information"?
- How are the training time and memory consumption of the supervised GNN models in the small- and medium-scale datasets?
- Please clarify which results are reproduced and which are sourced from other papers in other sections as in Section 5.1, and indicate how many epochs the models are trained in every case.
- I was curious about the behavior in convergence of the models until 300 and 100 epochs in Figure 4.
limitations: - As the authors mentioned, the group discrimination can depend on augmentation or corruption methods.
- It would be more persuasive if the experiments are conducted on the datasets from more various domains.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a novel approach to graph representation learning that learns informative node embeddings by projecting the embeddings obtained from the original graph using a GCN into scalars, and the embeddings obtained from a corrupted version of the graph using the same GCN into scalars, and optimizing the parameters of the model so that the embeddings of the original graph are all distinguishable from the embeddings of the corrupted version of the graph using a logistic classifier. The proposed method, Graph Group Discrimination, is shown to use much less memory, converge much faster, be more scalable, and to usually be more accurate than prior models when applied to node classification tasks.
strengths_and_weaknesses: Strengths: 
- The approach is novel, fast, and more scalable than previous graph contrastive learning methods.
- The method leads to embeddings that are accurate for node classification.
- The overview of GCL methods is informative and positions the contribution clearly.

Weaknesses:
- It is not stated how the learned graph representations are used to compute the node classifications show in the results.
- No explicit algorithm is stated, so some steps are ambiguous (e.g., it's never made clear how a "projection" and "aggregation" are in Figure 3; it's also not clear from that figure that the augmentation and corruption change between training iterations)
- I don't see why the claim in Theorem 1 about the GCN encoder leading to constant summaries is relevant: this result assumes that the encoder is not being trained, but it is, so the result does not accurately reflect what is going on in DGI. The empirical results of the author in 
Table 2 aren't explained by Theorem 1.

questions: - Explain how the graph representations learned by GGD and competing methods are actually used to compute the reported node classifications. E.g. are the same architectures used?

- State that the Figure 5 mentioned in Section 3 is in the appendix.

- The current description in Section 3 is not sufficient. Give an unambiguous loss function, and algorithm listing for minimizing this loss. Explain what the projector and aggregation steps are, and what architectures are used to parameterize/accomplish them, in the same way that it is explained a GCN is used for the encoder.

- It is mentioned that global information can be included in the GGD method in Section 3.2, but it is not clear whether this is used in any of the experiments from the main body of the paper. This should be clarified (i.e. no, none used global infromation), and the reader pointed to the Appendix for experiments using global information.

- I had questions about the data sets and specific architectures and hyperparameters used in the experiments. The former (data sets) were addressed in the Appendix; this should be pointed out in the paper. However, the experiments are not well specified because I do not see what the architectures or hyperparameters were used; please give this information in the Appendix as well. Also, what code did the authors use for the two baselines BGRL and GBT: did they reimplement them, or use existing code? This should be mentioned in section 5.1.

Overall, I thought the paper's contribution is significant, but the presentation needs work to address the above issues before I can give it a solid accept.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: I enjoyed reading this unusual paper. This paper proposes a new graph pretraining pre-text task -- group discrimination -- with low computational cost and fast convergence speed. The idea is simple -- pre-train a GNN to discriminate the node embeddings of original graphs and the nodes from corrupted graphs with randomly shuffled nodes. Besides group discrimination, the method also uses a diffusion operator to enhance the node embeddings. This method achieves state-of-the-art or comparable performances on node classification benchmarks. 

The method is inspired by an interesting observation in the previous DGI and MVGRL works. By analysis, the authors show that the supposedly contrastive objective in previous works is actually doing group discrimination. The authors then simplify the existing methods into this group discrimination pre-text task. The new pre-text task shows lower complexity and faster convergence speed.

strengths_and_weaknesses: The method is surprisingly simple but effective. It is also new and original in self-supervised learning for graphs. In experiments, the method shows significant improvement than baselines in terms of complexity and performance.

However, I find some incorrect claims of complexity and missing ablation study while reading. These weaknesses can hurt the credibility of the proposed method.



Weaknesses:

- The ablation study for the graph diffusion module is missing. Doe the group discrimination perform as good when the graph diffusion model is removed?
- Missing baseline: GraphCL. In Line218, the model uses the augmentation from GraphCL. How does GraphCL perform in the presented benchmarks?
- Some claims about the complexity are incorrect. I didn't verify all the complexity claims, but here are examples:
  - The complexity of the loss function for one node is actually O(d) because of the sum operator for node embeddings in Line173. The authors claim the complexity if O(1) throughout the paper. Despite the O(1) complexity, the theoretical improvement is weak. The time complexity of training GNNs is dominated by the GNN forward of at least O(EDL) (E is the edge number). Compared to GNN forward, the improvement on the time complexity of loss function is negligible.
  - The complexity of BarlowTwins loss is actually O(ND^2) for a batch of N nodes. This complexity is due to calculating the cross-correlation matrix using two matrices of shapes (D, N) and (N, D). However, the authors claim the complexity for one node is O(D) in Line50-51.
  - The complexity of GCN is actually O(LMD + LND^2) rather than the claimed value in Line648-649. See Cluster-GCN for the analysis of GCN's complexity.

- Some important experimental details are missing. 
  - It is unclear if the authors freeze the encoder while linear evaluation or do they fine-tune the entire model?
  - Is the learning rate in Table15 used for pre-training or fine-tuning?



Typos:

- Line123: are served --> serve
- Line142: Theorem and detailed --> Theorem

questions: Q1: graph diffusion models like SGC have shown promising feature extraction capability without any pre-training. GGD shares the similar spirit of using the power of adjacency matrix to obtain global embedding. How does GGD perform without the diffusion operator? How does the baseline perform if the diffusion operator is included?

Q2: how does GraphCL perform in the given benchmarks?

limitations: none
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on a way of scaling up GSL via the proposed group discrimination. The authors first analyze the actual reason that makes DGL work and then naturally use one scalar to summarize each node for discrimination in the MI max/min process compared to the usage of a d-dimensional vector in DGI. In summary, the proposed method is somewhat interesting and makes sense even though it is straightforward. However, the quality of this paper is still below the acceptance bar for NeurIPS due to its unclear contribution to the graph domain experts and the lack of theoretical analysis of the proposed method. 
strengths_and_weaknesses: Strengths 

The research question is significant. GSL is a hot term nowadays in the GNN domain. The demand for an efficient GSL method is increasing. This work fills this gap.

The logic of this work is quite clear. The authors first analyze why DGI works and how we can simplify it accordingly for efficiency. Then the term "group discrimination" is naturally introduced, and how it can be plugged into the paradigm of GSL is clearly elaborated.

The experiments are extensive and include several well-known large-scale ogbn datasets. The improvement in terms of efficiency is quite tremendous.



Weaknesses

The most critical issue of this work is that the proposed method is not specific to the graph domain. In other words, the proposed method seems to work perfectly even if the input is not the graph, say images. We can also apply the same idea of group discrimination for self-supervised learning on the image classification task. If the proposed method is motivated by the image domain or the general large-scale self-supervised learning, the authors should conduct experiments on the image data as well, as the proposed method is not domain-specific to graph data.

The motivation is unclear. The analysis of the DGI method is seemingly sound, but the mentioned phenomenon has nothing to do with graph contrastive learning. In fact, it is the over-smoothing issue involving GNN itself. Theorem 1 is also well-known[1]. The authors at least add some discussion on the relationship between oversmoothing issue in GNN and the graph contrastive learning. Otherwise, the motivation does not hold. It is not a rethinking of a GCL method but rather a reiteration of the well-known fact in GNN. If the motivation originates from the issue of DGI only, how can we ensure that the proposed method also works for other GCL methods except for DGI?

The proposed method is not quite exciting, and the group discrimination cannot be overexaggerated as a new GCL paradigm but more like a trick or technique. In fact, using one scalar to summarize the whole embedding vector is not quite new. For instance, in binarized GNN[2], we also aim to learn a binarized vector to quantize the embedding for scaling up GNN, which shares a similar idea.

Another critical issue is the lack of direct theoretical analysis. Since only one scalar is used for each node in GCL, how to make sure there is no information loss? Why does it perform even better than classic GCL methods? Are there any theoretical guarantees?


[1] Cai, Chen, and Yusu Wang. "A note on over-smoothing for graph neural networks." arXiv preprint arXiv:2006.13318 (2020).

[2] Wang, Hanchen, et al. "Binarized graph neural network." World Wide Web 24.3 (2021): 825-848.
questions: How is Theorem 1 related to the GCL specifically? It seems that it is just an oversmoothing issue in GNN, not caused by GCL. 

Why does the accuracy not drop when using group discrimination? How can we ensure this from a theoretical perspective? Since we only use one scalar now compared to the DGI, there should be some information loss.

How is the proposed method specific to the graph domain? It seems that the idea of group discrimination can be easily adapted to the image domain. If so, what is the performance in terms of efficiency and accuracy? 
limitations: The authors are suggested to elaborate more on the motivation part and the relationship with the oversmoothing issue in GNN.

The authors are suggested to provide a theoretical guarantee of the accuracy of the proposed method.
review_title: null

Meta-review: metareview: The paper presents a novel contrastive method of graph representation learning motivated by the observation that previous approaches to graph contrastive learning actually do group discrimination. The proposed method learns graph representations such that the representations of the original graph and a corrupted version are easily distinguishable when projected to a scalar space. The reviewers found the method to be simple and comparable in performance with state-of-the-art methods, while more scalable. 
confidence: Less certain
Decision: Accept