Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper systematically studies the graph contrastive learning approaches based on Data augmentations, Contrasting modes and objectives, and Negative mining strategies. The paper does not produce any surprising results but gives a very good view over different verticals of designing GraphCL methods. They also provide an easy-to-use library featuring modularized CL components, standardized evaluation, and experiment management. This can serve the community as a good tool to further investigate the GraphCL.
strengths: The paper systematically investigate different aspects of graph contrastive learning and studies such aspects on a good number of benchmarks. The released library to investigate such models fusther is also a nice tool that can help the community. All in all, I think this paper can be a good reference point for the community.
weaknesses: The only suggestion that I have is that I wish the authors had moved beyond the TUdataset and similar ones, and had run their evaluations on the open graph benchmark (OGB). I am aware that graph CL works haven't reported contrastive results on those datasets but evaluating them on those could have been a very good baseline for future directions. 
correctness: Good
clarity: Well-written and easy to follow
relation to prior work: The paper addresses the previous work.
additional feedback: NA
review_title: Sound paper which will help the community 

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: PyGCL attempts to study key design dimensions in graph contrastive learning by empirically evaluating the impact of a number of design choices on the accuracy of node and graph classification tasks for  a few small graph datasets. PyGCL is also available as a GitHub repo, which allows developers to attempt several different GCL methods and try different losses, serving as a resource for algorithm development. A number of observations are made as "contributions", though these observations do not generalize as no theoretical justification is provided and expansive experimentation is not conducted. From these results, it seems InfoNCE is a consistently useful loss, although may be impractical, given its compute requirements. Topological rather than feature augmentations are beneficial, mixing local-global augmentations is not a good idea, and negative mining is not worth the effort. However, these observations are tempered by completely different results from SSL for architectures that don't require negative samples so the work motivates future study using a clean, standardized framework like PyGCL to continue studying these design dimensions.

strengths: The demonstration that negative mining strategies generally don't improve performance and that InfoNCE is a consistent loss function for developers to start with is useful to the GNN community. In general, the authors conduct a large number of experiments across a variety of conditions in a standardized way. The field is quite diffuse and even though some newer stop grad methods are not included, these results are useful for researchers. 
weaknesses: Generally, the design space is too large for the approach the authors take to study it and they don't have an argument that justifies that they can make general claims. A particularly concerning discrepancy is that they only work with graphs that have fewer than 500 nodes. There are many publicly available medium to large scale graphs available as benchmarks so the data examined should at least be representative of the current state of research. There are some interesting results, e.g., that augmentations greatly improve node but not graph classification, that the authors don't comment on and in so omitting such results, fail to point out interesting trends in the field, e.g., that the design space also depends on the task. 
correctness: Some of the answers to the posed questions, e.g., "What is the most contributory component in an effective GCL algorithm" may differ for each architecture and dataset. The authors should justify why their empirical results should generalize to such a broad claim re: contrastive learning on graphs.

The problem formulation only considers graphs with node features and unweighted adjacency matrices. The authors should discuss how to incorporate and generalize their empirical study to graphs with edge features. Alternatively, they should clarify why they don't consider link prediction tasks.

Another equally valid speculation re:Table 2 results is that removing edges preserves true information, whereas adding edges adds noise. This highlights that many claims in the results are speculations and have scant theoretical justification. This would be fine if the entire space were explored, instead, the authors limit such experiments to one objective (InfoNCE) and contrasting modes. Would these results hold if a different objective were tried? To make the results more correct, specific details should be added, e.g., "Observation 1. With InfoNCE and local-local contrasting reps, ..."

In Table 2, only small graphs are used. It is incorrect to say that medium to large graphs are used, especially when OGB is available for millions of nodes.  

From Fig 2, it seems that the accuracy doesn't depend on any dropout probability at all, in fact, while the authors claim that there are significant observations to be drawn from these minuscule changes. It would help if the authors add error bars to their figure and test extreme cases (why not put dropout probability closer to 1 to see decay and 0 to recover the "None" case)?

clarity: The questions that the authors aim to address are clearly stated and the paper is well written. There are certainly details that could be slightly expanded upon but leaving the details to the appendix is appropriate, given the breadth of discrete variables per design dimension, and tastefully done. It is not clear what the data is and there should at least be a table in the main text (if possible) or a quick indication of number of edges and nodes in each dataset, to ensure that the datasets chosen are represent many scales.  

The experiment adding stochastic and deterministic augmentations is not clearly described and seems to do much worse than topology augmentations anyway so, for clarity, I would suggest Fig 4 and the corresponding section be deleted. The results just confuse and indicate something weird may be happening.

relation to prior work: There is at least one prior work (Design space of GNNs by Leskovec's group) that is similar in trying to understand how message passing network designs influence performance that the authors could either mention or build upon in standardizing some of the GCL methods. The authors could use that to disentangle architectural differences vs. impacts of dimensions they identify as sole "designs". In addition, section 2.2 and some of the comparisons covered in the methods are not quite extensively discussed, even though some papers, such as You et al. and the BYOL paper. This may be fine but it would be nice to see what datasets and performance each method achieves, since it remains unclear how this work relates to those benchmarks (as the authors originally constructed) or whether there are shared evaluations across GCL papers. Furthermore, some details are not explained, which should just be fixed, e.g., SP-JSD v. JSD. 

additional feedback: Given the authors say the InfoNCE require more negative samples, the authors should at least attempt to describe the computational scaling requirements of models using InfoNCE for comparable performance or compare the computational requirements for each method to achieve what are usually small marginal advantages over other methods, so the reader can understand whether fractions of a percent of accuracy are worth additional compute. Along those lines, the authors should at least run experiments on a larger GPU mem or mini-batch (e.g., cluster GCN) experiments that run OOM.. a failed experimental implementation should not be reported in the table. The intereptation of Observation 6 is not much of an argument at all but a speculation and should be removed to strengthen the paper. Instead, adding details on the datasets used might be helpful, and esp. to ensure that accuracy alone is fair, e.g., that there are not class imbalances.
review_title: Round 2 Review for An Empirical Study of Graph Contrastive Learning

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors first characterize GCL models in terms of four design dimensions, namely data augmentations, contrasting modes, contrastive objectives, and negative mining strategies. Then, they empirically evaluate how each design decision impacts the performance in downstream tasks in a systematic manner.
strengths: The experiments are extensive; they cover the design dimensions that the authors have set out to explore fully. Methods that generally work the best has been identified for each design dimension, which I think would be very useful for anyone looking to apply GCL algorithms for their own problem setting. The authors have also identified directions that require further research, which are summarized at the end of Section 4.
weaknesses: Some augmentation methods and contrasting modes have been left out due to computational limitations (as mentioned in Section 2.2). Please see my comments for correctness as well. Other weaknesses have already been acknowledged by the authors in Section 4.
correctness: I believe Observation 1 is not correct. MDK and PPR are edge-removing augmentation methods yet they generally perform worse than EA (PROTEINS and IMDB-M are the only two environments where both MDK and PPR outperform EA). I think the results presented in Table 2 are still interesting but Observation 1 needs to be either removed or rephrased (and the text accompanying it needs to be revised as well).


clarity: The paper is very well written; each design dimensions is defined clearly, experiments are summarized well through six observations (with the exception of Observation 1's inaccuracy), and limitations and future directions are outlined clearly in Section 4.
relation to prior work: The prior work is summarized in Table 1 using the proposed way of decomposing GCL methods into four design dimensions. I think this unified view of the prior work is a strength.
additional feedback: I think the performance of feature augmentations alone (FM/FD) should be plotted in Figure 3 as well (so that it is easier to see hybrid augmentations perform the best compared with all non-hybrid options). Similarly, the performance of stochastic augmentations alone (FD/ND) can be plotted in Figure 4.
review_title: Review of "an empirical study of graph contrastive learning"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper studies graph contrastive learning approaches and performs an empirical comparison and analysis. The authors introduce a taxonomy to categorize existing methods according to four dimensions: augmentation functions, contrasting modes, contrastive objectives, and negative mining strategies. Thanks to such problem decomposition, the authors can experiment with different components of GCL methods and draw several conclusions for future GCL research. The authors also publish an open-source library PyGCL, which allows building graph contrastive learning methods using various ready-to-use components (aligned with the four taxonomy dimensions).
strengths: + usage of popular benchmark datasets
+ extensive analysis according to four taxonomy dimensions
+ multiple downstream tasks
+ clear observations/conclusions that can be used in future GCL research
+ the experimental code is available and the open-sourced PyGCL library allows to easily build custom GCL models
weaknesses: **W1. Motivation for experimental setup**

For each experimental setup it would be beneficial to explain why the authors choose such a setting, e.g. in "3.1 Data Augmentations" why did the authors specifically use the InfoNCE objective and not another one? Maybe another objective would lead to different observations? 

Although subsection "3.2 Contrasting Modes and Contrastive Objectives" shows that InfoNCE is the best, we do not know whether this is due to the usage of specific data augmentations, which were the best performing in the previous experiment (again using the InfoNCE objective). 

I encourage the authors to address the experimental scenarios design choices, at least in the Appendix as there are space constraints for the main paper.

**W2. Missing details / clarity**
- Experiment "3.3 Negative Mining Strategies" lacks information about which augmentations were used
- In the main paper, there is no information about the used backbone GNN, optimization procedure or other training details; please redirect the reader directly to appendix B.
- I would suggest that the authors include a standardized way to denote the setup in each experiment, e.g. in the first line of each experiment/subsection I would include the following summary line (or something similar), where you explicitly denote which part is *evaluated* and which ones are fixed. This should allow saving some space, compared to textual descriptions.
`
**Augmentions:** ND + FM | **Contrasting mode:** L-L  |  **Contrasting objective:** InfoNCE |  **Negative mining strategy:** *evaluated*
`

**W3. Comparison protocol**

In appendix B, the authors mention that in every experiment they use a grid search to find the best GNN model architecture. I am not fully convinced that such a strategy leads to meaningful results. In my opinion, it would be better to use the same architecture across scenarios, e.g. when checking the contrastive objective for a particular dataset, you should use the exact same GNN encoder architecture, embedding dimension, learning rates, etc. In the current case, there are too many degrees of freedom to get consistent results.

EDIT: Clarifications for the concerns were provided by the authors

correctness: I have some doubts about sticking mostly to the InfoNCE objective. Otherwise, the experiments are well performed, including the popular linear evaluation scheme, as proposed in DGI and used in many follow-up papers. 
clarity: The paper is well written and easy to follow. There are a few minor grammar mistakes, but they should be easy to fix. I would suggest another proofreading round before the final revision.
relation to prior work: The paper considers multiple GCL methods, including older ones and the most recent approaches. The authors also refer to other self-supervised graph learning studies and denote the differences to their work.
additional feedback: Instead of leaving it in the appendix, more emphasis should be put on negative-sample-free methods, as negative sampling can be too expensive for large-scale real-world datasets.

I am willing to increase my score after clarifications from the authors.
review_title: Good paper, but some clarification needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper performs an extensive study on graph contrastive learning (GCL). The authors decompose the design components of GCL into four aspects, augmentation, contrasting modes, objectives, and negative sample mining. The authors further perform extensive experiments based on these four aspects. 

The contributions of this paper include: 
1. A good breakdown of the design choices in GCL into four major aspects. 
2. Extensive experiments on different datasets and tasks (node-level and graph-level). 

strengths: 1. A good breakdown of GCL methods into design components. 
2. Observations are insightful and guide designs of GCL methods. 
3. Clear discussion of drawbacks and future works. 
4. Documentation is clear. 
weaknesses: 1. Some details are missing, which makes understanding this paper hard. In the main paper, the authors briefly describe the design dimensions, but no details are stated. This makes understanding hard. For example, in the **data augmentations**, I find it hard to understand the difference between FM and FD. Also, in the negative mining strategies, few details are given (even in the appendix).
correctness: I think the observations and insights derived are correct and justified by experiments. 
clarity: The paper is well written and in general easy to follow. 
relation to prior work: I am not highly familiar with GCL. From my perspective, I think the discussion is OK. 
additional feedback: Section 2.1: Jenson-Shannon Divergence instead of Jason-Shannon Divergence.
review_title: An extensive study on GCL

Meta-review: The paper studies methods for contrastive learning on graphs, categorizing and performing extensive benchmarks on existing methods. They also provide an open-source package to facilitate further study of graph contrastive methods. Reviewers felt that this was an important topic to study, and that the detailed investigations and open-source library will be of use to the community. Reviewers wnP1 and rwpd initially had some concerns about details related to experimental setup and comparisons, but these were largely addressed by the comprehensive responses provided by the authors. Congratulations on having your paper accepted to the NeurIPS 2021 Datasets & Benchmarks Track! When preparing the final version of the paper, authors are strongly encouraged to take into account the issues raised by Reviewers wnP1 and rwpd.
Decision: Accept