Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a self-supervised graph representation learning algorithm named BGRL. BGRL employs similar architecture & training mechanism with BYOL and necessary adjustments (e.g., the augmentation methods, dropping the prejector module) to adapt to the characteristics of graph datasets. Given the nature of BYOL-based methods that negative samples are not used during training, BGRL scales only to O(n) when computing the objective function, which essentially breaks the computation limit incurred by the quadratically complex objective function adopted by SimCLR-based models (e.g., GRACE), hence makes the proposed model applicable to large graphs.

The major contributions of the paper are summarized as follows:

(1) The paper highlights and empirically verifies BYOL's advantages in not only boosting task performance but also enhancing model scalability, the second aspect is important but ignored by previous / concurrent BYOL-based methods.

(2) The paper tests the effect of introducing label supervision to the BYOL-based scheme on learning node representations for large graphs, which is not studied in previous / concurrent BYOL-based methods.

(3) The paper evaluated BGRL against other base models on various benchmarks, which not only provides multiple aspects of BGRL's advantages, but also enriches the baselines of unsupervised graph representation learning, especially on large graphs.
main_review: Strengths:

The developed method is technically sound. The paper is well-organized and easy to follow. The motivation of employing BYOL in tackling the scalability challenge is clearly stated and the empirical studies are sufficient to show the efficacy of BGRL in both producing high quality node representations and scaling to extremely large graphs. I appreciate the empirical results provided by the paper, which is useful to the graph representation learning community. The implementation specifications are recorded in detail. Other contributions are summarized as above.

Weaknesses:

(1) My major concern is the novelty of the paper. Although dropping projection network makes differences in the model architecture between BGRL and the original BYOL, its technical contribution is incremental. Besides, such technique is already discussed in SelfGNN [1].  

(2)  In experiments on large graphs, it would increase the significance if 

* an ablation study on the model choice of encoder is provided,
* more recent base model(s) with O(n) complexity in computing objective functions (e.g., MVGRL [2]) are compared against.



References

[1] Zekarias T. Kefato and Sarunas Girdzijauskas. Self-supervised graph neural networks without explicit
negative sampling. CoRR, abs/2103.14958, 2021

[2] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on
graphs. In International Conference on Machine Learning, 2020.
summary_of_the_review: In summary, although there are empirical contributions in the proposed method, the paper appears only to apply the well-developed BYOL scheme to graph representation learning tasks. The technical contribution is sort of incremental.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a novel graph contrastive learning model with bootstrapping latent objectives. Extensive experiments demonstrate the effectiveness of the proposed BGRL method.
main_review: # Strengths
+ The proposed method is easy to follow and the paper is well-organized.
+ The BGRL model is simple and effective.
+ Extensive experiments demonstrate the effectiveness and efficiency.

# Weaknesses
- The novelty of this work is a bit limited as this is a simple adaptation of BYOL on graph-structured data.

summary_of_the_review: It feels to me that no obvious weaknesses can be spotted. The BGRL method is easy to follow, is simple and also powerful, and the authors did an extensive set of experiments to verify the effectiveness and also efficiency of BGRL.

# Suggestions
* In Table 5, the memory consumption for both GRACE and BGRL should be given so that readers could understand how much memory is needed for sufficient number of negative samples to achieve similar performance as with BGRL.
* I believe some graph-level evaluations may further expand the applicability of BGRL. In this case, how to choose appropriate representations (e.g., graph- or node-level embeddings or some sort of context representations) for both branches is important.
* As the authors point out, theoretical analysis on the learning dynamics of the online and the offline branch [1] may be helpful for understanding the superior performance of BGRL.

[1] Y. Tian, X. Chen, and S. Ganguli, Understanding Self-Supervised Learning Dynamics without Contrastive Pairs, in ICML, 2021.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces Bootstrapped Graph Latents (BGRL) to learn graph representation by predicting alternative augmentations of the input.  The experiment results on a wide range of datasets exhibit appealing performance.
main_review: The paper can be evaluated mainly from two perspectives:

1. The novelty of the method and the conceptual contribution is extremely limited since the proposed model is a pretty straightforward application of the BYOL [1] model on graph data.  The difference of not having a projection head cannot be deemed as a contribution or novelty, since the reason is the low dimension of the node features in the graph datasets, as mentioned by the authors.

2. The experimental contribution is very solid. The model is evaluated on multiple large datasets and is also tested via the KDD CUP competition and promising results are obtained. This would be a significant contribution to the community and would facilitate future researches in this direction.

Besides the two main points raised above, the method section could be further improved by including explanations on the motivation of the designs, which is especially important for readers who are not familiar with BYOL. Currently, the paper just described the operations of the model without explanation on the operations are designed in this way. Explanations from the following points may be considered: 1. What is the difference between the online and the target branch. 2. What makes the target branch become a target for the online branch? 3. Why is the target branch updated through EMA? 4. What objective does the target branch (\phi) follows, as mentioned in the paragraph under Eq(3), in page 3. 

[1] Grill, Jean-Bastien, et al. "Bootstrap your own latent: A new approach to self-supervised learning." arXiv preprint arXiv:2006.07733 (2020).
summary_of_the_review: Considering the empirical contribution of the paper, I am inclined to accept this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes BRGL, a method for graph representation learning based on ‘bootstrapping’ (in the same sense BYOL is). Different from the prior art, which mainly relies on contrastive learning, BRGL naturally scales linearly with graph size.  An extensive suite of experiments shows that BRGL scales better than previous works while yielding state-of-the-art (SOTA) performance in node classification. 
main_review: # Strengths

* Different from contrastive methods, the cost of computing the loss for BRGL exactly scales only linearly with graph size. 
* An extensive suite of experiments shows BRGL achieves SOTA while avoiding the performance-memory tradeoff of contrastive methods.
* BRGL was validated in practice at the KDD cup 2021.

# Weaknesses
* BRGL is a direct analogue of BYOL for graphs (but without a projector network). Therefore, the novelty here is relatively limited.
* The authors provide no intuition on why their method works.

# Observations
*  It might help to show Table 1 as scatter plots nodes/edges versus memory. This way, we could ‘visualize’ the asymptotic difference between GRACE and BRGL.
* From the description in Section 4.2, GRACE-SUBSAMPLING scales linearly on $k$. If this is the case, $k=2$ asymptotically implies the same complexity as $k=2048$.  
summary_of_the_review: The paper shows good results, but i) the novelty is relatively limited; and ii) there is no intuition on why the method works. Therefore, I believe the paper is just below borderline.
review_title: null

Meta-review: To perform self-supervised graph representation learning that is scalable to large graphs, the authors propose Bootstrapped Graph Latents (BGRL) that learns its graph representation by predicting alternative augmentations of the input, avoiding the need to construct negative examples. The weakness of the paper lies in its novelty, as it can be considered as a direct adaptation of the BYOL method, whose success has been demonstrated on self-supervised visual representation learning, to learn graph node representations. While the novelty is limited, the paper has shown how to appropriately apply BYOL to graph representation learning, achieving state-of-the-art results on graph node representation learning on large-scale graphs. The overall assessment of the reviewers is that the empirical significance of the paper outweighs its shortcoming in novelty. The AC agrees with this assessment and hence recommends acceptance.
Decision: Accept (Poster)