Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method to train large scale graph neural networks containing up to billions of parameters, called Graph Parallelism. The method is used to train large scale versions of the DimeNet++ and GemNet models containing 10-20x more parameters that the vanilla versions. These large GNN models are evaluated on a set of tasks from the Open Catalyst 2020 (OC20) benchmark and show improved performance compared to the smaller baselines. 
main_review: Strengths:  

Overall the paper is well structured  

Methods that enable effective training of large scale GNNs on datasets containing many graphs could be very impactful  

The large GNN models are benchmarked on a set of tasks that are important in the real world (catalyst design), and show quite large improvements in performance compared to the smaller baselines  

Weaknesses:

Not sure how reproducible this work is without code  

Other comments/questions:  

Some additional information on the computational cost of Gemnet-XL and Dimenet++-XL models in the 3 tasks would be useful. Eg wall time, gpu hours, etc. Although methods to train very large GNN models can be very impactful, if the computational cost is prohibitive for a lot of people then that could limit its impactfulness. 

summary_of_the_review: Overall, I vote for acceptance. The paper proposes a method to train large scale graph neural networks and shows that very large GNN models can have quite large improvements in the Open Catalyst 2020 (OC20) benchmark
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an approach to train graph neural networks with many parameters across multiple GPUs. The approach is specifically demonstrated for graphs representing molecular structure and two graph neural networks of similar flavor that have previously been presented to learn from such structures. Higher-order interaction terms make these networks very compute intensive. Leveraging their parallel approach to scale up the existing GNNs, the authors demonstrate compelling results on the OpenCatalyst benchmark. 
main_review: The approach is well-motivated and well-explained. I am familiar with the DimeNet architecture but not very familiar with the recent literature on distributed GNN training, so it is hard for me to judge the novelty of the proposed parallel approach. The results on the OpenCatalyst benchmark are compelling.

Among graph neural networks for molecular structures, the DimeNet++ architecture is specifically memory and compute intensive (and as such naturally stands to specifically benefit from the proposed approach). It would be great if the authors could include a discussion of the applicability of their approach in the context of other graph neural networks for molecular structure, such as equivariant message passing in Jing et al. (ICLR, 2021) or Schuett et al. (ICML, 2021). 

The paper does not put the scaling results in Figure 2 in context with other approaches for parallel training. 
summary_of_the_review: I did not identify technical concerns with respect to the proposed method. I will defer to other reviewers regarding the technical novelty of the approach. The paper could be made stronger by explicitly discussing how this approach is applicable to other GNN architectures that have been designed for 3D molecular structure.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a distributed training method for large graph neural networks (GNN) up to billion parameters. The method first distributes the triplet update operations to multiple GPUs and aggregate the updated vectors by global synchronization. It then distributes the edge update operations to GPUs and aggregate the edge vectors (another global synchronization). Finally, it applies node update in parallel and global node aggregation at the end. The method is applied to GemNet and DimeNet up to 1.12 billion parameters and achieved state-of-the-art results on OC20 benchmark.
main_review: Strength:
 * Good empirical results: OC20 is an open, comprehensive benchmark and the method achieved decent improvement over strong baselines in the leaderboard.

Weakness and questions:
 * Since this paper is about distributed training, it should be compared with other distributed training methods, including data and model parallelism. Does the new method run faster than data/model parallelism? For data parallelism, one could consider splitting the batch to multiple GPUs. The paper states that model parallelism can be combined with the proposed approach, but I think this paper should compare with standard model parallelism, such as pytorch DDP. If memory is the issue, combining data and model parallelism should be a reasonable baseline.
 * I am not so sure about the novelty of the distributed training approach here. It simply splits the graph to different GPUs and apply "All Reduce" after each layer of computation. This strategy doesn't seem to be efficient. Using 8 GPUs, the method can only reduce run time by 50%.
summary_of_the_review: Despite the good empirical results, it is not clear what is the benefit of the proposed approach over standard distributed training techniques (data/model parallelism). Since this paper proposes a new distributed training algorithm for graphs, it is necessary to compare the speedup of the proposed method against standard distributed training techniques.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is based on the idea that most of the computation costs to extended graphs come from triplets. The authors proposed a way to parallelize the computation of triplets in a distributed way. The author also discussed two models that fit into this framework and showed their increase in performance due to the larger parameter count enabled by their parallel framework.
main_review: The paper is based on the idea that most of the computation costs to extended graphs come from triplets. The authors proposed a way to parallelize the computation of triplets in a distributed way. The author also discussed two models that fit into this framework and showed their increase in performance due to the larger parameter count enabled by their parallel framework.

The paper is well written and easy to follow. A minor part I am a bit confused about is that in the sentence “In many applications, the number of edges is one or two orders of magnitude larger than the number of nodes, while the number of triplets is one or two orders of magnitude larger still”, do you mean the triplets are one or two orders larger than the node or the edges?

In the experiment section, network structures and hyperparameters are given and the experiment should be reproducible. The results support the authors' claim that larger graph networks have better performance.

I do not know much about this field.
summary_of_the_review: The paper is based on the idea that most of the computation costs to extended graphs come from triplets. The authors proposed a way to parallelize the computation of triplets in a distributed way. The author also discussed two models that fit into this framework and showed their increase in performance due to the larger parameter count enabled by their parallel framework.

The paper is well written and easy to follow. A minor part I am a bit confused about is that in the sentence “In many applications, the number of edges is one or two orders of magnitude larger than the number of nodes, while the number of triplets is one or two orders of magnitude larger still”, do you mean the triplets are one or two orders larger than the node or the edges?

In the experiment section, network structures and hyperparameters are given and the experiment should be reproducible. The results support the authors' claim that larger graph networks have better performance.

I do not know much about this field.
review_title: null

Meta-review: The reviewers were split about this paper: on one hand they appreciated the clarity and the experimental improvments in the paper, on the other they were concerned about the novelty of the work. After going through it and the discussion I have decided to vote to accept this paper for the following reasons: (a) the potential impact of the work, (b) the simplicity of the idea, and (c) promise of release of open source code. I think these things make the paper a strong contribution to ICLR. The only thing I would like to see added, apart from the suggestions detailed by the reviewers, is a small discussion on the carbon footprint of training such largescale graph networks. The authors motivated the work by saying it could have a beneficial impact for modelling energy which is needed to combat climate change. However, we know from recent results that such large scale models also have a non-trivial emission footprint. So I'd like to see the authors specifically calculate the carbon footprints of the models they trained. There are tools to help with this such as: https://mlco2.github.io/impact/  With this addition I think this paper will not only make a large impact on graph network training but also start a discussion of how to responsibly decide training, taking environmental impact into account.
Decision: Accept (Poster)