Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to train graph neural networks on dense hardware such as TPUs. The method is motivated by an observation that connections in graphs have locality in some datasets.  Experiments show significant improvements in training speed compared to single-GPU training.

The overall score of this paper is slightly positive. There is a certain demand to perform training on hardware targeted to dense computations. Even though the applications of the proposed method is limited to data with low-bandwidth, the paper shows there are real applications of the method. The effectiveness of the proposed method is well-supported by the experiments.

Major comments:
Comparisons with single-GPU training can be unfair. The method in Ma et al. (2018) is indeed not easy to scale many GPUs because their target is processing extremely large graphs in parallel. Since the experiments in the submitted paper use relatively small graphs that fit in a single GPU memory, it will not be so challenging to scale many GPUs. At least, it is recommended to compare the results with training on several GPUs using data-parallel execution implemented in TensorFlow (or any other suitable frameworks). If it is difficult, please provide more specific reasons why it is challenging to perform multi-GPU training.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose a method to speed-up the time to validation accuracy for a particular class of graph neural networks:  Gated graph sequence neural networks (GGSNNs).

The paper is interesting in that it describes several operations and engineering considerations to speed up the processing of a GGSNN on TPUs. It is essentially a collection of engineering steps that improve the time to validation accuracy.

While I'm not an expert in (G)NN acceleration on TPUs, I have experience with GNNs and approaches to accelerate CNNs in GPUs. My assessment is that the scope of this work is far too narrow. It is specific to GGSNNs which is a small family of GNNs not widely used. It is also specific to TPUs and lacks evaluations of the proposed approach on other type of hardware.

It is for these reasons that I think the paper is not appropriate for ICLR. The scope has to be broadened both in terms of the NN models and the hardware types. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors present a framework to implement graph neural networks training
efficiently ---an inherently sparse task--- using "custom dense hardware", here
Tensor Processing Units (TPUs V2). The key steps are: (1) reordering the labels
to reduce the bandwidth of the adjacency matrix, (2) (Sometimes approximate)
Decomposition using block matrices for efficient storage and computations, and
(3) memory layout optimization. They evaluate their framework on the VarMisuse
dataset and compare the performance against a GPU implementation running on
Nvidia Tesla V100. In the best configuration, they were able to reach 78% test
accuracy in 13 minutes vs 19h for the baseline.

The paper is well structured and thorough. I was able to understand the
challenges and the solutions proposed, even without prior knowledge in the
architecture this work focuses on. I think it is sufficiently detailed to enable
an independent implementation without referring to the authors source code.

However, I feel that it might lack novelty. Indeed, as described in the Related
Work section, each component of the pipeline is well known and used very
frequently in the HPC community. Sometimes, knowing how arrange common
primitives is very powerful, and looking at the results of the experimental
section, this is enough to improve performance by almost two orders of
magnitude. However, I think it might be more due to some intrinsic properties of
the dataset than the method itself. As made clear by the title of section 5.1,
it is the data itself that has low bandwidth. If we consider a dataset that does
not satisfy this requirement, stage (1) has no effect, and it is impossible to
perform the decomposition done in (2). In that situation, the contributions of
this paper would be nullified.  It would be perfectly reasonable to think that
most datasets have low bandwidth, but this not a claim that made the authors.
This work would be considerably more impactful if it measured the bandwidth of
more well recognized datasets. My small exposition to these problem does not
allow me to make any meaningful suggestion.

Finally, I am puzzled by the "dense hardware"/GPU distinction. From my
experience, GPU devices *are* designed for and extremely efficient at dense linear
algebra. Sparse operations are historically performed on CPUs. While they are
possible on GPUs, they are usually much slower. For example CUSparse, the sparse
matrices library, part of NVidia CUDA toolkit, was only introduced in its
version 4. It's a clear indication that sparse operations are not a strength of
GPUs. According to my experience writing GPU code, I feel that this approach
would actually perform extremely well on GPUs as it does on TPUs. I think it is
thus important to compare this framework on GPUs too. Since these
optimizations are not TPU specific and have not been applied in the GPU based
GNN libraries referenced in this paper reinforce my concerns that they are
problem-specific.

Even though the performance gains demonstrated are sizeable, the fact the
approach does not seem TPU specific and is potentially problem specific makes me
lean towards rejection.

review_title: Official Blind Review #3

Meta-review: While there was some interest in the ideas presented, this paper was on the borderline, and was ultimately not able to be accepted for publication at ICLR.

Reviewers raised concerns as to the novelty, generality, and practicality of the approach, which could have been better demonstrated via experiments.
Decision: Reject