Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is about speeding up the GNN training with a new way for sampling which combines layer and neighbor sampling methods. 
strength_and_weaknesses: Strength:
The paper is about how to perform sampling GNN training. The sampling method proposed seems a reasonable way to combine the layer and neighbor sampling methods.

Weakness:

1: The proposed method aims to replace NS using a variance reduction sampling and achieves better convergence. The question is whether NS achieves SOTA performance on the standard GNNs datasets?   

2: My main concern is on the experiment part:

1) The paper is about sampling method to speed up GNN training. However, there are no result on wall time for training GNNs.

2) There is no accuracy reported for all the four datasets tested in the experiment part.

3) No comparison with latest GNN fast training methods.  

4) missing ablation studies. 
clarity,_quality,_novelty_and_reproducibility: The paper is with good writing. The sampling strategy seems novel to me. 

summary_of_the_review: I think the paper is interesting.  Due to limited experiment results, I am not fully convinced by this method for speeding up GNNs training.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a new sampling  strategy: LABOR, which employs the Poisson Sampling to achieve a better sampling variance. Furthermore, it introduces a fanout hyperparameter $k$ and  delicate importance sampling probability to sample much fewer vertices without sacrificing quality. 
strength_and_weaknesses: ## Strengths
 - Well organized and easy to follow. 
 - Detailed analyze about the sampling probability with   fanout hyperparameter $k$. 

## Weakness
 - Technical contribution is limited. Principally, This paper follow the layer-wise sampling paradigm and replace the uniform sampling with the Poisson sampling. The paper also show that this sampling strategy can obtain a smaller sampling variance. This point seems a marginal improvement over the existing sampling framework. Concretely, except for the smaller sampling variance, I can not figure out the benefits of adopting this sampling strategy in the large-scale GNN training from the analysis in Section 3. From the theoretical perspective, would it improve the convergence speed  or reduce the memory cost, or other benefits? 

 Since the efficient training 
 - Lack of comparison with other layer-wise baselines. As mentioned in Introduction, there are several layer-wise sampling algorithms, such as AS-GCN[Huang et al., 2018] and GNS[Dong et al., 2021] (Both of them have released the source code.). However, this paper only make the comparison with LADIES. This is not sufficient  considering both studies have released source codes. 

 - The experiments can not demonstrate the superiority of the proposed method. First, the paper only shows the training loss curve of all the methods. This is not sufficient since we can not get any cue about how the sampling strategy would affect the generalization ability unless the validation curves or the metrics on the test set are reported. Despite that, the improvement over the training loss is still very marginal and we can only observe its difference according to the zoom-in figure. 
 
 - Moreover, to demonstrate the effectiveness of the proposed method. It's better to make such comparisons:  
   - Vanilla GNN model with full-batch on the small data: to check the performance gain/drop of the proposed sampling method.
   - Other kinds of sampling strategies, e.g. GraphSAINT [1] and ClusterGCN [2].



 [1] Zeng, Hanqing, et al. "Graphsaint: Graph sampling based inductive learning method." arXiv preprint arXiv:1907.04931 (2019).

 [2] Chiang, Wei-Lin, et al. "Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks." Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear and written with a high quality. My main concerns are the novelty and the experimental justification.  I have no idea of the reproducibility since there is no source code released, but I think the experiments are easy to reproduce. 
summary_of_the_review: This paper is about a new sampling strategy of the layer-wise sampling in the GNN training. The details are clear and the analysis of sampling parameters is OK. However, the contribution of this  modification  looks marginal and the experiments failed to demonstrate the superiority of the proposed method. I vote the rejection for this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present LABOR, a method that experimentally shows to be better than LABOR neighbor sampling and layer sampling approaches.
strength_and_weaknesses: Strengths:
1. This work addresses the Neighborhood Explosion Phenomenon (NEP), the curse of dimensionality version of Graph Network Training.
Weaknesses:
1. The abstract states the method's superiority but not by how much or against what the method is compared.
2. The same happens in the main body of the work: it is not clearly stated how much better is the proposed pipeline.
3. The last point is important because the proposed method depends on a long series of assumptions and results that are true in general or valid in limit cases. Thus, the actual effectiveness of the method, which could be shown with experiments, is not indeed shown. Thus, it is challenging to state whether the method is suitable.
clarity,_quality,_novelty_and_reproducibility: The pipeline is well explained with a solid explanation of the theoretical foundation. However, the experimental results are not conclusive enough to state the usefulness of the method.
summary_of_the_review: It is an interesting paper with a great deal of theoretical justification. However, the fundaments rest on general assumptions whose usefulness needs to be assessed with experiments. But, as presented, these experiments are not conclusive enough to judge the pipeline benefits.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: While the reviewers appreciated the paper’s organization, clarity, and its hyperparameter analysis, their main concerns were with (a) the baselines compared against, (b) the technical contribution balanced against the overall improvement of the method. Specifically, for (a) the reviewers would have liked to see comparisons against other sampling algorithms (AS-GCN, GNS, GraphSAINT, ClusterGCN). The authors responded by pointing out that AS-GCN and GNS can be combined with their proposed method LABOR (AS-GCN reduces the variance of any sampler, and GNS requires access to a sampler, which could be LABOR). They add an experiment comparing against GraphSAINT and show LABOR converges faster (finally they point out that GraphSAINT improves over ClusterGCN). The issue is with the first two baselines: do AS-GCN and GNS still maintain their improvements when using LABOR instead of NS? Or do they actually harm LABOR’s result, making it worse than NS. This question is left unanswered. For (b), the reviewers point out that the improvements in training loss are marginal, often only observable when zooming into the validation-F1 or training-loss curves. The authors respond pointing out that their method converges more than 2x faster compared to NS and a 3% improvement in accuracy. First, I don’t know which accuracy result they are referring to: in Figure 1, NS often matches/improves over LABOR. Second, this isn’t a direct response to the core point the reviewers are making: the improvements are small and it’s not obvious why. It would be useful if the authors could explain why such improvements are important. As there is a relatively small methodological change made by this paper compared to prior work, the authors should convince us that the results aren’t similar because the change is minor. Given these open points, I believe this work should be rejected at this time. Once these things are clarified this paper will be improved.
justification_for_why_not_higher_score: See above metareview.
Decision: Reject