Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work proposes quantization methods for low bit training of GNNs. 
strength_and_weaknesses: Pro: The paper claims to improve training speed by using dynamic quantization. 

Con: The solution is complex, which limits its applicability. And technical novelty is rather limited. 

clarity,_quality,_novelty_and_reproducibility: The writing is reasonably clear, but some claims which was to justify the proposed method were not demonstrated. 

summary_of_the_review: The paper is not ready for publication. Additional works are needed to justify the claims and to simplify the algorithms.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes four factors that restrict the applicability of traditional quantization methods in GNNs. Based on that, a specialized and efficient quantization framework is proposed to improve forward, backward, optimizer, and loss functions with the dynamic quantization (DyQ), static quantization (StQ), and weight quantization (Qw ) operators. Experiments show the reduced memory and time complexity with comparable accuracy.

strength_and_weaknesses: **Strength** 
- The quantization of GNNs is an important problem, and the paper provides a comprehensive comparison of different quantization methods.
- Different quantization operators are proposed to handle real and pseudo overflows, and are plugged into the GNN workflow.
- The empirical results show improvement in efficiency.

**Weaknesses**
- Though I believe this paper has some nontrivial technical contributions, the writing of this paper needs to be greatly improved to make it accessible for the community. I have tried hard and spent a significant amount of time, but can still get a rough idea without understanding the details of why pseudo flow is a key bottleneck and how it is handled by DyQ. For example, Section 3.1 is hard to follow for a general reader like me: what exactly the pseudo overflow problem is (e.g. when and why $c$ cannot be represented by $2^n-1$), what is the correlation and difference between real and pseudo overflow. Figure 2 is almost impossible to understand for general readers too.
- It is not clearly justified why dequantization is not needed in this method.
- The claim that this method can accelerate inference is not supported by experiments.
- This method is not compared with existing works, for example, the memory reduction compared with EXACT.

clarity,_quality,_novelty_and_reproducibility: The writing of this paper should be improved for general readers.

Though the quality of writing is not satisfactory, I feel this paper contributes some technical novelty.

The code snippets are provided for reproducibility.

summary_of_the_review: This paper identifies some nontrivial challenges in quantization of GNNs. However the writing of this paper is hard to follow for a larger audience. I believe it is a work with good potential if it can be better presented.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a method to accelerate both training and inference in Graph Neural Networks by quantizing both the
weights and all the intermediate results (features, errors, gradients) to 8 bits. The proposed training and inference
pipeline can be executed on integer-only hardware.

To deal with the high dynamic range arising in GNNs due to various node degrees and feature types, this work proposes
to use a dynamic quantization technique to map 32-bit intermediate results from multiply-and-accumulate steps back to
8-bits shifting right with a dynamically-determined amount and clipping to 8 bits. The shift count is determined such
that the number of outliers resulting in overflow or underflow is minimized.

Some experiments are carried out on 8 GNN benchmarks, matching the accuracy of floating point counterparts. No
comparisons against state-of-the-art GNN quantization methods are provided. 
strength_and_weaknesses: * Strengths:

  * Due to the scalability issues and online-training demands for, integer-only training and inference is likely to have
  high practical impact.

* Weaknesses:

  * Technical contribution is incremental (see notes below on novelty).
  * Clarity could be improved a lot (see remarks below on clarity).
  * No comparison is provided against any competing method. The only baseline is the floating point pipeline.

clarity,_quality,_novelty_and_reproducibility: * Clarity:

  * I found the paper hard to follow, as explanations seem convoluted and structure needs improvements. Some examples:
  
    * On page 2, " a shaping model can be obtained directly". What do the authors understand by a "shaping" model?
    * On page 2, what do the authors understand by "quantification methodology"?
    * The authors refer in several places to "dequantization" from FP to INT (e.g. on page 3, 
    "the output [...] must be dequantized from the FP32 to INT16 format"). Shouldn't it be quantization that is needed
    here?
    * On page 4, "$x_{fn}$ represents the signed <i>bitn</i> to which $x$ is quantized". I am confused as to what a
    "bitn" is.
    * I guess $x_{in}$ and $x_{bn}$ are both the integer representation of $x$, just that the first is viewed as an
    integer in base 10, and the other is viewed in binary. It is confusing to use different mathematical notations
    for the exact same mathematical object (the same integer).
    * In Figure 3, they authors plot a histogram of a tensor multiplication output, but they do not state how was this
    histogram obtained. Were the tensors sampled from a certain distribution? Or is this obtained from a training/test
    sample?
    * The function $f$ appearing in equation (2) is never introducted (the only place it appears is on the vertical axis
    of figure 3).
    * The authors mention that "We utilize Kullback-Leibler divergence (Joyce, 2011) to prove that when g(k) in 
    Equation 2 gets the maximum, we may derive a 7-bit(k, k + 6) expression for this data set with reduced Kullback-
    Leibler divergence and more Shannon Entropy (Bromiley et al., 2004)", but no proof is provided, nor is it clear how
    KL divergence/Shannon entropy is related to maximizing g(k).
    * In Algorithm 1:
      * line 4, who is $j$? This variable is never iterated over nor assigned anywhere.
      * line 5, AdderTree has a height argument, but what output is it invoked over?
      * LoadAtoeachThreadGroup and AdderTree are never defined.
      * The vector $C$ is never initialized. 

* Novelty:

  * From a technical point of view, the framework is very similar to the NITI method, see
  <i>M. Wang, S. Rasoulinezhad, P. Leong and H. So. NITI: Training Integer Neural Networks Using
  Integer-Only Arithmetic,  IEEE Transactions on Parallel and Distributed Systems, 2020, DOI:10.1109/TPDS.2022.3149787
  </i>. While not applied to GNNs, their training pipeline works on full 8-bit integer arithmetic, and their dynamic
  shift and shift-and-round criterion is almost identical, with the only difference that they select the shift
  count to minimize overflow alone, while here the authors also take into account underflow. From a technical
  standpoint, this work is just an incremental step over their method.
  * There is no empirical novelty, as this work simply compares the integer-only training pipeline against the FP32
  baseline on standard GNN datasets.

* Quality:

  * Relevant baselines are not compared against (see baselines mentioned in section 2 and the NITI reference above)
  and no analysis of results is provided.
  * No theoretical justification for the criterion is provided (minimizing KL divergence is mentioned, but no proof is 
  given).
  * Overall, the scientific quality of this work, in its present form, is below acceptance standards for ICLR.

* Reproducibility:

  * The authors provide some information on their experimental procedure in the caption of Table 3.
  * Some code snippets are provided in the manuscript, but no intention to release publicly available code is
  mentioned. They do mention "implementing in existing framework" as a future intention, but it is not clear what
  framework they are referring to or whether this means the code will be public.
summary_of_the_review: This seems to be unfinished work with incremental technical novelty. The manuscript needs a major rewrite to
meet the presentation quality standards for publication (see notes on clarity above). At lease one relevant baseline is
missing (see e.g. NITI above), and no comparison against any competing work is provided. Overall,
in its current form, this work does not meet acceptance standards for ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes a method to accelerate GNNs during training and inference by using dynamic quantization. The method experimentally is effective, but no comparison is given with state-of-the-art competing methods, and the technical novelty is limited with respect to existing approaches to network quantization, e.g. the NITI method of Wang et al. in IEEE Transactions on Parallel and Distributed Systems, 2020.
justification_for_why_not_higher_score: All reviewers rejected
Decision: Reject