Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an aggregation-aware mixed-precision quantization method for GNNs, which quantizes different nodes features with different learnable quantization parameters, including bit-width and step-size. The paper also proposes a Nearest Neighbor Strategy to deal with the generalization on unseen graphs. Empirical evaluations on several node-level and graph-level datasets demonstrate the efficacy of the proposal.
strength_and_weaknesses: Strengths:

1. Given the increasing deployment of GNNs in edge devices, it is important to improve their energy efficiency. This paper tackles this timely problem, and is generally well-motivated.

Weaknesses:

1. My biggest concern is the limited novelty of the paper. It seems that the paper is applying known tricks in mixed-precision quantization to GNNs. The authors add a statement in section 2 "However,  due to the huge difference between GNNs and CNNs, it is difficult to use these methods on GNNs directly". However, I am not convinced with such an argument without concrete evidences. What happens if these techniques are naively applied to GNNs?
2. The paper only reports speedup with their quantization scheme. What about the energy/power consumption which is another important metric given the resource constraints of edge devices?
3. The authors do not quantify their accuracy-bitwidth trade-off with several related works they cite in section 2. Only DQ-int4 is considered.
4. Several related comparisons with quantized GNNs are missing, such as [1-3].  

[1] Brennan et al., "Not Half Bad: Exploring Half-Precision in Graph Convolutional Neural Networks", 2020

[2] Huang et al., "VEPQuant: A Graph Neural Network compression approach based on product quantization", 2022

[3] Zhao et al., "LEARNED LOW PRECISION GRAPH NEURAL NETWORKS", 2020
clarity,_quality,_novelty_and_reproducibility: The writing is clear. However, I have concerns regarding the novelty of the approach, as explained above. I have not tried to reproduce the results.
summary_of_the_review: See weaknesses above. I am giving a rating of 3 for now, but I can reconsider my rating based on the authors' rebuttal and discussions with the other reviewers.

*******************************

Review Update: Score changed to 8 after rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes Aggregation-Aware mixed-precision Quantization (A2Q) method to enable an adaptive learning of quantization parameters, which are innovatively linked to the topology of the graph, thus making more use of the graph information. A Local Gradient method is proposed to train the quantization parameters in semi- supervised learning tasks. For unseen nodes,  Nearest Neighbor Strategy to select quantization parameters. 
strength_and_weaknesses: Strength:

1. The idea of (in a more deterministic way) utilizing more of the topology information inside of the graph learning is novel, in my opinion. 

2. The experiments conducted are extensive.

3. Nice to see the evaluation using tailored hardware!

Weakness:

1. More clarification about the improvement on compression ratio and speed up. In the article, the compression ratio seems to represent the average bits used by the features as compared to FP32. It is better to show (1) more explicit definition of the metric and (2) the reduction in overall memory consumption of the whole inference/training process. For the speedup, it would be great for the authors to more clearly state which part is assumed to run on the accelerator. For instance, would the nearest neighbor part be executed on the accelerator? 

2. For NEAREST NEIGHBOR STRATEGY, it seems the FP32 copy of the features is necessary for "the feature with the largest absolute value fi in the node features is first selected, and then we find the nearest qmax"

3. The node-level task accuracy seems too low in Table 1? For instance, https://paperswithcode.com/sota/node-classification-on-cora 

4. The authors could provide more illustration on how their methods leverage the graph topology information, as it is a major part of the motivation.
clarity,_quality,_novelty_and_reproducibility: Overall the article is decently novel, but the effectiveness would also highly depend on the authors' further clarification.
summary_of_the_review: An article which is nicely motivated and target an unique angle of graph learning: how to more pointedly bake in the topology information to help the effectiveness and efficiency. However, a decent amount of clarification is also needed to further recommend this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn different quantization parameters for each node during GNN training, so as to reduce the quantization bit width and improve the accuracy. By taking the quantization error of each node as a part of the loss function, it solves the problem that many nodes have no gradient. By making statistics of the maximum value of each node feature on an unseen graph and matching the corresponding parameters, the migration between different graphs is realized without retraining.
strength_and_weaknesses: ==== Strength ====
* Quantizing the node features according to the topology of the graphs can retain higher algorithm performance is reasonable.

* By taking the quantization error of each node as a part of the loss function, the problem that many nodes have no gradient is solved. This is a new method to solve gradient sparsity in graph neural network transductive learning.

* Good to see the code is provided.

==== Weakness and Questions ====

I'm quite confused about Point 1 below. I'd like to see a thorough reply to it.

1. In which scenarios will the proposed method bring meaningful acceleration?

a) Most of the experiments in this paper are transductive learning tasks, which use the information of the whole graph (topology and input features) and labels of some nodes to learn and predict the labels of other nodes. I think for transductive GNN learning tasks, we only need to care about the cost of the training process, since when the training is completed, the original unlabeled nodes "have been" predicted, and there is no inference “after the training”. So, I don't quite understand why we need inference-time-quantization-aware training (QAT) here. The only case that may be useful is that after the training, the topology of the graph doesn’t change but the input features of nodes change, which requires re-prediction. But there are no such experiments. And also, it seems the paper doesn’t verify that there is no need to re-learn the quantization parameters in this case (that is, the quantization parameters only depend on the topology of the graph and are independent of the input features of the nodes of the graph).

b) For the inductive learning scenario, where the Nearest Neighbor Strategy proposed in this paper will be used, the model trained on one graph is used to predict the labels of nodes in other graphs. Because the topology information of other graphs cannot be obtained during training, the basic quantization algorithm proposed in this paper can no longer be used. And the proposed NNS strategy needs to perform floating point inference and determine quantization parameters for each node, to minimize the quantization error. I wonder if the float value of each node has been obtained in this process, the node label of this graph can already be predicted. Why do we need a quantization algorithm? 

2. The technical details are not clear enough.

a) The node-wise quantization scheme sets different quantization steps for each node. Then how to add the features of two nodes with different quantization steps during the aggregation? Do we need to operate with floating-point representation for all additions?

b) It's good to see Figure 2 shows how to perform matrix multiplication of features and weights in the node-wise quantization scheme. But the output y will be the input feature x of the next layer. So how can we transform y with different quantization parameters for each element into a format with different quantization parameters for each node? Also, this transformation is carried out online, and its cost should be counted as the cost of inference.

c) Other operations in GNN are not covered, such as the addition of self-feature and aggregation feature, and batch normalization or node degree normalization after aggregation. How are these operations quantized? For example, the normalization operation will have a greater impact on the data distribution after aggregation.

3. Learnable quantization parameters (bit-width, quantization step) are not new for CNN quantization, should refer to more papers on this.
clarity,_quality,_novelty_and_reproducibility: The clarity of technical details is not satisfying, should introduce the basic algorithm flow and also see the weakness section above.
The local gradient & NNS techniques are new.
The schemes of learnable quantization parameters and node-wise quantization are not new.
summary_of_the_review: My major concerns are listed in the weakness section 1/2, "I'm confused how this method could benefit actual scenario" and "unclear and important details". I hope the authors can explain those points.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Motivated by the values of node features that are highly related to the graph structure information such as in-degree, the authors propose and learnable quantization method, $A^{2}Q$, to learn the quantization step size and bitwidth for each node in the graph. Local gradient method is used to overcome the vanishing gradient problem for unlabeled nodes in the semi-supervised node level prediction and a Nearest Neighbor Strategy is used to deal with the generalization on unseen
for graph level prediction. Experiments show that $A^{2}Q$ achieves promising results with lower accuracy degradation compared to the previous SOTA on node-level and graph-level tasks.
strength_and_weaknesses: Strength:

+ The paper is well-written and easy to follow.

+ The proposed Aggregation-Aware Quantization, Local gradient method, and Nearest Neighbor Strategy techniques are well-motivated and novel.

+ Strong experimental results are shown on a variety of datasets on node-level and graph-level tasks.

Concerns and Questions:

- In Figure 1, the average feature of GAT is invariant to in-degree. Why could GAT benefit from Aggregation-Aware Quantization?

- The average feature depends on the chosen aggregation functions. It would be great to do an analysis and experiments with MPNN[1] or GraphSage [2] with different aggregation functions such as mean, max, and sum. I assume only the average feature of sum aggregation will be highly dependent on in-degree. The question is does GNNs with mean or max aggregations also benefit from Aggregation-Aware Quantization?

- The learnable node-wise quantization has significant computation overhead during the training phase. What is the runtime overhead compared to training a regular GNN?

- I am curious about the degradation of quantization for deep GNNs. The depths of GNNs are not mentioned in the experiment section. The quantization error would be accumulated with more layers. I wonder if this method could be used for deeper GNNs with skip connections.


[1] Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O. and Dahl, G.E., 2017, July. Neural message passing for quantum chemistry. In International conference on machine learning (pp. 1263-1272). PMLR.

[2] Hamilton, W., Ying, Z. and Leskovec, J., 2017. Inductive representation learning on large graphs. Advances in neural information processing systems, 30.

Reference:

A related work on GNN Quantization: VQ-GNN [3].

[3] VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization

============================================

 Post rebuttal:
Thanks for the authors' detailed rebuttal and added experiments. I keep my rating unchanged.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The proposed methods are well-motivated and novel. Given thorough details, the experiments should be easy to reproduce. 
summary_of_the_review: I would like to recommend a score of 7 for this paper. But there are not 7. So I recommend 8 for now. I will adjust my score depending on the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Quantization-Aware Training (QAT) method, which utilizes the GNN characteristics, e.g.,  topology and nodes attributes. 
The proposed method, called  Aggregation-Aware mixed-precision Quantization (A2Q), is automatically learned and assigned
to each node in the graph different bitwidth (number of bits), allowing heterogeneous allocation of different nodes embedding. Furthermore, to mitigate the vanishing gradient problem caused by sparse connections between nodes, the authors propose a local gradient strategy for exploiting the quantization error of the node features as the supervision during the training procedure. In addition, the authors introduce the Nearest Neighbor Strategy for choosing the appropriate quantization settings for unseen graph nodes. 
Extensive experiments on eight public node-level and graph-level datasets demonstrate the generality and robustness of our proposed method significantly outperform previous art GNN quantization schemes in terms of cpmp[ression ratio, speedup, and model accuracy. 


strength_and_weaknesses: *Strengths*

The paper is well-written, and the method is well-explained.  
The evaluation of the proposed method for quantizing graph-level and node-level tasks shows that the proposed method outperforms previous work in terms of compression ratio, computation complexity, and task performance. 
The proposed method can works also in a supervised/semi-supervised setting, which allows using the proposed procedure for large-scale real-world graphs. In addition, the authors propose a dedicated GNN accelerator architecture to evaluate the compression ratio and the speedup of the quantization. 

*Weaknesses*

Although authors provided evaluations on various graph datasets, all of them are homophilous and transductive. Please elaborate if the proposed method is scaled to non-homophilous or inductive graphs. Supporting experiments would be essential.  
The ablation study of how using a different number of GNN layers impacts the quantization performance is not appearing in the current version of the manuscript. I suggest performing this ablation study during the revision stage.
clarity,_quality,_novelty_and_reproducibility: The proposed method is well-explained, and all the implementations detail, including source code, are included. 
While some part of the proposed method is based on the previous QAT method, For example, the learning of the step size (section 3.1) was previously proposed by [1]. The entire method, including the non-homogenous bit allocation (different bitwidth for each node)  and prediction of the quantization parameters for unseen nodes, is novel. 

[1] https://openreview.net/pdf?id=rkgO66VKDS

summary_of_the_review: This paper proposes Quantization-Aware Training (QAT) method, which utilizes the GNN characteristics, e.g.,  topology and nodes attributes. 
The paper is well-written, and the method is well-explained.  
The evaluation of the proposed method for quantizing graph-level and node-level tasks shows that the proposed method outperforms previous works in terms of compression ratio, computation complexity, and task performance.
While some part of the proposed method is based on the previous QAT method, For example, the learning of the step size (section 3.1) was previously proposed by [1]. However, the entire method is novel, including the non-homogenous bit allocation (different bitwidth for each node)  and prediction of the quantization parameters for unseen nodes. 

While overall, I support paper acceptance, there are a few issues that I have pointed out in the weakness section, which I would ask the authors to address during the revision stage. 


[1] https://openreview.net/pdf?id=rkgO66VKDS

*****Post-rebuttal review*****
I appreciate the authors for providing a detailed and comprehensive rebuttal and fully responding to my concerns and other reviewers (in my opinion). Hence, I do not doubt that the revised manuscript will contribute to our community, and I would be very happy to see it accepted. My score is also increased to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers an important problem in the graph community, scalability during inference/deployment time. The authors propose an Aggregation-Aware mixed-precision Quantization, which quantizes different nodes features with different learnable quantization parameters, including bit-width and step-size. To mitigate the vanishing gradient problem caused by sparse connections between nodes, the authors propose a Local Gradient method to serve the quantization error of the node features as the supervision during training. The paper also proposes a Nearest Neighbor Strategy to deal with the generalization on unseen graphs.

The reviewers liked the idea of utilizing more of the topology information inside of the graph learning. However, reviewers also expressed concerns about the limited novelty of this paper, although the reviewers convinced the committee that there is vanishing gradient problem cause by sparse connections between nodes which is addressed by the proposed local gradient method. However, the justification of this local method is not clearly illustrated. 
 

justification_for_why_not_higher_score: Reviewers also expressed concerns about the limited novelty of this paper, although the reviewers convinced the committee that there is vanishing gradient problem cause by sparse connections between nodes which is addressed by the proposed local gradient method. However, the justification of this local method is not clearly illustrated. 
justification_for_why_not_lower_score: The reviewers liked the idea of utilizing more of the topology information inside of the graph learning. And the problem considered is timely.
Decision: Accept: poster