Degree-Quant: Quantization-Aware Training for Graph Neural Networks
Authors: Shyam Anil Tailor
Javier Fernandez-Marques
Nicholas Donald Lane
Venue: ICLR 2021 Poster (2021)
Keywords: Graph neural networks
quantization
benchmark

Abstract: Graph neural networks (GNNs) have demonstrated strong performance on a wide variety of tasks due to their ability to model non-uniform structured data. Despite their promise, there exists little research exploring methods to make them more efficient at inference time. In this work, we explore the viability of training quantized GNNs, enabling the usage of low precision integer arithmetic during inference. For GNNs seemingly unimportant choices in quantization implementation cause dramatic changes in performance. We identify the sources of error that uniquely arise when attempting to quantize GNNs, and propose an architecturally-agnostic and stable method, Degree-Quant, to improve performance over existing quantization-aware training baselines commonly used on other architectures, such as CNNs. We validate our method on six datasets and show, unlike previous quantization attempts, that models generalize to unseen graphs. Models trained with Degree-Quant for INT8 quantization perform as well as FP32 models in most cases; for INT4 models, we obtain up to 26% gains over the baselines. Our work enables up to 4.7x speedups on CPU when using INT8 arithmetic.

Published as a conference paper at ICLR 2021
DEGREE-QUANT QUANTIZATION-AWAR TRAINING FOR GRAPH NEURAL NETWORKS
Shyam A. Tailor
Javier Fernandez- Department of Computer Science University of Oxford
Department of Computer Science & Technology University of Cambridge
Nicholas D. Lane
Department of Computer Science and Technology University of Cambridge & Samsung AI Center
ABSTRACT
Graph neural networks (GNNs) have demonstrated strong performance on a wide variety of tasks due to their ability to model non-uniform structured data. Despite their promise, there exists little research exploring methods to make them more effi- cient at inference time. this work, we explore the viability training quantized GNNs, enabling the usage of low precision integer arithmetic during inference. For GNNs scemingly unimportant choices in quantization implementation cause dra- matic changes in performance. We identify the sources of error that uniquely arise when attempting to quantize GNNs, and propose an stable method, Degree-Quant, improve performance over existing quantization- aware training baselines commonly used on other architectures, such as CNNs. We validate our method on six datasets and show, unlike previous attempts, that models generalize to unseen graphs. Models trained with Degree- Quant for INT8 quantization perform as well as FP32 models in most cases; for INT4 models, we obtain up to 26% gains over the baselines. Our work enables up to 4.7x x speedups on CPU when using INT8 arithmetic.
and
INTRODUCTION
GNNs have received substantial attention in recent years due to their ability to model irregularly structured data. As result, they are extensively used for applications as diverse as molecular interactions (Duvenaud et al.) 2015 Wu et al.) 2017), social networks (Hamilton et al.) 2017). recommendation systems (van den Berg et al. 2017) or program understanding (Allamanis et al.. 2018). Recent advancements have centered around building more sophisticated models including new types of layers (Kipf & Welling 2017 Velickovic et al. 2018 Xu et al.) 2019) and better aggregation functions (Corso al. (020). However, despite GNNs having few model parameters, the compute required for each application remains tightly coupled to the input graph size. A 2-layer Graph Convolutional Network (GCN) model with 32 hidden units would result in model size of just 8IKB but requires 19 GigaOPs to process the entire Reddit graph. We illustrate this growth in fig.
One major challenge with graph architectures is therefore performing inference efficiently, which limits the applications they can be deployed for. For example, GNNs have been combined with CNNs for SLAM feature matching (Sarlin al. 2019), however it trivial to deploy this technique on smartphones, or even smaller devices, whose neural network accelerators often do not implement floating point arithmetic, and instead favour more efficient integer arithmetic. Integer quantization one way to lower the compute, memory and energy budget required to perform inference, without requiring modifications to the model architecture; this is also useful for model serving in data centers.
Although quantization has been well studied for CNNs and language models (Jacob et al. 2017] Wang et al. 2018 Zafrir et al. 2019 Prato et al. 2019) there remains relatively little work addressing
"Equal contribution. Correspondence to: Shyam Tailor <sat62 cam..acmcmmmmm cak.

---

Published as a conference paper at ICLR 2021
10a3
Layer
Layer /
0.13MB
0.0 0.2.3MB
1012
1.14MB
2-GAT
CNN
10
GCN
4 GCN
WideRcsNet101 (230MB)
h h² K
1020
ResNett1" (44MB)
10
MobileNetVE (13M9)
GNN
h
10a
single 3x224X224 image
Cora
Citeseer Pubmed
Reddit Amazon
h Yhh,, AuE lo(h', h euv)))
Figure 1: Despite GNN model sizes rarely exceeding IMB, the OPs needed for inference grows at least linearly with the size of the datasct and node features. GNNs with models sizes x00x smaller than popular CNNs require many OPs process large graphs.
Figure 2: While CNNs operate on regular grids, GNNs operate graphs with varying topology. A node's neighborhood size and ordering varies for GNNs. Both architectures use weight sharing.
GNN efficiency (Mukkara et al. 2018 Jia et al., 2020; Zeng & Prasanna 2020: Yan et al. 2020). To the best of our knowledge, there is no work explicitly characterising the issues that arise when quantizing GNNs or showing latency benefits of using low-precision arithmetic. The recent work of Wang et al. (2020) explores only binarized embeddings of single graph type (citation networks). In|Feng et al. (2020) a heterogeneous quantization framework assigns different bits to embedding and attention coefficients in each layer while maintaining the weights at full precision (FP32). Due to the mismatch in operands' bit-width the majority of the operations are performed at FP32 after data casting, making it impractical to use general purpose hardware such as CPUs or GPUs. In addition they do not demonstrate how to train networks which generalize unseen input graphs. Our framework relies upon uniform quantization applied to all elements in the network and uses bit-widths (8-bit and 4-bit) that supported by off-the-shelf hardware such CPUs and GPU for which efficient low-level operators for common operations found in GNNs exists.
This work considers the motivations and problems associated with quantization of graph architectures. and provides the following contributions:
The explanation of the sources of degradation in GNNs when using lower precision arith- metic. We show how the choice of srnight-thrrughh estimator (STE) implementation, node degree, and method for tracking quantization statistics significantly impacts performance.
An architecture-agnosse (DQ), which results in INT8 models often performing as well as their FP32 counterparts. At INT4, models trained with DQ typically outperform quantized baselines by over 20%. We show, unlike previous work, that models trained with DQ generalize to unseen graphs. We provide code this URL: https:::. : /tt://gitttt thub com caml sys hees//dgreegggeeee quantt
method for quantization-aware training graphs, Degree-Quant
We show that quantized networks achieve up to 4.7 x speedups CPU with INT8 arithmetic, relative to full precision floating point, with 4-8 x reductions in runtime memory usage.
BACKGROUND
2.1 MESSAGE PASSING NEURAL NETWORKS (MPNNs)
Many popular GNN architectures may be viewed generalizations of CNN architectures to irregular domain: at a high level, graph architectures attempt to build representations based on a node's neighhorhood (see 2). Unlike CNNs, however, this neighborhood does not have fixed ordering or size. This work considers GNN architectures conforming to the MPNN paradigm (Gilmer et al., 2017). graph G (V, E) has node features X E NxF, an incidence matrix E N2x and optionally D- dimensional edge features E E REXD The forward pass through an MPNN layer consists of message passing, aggregation and update phases: (.0)) (h) (AVeN() (6(())
()) h,9 ee)))) Messages from
2

---

Published as a conference paper at ICLR 2021
node node are calculated using function 6, and aggregated using permutation inniataiaiiiininiiinnnnn invariant function A. The features at e are subsequently updated using 7.
We focus three architectures with corresponding update rules:
1. Graph Convolution Network (GCN): h ( ΣSEN(OUU Vaid; Wh())
(iipf & Welling
2017), where d, refers to the degree of node
2. Graph Attention Network (GAT): h(t) = O Wh.
ESENCOJ(a Wh where a
represent attention coefficients (Velickovic et al. 2018).
3. Graph Isomorphism Network (GIN): h fe (1 )))
+ +EEEN
where s is h
learnable function (e.g. MLP) and is learnable constant (Xu et al. 2019
2.2 QUANTIZATION FOR NON-GRAPH NEURAL NETWORKS
Quantization allows for model size reduction and inference speedup without changing the model architecture. While there exists extensive studies of the impact of quantization at different bit- widths (Courbariaux et al. 2015 Han et al. 2015 Louizos al. 2017) and data formats (Micikevicius et al. Carmichael et al. 2018: Kalamkar et al. 2019), it is 8-bit integer (INT8) quantization that has attracted the most attention. This is due INT8 models reaching comparable accuracy levels to FP32 models (Krishnamorrhh]] [2018] Jacob et al.) 2017). offer 4x model compression, and result in inference speedups on off fff-the-shelf hardware as 8-bit arithmetic is widely supported.
Quantization-awar training (QAT) has become the de facto approach towards designing robust quantized models with low error (Wang et al. 2018 Zafrir et al. 2019; Wang et al. (0188). In their simplest forms, QAT schemes involve exposing the numerical errors introduced by quantization by simulating it on the forward pass Jacob et al. (2017) and make use of STE (Bengio al. 2013) to compute the gradients- as if no quantization had been applied. For integer QAT, the quantization of a tensor x during the forward pass is often implemented as: Ig min (qmax. max 9min /x/s =))), where Qmin and fmax the minimum and maximum representable values at given bit-width and signedness, is the scaling factor making x span the [Amin; (max range and, is the zero-point, which allows for the real value 0 to be representable in zg. Both and are scalars obtained at training time. hen, the tensor is dequantized as: f = (xt, z)s, where the resulting tensor f ~ x for a high enough bit- width. This similarity degrades lower bit-widths. Other variants of integer QAT are presented in Jacob et al. (2017) and Krishnamoorthi (2018).
max(quin-
To reach performance comparable FP32 models, QAT schemes often rely on other techniques such as gradient clipping, to mask gradient updates based on the largest representable value at given bit-width; stochastic, or noisy, QAT, which stochastically applies QAT to a portion of the weights at each training step (Fan et al. 2020) Dong et al. 2017); or the re-ordering of layers (Sheng et al.)
2018 Alizadeh et al. 2019).
3 QUANTIZATIONN FOR GNNS
In this section, we build intuition for why GNNs would fail with low precision arithmetic by identifying the sources of error that will disproportionately affect the accuracy of a low precision model. Using this insight, we propose our technique for QAT with GNNs, Degree- Quaee. Our analysis focuses on three models: GCN, GAT and GIN. This choice was made as we believe that these are among the most popular graph architectures, with strong performance on variety of tasks (Dwivedi et al. 2020), while also being representative of different trends in the literature.
3.1 SOURCES OF ERROR
QAT relies upon the STE to make an estimate of the gradient despite the non-differentiable rounding operation in the forward pass. If this approximation is inaccurate, however, then poor performance will be obtained. In GNN layers, we identify the aggregation phase, where nodes combine messages from a varying number of neighbors in a cermutation-invariaa fashion, as a source of substantial numerical error, especially nodes with high in-degree. Outputs from aggregation have magnitudes
3

---

Published as a conference paper at ICLR 2021
(Fggggggggattioo Output)
F Variance
Mean
12.5
GAT
GCN
2.5
7.5
GIN
0.0
5.0
-25
2.5
-5.0
-7.5
Figure Analysis values collected immediately after aggregation at final layer FP32 GNNs trained on Cora. Generated using channel data collected from 100 runs for each architecture. As in-degree grows, so does the mean and variance of channel values after aggregation.
that vary significantly depending node's 's in-degree: as it increases, the variance of aggregation values will increase Over the course of training Jmin and Gmaxx the quantization range statisties, become severely distorted by infrequent outliers, reducing the resolution for the vast majority of values observed. This reults in increased rounding error for nodes with smaller in-degrees. Controlling Amin and 4max hence becomes a trade-off balancing truncation error and rounding error.
We can derive how the mean and variance of the aggregation output values vary as node in-degree, n, increases for each of the three GNN layers. Suppose we model incoming message values for single output dimension with random variables X,, without making assumptions on their exact distribution or independence. Further, we use Y, the random variable representing the value of node output after the aggregation step. With GIN layers, we have Y, (1 + x)Xo K + >i X;. It is trivial to prove that (E,)),)) = O()). The variance of the aggregation output also O(n) the case that that Eijy Cov(X,, X₁) < E; Var(X;). We note that if Σi Cov(X, Xx) is large then it implies that the network has learned highly redundant features, and may be sign of over-fitting. Similar arguments can be made for GCN and GAT layers; we would expect GCN aggregation values to grow like OV) and GAT aggregation values remain constant (1(1)) due to the attention coefficients.
We empirically validate these predictions on GNNs trained on Cora; results are plotted in fig. We that the aggregation values do follow the trends predicted, and that for the values of in-degree in the plot (up to 168) the covariance terms can be neglected. As expected, the variance and mean the aggregated output grow fastest for GIN, and are roughly constant for GAT as in-degree increases. From this empirical evidence, it would be expected that GIN layers are most affected by quantization.
By using GIN and GCN examples, updates. Suppose we consider GIN layer incorporating one weight matrix in the update function (i)
see how aggregation error causes error in weight
iee h /(Wy(l)) GIN), where is activation function, YGIN (1 +)h? (2) + EjEN(() hf"", and (i) denotes the in-neighbors of node i. Writing y YCN EkEN(i) Vd,da Wh)"), we see that the
(i)
derivatives of the loss with respect to the weights for GCN and GIN are:
GIN
GCN
M
ac ow
OC Oh!
('
( YON
Oc
OC @
I ('(y(ew))
aw
ddd
F= (E)(((
(6)
The larger the error in y GIN caused by aggregation error- the greater the emrrr in the weight gradients for GIN, which results in poorly performing models being obtained. The same argument
applies GCN, with the h())" and (5)
yee terms introducing aggregation error into the weight updates.
3.2 OUR METHOD: DEGREE- QUANT
To address these sources of error we propose Degree-Quan (DQ), a method for QAT with GNNs. We consider both inaccurate weight updates and unrepresentative quantization ranges.
'The reader should note that are not referring to the concept of estimator variance, which is the subject of sampling hased approaches-aaeee we are exclusively discussing the variance of values immediately after aggregation.
4

---

Published as a conference paper at ICLR 2021
Protect Quantize
Aggregate Update
Figure 4: High-level view of the stochastic element of Degree- Quaett. Protected (high in-degree) nodes, in blue, operate at full precision, while unprotected nodes (red) operate reduced precision. High in-degree nodes contribute most to poor gradient estimates, hence they stochastically protected from quantization more often.
Algorithm Degree-Quald (DQ). Functions accepting a protective mask perform only the masked computa- tions at full precision: intermediate tensors are not quantized. At test time protective masking is disabled. In fig. [1 (in the Appendix) we show with a diagram how a GCN layers makes use of DQ
p) 2: D Calculate mask and quantized weights, O', which all operations share
I: procedure TRAINFORWARDPASSS
3:
In <- BERNOULLI(P)
4:
O' +- QUANTIZE(
Messages with masked sources are at full precision (excluding weights)
6:
.M +- -MESSAGECALLULATEE O', m)
7:
X < QUANTIZEAAGGEEENTEEEM O', m), m)
b No quantization for masked nodes Quantized weights always used
8:
return UPDATE(X, 0".
9:
end procedure
Stochastic Protection from Quantization to Improve Weight Update Accuracy. DQ aims encourage more accurate weight updates by stochastically protecting nodes in the network from quantization. each layer protective node mask is generated; all masked nodes have the phases of the message passing, aggregation and update performed at full precision. This includes messages sent by protected nodes to other nodes, shown in fig. (a detailed diagram is shown in fig. TT It also important to note that the weights used at all nodes are the same quantized weights; this is motivated by the fact that our method is used encourage more accurate gradients to flow back the weights through high in-degree nodes. At test time protection is disabled: all nodes operate at low precision.
To generate the mask, we pre-process each graph before training and create vector of probabilities with length equal to the number of nodes. At training time, mask m is generated by sampling using the Bernoulli distribution: m ~ Bernoull(p).. is large, as we find empirically that high in-degree nodes contribute most towards error in weight updates. We use a scheme with two hyperparameters, Pmin and Pmaxi nodes with the maximum in-degree assigned Pmaxx their masking probability, with all other nodes assigned a probability calculated by interpolating between Pmin and Pmax based on their in-degree ranking in the graph.
our scheme p: is higher if the in-degree of node
Percentile Tracking of Quantization Ranges. Figure demonstrates large fluctuations in the variance the aggregation output as in-degree increases. Since these can disproportionnatll affect the ranges found by using min-max or coomentum-baset quantization, we propose using percentiles. While percentiles have been used for post-training quantization (Wu et al. 2020), we are the first (to the best of knowledge) to propose making it core part of QAT; we find to be key contributor to achieving consistent results with graphs. Using percentiles involves ordering the values in the tensor and clipping a fraction of the values at both ends of the distribution. The fraction to clip is a hyperparameter. We are more aggressive than existing literature on the quantity we discard: we clip the top and bottom 0. %, rather than 0.01 %, as we observe the fluctuations to be larger issue with GNNs than with CNNs or DNNs. Quantization ranges are more representative of the vast majority of values this scheme, resulting in less rounding error.
We emphasize that core contribution of DQ is that it is architecture-agnosse Our method enables wide variety of architectures to use low precision arithmetic at inference time. Our method is also or- thogonal-and and complementary- such as sampling based methods which are used to reduce memory consumption (Zeng et al. 2020). weight pruning (Blalock al. 2020) approaches to achieve further model compression.
other techniques for decreasing GNN computation requirements,
5

---

Published as a conference paper at ICLR 2021
vanillo STE
STE: with Gindiont Clipping
Dataset
Model Arch.
min/max
momestum
minimax
momennum
W8AS
W4,4
W8A8
W444
WSAS
W4A4
WBAS
W4A4
GCN
$1.0 0.7
65.3 4.9
42.3 11.1
19.44.8.5
50.8 4: 0.8
62.3
66.9 18.2
77.2
(Acc: 1
GAT
GIN
69.9 1.9
25.9
19.2 10.2
81.7 1.
42.8 4.0
69.2 1: 2.3
76.4 16.41.2.6
3.5
15.4
75.1 = 1.1
81.9 0.7
47.41...0
5.0
MNIST
(Acc: 1
GCN
GAT
GIN
90.4 0.2
95.8 0.1
62.4
7.
20.1
96.7
90.1
96.7 0 0.3
70.6 2.4
91.0
67.4 4:2.
90.1 0.3
95.1 +:
96.4
30,2 7.4
5188
19.5
75.3
95.7 2.3
90.2 0.4
76.3
10.3
ZINC (Lo88)
GCN
GAT
0.480
0.071
1: 0.01
1: 0.01
0.740
0.747
0.02
0.02
0.571
0.509 0.01
1: 0.03
0.692 + 0.06
0.710 0.05
0. 1: 0.01
0.495 1. 0.01
0.739 = 0.04
0.706 =: 0.02
0.483 4: 0.01
0.463
#: 0.01
0.692 +: 0.01
0.717 1 0.03
GIN
0.395 1: 0.02
1,206
0:27
0.386 + 0.05
0.572 + 0.02
0.300 4: 0.02
1,669 = 0.30
0.388 :: 0.02
0.973 1 0.24
Table Impact on performance of four typical quantization implementations for INT8 and INT4. The configuration that resulted in best performing models for each dataset-model pair is bolded. Hyperparameters for each experiment were fine-tuned independently. As expected, adding clipping does not change performance with min/max but does with momentum. A major contribution of this work is identifying that seemingly unimportant choices in quantization implementation cause dramatic changes performance.
4 EXPERIMENTS
In this section we first analyse how the choice of quantization implementation affects performance of GNNs. We subsequently evaluate Degree- Begree-Quant against the strong baselines of: FP32, INT8-QAY and, INT8-QAT with stochastic masking of weights (Fan et al. (020). We refer to this last approach as noisy QAT nQAT. To make explicit that are quantizing both weights and activations, we the notation W8A8, We repeat the experiments at INT4. Our study evaluates performance on six datasets and includes both node-level and graph-level tasks. The datasets used were Cora, CiteSeer, ZINC, MNIST and CIFAR10 superpixels, and REDDIT-BINARY. Across all datasets INT8 models trained with Degree-Quan manage to recover most of the accuracy lost as result of quantization. In some instances. DQ-INT8 outperform the extensively tuned FP32 baselines. For INT4, DQ outperforms all QAT baselines and results in double digits improvements over QAT-INT4 in some settings. Details about each dataset and our experimental setup can be found in appendix A.
4.1 IMPACT OF QUANTIZATION GRADIENT ESTIMATOR ON CONVERGENCE
The STE is a workaround for when the forward pass contains con-differentiable operations (e.g. round- ing in QAT) that has been widely adopted in practice. While the choice of STE implementation generally results in marginal differences for CNNs even for binary networks (Alizadeh et al. 2019) unclear whether only marginal differences will also be observed for GNNs. Motivated by this, we study the impact of four off- off-the-shelve quantization procedures on the three architectures evaluated for each type of dataset; the implementation details of each one is described in appendix[A.3 We perform this experiment to ensure that we have the strongest possible QAT baselines. Results are shown in table [ We found the choice quantization implementation to be highly dependent on the model architecture and type of problem to be solved: we see a much larger variance than is observed with CNNs; this is an important discovery for future work building on our study.
We observe general trend in all INT4 experiments benefiting from momentum as it helps smoothing out the quantization statistics for the inherently noisy training stage at low bitwidths. This trend applies as well for the majority of INT8 experiments, while exhibiting little impact on MNIST. For INT8 Cora-GCN large gradient norm values the early stages of training (see fig. (5) mean that these models not benefit from momentum as quantization ranges fail to keep up with the rate of changes in tensor values; higher momentum can help but also leads to instability. In contrast, GAT has stable initial training dynamics, and hence obtains better results with momentum. For the molecules dataset ZINC, we consistently obtained lower regression loss when using momentum. We note that GIN models often suffer from higher performance degradation (as was first noted in fig. 3 specially at W4A4. This is not the case however for image datasets using superpixels. We believe that datasets with Gaussian- like node degree distributions (see fig. (9) are more tolerant of the imprecision introduced by quantization, compared to datasets with tailed distributions. We leave more in-depth analysis of how graph topology affects quantization as future work.

---

Published as a conference paper at ICLR 2021
Quant
Model Arch.
Node Classification (Accuracy
Gruph Classiscation (Accuracy
Gruph Regression (Loss)
Scheme
Cora
Citeseer
MNIST 1
CIFAR-10
ZINC
GCN
81.4
+ 0.7
+ 0.7
90.0
54.5 +
0.469 0.002
Ref.
GAT
83.1
+ 0.4
72.5 = 0.7
95.6 0.1
65.4 + 0.4
0.463 + 0.002
(FP32)
GIN
77.6 + 1.1
66.1 = 0.9
93.9 0.6
53.3 : 3.7
0.111 = 0.009
GCN
81.2
1: 0.6
71.4 +: 0.9
90.9 0.4
58.4
0.450 0.008
Ours
GAT
83.2
1 0.3
72.4 1 0.8
95.8 1. 0.4
65.1 1. 0.8
0.455 1. 0.006
(FP32)
GIN
77.9 +
65.8 + 1.5
96.4 0.4
57.4 0.7
0.334 + 0.024
GON
81.0 0.7
71.3 = 1.0
90.9 1 0.2
66.4 0.5
0.481 + 0.029
QAT
GAT
$1.9
71.2 =
95.8
66.3
0.460 0.005
(W8A8)
96.7
0.386
GCN
$1.0 +
70.7 = 0.8
91.1 0.1
56.2 0.5
0.472 0.015
IQAT
0.459
(W8A8)
GIN
77.4 1 1.3
65.1
96.4 0.3
52.7 1.4
0.405 0.016
81.7
(40.7)
71.0 0.9 (-0.8)
90.9 (-0..2
56.3 0.1 (-0.1)
+ 0.009 (-9.8)
DQ
71.6
95.8 0.4
67.7
0.456 0.005 (-0.9)
(W8A8)
GIN
78.7
(+1.3)
67.5 (+2.4)
96.6 (-0.1)
0.6 (-2.8)
0.357 + (-7.5)
GCN
77.2 2.5
61.1 = 4.1
70.6
1.6
0.692 + 0.013
GAT
55.6 5.4
65.3 + 1.9
76.3 + 1.2
41.0 1.1
0.655 + 0.032
(W4A4)
GIN
42.5 1 4.5
18.6 2.9
91.0 0.6
45.6 1 3.6
0.572 1 0.02
GCN
1.5
65.8 2.6
70.9 1.5
40.1 + 0.7
0.669 + 0.128
RQAT
GAT
54.9 5.6
65.5 + 1:7
78.4 1.5
41.0
0.637 + 0.012
(W4A4)
45.0
5.0
34.6 3.8
91.3 0.5
48.7
0 0.561 0.068
GCN
78.3 = 1.7 (+0.2)
06.9 (.4+111))
84.4 1.3
51.1 0.7
0.536 = 0.011 (+26.2)
IQQ
GAT
71.2 1: (.916.33)
67.6
1.5
93.1
(-3
56.5
1: 0.6 (+15.)
0.520
0.021 (+20.6)
(W4A4)
(+2449)
(-26.2)
0.4 (+4.2)
(-6
(+23.2)
Table 2: This table is divided into three sets of rows with FP32 baselines the top. provide two baselines for INTS and INT4: standard QAT and stochastic QAT (nQAT). Both are informed by the analysis in with nQAT achieving better performance in some cases. Models trained with Degree-Quant (DQ) are always comparable to baselines, and usually substantially better, especially for INT4. DQ is stable method which requires little tuning obtain excellent results across variety of architectures and datasets.
4.2 OBTAINING QUANTIZATION BASELINES
Our FP32 results, which we obtain after extensive hyperparameter tuning. and those from the baselines are shown at the top of table 2 We observed large gains on MNIST, CIFAR 10 and, ZINC.
For our QAT-INTS and QAT-INT4 baselines, use the quantization configurations informed by our analysis in section 4.1 For Citeseer we use the best resulting setup analysed for Cora, and for CIFAR- 10 that from MNIST. Then, the hyperparameters for each experiment were fine tuned individually. including noise rate E [0.5, 0.95) for nQAT experiments. QAT-INTS and QAT-INT4 results in ann QAT-INT4, with the exception of MNIST (an easy classify dataset), corroborate our hypothesis that GIN layers are less resilient to quantization. This was first observed in fig. In the case of ZINC, while all models results noticeable degradation, GIN sees more severe 16% increase of regression loss compared to our FP32 baseline. For QAT W4A4 accuracy drop of over 35% and 47% observed for Cora and Citeseer respectively. The stochasticity induced by nQAT helped in recovering some of the accuracy lost result of quantization for citation networks (both INT8 and INT4) but had little impact on other datasets and harmed performance in some cases.
4.3 COMPARISONS OF DEGREF- -UGREE-QUANT WITH EXISTING QUANTIZATION APPROACHES
Degree- Quaett provides superior quantization for all GNN datasets and architectures. Our results with DQ are highlighted in gray in table and table Citation networks trained with DQ for W8A8 manage to recover most of the accuracy lost as a result QAT and outperform most of nQAT baselines. In some instances DQ-W8AR models outperform the reference FP32 baselines. At 4-bits, DQ results in even larger gains compared to W4A4 baselines. We see DQ being more effective for GIN layers, outperforming INT4 baselines for Cora (+24.9%), Citeseer (+26.2% and REDDIT- BINARY (+23.0%) by large margins. Models trained with DQ at W4A4 for graph classification and graph regression also exhibit large performance gains (of over 10%) in most cases. For ZINC, all
7

---

Published as a conference paper at ICLR 2021
Quantization
Model
REDDIT-BIN 96)
Zinc (Batch- 10K)
Reddit WKA8
Arch.
Ref. (FP32)
GIN
92.2 = 2.3
FP33
W8A8
Speedup
HP32
Speedup
Ours (FP32)
GIN
92.0 + 1.5
GCN
42ms
13.18
4.2x
QAT-W8A8
GIN
76.1 7.5
CPU
GAT
190ms
50ms
3.8x
13.1s
2.8s
4.7x
W8AS
GIN
77.5 3.4
GIN
182ms
43ms
4.2x
13.18
3.18
4.2%
DQ-W8A8
GIN
91.8 2.3 (+14.33
GCN
39ms
31ms
1.3x
191ms
176ms
1.1x
QAT-W4A4
54.4
GPU
GAT
17ms
15ms
L1x
00M
mQAI-W444
58.0
GIN
31ms
DQ-W4A4
81.3 + 4.4 (+23.0)
Table INT8 latency results rum on 22 core 2.1GHz Intel Xeon Gold 6152 and, on GTX 1080Ti GPU. Quantization provides large speedups variety of graphs for CPU and non-negligihle speedups with unoptimized INT8 GPU kernels.
Table 3: Results for DQ-INT8 GIN models perform nearly as well as at FP32. INT4, DQ offers significant increase in accuracy.
models achieve over 20% lower regression loss. Among the top performmn models using DQ, ratios of Pmin and Pmax in [0.0, 0.2] were the most common. Figure 10 in the appendix shows validation loss curves for GIN models trained using different DQ probabilities on the REDDIT-BINARY dataset.
5 DISCUSSION
Latency and Memory Implications. In addition to offering significantly lower memory usage (4x with INT8), quantization can reduce latency-especially on CPUs. We found that with INT8 arithmetic we could accelerate inference by up to x.7x. We note that the latency benefit depends on the graph topology and feature dimension, therefore we ran benchmarks on variety of graph datasets, including Reddi Zinc, Cora, Citeseer, and CIFAR- 10; Zinc and Reddit results shown in table 4 with further results given in the appendix. For a GCN layer with in- and out-dimension of 128, get speed-ups of: 4.3x on Reddit, 2.5x Zinc, 1.3x Cora, 1.3x Citeseer and, 2.1x x on CIFAR-10. It also worth emphasizing that quantized networks are necessary to efficiently use accelerators deployed in smartphones and smaller devices as they primarily accelerate integer arithmetic, and that CPUs remain a common choice for model serving on servers. The decrease in latency CPUs is due to improved cache performance for the sparse operations; GPUs, however, see less benefit due to their massively -araallll nature which relies on mechanisms other than caching hide slow random memory accesses, which are unavoidable this application.
Quantzed Max Range
(Second Layer (mpuls)
0.20
0.30
0.25
Gradent Nomm
[Second Layer Weights)
Test Aouuracy
0.80
Weights
Features Atwntion W
Posi-Apg--piaen spation
Messages
Aipha (pro-aofira))
Poes-ftte
Percentie
3-
0.15
Absolute Max
2-
0.10
0.05
0.70
0.00
100
Epoch
Test Loss
Figure 5: Qmax with absolute min/max and percentile ranges, applied to INT8 GCN training on Cora. We ob- serve that the percentile max is half that of the absolute, doubling resolution for the majority of values.
Figure 6: Analysis of how INT8 GAT performance degrades Cora individual elements reduced 4-bit precision without DQ. For GAT message elements are crucial to classification performance.
Ablation Study: Benefits of Percentile Ranges. Figure 5 shows the value of percentiles during training. We see that when using absolute min/max the upper range grows to over double the range required for 99.9% of values, effectively halving the resolution of the quantized values. DQ is more stable, and we obtained strong results with an order of magnitude less tuning relative to the baselines.
Ablation Study: Source of Degradation at INT4. alssesses how INT8 GAT (without DQ) degrades as single elements are converted to INT4, in order to understand the precipitous drop in
"The largest graph commonly benchmarked in the GNN literature
8

---

Published as a conference paper at ICLR 2021
accuracy in the INT4 baselines; further plots for GCN and GIN are included in the appendix. We observe that most elements cause only modest performance losses relative to a full INT8 model. DQ is most important to apply to elements which are constrained by numerical precision, such as the aggregation and message elements in GAT. Weight elements, however, are consistently unaffected.
Ablation Study: Effect of Stochastic Element in Degree-Quant. We observe that the stochastic protective masking in DQ alone often achieves most of the performance gain over the QAT baseline; results are given in table|jin the appendix. The benefit of the percentile-based quantization ranges stability, although it can yield some performance gains. The full DQ method provides consistently good results on all architectures and datasets, without requiring an extensive analysis as in 4.1
CONCLUSION
This work has presented Degree-Quant, an architecture-agnosiic and stable method for training quantized GNN models that can be accelerated using off-the-shelf hardware. With 4-bit weights and activations we achieve 8x compression while surpassing strong baselines by margins regularly exceeding 20%. At 8-bits, models trained with DQ perform on par better than the baselines while achieving up to 4.7% x lower latency than FP32 models. Our work offers a comprehensive foundation for future work in this area and is first step towards enabling GNNs to be deployed more widely, including to resource constrained devices such as smartphones.
ACKNOWLEDGEMENT
This work was supported by Samsung AI and by the UK's Engineering and Physical Sciences Research Council (EPSRC) with grants EP/M50659X/1 and EP/S001530/1 (the MOA project) and the European Research Council via the REDIAL project.
REFERENCES
R. Achanta, A. Shaji. Smith, A. Lucchi, Fua, and Süsstrunk. Slie superpixels compared to state-of- oh- superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274-2) 2012.
Milad Alizadeh, Javier Fernandez-Marquds, Nicholas D. Lane, and Yarin Gal. A empirical study of binary neural networks' optimisation. In International Conference Learning Representations, 2019. URL
https:///peereeviee.. net wttp://w.eee..roeoeeeeeeeee Firrrmmmm2md-FFFuRRSRS
Milad Alizadeh, Arash Behboodi, Mart Baalen, Christos Louizos, Tijmen Blankevoort, and Max Welling. Gradient ε regularization for quantization robustness. arXiv preprint arXiv:2002.075200 2020.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Leaming to represent programs with graphs In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. [tttsts:t:/ /ttttps://opnrreviww net forum 2 O-RJOFFTXX
Yoshua Bengio, Nicholas Lčonard, and Aaron Courville, Estimating propagating gradients through stochastic neurons for conditional computation, 2013.
Davis Blalock, Jose Javier Gonzalez. Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning?, 2020.
Zachariah Carmichael, Hamed F. Langroudi, Char Khazanov, Jeffrey Lillie, John Gustafson, and Dhireesha Kudithipudi. Deep positron: A deep neural network using the posit number system, 2018.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets, 2020.
Matthicu Courbariaux, Yoshua Bengio, and Jean-Pierre David, Binaryconnect: Training deep neural networks with binary weights during propagations, 2015.
Yinpeng Dong Renkun Ni, Jianguo L., Yurong Chen, Jun Zhu, and Hang Su. Learning accurate low-bit deep neural networks with stochastic quantization, 2017.

---

Published as a conference paper at ICLR 2021
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru- Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224-22222222 2232, 2015.
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmark- ing graph neural networks, 2020.
Steven K. Esser, Jeffrey McKinstry, Deepika Bablani, Rathinakumar Appuswamy, Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020. URL htttps:/epenreviiet net http//ormmmmmmrkgooaa
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression, 2020.
Boyuan Feng, Yuke Wang, Xu Li, Shu Yang. Xueqiao Peng. and Yufei Ding. Sgquant: Squeezing the last bit on graph neural networks with specialized quantization, 2020.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric, 2019.
Justin Gilmer, Samuel S. Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. CoRR. abs/1704.01212, 2017.
01212
William I.. Hamilton, Rex Ying, and Jure Leskovee. Inductive representation learning on large graphs, 2017.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2015.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks efficient integer-arithmetic-only inference, 2017.
Zhihao Jia, Sina Lin, Mingyu Gao, Matei Zaharia, and Alex Aiken. Improving the accuracy, scalability, and performance of graph neural networks with roc. In Proceedings of Machine Learning and Systems 2020, pp. 188-198. 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation, 2018.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang. Hector Yuen, Jiyan Yang. Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. study of 6 for deep learning training, 2019.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpp:://pperreiiee net. forum?
Id=SJU4ayYq1
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. 2018.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh. Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. In Proceedings of Machine Learning and Systems 2020, pp. 230-246. 2020.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben. Gregory Diamos, Erich Elsen. David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2017.
Anurag Mukkara, Nathan Beckmann, Maleen Abeydeera, Xiaosong Ma, and Daniel Sanchez. Exploiting locality in graph analytics through hardware-accelernan traversal scheduling. In Proceedings of the 51st Annual IEEE/ACM International Symposiun on Microarchieectur, MICRO-51, pp. 1-14. IEEE Press, 2018
ISBN 9781538662403. doi: 10.1 109/MICRO.20 201 8.00010. URL https:: //0oi .org/ 0. W09/MICCC
2018 8 00010
10

---

Published as a conference paper at ICLR 2021
Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh, Fully quantized transformer for machine translation. 2019.
Yu Rong. Wenbing Huang. Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional
networks on node classification. In International Conference on Learning Representations, 2020. URL
httpps://pperevviee www.teeuttteeeeeeeeeeeee ooe qkrKPT
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. arXiv preprint 1rXiv:1111.1163, 2019.
Sheng. Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. quantization- friendly separable convolution for mobilenets. 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), Mar 2018. doi: 10.1 (09/cmc2.2018.00000 URL
http: //dx. dci. org wwt.///c......e. 2018 00011
Moran Shkolnik, Brian Chmiel, Ron Banner, Gil Shomron, Yuri Nahshan, Alex Bronstein, and Uri Weiser. Robust quantization: One model to rule them all. arXiv preprint wxXxx:2002.07660 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
Rianne den Berg, Thomas N. Kipf, and Max Welling. Graph convolutional matrix completion, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL. https://cpenrest ew, net fort/ dit.rrmmmmiiuuuar r kCZ
Hanchen Wang. Defu Lian, Ying Zhang, Lu Qin, Xiangjian He, Yiguang Lin, and Xuemin Lin. Binarized graph neural network, 2020.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-awarc automated quantization with mixed precision, 2018.
Hao Wu, Patrick Judd. Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius, Integer quantization deep learning inference: Principles and empirical evaluation, 2020.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Calcb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: A benchmark for molccular machine Icarning, 2017.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htRtstts:: http::::///oeeeeveeeeeee net www.mmrrmm.oooooommooo --mmmmidd-yyaa A1A55m
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. Tinygnm: Learning efficient graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '20. pp. 1848-1856, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 4503799944 doi: 10. 144/3394486.3403223 URL CRR.:htts:://// orrg/00
(145/3394486.) 343336
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert, 2019.
Hanqing Zeng and Viktor Prasanna. Graphact The 2020 ACM/SIGDA International Symposium Field- Programmable Gate Arrays, Feh 2020. doi: 10.1 N0.1145/377388733333122
1555/3373387.337533
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method, 2020.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
Readers seeking advice on implementation will find appendix A.5 especially useful. We provide significant advice surrounding best practices on quantization for GNNs, along with techniques which we believe can boost our methods beyond the performance described in this paper, but for which we did not have time to fully evaluate.
A.1 EXPERIMENTAL SETUP
As baselines we use the architectures and results reported by Fey & Lenssen (0019) for citation networks, Dwivedi al. (220) for MNIST, CIFAR-I0 and ZINC and,[Xut et al. (0199] for REDDIT BINARY. We re-implemented the architectures and datasets used in these publications and replicated the results reported at FP32. Models using GIN layers learn parameter 6. These models often referred to as GIN-e. The high-level description of these architectures is shown in table 5 The number parameters for each architecture-datase in this work are shown in table
Our infrastructure was implemented using PyTorch Geometric (PyG) (Fey & Lenssen 2019). We generate candidate hyperparameters using random search, and prune trials using the asynchronous hyperband algorithm (Li et al., 2020). Hyperparameters searched over were learning rate, weight decay, and dropout (Srivastava et al. 2014) and drop-edge (Rong et al., 2020) probabilities. The search ranges were initialized centered at the values used in the reference implementations of the baselines. Degree-Ques requires searching for two additional hyperparameters, Pmin and Punax these were tuned in grid-search fashion. We report our results using the hyperparameters which achieved the best validation loss over 100 runs on the Cora and Citeseer datasets, 10 runs for MNIST, CIFAR-10 10 and ZINC, and 10-fold cross- volss-valiationn for REDDIT- BINARY...
We generally used fewer hyperparameter runs for our DQ runs than we did for baselines even ignoring the searches over the various STE configs. As method is more stable, finding reasonable set parameters was easier than before. As is usual with quantization experiments, we found useful decrease the learning rate relative the FP32 baseline.
Our experiments ran on several machines in our SLURM cluster using Intel CPUs and NVIDIA GPUs. Each machine running Ubuntu 18.04. The GPU models in our cluster were: V100, RTX 2080Ti and GTX 1080Ti.
Model
Layers C
# Hidden Units
Residual
Output
Arch.
Cit
M
R
M
Z.
R
Cit
C
Cit
M
Z
R
GCN
2
4
4
16
146
146
145
x
GAT
2
19
18
X
110
Table 5: High level description of architectures evaluated for citation networks (Cit), MNIST (M), CIFAR-10 (C), ZINC (Z) and REDDIT-BINARY (R). We relied on Adam optimizer for all experiments. For all hatched experiments, we used 128 batch-sizes All GAT models used 8 attention heads. All GIN architectures used 2-layer MLPs, except those for citation networks which used single linear layer.
Model Arch.
Node Classification Cora
Graph Classification CIFAR-10
Graph Regression ZINC
Citeseer
MNIST
REDDIT-BIN
GCN
23063
59366
103889
104181
105454
GAT
92373
237586
113706
114010
105044
GIN
23216
59536
104554
104774
42503
102088
Table 6: Number of parameters for each the evaluated architectures
For QAT experiments, all elements of each network are quantized: inputs to each layer, the weights, the messages sent between nodes, the inputs to aggregation stage and its outputs and, the outputs of the update stage (which are the outputs of the GNN layer before activation). In this way, all intermediate tensors GNNs are quantized with the exception of the attention mechanism GAT; do not quantize after the softmax calculation, due to the numerical precision required this
12

---

Published as a conference paper at ICLR 2021
stage. With the exception of Cora and Citeseer, the models evaluated in this work make of Batch Normalization (loffe & Szegedy 2015). For deployments of quantized models, Batch Normalization layers often folded with the weights [Krishnamoorthi] 2018). This is to ensure the input the next layer is within the expected [/min, qmax ranges. In this work, for both QAT baselines and QAT+DQ, we left BN layers unfolded but ensure the inputs and outputs were quantized to the appropriate number of bits (i.e. INT8 or INT4) before getting multiplied with the layer weights, We leave as future work proposing BN folding mechanism applicable for GNNs and studying its impact for deployments of quantized GNNs.
The GIN models evaluated on REDDIT-BINAR used QAT for all layers with the exception of the input layer of the MLP in the first GIN layer. This compromise was needed to overcome the severe degradation introduced by quantization when operating on nodes with a single scalar as feature.
A.2 DATASETS
We show in Table the statistics for each dataset either used or referred to in this work. For Cora and Citeseer datasets, nodes correspond documents and edges to citations between these. Node features are bag-of-words representation of the document. The task is to classify each node in the graph (i.e. each document) correctly. The MNIST and CIFAR- 10 datasets (commonly used for image classification) are transformed using SLIC (Achanta et al. 2012) into graphs where each node represents cluster of perceptually similar pixels or superpixels. The task is to classify each image using their superpixels graph representation. The ZINC dataset contains graphs representing molecules, were each node is an atom. The task is to regress a molecular property (constrained solubility (Jin et al., 2018) given the graph representation of the molecule. Nodes in graphs of the REDDIT-BINARE dataset represent users of a Reddit thread with edges drawn between a pair of nodes if these interacted. This dataset contains graphs of types of communities: question-answer threads and discussion threads. The task is to determine if a given graph is from question-answer thread discussion thread.
We use standard splits for MNIST, CIFAR-10 and ZINC. For citation datasets (Cora and Citeseer), we use the splits used by Kipf & Welling 2017). For REDDIT-BINARY we use 10-fold cross validation.
Dataset
Graphs
Nodes
Edges
Features
Labels
Cora
2,708
5,278
1,433
Citeseer
3,327
4,552
3,703
Puhmed
1
19,717
44,124
500
3
MNIST
70K
40-75
564.53 (avg)
10
CIFARIO
60K
85-150
941.07 (avg)
10
ZINC
12K
9.37
49.83 (avg)
REDDIT-BINARY
2K
429.63 (avg)
497.75 (avg)
2
Reddit
232,965
114,848,857
41
Amazon
9,430,088
231,594,310
Table Statistics for each dataset used in the paper. Some datasets are only referred in fig.
A.3 QUANTIZATION IMPLEMENTATIONS
In section we analyse different readily available quantization implementations and how they impact in QAT results. First, vanilla STE, which is the reference STE (Bengio et al.) 2013)) that lets the gradients pass unchanged; and gradient clipping (GC), which clips the gradients based on the maximum representable value for a given quantization level. Or in other words, GC limits gradients the tensor's magnitudes are outside the [gmin: [max] range.
(min(X) if step = 0 otherwise
Trin
Ahinnxxminvie X)
(1)
XX) Emin
if step 0 otherwise
(2)
(1 c) + e emin(X)
13

---

Published as a conference paper at ICLR 2021
The quantization modules keep track of the input tensor's min and max values, Emin and mmax. which are then used to compute Jmin- Qmax, zero-point and scale parameters. For both vanilla STE and GC, study two popular ways of keeping track of these statistics: min/max, which tracks the min/max tensor values observed over the course of training; and momentum, which computes the moving averages of those statistic during training. The update rules for Tmin for STE min/max and STE momentum are presented in eq. (I) and eq. (2) respectively, where X is the tensor to be quantized and the momentum hyperparameter, which all our experiments set to its default 0.01. Equivalent rules apply when updating mmax (omitted).
For stochastic QAT we followed the implementation described in Fan et al. (2020), where each training step a binary mask sampled from Bernoulli distribution is used to specify which elements of the weight tensor will be quantized and which will left at full precision. We experimented with block sizes larger than one (i.e. single scalar) but often resulted in a sever drop in performance. All the reported results use block size of one.
A.4 DEGREE-QUAN QUANT AND GRAPH LEVEL SUMMARIZATION
The percentile operation in our quantization scheme remains important for summarizing the graph when doing graph-level tasks, such as graph regression (Zinc) or graph classification (MNIST, CIFAR- 10 and REDDIT-BINARY) Since the number of nodes in each input graph is not constant, this can cause the summarized representation produced from the final graph layer to have a more tailed distribution than would be seen with other types of architectures (c.g. CNN). Adding the percentile operation reduces the impact of these extreme tails in the fully connected graph-summarizatto thereby increasing overall performance. The arguments regarding weight update accuracy also still apply, as the BC term in the equations for the GCN and GIN should be more accurate compared to 2h,2,
layers,
when the activations are always quantized before the summarization. This phenomenon is also noted by Fan et al. (202)).
A.5 IMPLEMENTATION ADVICE
We provide details that will be useful for others working in the area, including suggestions that should boost the performance of our results and accelerate training. We release code on GitHub; this code is clean implementation of the paper, suitable for users in downstream works.
A.5.1 QUANTIZATION SETUP
As our work studies the pitfalls of quantization for GNNs, we were more aggressive in our imple- mentation than is absolutely necessary: everything (where reasonably possible) in networks is quantized. In practice, this leaves low-hanging fruit for improvements in accuracy:
Not quantizing the final layer (as is common practice for CNNs and Transformers) helps with accuracy, especially at INT4. A similar practice at the first layer will also be useful.
Using higher precision for the "summarization" stages of the model, which contributes little towards the runtime in most cases.
Taking advantage of mixed precision: since the benefits of quantization are primarily in the message passing phase (discussed below), one technique to boost accuracy is to only make the messages low precision.
We advise choosing a more realistic (less aggressive) convention than used in this work. The first two items would be appropriate.
A.5.2 RELATIVE VALUE OF PERCENTILES COMPARED TO PROTECTIVE MASKING
There are two components to our proposed technique: stochastic, topology-awer aawae, masking and percentile-base range observers for quantizers. We believe that percentiles provide more immediate value, especially at INT4. We find that they are useful purely from the perspective of stabilizing the optimization and reducing the sensitivity to hyperparameters.
14

---

Published as a conference paper at ICLR 2021
However, adding the masking does improve performance further. This is evident from table In fact, performance may be degraded slightly when percentiles are also applied: this can be observed by comparing table to the main results in the paper, table 2
A.5.3 PERCENTILES
The key downside with applying percentiles for range observers is that the operation can take significant time. Training with DQ is slower than before-howevee since there is less sensitivity to hyperparameters, fewer runs end up being needed. We are confident that an effective way to speed up this operation to use sampling. We expect 10% of the data should be adequate, however we believe that even I % of the data may be sufficient (dataset and model dependent). However, have not evaluated this setup in the paper; it is provided in the code release for experimentation.
A.5.4 IMPROVING PERCENTILES
We believe that it possible to significantly boost the performance GNN quantization by employing learned step size approach. Although we used percentiles in this paper illustrate the range- precision trade-off for GNNs, we expect that learning the ranges will lead to better results. This approach, pioneered by works such Esser et al. (2020), has been highly effective on CNNs even down to bit quantization.
Another approach would be to robust quantization: the ideas in these works reduce the impact of changing quantization ranges i.e. making the architecture more robust to quantization. Works in this area include Alizadeh et al. (2020) and Shkollik al. (2020).
A.5.5 IMPROVING LATENCY
The slowest step of GNN inference is typically the sparse operations. It is therefore best to minimize the sizes of the messages between nodes i.e. quantize the message phase most aggressively. This makes the biggest impact CPUs which are dependent caches to obtain good performance.
We evaluated our code on CPU using Numpy and Scipy routines. For the GPU, we used implemen- tations from PyTorch and PyTorch Geometric and lightly modified them support INT8 where necessary. These results, while useful for illustrating the benefits of quantization, are by no means optimal: did not devote significant time to improving latency. We believe better results can be obtained by taking ad advantage of techniques such as cache blocking or kernel fusion.
A.5.6 PITFALLS
Training these models can be highly unstable: some experiments in the paper had standard deviations large as 18%. We observed this affect citation network experiments to the extent that they would not converge on GPUs: all these experiments had to be run on CPUs.
A.6 DEGRADATION STUDIES
Figures 7 and 8 show the results of the ablation study conducted in section for GCN and GIN. We observe that GCN is more tolerant to INT4 quantization than other architectures. GIN, however, requires accurate representations after the update stage, and heavily suffers from further quantization like GAT. The idea of performing different stages of inference different precisions has been proposed, although it is uncommon (Wang et al. 2018).
15

---

Published as a conference paper at ICLR 2021
0.88
Reference Weights
Messages Pool-Aggrggation
0.86
Text Annurary
Accuracy
0.84
0.8
0.82
0.5
0.80
0,4
Weights
Reference
Updato
Agpregation
0.78
0.3
Messagua
0.64
0.66
0.68
0.70
0.72
0.74
0.8
1.0
1.2 Test Loss
1.4
1.6
Test Loss
Figure Degradation of INT8 GCN Cora indi- vidual elements converted to INT4 without Degree- Quant.
Figure 8: Degradation of INT8 GIN Cora indi-
vidual elements are converted INT4 without Degree- Quant.
MNIST Superpixel
CIFAR10 Superpixel
3.0e+06
1.5e+06
Node Count
2.0e+06
1.0e+06
5.0e+05
1.0e+06
0,0e+00
0.0e+00
10
ZINC
Cora
10
10"
Node Count
10
10
10
10
4
50
100
150
Citeseer
Reddit-Binary
10
10"
Node Count
10
10
10
10
0
20
40
60
80
100
0
1000
2000
3000
In-Degree
In-Degree
Figure In-degree distribution for each of the six datascts assessed. Note that log y-axis is used for all datasets except for MNIST and CIFAR-10.
16

---

Published as a conference paper at ICLR 2021
1.2
FP32
DQ-INTB (0.0.0.1)
DQ-INTB (0.1.0.2)
DQ-INTB (0.2.0.2)
Quantization
Model
REDDIT-BIN
Val Val Loss
DQ-INT8 (0.2.0.3)
Ref. (FP32)
GIN
92.2 + 2.3
Ours (FP32)
92.0 1.5
0.6
DQ-INTS (0.0, 0.1)
GIN
91.8 1: 2.3
0.4
DQ-INTS (0.1, 0.2)
GIN
90.1 + 2.5
DQ-INTS (0.2, 0.2)
GIN
89.0 + 3.0
0.2
DQ-INTS
GIN
88.1 3.0
0
25
50
100
125
150
175
200
Epoch
Table 8: Final test accuracies for FP32 and DQ-INT8 models whose validation loss curves shown in fig. TO
Figure 10: Validation loss curves for GIN models evaluated REDDIT-BINARY. Results averaged across 10-fold cross- validation. We show four DQ-INT8 experiments each with differ- ent values for (Pmin-Pmax) and our FP32 baseline.
Quantization
Model
Node Classification
Graph Regression
Schemne
Arch.
Cora 1
Citeseer
ZINC J
GCN
81.1 + 0.6
71.0 + 0.7
0.468 + 0.011
QAT-INTS DQ Masking
GAT
82.1 0.1
71.4 + 0.8
0.462 + 0.005
GIN
78.9 + 1.2
67.1 + 1.7
0.347 1 0.028
GCN
78.5 + 1.4
62.8
0.599 + 0.015
QAT- Masking
GAT
64.4 9.3
68.9 1:
0.529 1 0.008
GIN
71.2
56.7 = 3.8
0.427 = 0.010
GCN
75.6
64.8 3.8
0.633 + 0.012
mQAT-INT4 Percentile
51.4
GIN
03.5 2.0
46.3
0.771 = 0.058
Table 9: Ablation study against the elements of Degree-Quam (DQ). The first two rows of results obtained with only the stochastic element of Degree-Quam enabled for INT8 and INT4. Percentile- based quantization ranges are disabled in these experiments. The bottom row of results were obtained with noisy quantization (nQAT) at INT4 with the use of percentiles. DQ masking alone is often sufficient to achieve excellent results, but the addition of percentile-basee range tracking be beneficial increase stability. We can see that using nQAT with percentiles
sufficient achieve results of the quality DQ provides.
Device
Arch.
FF32
W8A8
Speedup
FP32
W8A8
Speeduup
Speedup
GCN
182ms
88ms
2.1x
0.94ms
0.74ms
1.3x
0.97ms
0.76ms
1.3x
CPU
GAT
500ms
496ms
1.0x
0.86ms
0.78ms
1.1x
0.99ms
0.88ms
1.1x
GIN
144ms
44ms
3.3x
0.85ms
0.68ms
1.3x
0.95ms
0.55ms
1.7x
GCN
2.
1.6ms
1.3x
0.08ms
0.09ms
0.9x
0.09ms
0.09ms
1.0x
GPU
GAT
30.0ms
27.1ms
1.1x
0.57ms
0.64ms
0.9x
0.56ms
0.64ms
0.9x
GIN
20.9ms
16.2ms
1.2x
0.09ms
0.07ms
1.3x
0.09ms
0.07ms
1.3x
Table 10: INT8 latency results 22 core 2.1 Intel Xeon Gold 6152 and, on GTX 1080Ti GPU, All layers have 128 in/out features. For CIFAR-10 used batch size of 1K graphs.
17

---

Published as a conference paper at ICLR 2021
training stap Nodes
Node Activations
(quantized in the previous layer)
Matmul
Quantize
All Weights are uniformly quantized INT-NN
DO
Retain for bacloaard pass
Quantize
INT-N FP32
Node Messages
Intermadiate
aggregation output
Each node gathers the messages from neighbour nodes (thee be FP32 onland INT-N), The intermediate sggregation always FP32. the of aggregation quantized depending on the DO mask the current training step.
at
DQ Mask
PP12
PFNN
Aggregated
F
Messages
Update
DQ Mask
Quantize
Layer Output
Figure 11: Diagram representing how DQ makes use of a topology-aware quantization strategy that is hetter
suited for GNNs. The diagram illustrates this for GCN layer. At every training stage, degree-hased mask generated. This mask used in quantization layers located after each the stages the message-passing pipeline. By retaining at FP32 nodes with higher-degree more often, the noisy updates during training have lesser impact and therefore models perform better, even at INT4.
18

---

Published as a conference paper at ICLR 2021
Node Activations (quantized the previous layer)
Matmul
Quantize with noise
Retain pass
Quantize
INT-N FP32
Node Messages
Aggregate
Unlike with Degre:Quant, all stages make uniform quantization intraducing topology aware masking
Quantize
Aggregated
Messages
Update Stage
Quantize
Layer Output
Diagram representing how nQAT is implemented for GNNs. The diagram illustrates this for a GCN layer. The stochastic stage only takes place when quantizing the weights, the remaining of the quantization modules happen following standard QAT strategy. A QAT diagram would be similar to this one but fully quantizing the weights.
Figure 12:
19

---

Published as a conference paper at ICLR 2021
Graph summarization output stage for graph-level tasks using DQ
Graph summarization output stage for graph-ievel tasks using NQAt
Layer Output
Final Layer Outpul
Summarization
Summertzation
percentie es Da made
MLP
MLP
Morhweloally masked
Graph-level Outgut
Graph-level Oulput
Figure 13: Diagrams representing how the output graph-summarization stages for graph-level tasks (e.g. graph classification, graph regression) are implemented when making use of DQ (left) and nQAT (right). GNNs making use of DQ during the aode-aggregation stages (see fig. [1]. do not use the stochastic element of DQ in the output MLP layers but still make of percentiles. For models making of nQAT, the final MLP still makes stochastic quantization weights.
20

---

Published as a conference paper at ICLR 2021
B CODE LISTINGS
Our code depends on PyTorch Geometric (Fey & Lenssen 2019). These snippets should be compatible with Python 3.7 and PyTorch Geometric version 1.4.3. You can see the full code on GitHub:
hetps://githie comm//oomlssy//esrree-uuuaa
B.1 MASK GENERATION
class Sogeabiiiettttttgheeeeeeett
def iiitt__ (self, lcw_prcb, high_prob, par_graphetrua); selr. low_prob
self.bigh_prob high_prob
self.per_grape per_graph
def _pcccess_ssep (sepces,, graph): that:
# 1. The probabiiity of being meaked increases as the indegres incresses
bernoullf p # 3. you can set this such that all nodes have some probability of being masked
with
Indegree
siaphh.num_zode indegres degree eegeeeedeee_eeee torch. bincount (indagrec)
n, dtypestorch.
step_size [eelC.high_peae
sell.iow_prob)
indegres_ps counts step_size
indegree_ps corch.cumaure (indegree_ps, dim=0)
indegres_ps +* self.lew_pre
proph.prob_mack stdegree_ps
return graph
def _call_ (self, data):
if Sull.per_grapp and isinstance (data, Batch)
graphs data.to_dst__lit(()
processed = []
grapha: pcoces__ccces_araalll
processed.appente
return #atch. List (processed)
else:
sel: processsggraphhees....
def evaluate prob_med mask (data):
torch.bernoula cdats.prrb_nne ttt (torcch.bbol)
B.2 MESSAGE PASSING WITH DEGREE- QUANT
Here we provide code to implement the layers as used by our proposal. These are heavily based off of the classes provided by Py PyTorch Geometric, with only minor modifications to insert the quantization steps where necessary. The normal quantized versions are similar, except without any concept of high/low masking.
class PassagePasaingtin Quant (nn. e))ulle): *eathia class is Lightweeight modi flcation of the default PyToron Geometric MessagePaseing class en*
izrelevant
removed
def propagata (self, edge_index, mask, size=None, *ekwargs);
$ some Jinas skipped
mag self. mels.mesesge (wamsq_kwazgs) if self.treining:
This da for the masking of messages:
edge_mesk torch. index_eeiect (nask, 0, edge_index(O0) cut tarch, empty_like torch.empty out [adge_mask]] www...mp.pp..ece.e. _m__quantizerss (mmossage (iigghgg__hhhh" (mag [odgo_mask])
(msg)
[=edge_maak] = self. mol...m__uantizer.. ["mesaage_low"]
nag [ag[=edge_nask]
else:
gelf.mp_quantizent ( ["nessage_ (mag)
aggr aggr_kwarge kwargs self-_ dlatributa (salf. _aagg__aaaaaaaa kwargs)
21

---

Published as a conference paper at ICLR 2021
aggra self. arg.l..gggggteeeeee (out, *eagge_kwargg)
if celf.training
out torch. terch.empty_ (ikerygr))
out [pask] self. mei...p_quantiieee qug_u ("gggrggeggggeeeeeeaa (agges(mask]
out [-mask] self.mp.quantizer: wttpguuaiirerrsssgeeeeeee "aggrepateeoo (aggrs[-maske
else:
out self pelr.mp_qqante qu izers ( ("aggregate_"ov" low"] (aggra)
update_ kwargs = sel sel". distriaute (self. update_params___ (warga)
updales upllte **upoate_kwazgg)
if telf.treining:
out terch, terch.cmpty_ eap
[mask] sal mmm__qunnnt ("update_high") (updaties [maak])
out [*maak] = selt: ["update_]ral 1ow*] [=pask
out self, selt.mp_quäntizers ["opdate_oow"] (updates)
B.2.1 GCN
class GCMConvMult iQuant (MeasagePaast ngMul Quantt)
Some methods missed...
def (seward x, edge_index, mask, sege__eeghh:Neee)) i quantizing inprt
self.tre teli.ttainnn::
tarch.empty emppyeetttyy_keeee
x_g[mask] = sal sals.
"inputs_hiig hhigh"" (x[mask])
x_a[-maak] self. rpputaa__oo (x[=mask])
else:
x_9 Lae.elaaye__uee Layereye ("inputs_iow"]
guant layer weights
m_g lage...ayys_.gatiieeeee (wa.eeerrraiattaaaaaaaa (sels.
if self.training:
torch.empty ((__ sh)pe[0], (001 M_9, shape (1])) (o_g.. deviceee
x_tmp marcch. (_9)
x[mask] seet (" (x_=mp [mask])
x[-mask] self, Lalf..aayr_guuaiizzerr features_ (x_ [=mask])
else:
('faatures_oo") (torch. matmul (x_g,
nolmmnozmalss::
idge_index, narm seif.norm(
adge_index,
x, (sel nodddddddd_)) edge_weight, ael improoee, x.dtype,
else:
norm Sogghtoight
noza self, lay.r_yyeeratttteee snver_guantizer "ilantiierss"nni ["norn"] (nozm)
salf prrppprpaaaaee sadge_index, xex, normenore, maskemask)
22


Reviews:
[{"review_id": "s0XBmeLAO4r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new technique for quantization aware training of neural networks that is specially suited for graph neural networks. They do a good job of motivating the problem by demonstrating that the large variation of input degree in GNNs can lead to unique challenges for numerical precision, forcing a compromise between truncation error and rounding error. Th proposed technique incorporates stochastic masking and quantization proportional to the input degree to allow higher input-degree nodes to operate at higher resolution on average. \n\nThe authors demonstrate strong improvements over quantization aware training that treats all nodes equally, achieving relatively small drops in accuracy for a large compression and speedup of GNN inference. \n\nThe work is presented in a straightforward and clear manner, with clear applications to important problems. \n\nTwo small things that could improve the paper.\n* Percentile tracking is a component to the methods, but relies on a reference for full explanation. A more precise statement of this part of the method in the paper itself would help clarify for readers.\n* Minor nit, but some acronyms are used before they are defined (such as GCN). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nG5zn6dcQTG", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Let me note that I have very little expertise in quantization and so cannot really judge the significance of such contributions. I am, however very familiar with the GNN literature.\n\nSummary\n-------------\nA method to train GNNs such that later quantization works well is presented. The authors first analyse the message passing definition to identify those computation steps whose results show the largest variance, and hence suffers most from the imprecision introduced by quantization. Consequently, hey focus on the message aggregation phase of message passing. \nThey then propose two improvements to more standard quantization-aware training (QAT): (1) applying quantization during the forward pass only on message aggregation outputs (and doing it more often on nodes that receive many messages); and (2) using percentile-based statistics for determining the ranges of values considered during quantization.\nFinally, experiments show that the resulting training procedure works well for GNNs on a number of datasets, matching or slightly improving the baseline performances. In most cases, the proposed Degree-Quant method also outperforms baseline QAT methods.\n\nStrong/Weak Points\n-------------\n* (+) Empirical results show moderate gains over the baseline QAT methods for int8 quantization, and substantial gains for very coarse quantization to int4.\n* (+) Thoughtful experimental ablations study the effect of the two improvements separately, and further empirically verify the theoretical analysis of sources of errors.\n* (-) The paper is not self-contained and hence not easily readable for people without background knowledge in quantization. While GNNs are fully (though very densely) defined in Sect. 2.1, no technical details on quantization are provided in Sect. 2.2. I ended up skimming some of the cited papers to even understand how values are practically mapped between fp32 and int8. Consequently, Sect. 3.2 is discussing extensions and alternatives to concepts that are simply not explained in the paper.\n\nRecommendation\n-------------\nI think this paper can be accepted and would be useful for the very narrow segment of people interested and knowledgeable in GNNs and quantization. However, in the current form, it is inaccessible to a wider audience and I believe that it could be significantly improved in that regard.\n\nQuestions\n-------------\n(1) Message aggregation is identified as a key source of quantization error due to the variance in the number of messages. For graph-level tasks (such as MNIST, CIFAR and ZINC), the aggregation of node representations to a graph representation should lead to a similar problem. Do you have deeper analysis on this aspect?\n\nDetail Feedback\n-------------\n* Sect. 3.1, end: the mixing of GCN and GIN is somehow confusing and it would be wortwhile to restructure this. (i.e., $\\mathbf{y}_{GIN}^{(i)}$ is defined before the equation its used in, but $\\mathbf{y}_{GCN}^{(i)}$ after, etc.)\n* Sect 3.2 / Alg. 1: I found the use of \"mask\" / \"masking\" here highly confusing, as I associate it with removing a value (as in masking of loss components, dropout masks, hiding a human face behind a cat mask, ...), but here the semantics is inverted: masks determine which values are \"more visible\" (by not applying the quantization to them). Unless this term is already in standard related use in the quantization literature, I would strongly recommend to use a different term here (e.g. \"preserved\", \"protected\", ...) \n* Fig 5/6 are not readable for colorblind people.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SbX2Pwk21QL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(Edit: Sorry, the previous review was for a different paper that ended up in here due to a copy-paste issue)\n\nThis paper uses quantization and Quantization Aware Training (QAT) to improve the speed performance of GNN inference for three types of GNN modes: GIN, GCN and GAT. The paper identifies the aggregation step to be where quantization introduces the most numerical error, and use stochastic masking and clipping the top/bottom values to mitigate the issue. \n\nThis topic is very relevant and interesting, and novel to the best of my knowledge---although I'm not familiar with the literature surrounding quantized neural networks. \n\nThere are places where the writing can be more careful. For example, in the abstract the authors write: \"little research exploring methods to make GNN more efficient at inference time\". However, there has been research focusing on both hardware acceleration [1] and making GNN models smaller [2]. Quantization isn't the only approach to make GNN inference faster. Claims like \"it is not possible to deploy this technique on smartphones\" (from intro paragraph 2) should be supported, since it's difficult for a reader\nto verify such a claim.\n\nSome of the claims, like the one bolded in Table 1, should be in the abstract. I'm not sure if this is typical in the quantization literature, but a wallclock time comparison would be useful in Table 2 to compare the time speedup against the baseline.\n\nOne other presentation feedback: in figure 1, the x-axis is not continuous. A line chart is not appropriate since the slope of the line segments in the chart is meaningless. Removing the lines connecting the dots would make more sense.\n\n[1] Zeng and Prasanna. (2020) GraphACT: Accelerating GCN training on CPU-FPGA heterogeneous platform https://arxiv.org/abs/2001.02498\n[2] Yan et al. (2020) TinyGNN: Learning Efficient Graph Neural Networks https://dl.acm.org/doi/abs/10.1145/3394486.3403236", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a quantization aware training method for GNNs. The problem is very well motivated, the method is well-executed, and experiments are also well designed. The paper does seem relatively low on technical novelty. 

All the reviewers are positive about the paper, and the paper has certainly improved significantly over the rebuttal phase. 

So, we would like to see the paper accepted at ICLR. 
Decision: Accept (Poster)