GCINT: Dynamic Quantization Algorithm for Training Graph Convolution Neural Networks Using Only Integers
Authors: Qizhe Wu
Letian Zhao
Huawen Liang
Xiaotian Wang
LinFeng Tao
Teng Tian
Tingxin Wang
Zerong He
Wei Wu
Xi Jin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Quantization
Graph Neural Networks
Acceleration Training
Integers Networks

Abstract: Quantization approaches can minimize storage costs while decreasing the computational complexity of a model, although there is minimal study in the GNN field on quantization networks. We studied the four primary reasons why existing quantization approaches cannot be employed extensively with GNNs: (1)Quantifying the distinctions between data sources; (2)Quantifying the distinctions between data streams; (3)Quantifying the distinctions between concentrations; (4)QAT’s Limitations. Based on this, we propose GCINT, which is an efficient quantization framework prepared for GNN training. The entire forward, backward, optimizer, and loss functions are calculated using integer data. We achieved a training acceleration ratio of nearly 10× compared to FP32 Cuda Core in RTX 2080TI INT8 Tensor Core. Our quantization is independent of the dataset and weight distribution, and more than 2,000 randomized trials have been undertaken on the 8 popular GNN benchmark datasets, with all achieving errors within 1% of the FP32.

Under review as a conference paper at ICLR 2023
GYINT:DYNAMIC QUANTIZATION ALGORITHM FOR TRAINING GRAPH CONVOLUTION NEURAL NETWORKS USING ONLY INTEGERS
Anonymous authors Paper under double-bling review
ABSTRACT
Quantization approaches can minimize storage costs while decreasing the compu- tational complexity of a model, although there is minimal study in the GNN field quantization networks. We studied the four primary reasons why existing quan- tization approaches cannot be employed extensively with GNNs: (1)Quantifying the distinctions between data sources; (2)Quantifying the distinctions between data streams; (3) [3)tttniiy the distinctions between concentrations; (4)QAT's Limitations. Based on this, propose GCINT, which is efficient quantization framework prepared for GNN training. The entire forward, backward, optimizer, and loss functions are calculated using integer data. We achieved a training ac- celeration ratio of nearly 10x compared to FP32 Cuda Core in RTX 2080TI INT8 Tensor Core. Our quantization is independent of the dataset and weight distribu- tion, and more than 2,000 randomized trials have been undertaken on the popular GNN benchmark datasets, with all achieving errors within % of the FP32.
INTRODUCTION
There is an abundance of graph-struetured data in the natural and social sciences. In fields such as social networks (Fan et al., 2019), recommender systems (Wu et al. (2200), traffic networks (Jiang & Luo 2022), molecular prediction (Mansimov et al. 2019). and drug discovery (Zhang et al.) 2022). Graph Neural Networks (GNNs) representative deep learning systems for graph data learning, infer- ence, and generalization have produced superior outcomes, As graph learning applications increase and graph data expands, the training of GNNs becomes inefficient due to two significant obstacles: (1) Storage expenses. Since training needs recording the outputs of several layers in forward prop- agation for backward propagation calculation, extremely large scale graph data is frequently saved utilizing distributed CPU-centric memory storage by distributed GPU clusters employing mini- batch training technique. Common acceleration devices such as GPUs and FPGAs with on-chip storage and bandwidth no longer match the demand for training large GNNs and are too depen- dent on sampling techniques to train on a device with a limited batch size for each training session Yang 2019). (2) Calculated expenses. Training single epoch on the Reddit dataset generally requires tens of TFLOPS, even for KB-sized GNN models.
lower storage costs while decreasing the model's computa- tional complexity Nagel et al. 2021). Although quantization is widely used in CNNs, research on quantized networks for GNNs is scarce, we believe the following factors primarily restrict the applicability of quantization approaches in GNNs: (1) Quantifying the distinctions between data sources. During CNN training, the RGB images of UINTS are normalized and sent to the net- work. In contrast, when using GNN models, the node features of the network are frequently not the consequence of normalization, and the distribution of node features will shift as the graph changes and embedding methods are employed. The information contained in the image could have been represented by UINTS, whereas the embedding vectors of graph nodes are typically in FP32 data format, which contains significantly more information than UINT8. Therefore, it is essential for GNN to quantize the dataset, which must represent a large amount of information in the dataset with limited number of bits. (2) Quantifying the distinctions between data streams. The calculation in each layer of CNN that maps to the GPU is typically General Matrix Multiplication (GEMM), and the activation distribution in each layer of GNN is strongly tied to the graph topology, when the
Quantization YYang al.) 2019

---

Under review as a conference paper at ICLR 2023
average degree of the graph is high, the aggregation process of the integer domain is more suscep- tible to data overflow. Conversely, when the average degree of the graph is low, the distribution of activation in each layer of the network will be focused in the low bit data range. This brings uncer- tainty into quantitative training, and typical quantization approaches for CNN will not be employed directly in GNN models. (3) Quantifying the distinctions between concentrations, Due to the fact that DNN models generally include millions of parameters, it is required to decrease the com- plexity of storage and processing by quantizing and compressing the weights, such as BNN (Tang et al. 2017) and XNOR-Net (Rastegari et al. (2016) quantize the weights to binary. However, GNN models are typically the KB order of magnitude, and the gains from compressing the weights are not substantial. (4) QAT's Limitations. The majority of CNN quantization is based the re- search of QAT (Wang et al. 2019) quantization operators, and this design strategy has evolved into a robust, low-error quantization model. QAT conducts low-bit quantization of weights and activa- tion during forward propagation, then reduces the noise and loss induced by forward quantization using FP32 back propagation, and lastly dequantizes the model to integer for inference accelera- tion during model deployment (Krishnamoorthi 2018). The restriction of the QAT is that it cannot be utilized for accelerated training since the data format after quantization is still FP32 during the training process. After training a CNN model for certain class of tasks with a significant quantity of data, they seldom need to be retrained after deployment, hence QAT provides CNN models with extraordinarily high advantages. GNNs tend to be dynamic graphs in the real world and require fine- tuning or retraining of the model; hence, speeding the training process of GNNs is more relevant than accelerating the inference process, which QAT cannot perform for accelerated training.
This work considers the motivations and problems associated with quantization of graph architec- tures, and provides the following contributions:
We employ a top-down quantification study methodology. The vast majority of prior quan- tization investigations have been bottom-up procedures, i.e., beginning with the FP32 ten- sor, quantizing to the FP32 tensor that can be mapped by integers one by one, and then dequantizing to the integer, which we consider a superfluous operation. Consequently. we investigate the quantization strategy for integer tensor computation directly from differ- ent angle, which has the advantage that a shaping model can be obtained directly without dequantization and can be used directly for inference and training speedup in fixed-point hardware, such as GPU INT Tensor Core.
We propose a novel quantization training algorithm for graphs as an alternative to the tradi- tional QAT method to accelerate the GNN training process. This algorithm can adaptively adjust the quantization range according to the sparsity of the graph data and can accom- modate unevenly distributed data during training. The entire training forward, backward, optimizer, and loss functions are calculated using integer data, which can be directly ac- celerated by using the INT Tensor Core of GPU for GNN training. We achieved a training acceleration ratio of nearly 10x compared to FP32 Cuda Core in RTX 2080TI INT8 Tensor Core, and can train a larger subgraph than the original one with limited memory.
Our quantization independent of the dataset and weight distribution, and more than 2,000 randomized trials have been undertaken on the 8 popular GNN benchmark datasets, with all achieving errors within 1% of the FP32 baseline, and without fine-tuning hyperparameters.
2 BACKGROUND AND RELATED WORK
Table/[present the quantitative studies published in recent years. WAGE (Wu al. 2018) provides a low-bit quantization approach for weight, activation, gradient, and error CNN training. Data is linearly mapped using integers to FP32, and all training data must be dequantized to the integer domain before being utilized for training acceleration. The authors believe that following matrix multiplication of the bita tensor with the bitw tensor, an bit(a + w 1] tensor is formed, which is then quantized to the bita tensor and sent to the next network layer. We feel that basing the quantization approach on this assumption is not rigorous since it disregards the dimensionality of the tensor. Assuming that the dimensionality of the tensor is n. the data bit-width range of the output tensor should be |a+ w- 1+logyn]. However, because the GEMM and SPMM(Sparse. Dense Matrix Multiplication) dimensions in GNN are typically large, the output data bit-width range will overflow to varying degrees, rendering the method inapplicable to the training speedup of GNN.
2

---

Under review as a conference paper at ICLR 2023
Table I: Comparing the quantization precision of several methods. PUB indicates the origin of the algorithm and the date of its publication. Following is the number of W(Weight) A A(Activation), G(Gradientt, E(Error) and OPTIM(Qptimizer) bits after the respective quantization. SFB(SSve For Backward) is the number of bits of the forward that are saved to compute the back propagation during training. Where 32(n) represents a floating- point number quantized to bitn(stored and calculated in FP32 format, the number will be dequantized to yield INT integers). TRAIN shows if training may be expedited (forward and backward). INFER specifies whether or not inference may be accelerated. DEQ(Dequantizatinn, FP to INT) specifies whether dequantization is required by the algorithm. DOMAIN specifies the algorithm's 's application space and the scope of the experiment.
METHOD
PUB
W
A
G
E
SFB
LOSS
OPTIM
DEQ
TRAIN
INFER
DOMAIN
WAGE
ICLR2018
32(2)
32(8)
32(8)
32(8)
32(8)
FP32
FP32
CNN
DOREFA
CORR2018
32(1)
32(4)
FP32
FP32
FP32
FP32
FP32
CNN
DEP
ICLR2018
16
16
HP32
FP32
CNN
DQ
ICLR2021
32(8)
32(8)
FY32
FP52
FP32
FP32
FP32
GNN
EXACT
ICLR2022
FP32
FP32
FP32
FP32
FP32
FP32
x
GNN
GCINT
THIS WORK
8
8
8
8
8
x
GNN
DoReFa-Net (Zhou et al. 2016) proposes lower-bit weight, activation quantization scheme, which introduces nonlinear activation function in the computation to improve the state representation of the network for overflow data. However, the process is still presented in FP32, and dequantization becomes more complex due to nonlinear activation. In general, it is difficult to directly implement DoReFa-Net training on integer- based hardware, although it has the potential to be utilized in the exploration of high-dimensional discrete spaces with discrete gradient descent directions.
DFP Das et al. 2018) proposes the method of employing mixed precision training with integers in CNNs that precision-criticat operations (such as the optimizer, normalization, and loss function) are maintained at FP32 while computationally intensive processes are maintained at INT16. Since the result of the multiplier output of I will create up 30 bits of data with 1 bit of sign, the time of MAC, at least the accumulation unit of INT32 is necessary to prevent data overflow. The computation result is transformed into an FP32 tensor output the DFP. Before the next layer of input, the FP32 tensor must be dequantized into an INT16 tensor. This ensures that the accumulation of data will not result a massive excess of integers. To participate in the subsequent phase of train- ing, the output of the loss calculation and optimizer parameter update phase must be dequantized from the FP32 tensor to the INT16 data format. DFP investigates the concept of integer training, which enables intense computing to be directly utilized in integer-based hardware. The training is still comprised of a substantial number of FP32 dequantization and is not an entirely integer training stream.
DQ (Tailor et al. 2020 proposes QAT method for GNN. The authors believe that the largest source is the portion of the node with a higher degree, because the node degree in the graph exhibits a power-law distribution, resulting in a larger output in the aggregation of the portion of the node with higher degree, which necessitates a larger bit width, leading to a greater variance of the entire batch of data and very large error when the bit width is converted downward. DQ has effectively analyzed the issues that occur from quantizing GNN, assigning the high-degree node to the FP32 Tensor Core and the low-degree node's data to the INT Tensor Core, and the task assignment of hardware resources will vary for different graph as memory management changes with the degree, makes it challenging to deploy the actual solution to the GPU side for inference acceleration. On the other hand, large graphs are frequently trained with a fixed node sampling method for MiniBatch, in which case there is variation in the degree of each node, rendering DQ inapplicable; hence, DQ is incompatible with existing graph sampling algorithms.
of quantization
EXACT (Liu et al. 2021) provides a low-memory training approach for GNNs, and the author's concept is truly groundbreaking. EXACT utilize FP32 format for both forward and backward prop- agation; however, a portion of the backward propagation computation must utilize some of the activation saved in the forward pass. The authors quantize the data to a very low level of precision and store it in the memory before dequantization to FP32 in order to perform the operation with the
3

---

Under review as a conference paper at ICLR 2023
tensor in FP32 format. This drastically reduces the amount of space required for training, but cannot be utilized to speed up training.
We propose GCINT, quantization approach that accelerates graph training and reduces memory consumption. It may be utilized directly in the GPU's INT Tensor Core to accelerate training without the need for dequantization procedures.
0
1
2.3
Bec 26:042283664567
be=0 6GG-666062922266
Software simulation
Hardware computing overflow region Pseudo overflow region
cRamb
Step3
12 11
122-1)
Step1
(x)
Non-linear activation
(x)
?
0o 7412322855
Wem 222344667
6062992126
2
x127 T 1-127
()(c
Fita
+127
x1227
Binary
Binary
Step2
4s30000 10010
Duss 0001 0111001
Rap
Figure I: (a) and (b) depict the fundamental quantization and dequantization procedures. The green region represents the floating-point data stream for simulated quantization, the purple region repre- sents the integer data stream for actual fixed-pos point hardware execution, the blue region represents the real overflow, and the red region represents the pseudo overflow.
METHODOLOGY
3.1 THE IGNORED PSEUDO-OVERFLOO ISSUE
Before proposing the solution, we first generate the pseudo-overflow problem, which is not counted for in conventional quantization algorithms, and discuss how nonlinear activation influences dequantization. We use I to represent any arbitrary scalar that is compatible with the FP32 data for- mat. Ifn represents the number of signed bitn to which is quantized (as floating-pon poonn value), Jin represents the integer representation of bitn, and Tbo represents the binary representation of bitn. From Figure aa) ys. although it is a floating-point representation, that data falls squarely on the data endpoint that divides the 0-1 interval into 127 copics, indicating that zin multiplied by 2n -1 must be integer. Figure Ta) a) depicts the typical scheme's quantization and dequantization data flows. The crepuuautttionn quantization and post-computation quantization of a batch of data typically need three phases, and INT8 quantization serves function will first remap the data to the interval [-1,1], and the second step quantifies the æ in the interval its nearest endpoint split by area 127, i.e., the & to 2/8- In the third phase, all 2yss uti- lized to do the necessary computation. If the computation involves vectors or matrices, the outcome may be more than 1. So, following each calculation, stepl will be assigned a nonlinear mapping to the interval [-1,1], and then re-quantized.
illustration. A nonlinear activation
In conventional quantization, the fn (green region) often employed for training or inference, To really accomplish computational acceleration with fixed-point hardware, it is frequently essential dequantize the data in the green area the integer domain in order to generate data stream consisting of just integers (the purple region is the data executed in the fixed-point hardware). In Figure aa), step3 we require the following stage of input as Cys- Quantizing to Cfs corresponds to shifting the binary field by 7 bits and then plus (Cb15 to cas), regardless of the bit-width of the accumulator, they will round off the lowest 7 bits of the result and keep the next lowest 7 bits to INT8 form new data (red portion of cb15). plus sign bit (orange portion of co1s) to form integer.
Real overflow: refers to the calculated data portions that exceed 1 (blue area in stepl). Existing quantization often resolves the problem of genuine overflow using a variety of nonlinear activation functions, such clamp( (X, 1) in the Google White Paper (Krishnamoorthi 2018), tanh in
4

---

Under review as a conference paper at ICLR 2023
X 10
Transpose and
GEMM
F
Gradient Stage
W
132
A
SGD
DyQ
Optimize Stage
X00
122
FO
F (2)
132
18
(1+1)
DyQ
SPMM
DyQ
Forward Stage
Transpose
Transpase Save
W
EO
F
F (3) 132
F
132
18
StQ
GEMM
S1Q
SPMM
Backward Stage
Figure 2: GCINT's network layer. For the sake of clarity, the ReLU and Dropout layers omit- wrO
ted from the figure. x-)) in in E!! G!l) A(!) p(k) are tensors in which features, weights, er- rors, gradients, and normalized adjacency matrices are represented INT n signed integers. F((k) denotes the tensor representation inside the 1sh layer. At the moment of initialization, the inte- gerized embedding tensor x-0) and (1) (dataset) are kept in memory. The mode initiation:
x
man ( (2 1)1. A [A() (2" 1)].
(0)
DoreFa-Net (Zouu et al. 2016). etc. The primary issues follows: (I) Universal. Quantiza- tion is optimal when the majority of the data distribution is concentrated in the area with the highest activation function derivative, there are more state representations at this point. No totally univer- sal nonlinear activation function exists to handle the actual overflow problem for diverse data sets or for data streams from different network tiers. (2) Whether the dequantization function has a solution is an open question. After computing e in step3, as seen in Figure lb), the algorithm will employ nonlinear activation remap the data Φ(c) and then quantize it. Since the hardware output is Can, the dequantization function F(x) of ((x) must satisfy the following conditions, o(c) equal to the quotient of F(c) and another integer, and the computation in F(zz) cannot contain a floating-poin point operation; otherwise, the one-to-one mapping between Cfn and is not satisfied after the quantization or shift operation. Therefore, when the quantization interval nonlinear, is frequently challenging to directly dequantize to an integer, and low-bit computational speedup cannot be achieved because the dequantization of (xx) does not necessarily have solution to prove the existence nonexistence of F(x).
Pseudo overflow: Here, we define pseudo-overflow as the occurrence which the result of mul- tiplying and adding the fn cannot be stated using 2" (Figure T(a) red area). As for ass and bys. the result e does not fall on the endpoints divided into 127 copies with a 0-1 interval, indicating that finer division of the 0-1 interval (more data blocks larger than 127) is required to express directly, such that e needs to divide 0-1 into 16129 131 < 16129 < 244 - 1) parts, it requires least 14 bits. Therefore, despite the fact that the data do exceed the 0-1 interval, severe overflow has occurred. Pseudo-overflow in the floating-point is not as intuitive as actual overflow and is frequently overlooked by algorithm researchers. However, since substantial quantity of data is frequently concentrated in the 0-1 calculation interval, pseudo-overflow can introduce extremely large errors to quantization.
3.2 INT8 DYNAMIC AND STATIC QUANTIZATION OPERATORS
Our works is based on the basic GCN Kipf & Welling 2016 layer, (() is the node embedding matrix at the th layer, (()) is the weight matrix of the thh layer, A= - AD- is the normalized adjacency matrix, where 7 is the degree matrix of A + 7, Since Á is often sparse and feature (!)

---

Under review as a conference paper at ICLR 2023
is typically dense, the GCN layer is expressed as the expression of GEMM and SPMM (Fey & Lenssen 2019). see Equation !..
x ((+1) ReLU (SPMM (A", (x"), (WW)))
(1)
We define the dynamic quantization (DyQ), static quantization (SiQ), and weight quantization (Q operators. GCINT's network layer is shown Figure The distinction between DyQ and StQ is that DyQ modifies the quantization interval based on the distribution of matrix, whereas StQ dis- regards. An all-integer network exhibits a pseudo-overflow phenomena, according to our findings. Multiplication of two 2ys integers requires at least 15 bits (14 data and I sign) to completely de- scribe the result without degradation, the result is deemed to have pseudo overflow. From Figure 3 clearly, when matrix multiplication is conducted using integers, the results(absolutt value) in the region [27, 21$ correspond to the floating point pscudo overflow, whilst the real overflow occurs in the region beyond 2 which corresponds to the portion of the data where the floating point is beyond than
f(X)
Wenden
By0 area
M122
21 2 21. 2/...
2 20.5
Figure 3: histogram of the output tensor of the matrix multiplication of two INT8s, where f(2²) represents the number of data in [2 22,22*1). The process of converting INT32 accumulation to INT8 under DyQ and StQ seen on the right one.
We utilize Kullback-Leible divergence (Jycce 2011 to prove that when g(k) Equation gets the maximum, we may derive 7-bit( (k,, k + 6)) expression for this data set with reduced Kullback- Leibler divergence and more Shannon Entropy (Bromiley et al., (2004). From Figure when quanti- fying to INT8, two windows of length are required. Each time, they take Before each step, the region encompassed by the window and the current step value are recorded. When the slide traverses each of the 25 steps, the step with the biggest area is selected as the k. Large area equals more data, therefore the 7 bits from the [k: & + 6) may effectively represent the major- ity of the matrix's data and can harvest more expressive status and distinction. If the conventional direct quantification method outlined in Figure I by default, the corresponding integers use the 7th 13th bits as the expression of the whole batch of data. During the training process of GNN, the sparsity of the graph and the initialization of the weights are different, causing GEMM and SPMM be heavily distributed between the real overflow region (> 214) and the pseudo overflow region (< 214), or between the real and pseudo, and if fixed truncation pattern is adopted >> 2 127, < 27 takes 0), resulting in a poor quantization effect when the data is unevenly distributed.
step the other way.
g(k) Σ f (2k+1) + + (-(22++)) k 0, ..... 25
In Equationn]ann Equation we display both the binary and decimal representations of the dynamic quantization function. Since computers utilize binary storage and processing, our GPU implementa- tion utilizes Equation 3 operations mostly in the form of bitwise operations. Where 7'b represents 7- binary and "I" represents the bit-by-bit "OR" operation on consecutive binary. The StQ func- tion is equivalent to the case where & equals 7 in DyQ. Algorithm [ code demonstrates the GPU computation of k. whose time complexity significantly less than that of matrix multiplication (Ollgg)) < O (n:)) ) and whose high parallelism facilitates parallel processing in the GPU.

---

Under review as a conference paper at ICLR 2023
[csscat(Xis2(31)] (b1111111111111
[362 + otherwise
Binary: DyQ
(oon [31], Xb32 [A: + 6: k])
(3)
Decimal: DxQ(()s2) clamp (i32,, - 2
(4)
2
Algorithm 1: Calculate Parameter k of DyQ on GPU. Maxbit(Xua2): Remove the sign bit and return the location the first occurrence of from the high bit the low bit for each element. (hreadGrroup(32,n33 Allocate 32 xn threads on the GPU.
Input:
X,32 E Raxn
Output:
InitializeMatrix (Ass E xx? Ba32 E Rt x31311 Cha2 E n x2
A + Maxbit(X6st) flatten() then LoadAAooeahhhhraadGooo for i=0 31 each ThreadGroup(i) Parallel do
if' ('TTrradGrruupillii--== then ThreadGroupfilll]] 1 else ThreadGrouplillii-- 0
B[i] Adder (HH:GGH1o end for
q=0 to 25 Parallel do for q=0 do
C[g] Clql +B[ end
10
11 end
12 1 Argmax(())
13 return K
3.3 QUANTIZATION OF ACTIVATIONS
In forward propagation, each matrix multiplication is followed by dynamic quantization using Equation 5 computing process. In contrast to back propagation, the output tensor of each layer in the forward direction must ensure a high degree of discrimination and Shannon Entropy, and the data quantization interval should fall as close as possible to position with higher derivative of the quantization function so that the data can acquire more state bits. Stability the greatest challenge of quantitative training. which is mostly caused by the disparity between graph distributions and weight initialization. DyQ can resolve this issue by using the findings of GEMM and SPMM to alter the quantization interval. As seen in Table 2 we initialize the matrices to be quantized by INT8 of the two quantization techniques with different distributions and determine the Shannon Entropy of the output matrix. We greater Shannon Entropy than classical(Figure I direct quantization for nearly all distributions, and achieves nearly the maximum amount of information that can be conveyed by 8-bit state bits under uniform distribution.
that DyQ handles uneven distributions effectively, achieves
Forward: ((+1) = DyQ (PPM
(1) AA
DyQ (GEMM (x()
w(!) 18
)))
(5)
i8
3.4 QUANTIZATION OF WEIGHTS
To assure the integerization of the complete stage, the computational portion of the optimizer must be as basic as feasible, and it must not contain any additional nonlinear functions; hence, we choose SGD (Bottou, 2012) as our optimizer((Euuatinnn ()). We must employ integer weights and gradient tensor throughout the calculation of the optimizer. We continue to compute the difference between the weights and the gradient using the cumulative unit INT32. After obtaining an updated weight, we normalize new set of weights using an unsaturated quantization approach(Equation]
Optimize:
W
(!) 18
= Qui
(w(!) i8
x (1) 18
(6)

---

Under review as a conference paper at ICLR 2023
Table 2:
N(A.B) denotes normal distribution with A and standard deviation B, U(+T)
denotes uniform distribution in the interval [-T.T], and H(x) is the Shannon Entropy where (x) E p (x+) log p (z:).
Matrix Distrubation 2
NOO, 2
N(2",2 22) N(2?, :2)
N(218, 2 N(218 N2248,244)
N(244,244
N(245 2/4) N(246 2
((Dy((X)
6.993
7.007
6.991
6,992
7,046
7.053
7,033
6.053
5.048
H (TTadd(X))
6.647
2.098
2.106
6.525
1.826
0.023
Matrix Distrubation
U(=2")
Ur::2")
U(::21))
U(d:21))
U(::2*
U(::2
UC::22
U(::225)
U(::2"1)
H (DyQ(X)
7.997
7,813
7.990
8,000
8,000
8,000
8,000
8,000
8,000
H (Trad(())
1.498
2.252
4.064
6.015
8.000
3,564
1.228
1,008
1,004
Qw
W
(
432
X
127
432
(
max
:32
3.5 QUANTIZATION OF ERROR
In contrast to forward propagation, all errors quantized statically, The eтror often does not require as much differentiation as the activation since the error tensor represents the direction and magnitude of each layer's offset. Even with minimal data bits, the weights can be transferred in the direction of the gradient if the erroe is in the correct direction. The error quantization less intimately related when the етгог surpasses the maximum number of bits that can be represented, the network is still capable of learning; however, the gradient does not fall as rapidly as previously, therefore dynamic quantization of the error is unnecessary.
by the Equation S
own distribution than the activation value. Even
E = StQ GEMM SIQ SPMM ( (e(LTt)))
(W!!) T
Backward:
3.6 QUANTIZATION OF GRADIENTS
Before entering the optimizer, the results of the most recent GEMM computation of radients are sub- jected to a dynamic quantization technique. As the gradient reduces during training, the error tensor eventually approaches the lower integer region, causing the estimated gradient after quantization to converge on 0. This phenomenon is especially evident in the backpropagation of the first layer, where the error of the first layer is always smaller in magnitude than the eтror of the second layer, and the transmission of the error in the first layer results in a very severe gradient disappearance as training continues. Consequently, we employ DyQ for the final GEMM of the calculated gradient to invert the offset of the error distribution throughout the training phase, as given in the Equation 9 and this alteration yields excellent results.
((x))* StQ (PMM
( ((4))
DyQ GEEM
E (1+1)
(9)
i8
3.7
LOsS FUNCTION
The Softmax layer and cross-entropy criteria are extensively used in classification tasks, although the computation of e cannot be utilized in instances of low-bitwidth linear mapping. We omit the Softmax layer and use the mean- mnean-squareeery criteria instead (MSE), The back propagation of the loss function layer is shown in the Equation TO
Backward: (last)
OL
x (last) 48
Yi8
(Yi Yonehot X 127)
(10)
48
x (last)
is
8

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
Table 3: Comparison on the test accuracy/FI-micro, memory saving and training time.
Node
89K
232K
716K
1.5M
Edge
899K
114M
13M
264M
Dutaset
Flickr
Reckit
Yelp
Amazon
Model
Minibatch-GCN
Method
FP32
GCINT
FP32
GCINT
IP32
GCINT
FP32
GCINT
A)c(%)
50.10:1:0.22
49.98:0,34
93.06=0.08
92.91:10.12
38.1410.46
38.09
29.34:10.03
29.08
Mem
1.52GB
0.39GB
6.93B
1.79GB
2.75GB
0,70GB
5,57GB
148GB
Timelep
1.99s
0.08s
18.40s
2.12s
27.90s
1.93s
70.0%
5.11s
Speedup
13.70
Nade
2.7K
Edge
10K
9K
BSK
163K
Dataset
Cora
Citosecr
Pubmed
CS
Model
Fullbatch-GCN
Method
FP32
GCINT
FP32
GCINT
FP32
GCINT
FP32
GCINT
Acc(%)
81.50:10.13
83.16:00.37
71.26=0.14
7168::0.43
79.02:20.17
78.8620.39
91.10:00.55
91.50:10.67
Mem
0.08GB
0.02GB
083GB
0.23GB
0.82GB
0.21GB
1.36BB
0.35GB
Time/ep
10ms
0.32ms
12.20ms
0.379ms
80ms
3.69ms
104ms
3.62ms
Speedup
1x
x0.94
1x
32.19
1x
x
1x
28.73
Table 3 Comparison the test accuracy/F1-miero. memory saving and training time for single epoch on the popular GNN benchmark datasets, more than 2,000 randomized trials have been undertaken. For the large graph, we employ the sampling-based mini-batch training method, where Flickr, Reddit, Yelp, and Amazon all use the hidden layer with 256 dimensions, 2 GCN layers and the GraphSage [aamltoo al. (2017) sampling method to fix 25 first-order neighbors and 10 second-order neighbors, and each dataset is randomly tested 50 times to determine the mean and standard deviation. Cora, Citeseer, Pubmed, and CS employ the full-batch training technique with a 16-dimensional hidden layer, GCN layers and 500 random trials are conducted for each dataset to determine the mean and standard deviation. On the CUDA C++ side, the training time GCINT is measured by invoking INT8 Tensor Core. Our method enables the GNN training process to take use of the GPU INT Tensor Core's considerable computational capacity, enabling relatively large-scale GNN models to be trained fast and affordably. Our code segment is in the Appendix.
5 CONCLUSION AND FUTURE WORK
In this work, propose GCINT, dynamic quantization training approach that adjusts the quanti- zation range based on the sparsity of the graph. It can be employed in the GPU's INT Tensor Core accelerate GCN training and reduce memory consumption without dequantization procedures. Meanwhile, GCINT is independent of the dataset and weight distribution. Experiments show that GCINT has least 10x speedup and 4x memory consumption reduction with negligible accuracy loss compared with training in FP32 Cuda Core. Future work includes (1) implementing GCINT in existing framework; (2) combing GCINT with other memory-efficient training frameworks; (3) evaluating GCINT under other hardware accelerators; (4) evaluating GCINT under other neural networks.
REFERENCES
Léon Bottou. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade, pp. 421-
436. Springer, 2012.

---

Under review as a conference paper at ICLR 2023
PA Bromiley, NA Thacker, and E Souuova-Thacker Shannon entropy, renyi entropy, and informa- tion. Statistics and Inf. Series (2004-004), 9:10 42, 2004.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, et al. Mixed precision training of convolutional neural networks using integer operations. arXiv preprint arXiv:
2018.
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pp. 417-426, 2019.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric.
arXiv preprint 1XXv:1903.02228 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Weiwei Jiang and Jiayun Luo. Graph neural network for traffic forecasting: A survey, Expert Systems with Applications, pp. 117921, 2022.
James M Joyce. Kullback-leibler divergence. In International encyclopedia of statistical science, pp. 720-722. Springer, 2011.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 2016.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 1rXiv:/806.0832 2018.
Zirui Liu, Kaixiong Zhou, Fan Yang. Li Li, Rui Chen, and Xia Hu. Exact: Scalable graph neural networks training via extreme activation compression. In International Conference on Learning Representations, 2021.
Elman Mansimov, Omar Mahmood, Seokho Kang, and Kyunghyun Cho. Molecular geometry pre- diction using deep generative graph neural network. Scientific reports, 91)::111,, 2019.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint (rXiv:2/06.0829: 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Shyam A Tailor, Javier Fernandez-Marquee and Nicholas D Lane. Degree-quant: Quantization- aware training for graph neural networks. arXiv preprint w:Xiv:2008.05000 2020.
Wei Tang, Gang Hua, and Liang Wang. How to train compact binary neural network with high accuracy? In Thirty-First AAAI conference on artificial intelligence, 2017.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys (CSUR), 2020.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. arXiv preprint arXivv1802.04688) 2018.
Hongxia Yang. Aligraph: A comprehensive graph neural network platform. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3165- 3166, 2019.
10

---

Under review as a conference paper at ICLR 2023
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian- sheng Hua. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7308-7316, 2019.
Zehong Zhang, Lifan Chen, Feisheng Zhong, Dingyan Wang, Jiaxin Jiang, Sulin Zhang, Hualiang Jiang, Mingyue Zheng, and Xutong Li. Graph neural network approaches for drug-target interac- tions. Current Opinion Structural Biology, 73:102327, 2022.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: (6006006660 2016.
A APPENDIX
Folass GCINT 1о GEMM (torch. wtw.oscc.cutogre..Peaa....
8otaticme thoo'
def forward (ctx,, input, weight, abit_K, nbit A, abit_G, mbit Int, Int,,bit_ess,ie
#et ght 0 (weight, nbit_l)
output
(input, Weight) outpot. DyO (ontput.)
sev. save for be (Weight,, imput, G[0]_ nbol_B[0], Inl [nL (0lynbit_ssa [0l,nbit__eeeeee [w]) actput.
8atar CilirElon Lomethod backward (ctx, grad output)):
weight input, nbit 1,, abit ,int, abira,nbitw etx.saved tensors abitA int (nbittA(0))
(mbitw[01]
IntPoint int (Int [1nt
ubit G int (mbit G[O])
int (abit k(0])
input_ T =
(input, 1)
deight torca.tsanapese (N
overall_grte torch. (grad output, Weight_ litt N mescch.m((nnuut.. T, grad output) $0.35s
IntPoint:
niff (yQ W]
eveiall_uzal SLQ ubil_ E, ubit_
return aversll sverall_grad, Diff w, None, None, None, None, None, None, None, None
Rolass GCINT 1O AGOREGATE (torch. atotcaac Functio))
@statioms thoa
daf (cLx, imput_ a, input_ b, abit A, A,nti E, Int, molt_aa,nbite int [0])
torch.mm (input a,
if InLPoinL:
output = (yQ(out))))
cax.save for backward (input_a, a, input_b, b, nbit Int [n] nbbit aa [0] ,nbit_ G(o])
return ouLpaL
Sstati
(etx,
o
input a,Intut b, nbit E, Int .nbit aa,nbit G= ctx. saved tensors
nBit à - int (ubit [0])
IntPaint int (Tn):[0])
nbit E (nbit B(U])
abit (ubit G[0])
1) overall qrad = torch (inpuut T, qrad output.)
input torch. torchh
1f IntPoint:
overall StQ(overs') graad, nbit: E, nBit, A) None, overall perall_grad, None, None, None, None,None, Sone,Non.
11

---

Under review as a conference paper at ICLR 2023
olass GCINT L1 (eM((oors..uutogge wtw..aaa.noord.....iiiio. Function):
feethod
der foruand (atx, input, weight, nbat nbit R, nbit nhát R, Int, an,nnhhtt ww,
Keight 2 (weniggh,, phir
Luzch.mm (input, Weight)
output (output))
bWesght input, G[o], R[O], Int[o),rbit raturn eutper
def backwerd(etx, opad_output);
Keigat input, nbit 6, noit ,in,, nostk, ctx.daved
nbitA (nt(nhttA[0])
sbit int (nbitw((0))
Intnt int iint [0])
moit 5 int (ntit G(V1]
(ntttttt_ [(0])
input = torch. tranapose (Anput, 1)
Reight_"" Lorch. Corch. (Reiaghtt,,
overal- qrad torch. tareh.mm(qred output, Weight 2)
torch.mm (input grad output) if TntPoint::
Dilr_N a DyO
overall grad St@(ovezall gead, nbit nbit N)
return overall grad, DifE N, None, None, None, None, None, Nono, None
olass GCINT L1 AGEREGATE (Lorch..ulogreee.. Function):
Jemethodd
def forward input a, input_b, abit n,bitt,, Int, nbit abit_aa,abit
IntPoint int([nt[[))
culput - Lorch.ma (input_a, imput b)
if Int.Pint:
output (ou(tuu)tt)
clz.seve (inpal a, impul b, InL[o] mbil sa[0] mbil G[O1)
zeturn output
@staticmat.od
def backward (ctx, grad output):
input_=,,npi input b, nbit E, Int jubit nh,n G- cbx.saved tensors
nRit_ int anbit_a IntPoint int(Int(o))
nbit I = 1nt (mbit B[O])
bbit_ = (nbit__((O)
input_" T torch. (impttt 0, overall ggad
monch.t grad grad_output) output)
overall_grad (1verralll grad, nbil_K, n8it_A)
return None, overall grad, None, None, None, None, None
ccaas MyDropout awt.orch.utorrrddd. Function):
@staticn ceetiichhhh
def forward (ctx, input, rate):
Rate tensor (rate*torch, random tensor torch. rand shapet saput.shaped [1]) .udda ()
(input.shape) [0], input. [1])) cuda
Mask Mask_tenson tensor torch.ge (random tensor, Rate_tensor) cuda ()
seniooomeeenrrrrro output torch.mi mul (Mask tensor, input) cax. for backward tensoor return output
@staticmethod
def backward (ctx, grad output)
result ctx.saved tensors grad result[0]
overall grad torch.mul (grad output, grad) return overall_graa grad, None
12

---

Under review as a conference paper at ICLR 2023
claass
MyReIu (torch (torch www.agra...ontttiioo....
def forward input, nbit E, Int):
output F. F.relu (input)
ctx.save_ for _oo__aackwardd (output, nbit__[0] Int [n]]
return output @staticmethod
def backward (ctx, grad output):
output ,bbt E, Int= teree tensors
IntPoint (Int [0])
nbit int (nbit_EfDO)) E(o])
output
grad = torch. zeros shape [0] data.. shape [1]) cuda ()
grad = torch.ne ne (grad, Inot equal
overall @eeaallggrad_ torch.mul (grad output, grad)
if IntPoint:
overall grad = overall grad
return overall grad None
def W (w, nbit):
torch. round (w*127) / (torch. max (torch.abe (w) return
def StQ (w, mbitA, nbitt)
A ((2** (nbitA+nbitW))
W torch. clamp (w, A)
torch. round (w/ (2** (nbitw)))
re return
def
DyQ (input) :
kw check (input)
A = int ** (kw) )
torch. clamp (input, -A, A) W torch.round (w / (2 ** (kw ))))
return
def check (input):
reg = []
reg_ nub = []
input = abs (input)
for i in range (1,32)
temp torch.ones simput..ahape [0] input..aaap [1]) (2++i)
temp temp.to ('cuda')
mid = torch. (temp, input)
reg. appand (int (torch. sum (mid)))
if = 1:
peg__uub_... append (xeg
else:
reg nub append [reg[i-1] regli 21)
local pum []
for k in range (7, 72)2)
local pomm.appert (sum (reg mab(k-7:k1)
ar.argmax (local pum)
raturn
13


Reviews:
[{"review_id": "wgFQf86aSjz", "reviewer": "Reviewer_5yWh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes four factors that restrict the applicability of traditional quantization methods in GNNs. Based on that, a specialized and efficient quantization framework is proposed to improve forward, backward, optimizer, and loss functions with the dynamic quantization (DyQ), static quantization (StQ), and weight quantization (Qw ) operators. Experiments show the reduced memory and time complexity with comparable accuracy.\n", "summary_of_the_review": "This paper identifies some nontrivial challenges in quantization of GNNs. However the writing of this paper is hard to follow for a larger audience. I believe it is a work with good potential if it can be better presented. \n", "strength_and_weaknesses": "**Strength** \n- The quantization of GNNs is an important problem, and the paper provides a comprehensive comparison of different quantization methods.\n- Different quantization operators are proposed to handle real and pseudo overflows, and are plugged into the GNN workflow.\n- The empirical results show improvement in efficiency.\n\n**Weaknesses**\n- Though I believe this paper has some nontrivial technical contributions, the writing of this paper needs to be greatly improved to make it accessible for the community. I have tried hard and spent a significant amount of time, but can still get a rough idea without understanding the details of why pseudo flow is a key bottleneck and how it is handled by DyQ. For example, Section 3.1 is hard to follow for a general reader like me: what exactly the pseudo overflow problem is (e.g. when and why $c$ cannot be represented by $2^n-1$), what is the correlation and difference between real and pseudo overflow. Figure 2 is almost impossible to understand for general readers too.\n- It is not clearly justified why dequantization is not needed in this method.\n- The claim that this method can accelerate inference is not supported by experiments.\n- This method is not compared with existing works, for example, the memory reduction compared with EXACT.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JzGC2iEo1mP", "reviewer": "Reviewer_Po7t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to accelerate both training and inference in Graph Neural Networks by quantizing both the\nweights and all the intermediate results (features, errors, gradients) to 8 bits. The proposed training and inference\npipeline can be executed on integer-only hardware.\n\nTo deal with the high dynamic range arising in GNNs due to various node degrees and feature types, this work proposes\nto use a dynamic quantization technique to map 32-bit intermediate results from multiply-and-accumulate steps back to\n8-bits shifting right with a dynamically-determined amount and clipping to 8 bits. The shift count is determined such\nthat the number of outliers resulting in overflow or underflow is minimized.\n\nSome experiments are carried out on 8 GNN benchmarks, matching the accuracy of floating point counterparts. No\ncomparisons against state-of-the-art GNN quantization methods are provided. ", "summary_of_the_review": "This seems to be unfinished work with incremental technical novelty. The manuscript needs a major rewrite to\nmeet the presentation quality standards for publication (see notes on clarity above). At lease one relevant baseline is\nmissing (see e.g. NITI above), and no comparison against any competing work is provided. Overall,\nin its current form, this work does not meet acceptance standards for ICLR.\n", "strength_and_weaknesses": "* Strengths:\n\n  * Due to the scalability issues and online-training demands for, integer-only training and inference is likely to have\n  high practical impact.\n\n* Weaknesses:\n\n  * Technical contribution is incremental (see notes below on novelty).\n  * Clarity could be improved a lot (see remarks below on clarity).\n  * No comparison is provided against any competing method. The only baseline is the floating point pipeline.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "F19p0rDH7I", "reviewer": "Reviewer_o9Df", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work proposes quantization methods for low bit training of GNNs. ", "summary_of_the_review": "The paper is not ready for publication. Additional works are needed to justify the claims and to simplify the algorithms. \n", "strength_and_weaknesses": "Pro: The paper claims to improve training speed by using dynamic quantization. \n\nCon: The solution is complex, which limits its applicability. And technical novelty is rather limited. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject