Neural gradients are near-lognormal: improved quantized  and sparse training
Authors: Brian Chmiel
Liad Ben-Uri
Moran Shkolnik
Elad Hoffer
Ron Banner
Daniel Soudry
Venue: ICLR 2021 Poster (2021)

Abstract: While training can mostly be accelerated by reducing the time needed to propagate neural gradients (loss gradients with respect to the intermediate neural layer outputs) back throughout the model, most previous works focus on the quantization/pruning of weights and activations. These methods are often not applicable to neural gradients, which have very different statistical properties. Distinguished from weights and activations, we find that the distribution of neural gradients is approximately lognormal. Considering this, we suggest two closed-form analytical methods to reduce the computational and memory burdens of neural gradients. The first method optimizes the floating-point format and scale of the gradients. The second method accurately sets sparsity thresholds for gradient pruning.  Each method achieves state-of-the-art results on ImageNet. To the best of our knowledge, this paper is the first to (1) quantize the gradients to 6-bit floating-point formats, or (2) achieve up to 85% gradient sparsity --- in each case without accuracy degradation.
Reference implementation accompanies the paper in the supplementary material.

Published as a conference paper at ICLR 2021
NEURAL GRADIENTS ARE NEAR-LOGNORMAN IMPROVED QUANTIZED AND SPARSE TRAINING
Brian Chmiel for Liad Ben-Uri Moran Shkolnik to
Elad Hoffer
Ron Banner
Daniel Soudry
Haabnna Labs An Intel company, Caesarea, Israel. *Department of Electrical Engineering
Technion, Haifa, Israel
bcchiel
rahkolnik
eho ffer
rbanner @habana a
1iadge2
daniel. soudry ry @egmal i com
ABSTRACT
While training can mostly be accelerated by reducing the time needed propa- gate neural gradients (loss gradients with respect to the intermediate neural layer outputs) back throughout the model, most previous works focus on the quantiza- tion/pruning of weights and activations. These methods are often not applicable to neural gradients, which have very different statistical properties. Distinguished from weights and activations, we find that the distribution of neural gradients is approximately lognormal. Considering this, we suggest two closed-form analytical methods to reduce the computational and memory burdens of neural gradients. The first method optimizes the floating-poin format and scale the gradients. The sec- ond method accurately sets sparsity thresholds for gradient pruming. Each method achieves state-of-ther of-thc-arttttttt results on ImageNet. To the best of our knowledge, this paper is the first (1) quantize the gradients 6-bit doating-point formats, (2) achieve up to 85% % gradient sparsity - in each case without accuracy degradation. Reference implementation accompanies the paper in the supplementary material.
INTRODUCTION
Neural gradients are used in the training process of deep networks backpropagate the error-gradient throughout the model, thus allowing to compute the required weight updates. As these neural gradients are needed for substantial ratio of the underlying computations (about 3), compressing them can alleviate throughhou requirements and accelcrate the training process.
Compression of neural gradients reduce the memory footprint for the intermediate calculation and the bandwidth of data transfer inside the HW accelerator. Moreover, in term of distributed training in model parallelism the neural gradients are one of the main bottlenecks that need to be transferred
between devices (Rong et al. 2020 Gupta et al. 2020).
Many previous works (Banner al. 2019 Fang et al. 2020) compress tensors such as weights and activations by approximating their distributions using an analytically tractable density. These works often assume a bell-shaped distribution such as Gaussian or Laplace distributions, which have been reported to fail for neural gradients (Ye et al.) 2019]. One key observation in this paper is that neural gradient distributions are heavy-tailed, fundamentally different from the light-tailed distributions of weights and activations. Further statistical and distributional tests reveal gradient magnitudes follow a lognormal distribution.
Adopting this lognormal observation, paper suggests two main applications quantization and pruning, used to reduce the computational and memory burden of neural gradicnts. To tackle these challenges, first formalize the problems and find closed-form expressions that enable predict the optimal quantization and pruning policies. These measures are easy to use and depend only on the estimated lognormal parameters.
"qqual contribution.

---

Published as a conference paper at ICLR 2021
In Figure summarize these applications and their derivation. The first application the lognormal prior to enabling low-preete now-precision doating-point point (FP) quantization the gradients. Here we optimize two tasks. The first task is to find partition between mantissa and exponent bit-widths that minimizes quantization noise for a given n bit FP gradient representation. The second task is scale these gradients so that they would be properly represented within limited dynamic range (distance between the maximum and minimum that FP format can represent). We provide useful insights that make empirically-bai based heuristics such as loss scaling (Micikevicius et 2018) more grounded approach with theoretical basis. Optimizing both tasks we obtain state-of- ofte-of-the-ese ff--the-rt art results for FP quantization of the neural gradients. The second application performs accurate and predictable stochastic pruning of gradients on the fly. which results in two state-of-the- pruning schemes. The first translates desired sparsity level into an accurate threshold, and the other enables combined use of different sparsity levels at different layers (heterogeneous sparsity).
Neural gradients distribution (lognormal)
Empirical estimate:
Quantization
Pruning
Analytically
derived measures:
Relative
Cosine similarity
Sparsity
error
Applications, by optimizing measures:
Mantissa-
Gradient scaling
Stochastic pruning
Heterogenous
exponent partitioning
pruning
Figure/ High-level overview of the different methods developed in this work. Adopting the lognormal observation of the neural gradients, we suggest three different measures we can analytically optimize: two for pruning and one for quantization. These analytical measures are used in four different schemes.
2 RELATED WORK
Quantization and pruning of neural networks have seen tremendous amount of works (e.g., Nahshan
al. (2019); Choi al. (018): Baskin al. (2018); Louizos al. (2018); Franklee & Carbin (2018);
Cambier et al. 22020) aiming reduce both bandwidth and memory footprint, as well as computation
time. Most of these methods focus on the quantization or pruning of the weights activations
forward path (Banner et al. 2019 Nahshan et al. 2019) or the weight gradients (Bernstein al.
2018: Alistarh et al. 2016) in the backward path. So far, neural gradients got less attention. Some of systemic and rigorous statistical approach to optimize various distortion measures. For example, [Banner et al. (2019) used the normal distributional assumption (of weights and activations) analytically minimize the mean-squared quantization error. Our work follows a similar line to rigorously optimize similar performance measures for quantization and pruning of gradient distributions, which are different from that of the weights and activations.
these methods (Banner al. 2019 Ye et al., 2019 Fang et al.) 2020)
Gradient Quantization. While a lot of research focused on the quantization of weights and activa- tions for inference (Krishnamoorthi] 2018 Choi et al., 2018) Jain et al.) 22020), there were also major advances in quantization during training, many them faced difficulty trying to represent the high dynamic range of the gradients (Banner et al. 2018) Wu et al. 2018). Cambier et al. (2020) suggests keeping for each tensor shift and scale in full precision numbers to make them in FPS format dynamic range. Mantissa vs exponent allocation of the available bits has proven crucial in deep learning workloads, where for example, BF16 (1-8-7: sign-exponent cean--epooett-mantiisttt has shown greater success compared to traditional FP16 (1-5-10) format due to wider dynamic range (Henry et al. 2019 Kalamkar et al.,, 2019). Research over required format and trade-offs in exponent versus mantissa is on-going with growing interest over lower precision representations such FP8. Some works have
2

---

Published as a conference paper at ICLR 2021
explored using different FP8 formats: Wang et al. (2018) has used (1-5-2), while (Sun al.
2019)
suggests using one type of FP8 format for the forward (1-4-3), and a different FP8 format (1-3-2) for the backward pass, after empirically assessing the different possible formats. Additionally, with
the growing usage of FP16 mixed-precision training, researchers and practitioners faced the need to
loss scaling (Mieikevicius al., 2018), adjusting the tensor distribution avoid over/under-flow. This procedure required, in some cases, an intensive parameter search and was not guaranteed to succeed in all cases.
back-propagation "meProp" (Sun al., 2017) prunes the smallest absolute-valuce valued entries of the neural gradients on the fly, using the top-k algorithm. Works following it, replaced pruning with quantization to induce sparsity (Wiedemann al. (020), or used top-k pruning as well on the copies of weights and activations used in back-prop (Aamir Raihan & Aamodi 2020). Ye et al. (2019), inspired by conventional unbiased estimators like stochastic rounding, suggested "stochastic pruning". reaching higher sparsity levels. Yet, the authors assumed the gradients are normally distributed, leading to incorrect estimation of the threshold and a large difference between the required sparsity and the one obtained. As we shall see later, using the correct statistical distribution model is essential to determine the proper threshold.
Gradient pruning. Focusing on the computationally-inneee
3 NEURAL GRADIENTS DISTRIBUTION
Many prior works (Banner al. 2019 Bernstein et al. 2018) take the assumption that tensor data e.g.. weights (W) and activations (AA) is sampled from a Gaussian distribution. Recently, Ye et al. (2019); Wiedemann et al. (2020) used the same assumption for the distribution of neural gradients VA. In
the section, we discuss this assumption. We show that neural gradients are better approximated by lognormal distributions, i.c., the gradient logarithm values are normally distributed, as opposed the gradients themselves.
In Fig. 2 we plot the histogram of neural gradient magnitudes linear and log scales in one layer of 8 ImageNet dataset. At a linear scale, the distribution has few gradients of huge magnitude and great many gradients of small magnitudes (Fig 2a). Plotting the histogram on a logarithmic scale reveals distribution close to a symmetric normal (Gaussian) distribution (Fig 2b). This is the hallmark of the lognormal distribution. Finally, when plotting the theoretical quantiles of the normal distribution against the quantiles of the gradient distribution (Q-Q plot), we see that the points follow a strongly nonlinear pattern in Fig 2c, suggesting that the data is not distributed as a standard normal distribution. Note that in the Q-Q plot for lognormal distribution (Fig 2d), almost all points lie on a straight line.
We further estimate the goodness of fit of the neural gradients to normal and lognormal distributions. To that end, we measure the static distance (largest vertical line) between the cumulative distribution function (CDF) of the empirically observed distribution and the CDF of the reference distribution (also known as Kolmogorov- -miirnov--minovvvvoooo test (Smirnov) 1948)). For each model and dataset in Table [] we calculate the average (across all layers) of the static distance to normal and lognormal distributions. The analysis is performed on the absolute value of the gradients, excluding the zero-valued entries. Note that lognormal distribution gets better fit. Additional statistical distributions in Appendix A.
Normal
Data quantiles E
Lognormal
Daza 
80
20
20
0.001 0.022
0.03
2- 2-15 2-30 2-16
2-10 2
50
100
Neural gradients
Log (neural gradients
Theoretical quantiles
Theoretical quantiles
Figure 2: Identifying the distribution of neural gradients (normal vs. lognormal): (a) probability density function of gradient magnitudes; (b) the same histogram a logarithmic scale notice that the shape looks very much like the symmetrical shape of the regular normal distribution; (c) quantiles of gradient magnitudes against the quantiles of a normal distribution; and (d) quantiles of gradient magnitudes against the quantiles of a lognormal distribution; Additional layers' distribution in Fig. A.2] In order emphasize the difference between neural gradients and weights gradients we show the quantiles of weights gradient in Fig. A.23
3

---

Published as a conference paper at ICLR 2021
Table Mean + std (p-value) over all layers over time of KS on different models and datasets for normal and lognormal distribution. Notice the lognormal distribution gets the higher fit across all models. We compare additional distributions such as Laplace, uniform, Cauchy and loglaplace in [n.e]dii in Table A.2 we show there is good match to the lognormal through training (not only on average), and in Fig. A.3 we show this holds also for specific layers over time.
Model
Distribution
(Dataset)
BERT (CoLa)
BERT (MRPC)
ResNet18 (ImageNet)
MobileNetV2 (ImageNet)
VGG16 (ImageNet)
DenseNet1 21 (ImageNet)
0.46=0.02 (2 10-1)
0.39:+0.04
0.38-10.1
0.22-40.09
0.35
0.334-0.1
Normal
(5 (5-10-9)
(3 10
(5 10-00-)
(3 10
(5 10-10-9)
0.05++0.002 (0.28)
0.04+-0.002
0.02++0.002
0.07+00003
0.06++0.002
0.05++0.011
Lognormal
(0.23)
(0.26)
(0.18)
(0.31)
(0.29)
APPLICATION I OPTIMAL FLOATING-POINT QUANTIZATION
Pooating-point point (FP) representations cover very wide dynamic range with relatively small number of digits. The mantissa-exponent tradeoff control the balance between the dynamic range of numbers that can be represented (exponent) the precision of these numbers (mantissa). This dynamic range is especially important for the heavy-tailed distributions that characterize neural gradients. In this section, study the optimal characteristics of the FP quantizer.
Floating
4.1 PROBLEM FORMULATION
We can decompose any positive real value a E R as follows:
1,
Rea
x = 2 2 21 [nx
[n
(I)
where M E 2) is the mantissa and E E Z the exponent. Given N bits, we allocate 1 bit for the sign and seek the optimal allocation of nı bits to M and m2 bits to E, such that n1 + n2 N Accordingly, we define the quantized 2g as:
Faaax
E Emax
Fig
Mg 2
-Emax S E < Emax
F. Fmux
where Emax 2n2-1 and M, is the quantized mantissa with the range [1, 2) divided into n quantization levels with spacing of A
2
Finally, we measure the relative error between the FP number xg and the real number x, which is simply the difference between the two numbers divided by the real number (Widrow & Kollár 2008):
Fig
17(11. R22)
(3)
4.2 ANALYTICAL DERIVATION OF THE RELATIVE ERROR
We assume that Lognormal(p,, o²). Note that E = [n R In Nu,, o²). In Appendix A.4
we split the range into three parts according to E: (i) Emax S E < Emaxi (ii) E 2 Emaxi (iii)
E < Emax. and calculate the expected contribution for each term. A closed-form formula for the expected relative error could be obtained as follows:
E (1 n22)
24 [maax)
+ 2 Fmaar
e
2 (22(2)
erf
a lu2
+
Emax
-1)
8 8 (2) (2 - 1)
V2
/20
(4)
Emax
Emax
erf
+
/2a
2
4

---

Published as a conference paper at ICLR 2021
where ((x) is the CDF of (O,,)) 1). In Fig. 3a show that analytical results stated by Eq. (4 are in good agreement with simulations for FP8 with various number of exponent bits. Simulations were obtained by quantizing 10,000 values, generated from lognormal distribution with = 1,3,5. 5.
4.3 THE OPTIMAL MANTISSA SENISSSA--XPONEEEE EXPONENT REPRESENTATION
The relative error in Eq. depends on the scale parameter o, the number of bits of the mantissa an and exponent 122, respectively (the latter through Fmaa = 2n2-1). Given any N-bit FP format, we wish find a mantissa-exponent partition that minimizes the expected relative error such that 71 + = N 1. Minimizing Eq. yields this optimal partition. To do so we set equate the derivative to zero and solve. The computational cost such solution is negligible (details are in Appendix A.4.4). This allocation depends on N and In Fig. 3b we show the optimal allocations for N = 5, 6, 7, bits and a E 8]. In Fig. 3d we show the FP format obtained by solving Eq. 3 with normal distribution assumption for neural gradients, which leads sub-optimal format as shown in Table 3
N R2 1,
The full solution can be found in Appendix A.5
Lognormal
Normal
Relative Error
simulation analysis
0.6
5
5
1p5
1p5
0.2
1p6
Tpc
>
fp7
lp7
0.0
Imagelet
fp8
1
ImageNet
tpé
Exponent bits (nz)
7
8
4
5
6
7
(a)
(b)
(c)
Figure 3: (a) Expected relative error function of n2 for FP8 with lognormal distribution. The simulation in good agreement with the analytical results stated by Eq. (4). (b) Ideal bit allocation for the exponent (E*) as function of a, using Eq. Notice that a varies between datasets, imposing different optimal allocations. (c) Ideal bit allocation assuming a normal distribution, which leads wrong FP formats, e.g., the number of bits assigned to the exponent decreases when (dynamic range) increases.
Table summarizes the statistical analysis applied for various floating point formats. Empirical observations prove that gradient distributions have a range of [3,5.5] and [2.5, 4.5] in ImageNet and determine the optimal mantissa-exponen partition for FP4-FP8. In Section [6, we use these partitions to train ImageNet and Cifar models at reduced precision (bit-width lower than 8), and show empirically that these partitions provide the best results.
Cifar100 datasets respectively, which
Table 2: Ideal sign-exponent segn--eponeett-antiise mantissa bit allocations based on the optimization of Equation and the gradient statisties of ResNet18. ResNet101 and SqueezeNet in Cifar100 and ImageNet datasets. The analysis matches the previously suggested allocation by Sun et al. (2019) for FP8, showing a clear benefit of 1-5-2 for gradient quantization.
Models
Dataset
a Range
FP4
FP5
FP6
FP7
FP8
ResNet18, ResNet101
Cifar1
2.5-4.5
1-3-0
1-4-0
1-4-1
1-4-2
1-5-2
ResNet18, SqueezeNet
ImageNet
3-5.5
1-3-0
1-4-0
1-5-0
1-5-1
1-5-2
4.4 PER-LAYER GRADIENT SCALING
The use of loss scaling Micikeviiius et al. 2018 key to the quantization of gradients using low precision FP formats. The idea to shift the neural gradients' dynamic range to fit the floating-poin point range thus avoiding possible underflows. Loss scaling is usually performed by multiplying the loss value by large constant and then dividing weight gradients by the same value after back back-propagatioo and before any update has taken place. As gradient distribution changes across training. dynamic loss scaling is sometimes needed. In this setting, gradients are monitored for any overflow underflow
5

---

Published as a conference paper at ICLR 2021
that may occur. Upon this occurrence, gradients discarded for that step and loss scale value is changed to combat the phenomena (either increased or decreased heuristically).
Choosing either fixed dynamic global loss scale that can fit across all the layers is challenging and may prove impossible for low precision FP formats that further reduce the dynamic range. Fig. A..5 show the standard deviation of the gradient the log scale for different transformer layers. The high variability in a between the layers makes the choice of one global loss scale unlikely. The variation in gradient statistics across layers may also explain the need for previous hybrid approaches that kept some layers at higher precision. For example, [Snn et al. (2019) reported that when training ResNet18 18 models using FP8, they had to keep the first and last layers FP16 precision. Fig. A.6 suggests a reason - the first and last layers exhibit std that is very different from the other layers (i.e., they require different gradient scaling) ). Cambier et al. (2020) showed the low results achieved using global loss scale in FP8 format in transformer and suggested full precision (expensive) computational operation of rescaling and shifting the neural gradients to FP8 dynamic range.
We therefore suggest to use per-layer gradient scale, instead of global loss scale. As detailed in Appendix A.6] our gradient scaling method keeps the largest gradients representable but sacrifices the smallest gradients. These tiny gradients can be pruned without significantly distorting the original tensor because: (I) the lognormal distribution suggests that gradients of tiny magnitude are relatively scarce; (2) such tiny gradients typically less significant in training compared to the larger gradients. Pseudo-code appears Algorithm
5 APPLICATION II STOCHASTIC PRUNING
Inspired by conventional unbiased estimators for quantization such as "stochastic rounding", re- searchers have recently proposed "stochastic pruning" (Ye et al. 2019), unbiased pruning method that introduces zero bias on expectation.
Given threshold a, sample uniform variable U10, 1] and prune follows:
( a
|x| >
Ta.s (I) = sign(x)
a S |x|| S
(5)
|x| < a
The method is graphically illustrated in Figure 4 Note that all values in the range [-o, al need to be mapped to only one of the three possible values (0, +o)). Their increased frequency of occurrence can be used to design a custom encoding method to compress their representation. Appendix[A.10 we propose encoding method with compression ratio equivalent quantizing 4 bits 80% sparsity or to only 2 bits at 90% sparsity.
5.1 PROBLEM FORMULATION
Our goal would be find an analytical expression for proper threshold a that induces the desired sparsity S using stochastic pruning. Specifically, let be random variable with known distribution and S given sparsity level (0 S < 1). Using stochastic pruning, with U[0 1]. we obtain:
S= Eg /
f(x) dx
5.2 ANALYTICAL DERIVATION OF SPARSITY
Understanding the lognormal distribution of the gradients, Eq. (6) is simply the CDF of lognormal distribution a 6, that
1 eee erf S = 2
In (a (mnaa [0t(
1
In (T)
ex
erf
dr
V2o
2
/2a
a
The complete solution for this integral can be found in Appendix
A.3
resulting in:
1
e
σ
In ((())
(a)
a
In a
S = 2
erf
+
erf
e
(8)
2x
/2
V20
c
/20

---

Published as a conference paper at ICLR 2021
To. (Vx)
Stachastic praning
t- [0.1]
Figure 4: Effect of stochastic pruning on a lognormally distributed tensor. The threshold can be found from Eq. (6) according to the gradients' distribution. is the ratio of values mapped to 0, notice that a large fraction of values, all values ar e S x S are mapped to ta two values that can have special encoding. thus reducing the tensors' memory footprint - details in Appendix A.
Eq. (8) can easily be solved numerically to find a. As shown Fig. A. the lognormal parameters and of the gradients' distribution at each layer are pretty stable throughout the training, which allows sampling a, f and calculate the threshold a not very frequently. In practice, we perform this procedure only once per epoch while achieving stable sparsity levels throughout the training. Moreover, the computational complexity of solving Eq. (8) is negligible (empirically it converges in few iterations); further details are found in Appendix A.7
5.3 HETEROGENEOLS SPARSITY ALLOCATION
We show in Fig. A.9a that the angle between the tensors before and after the stochastic pruning (measured by the cosine similarity) can serve an important proxy to the overall validation accuracy achieved. Interestingly, using the cosine similarity, we observed that stochastic pruning takes different toll from the different layers, i.e. pruning all layers to the same sparsity level damages some of them more than others. This phenomenon can be explained and assessed by analyzing the cosine similarity of lognormal distribution, where the difference between the layers is the parameters the distribution. We derive the cosine similarity another analytical measure and propose algorithm that better preserves the cosine similarity of some of the layers, by decreasing their sparsity level, while increasing the sparsity level of other layers maintaining the overall sparsity budget (mean sparsity of all the layers). Further details can be found in Appendix AA.9 This allows us to increase the sparsity level while preserving accuracy, results be seen in Section 6
6 EXPERIMENTS
In this section, we evaluate the methods and predictions above, all stemming from the lognormal dis- tribution of the neural gradients, for the two suggested applications: floating point format quantization and stochastic pruning. Experiments, details, and additional results appear in Appendix A.
Floating point format. Table 3 we show the results of different allocations between exponent and mantissa for different FP formats in Cifar100 and ImageNet dataset. We quantize all convolutional layers' gradients, unlike previous methods (Sun et al., 2019) that keep part of them in FP32. All results were achieved using the suggested gradient scaling, where the mean is sampled once every epoch. For all FP formats, the results fit the analysis in Table In contrast, we get sub-optimal FP format if instead solve Eq. (3) with normal distribution assumption.
Per-layer gradient scaling In Table 4 we compare the suggested gradient scaling with static and dynamic (Micikevicius et al.) 2018) global loss scaling. We clipped the values to maximum/ maximum//minimmm FP representation for our method and the "static" method to avoid overflow/underflow, respectively. On the other hand, the dynamic method (Micikevicius et al. 2018) clips the values to adjust the scale in response to overflow/underrloww in the weights updates. Our suggested layer-wise gradient scaling achieves better results, highlighting that one global loss scale for the entire network might be too restrictive practice. In Fig. A.7 we show the variability of the suggested gradient scale across different layers. Lastly, to verify that gradient quantization is the main bottleneck, we also quantized the weights and activations to INT4 and found a minor degradation in accuracy (0.3%).
7

---

Published as a conference paper at ICLR 2021
Table Floating-poin point formats for neural gradient representations. The table reports the validation accuracy of ResNet1 8 on ImageNet & 100,r100, ResNet101 on Cifar100 and SqueezeNet on Ima- geNet. E" denotes the optimal exponent bit-width, which minimizes the expected relative error of Eq. 4 N/A refers to cases of invalid format (e.g.. 1-5-0 is the optimal FP6 format for ImageNet, but FP6 cannot have E = 6 exponent bits). Note that all cases, the formats that are predicted by Eq. (4 to minimize the relative error also provide the highest accuracies (bolded), We marked (by !) the results for the optimal format assuming normal distribution for neural gradients in Eq.
Dataset
Model
a Range
Bascline
FP
E*
E*+1
E"-1
E*
FP5
64.0%
N/A
58.9%
26.6%
ResNet18
2.5 4.5
64.9%
FP6
64.9% %
64.6%
59.7%
28.6%
C0
FP5
70.4%
N/A
66.5%
35%
ResNet101
2.5-4.5
71.3%
FP6
70.97 %
70.82%
67.5%
42.7%
FP6
70.0%
N/A
67.1%
30.8%
ResNet18
5.5
70.4%
FP7
70.4%
70.1%
66.7%
47.5%
ImageNet
FP5
55.2%
N/A
47.3%
33.2%
SqueezeNet
3 5,5
58.19
FP6
57.8%
N/A
56.1% %
54.3%
Table Comparison the suggested gradient scaling against different global fixed loss scaling and global dynamic loss scaling (Micikevicius et al. 2018) in ResNet18, on the ImageNet dataset for the optimal format in FP4 (1-3-0).
Gradient
Dynamic Global
Static (Global)
Baseline
scaling (Ours)
213
2
2
26
2
2
2
70.4%
64.8% %
3%
26.7%
26.4%
38.7%
41.5%
42.2%
54.9%
53.9%
Stochastic pruning. In Section 6 we compare the proposed methods for homogeneous and hetero- geneous stochastic pruning against SAW (Aamir Raihan & Aamodi 2020) that uses "top-k" pruning and ATP Ye et al.] 2019) that also uses stochastic pruning but assumes the gradients are normally distributed. Notice that stochastic pruning outperforms the non non-stochastic top-k, and that the heteroge- neous version surpasses the homogeneous one. The validation accuracy during training for different sparsity levels and different datasets can be found Fig. A. In Section we demonstrate our methods' ability to produce the required sparsity level, for both the homogeneous and heterogeneous versions. In contrast, the sparsity is not accurate for the baseline methods: (1) finding the threshold using top-k and then applying stochastic pruning, and (2) using ATP (Ye et al. 2019), which assumes a normal distribution. In Fig. A. we see how the sparsity inaccuracy occurs at all layers, and in Fig. A.la we see how other distributions (not lognormal) cause inaccuracy. This strengthens the importance of using the correct distribution of the neural gradients in Eq. (6).
7 SUMMARY
We evaluated the distribution of neural gradients and showed they can be well-approximated as lognormal distribution. We use this distribution to analytically derive accurate measures (e.g., sparsity and local distortion metrics), useful for the two following applications:
Quantization. We found the optimal bit allocation to the mantissa and exponent for floating gradient representation, explaining prior results for FP8 and paving the way towards lower accuracy representations. We suggest using per-layer gradient scale and find its optimal value, preventing under/over-flor in scenarios that challenged prior methods required intensive parameter search. Combining both methods, we trained using low precision neural gradients on ImageNet and achieved, for the first time, noticeable validation accuracy degradation with FP7 and FP6.
8

---

Published as a conference paper at ICLR 2021
76
95
90
[Lognormal
Accuracy (%6
70
72
Rchieved sparsity
80
85
-X- (Norma))
(ognermal)
Dura Heterogenouu
ATP
68
75
-
Topk stochactic
statittiall proo)
70
66
65
64
(sochaskc) (sactasid ATP clotheer))
60
62
569 (dete minetto
55
60
50
40
50
60
70 Sparsity (%)
80
90
50
55
60
65
70
75
80
85
90
95
Required sparsity (%)
Figure 5: (Left) Comparison of the homogeneous and heterogeneous stochastic pruning methods against pruning with SAW (Aamir Raihan & Aamodi 2020), which uses the "top-k" method and stochastic pruning, but assumes the neural gradients
ATP (Ye et al. 2019), which also normally distributed. These are compared on 3 different architectures (ResNet18, VGG16 and DenseNet121). Our proposed methods achieve higher sparsity while maintaining baseline validation accuracy. (Right) Comparison of the required and achieved sparsity of our method, ATP (Ye et al. 2019) and applying "top-k" followed by stochastic pruning for ResNet-15 18 (additional models in Fig. A.244.. The validation accuracy for our method is never less than the accuracy of the other methods for the same achieved sparsity. Notice the large deviation in the other methods between the achieved and required sparsity. This emphasizes the importance of using the correct distribution in order to both enjoy the improved performance of stochastic pruming over regular "top-k" and maintain the ability to fully control the achieved sparsity. Additional details and results in Appendix A.12
Pruning. We can stochastic pruning prune the neural gradients during training precisely to a predetermined sparsity level, with minimal overhead computation. We show that this form of stochastic pruning superior deterministic pruning. Specifically, we have achieved up 80% gradient sparsity without hurting validation accuracy (ResNet18 on ImageNet) using homogeneous sparsity level for all layers. We also show that the uniform sparsity method sub-optimal with respect to an analytical еттоr measure we derive (the cosine similarity), and suggest allocating different sparsity levels to the different layers to preserve better. We suggest and test an algorithm for this allocation allowing for more aggressive overall pruning, achieving 85% gradient sparsity while preserving baseline accuracy, and reaching nearly 90% with less than 0.3% accuracy degradation.
Why we get lognormal distribution? From the central limit theorem, normal distributions uni- versally arise from the sum of many random variables, while lognormal distributions universally arise as the product of many random variables. This might suggest that, we backpropgate the neural gradients through the network, these gradients have a few dominant paths, so most significant operations are products, rather than summations (i.e., along these paths: effective depth >> effective width). Recent work (Hanin & Nica 2018) made this explanation rigorous at initialization, in the limit where both the width and depth of the neural network jointly go to infinity. However, suspect this explanation is only part of a more nuanced picture. Understanding this is an interesting direction for future research.
REFERENCES
Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training. arXiv:2001.01969, 2020.
arXiv preprint
Dan Alistarh, Demjan Grubic, Jungshian Li, Ryota Tomioka, and M. Vojnovic. Qsgd: Communication- optimal stochastic gradient descent, with applications to training neural networks. 2016.
R. Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In NewrIPS, 2019.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NewrIPS, 2018.

---

Published as a conference paper at ICLR 2021
Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii. Eli Schwartz, Raja Girayes, Avi Mendelson, and Alexander M Bronstein. Nice: Noise injection and clamping estimation for neural network quantization. arXiv preprint arX1v:1810.00162,, 2018. URL http://aaxiv..oe
abs/ /tt/18100 60162
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: compressed optimisation for non-convex problems. In ICML, 2018.
Gerard Biau and David M Mason. High-dimensional norms, In Mathematical Statistics and Limit Theorems, pp. 21-40. Springer, 2015.
Léopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz H. Elibol, and Hanlin Tang. Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. In ICLR, 2020.
Jungwook Choi, Pierce I-Jen Chuang. Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN). arXiv preprint arXlv:1807.06964 2018. URL 06964
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arviv: 1805.0688, 2018. URL httt: rxviv wttg/bb///ii 06085
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jun Fang, Ali Shafiee, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph Hassoun. Post-training piecewise linear quantization for deep neural networks. In ECCV, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018.
Vipul Gupta, Dhruv Choudhary, P. Tang. Xiaohan Wei, X. Wang. Yuzhen Huang, A. Kejariwal, K. Ramchandran, and M. W. Mahoney. Fast distributed training of deep neural networks: Dynamic communication thresholding for model and data parallelism. ArXiv, htss/200000889, 2020.
B. Hanin and M. Nica. Products of many large random matrices and gradients in deep neural networks. Communications Mathematical Physics, 376:287-322, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Greg Henry, Ping Tak Peter Tang, and Alexander Heinecke. Leveraging the bfloat16 artificial intelligence datatype for higher-precision computations. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp. 69-76. IEEE, 2019.
C. A. R. Hoare. Algorithm 65: Find. Commun. ACM, 4(7:3112-233 July 1961. ISSN 0001-0782. doi: 10.1145/366222.366447 URL httttt:://oi..rrret 1ttp:/11..111333663313 0111662226666441 366647
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269 2017.
Sambhav Jain, Albert Gural, Michael Wu, and Chris H. Dick. Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. In Machine Learning and Systems, 2020.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint arXiv: 12322, 2019. URL http::/aaae or. abs 905. 12322
10

---

Published as a conference paper at ICLR 2021
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018. URL htttt:://aaii.../// org/ abs/1806.083
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. In ICLR, 2018.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In ICLR, 2018.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint arXiv:1911.07/90 2019. / wbwpor///aaaaaaaaaaa /bt.19....11190
Haidong Rong. Yangzihao Wang, F. Zhou, Junjie Zhai, Haiyang Wu, R. Lan, F. Li, H. Zhang,
Y. Yang, Zhenyu Guo, and Wang. Distributed equivalent substitution training for large-scale recommender systems. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development Information Retrieval, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs//409. 1556, 2015.
N. Smirnov. Table for estimating the goodness of fit of empirical distributions. Ann. Math. Statist. 19 (1948), no. 2, 279-281., 1948.
1177730256]
Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In NeurIPS, 2019.
Xu Sun, Xuancheng Ren, Shuming Ma, and Houfeng Wang. meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In ICML, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Naigang Wang, Jungwook Choi, D. Brand, Chia-Yo Yu Chen, and K. Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In NeurlPS, 2018
Bernard Widrow and Istvá Kollár. Quantization noise: roundoff emror in digital computation, signal processing, control, and communications, 2008, 2008
S. Wiedemann, Temesgen Mehari, Kevin Kepp, and W. Samek. Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp- 3096-3104, 2020.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. ICLR, 2018.
Xucheng Ye, P. Dai, Luo, X. Guo, Y. Qi. Jianlei Yang, and Yiran Chen. Accelerating cnn training by pruning activation gradients. arXiv preprint arXiv: 1008.099000073. 2019. URL http: 7/arxiv. wwwg/bbb/190..... 00173


Reviews:
[{"review_id": "ZEHQF8Aji10", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors show that gradients from backpropagation are lognormally distributed empirically across several popular architectures like (Bert, ResNet18, MobileNetV2, VGG16, DenseNet121) over widely used datasets (CoLa, MRPC, ImageNet and CIFAR100). Using this result, they propose schemes for gradient quantization and gradient pruning that are theoretically principled and outperform existing methods in the literature. \n\nI was very impressed by the principled nature of the authors' approach. Proving the lognormal distribution using a KS test and using it elegantly to find analytical formulations for optimal bit division in quantization and threshold parameter for sparsity in gradient pruning was masterful. \n\n* In the appendix, the KS test shows that logLaplace distribution is also a good fit for the gradient distribution. Can the authors provide any intuition as to why a log normal might be better? \n\n* Clearly for pruning, logNormal distribution seems to work better than logLaplace. It’d be interesting to see the results of using LogLaplace distribution for quantization and verify that it yields worse results there too.\n\n* Can the authors elaborate on time/memory complexity, and how well their methods made improvements for training time?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VnCDbkewfqV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Following the author response, I have updated my score to an accept. The key contribution is likely to be of interest to the ICLR community and the paper is well executed. \n\n---\n\nThe paper proposes a principled approach to the compression of intermediate gradients by fitting a log-normal distribution to the individual derivatives, which they show is a good fit for the studied models. Although simple a posteriori, the proposed approach is elegant and the arguments are presently clearly and well illustrated.\n\nThe weak points of the submission are a lack of clarity for some of the assertions and a lack of motivating example or application. The current writing takes for granted that quantizing intermediate gradients is good. The claimed benefits of quantization is to \"alleviate data-throughput requirements and accelerate the training process\"\nand \"reduce both bandwidth and memory footprint, as well as computation time\". Those benefits are more obvious in a distributed setting, where the main bottleneck to training is the communication of gradients. This reasoning does not directly apply to intermediate gradients. Quantization and low-prevision might very well provide those benefits in the non-distributed setting, but the current exposition does not make it concrete, which might limit its impact.\n\nMy initial recommendation is towards acceptance if those points and some smaller points below can be adressed during the discussion period. The submission is well executed and its contribution is likely to be of further theoretical and practical interest.\n\nSmaller points;\n- The introduction mentions that 2/3 of the underlying computation involve intermediate gradients. Please make this number concrete through an example, as this seems highly dependent on the architecture and data type.\n- One of the main points of the introduction is that the intermediate gradients are fundamentally different from weight gradients. This assertion should be supported by data in the submission, maybe as an addition to Figure 2.\n- The term `neural gradient` being undefined in the abstract and title is an issue. While its meaning can be inferred, it is not standard terminology and needs to be defined in the abstract or replaced.\n- The term `dynamic range` would benefit from a technical definition.\n\n\n\n\nMinor points, not part of the main decision\n- Something went wrong in Equation (7).  $\\tau$ seems to be used instead of what was originally planed as $\\epsilon'$?\n- Some `\\refs` seem broken in Section 4.3 as they point _to_ section 4.3\n- The acronym `BF16` does not seem defined in text.\n- The value of the Kolmogorov–Smirnov test and Table 1 is limited by a lack of interpretability, since it is not aimed at a p-value. Adding more candidate distributions to communicate the range of possible values would help. \n- A short definition of the tradeoff between mantissa and exponent might help a general audience.\n- Some of the references point to preprint versions when the work has been published, for example [Cambier et al. 2020], or does not give appropriate bibliographical information, for example [Choi et al. 2018a]. Please take the time ", "extra_scores": {}, "extra_text": {}}, {"review_id": "S3mlUgIugA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis work proposed a very interesting idea that the back-propagated errors have log-normal distributions. The authors could extend this intriguing observation into computation efficient algorithms; reduced-precision floating-point quantization or the pruning of the back-prop error, which are very interesting. The authors also provide details of derivations as well as the data analysis along with several small ideas (in Appendix) to strongly support their ideas.\n\nThe biggest shortcoming would be the relatively shallow evaluation of the proposed method. In particular, since the main ideas of this work relies on a very strong assumption that the back-propagated errors are log-normal distributions, (and I guess it would not be possible to analytically prove this assumption) it would be necessary to provide thorough investigations on various deep learning applications to validate this assumption. Table 1 seems to be a good start, but it is a little disappointing to see that a very limited set of image classification models are tested with the proposed method in Table 3 and Fig 5. \n\nIn fact, one concern about the proposed method is that the back-prop error might evolve drastically throughout the training periods, and there is no guarantee for it to maintain its shape. Therefore, it would be critical for the authors to convince that the proposed method work in a wide range of applications (at least the kinds shown in Table 1, since this table claims that the evaluated models here show log-normal shapes).\n\nIn summary, this work seems to be interesting, but it should be supported by a much more thorough evaluation.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4pX8DZCLRN2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors suggest an interesting finding where the gradient distribution in each layer is close to log-normal distribution instead of normal distribution. Given the suggestion, the authors propose two closed-form analytical methods to produce a better low-precision floating-point format and an optimized sparsity threshold for gradient pruning. While such distribution assumption with associated analytical methods seems promising, there are a few suggestions and questions I want to address for the paper:\n\n1. There are many previous studies on quantizing gradients for efficient distributed training, such as QSGD [1] and signSGD [2]. Although most of those studies do not reduce the computational complexity for backpropagation, they still serve as well-performed gradient quantitation methods. I think it's better to distinguish from those previous studies explicitly in the paper.\n\n2. It's better to involve more motivations and explanations about why gradient quantization is important for reducing the computational and memory burden of neural gradients.\n\n3. In the context of the real hardware design, it would be hard to get a computational reduction by using the proposed method which determines the optimal floating-point representation. Real hardware design (e.g. GPU architecture) requires a global floating representation across all datasets so it is impossible to design hardware with a specific floating representation for a specific dataset. Therefore, I don't think determining the best floating-point format for each dataset is useful in practice.\n\n4. Also I would like to address some technical questions regarding the setting and experiment:\n\na. How does weight update work with the quantized gradient? Do you perform the quantized gradient over the full-precision weight?\n\nb. What kind of optimizer you used for the experiments?\n\nc. How do you handle the precision of the error term (defined in WAGE paper [3])? \n\nOverall, I think the paper is interesting and it would be worth being accepted after addressing the suggestions above.\n\n\n[1] Alistarh, Dan, et al. \"QSGD: Communication-efficient SGD via gradient quantization and encoding.\" Advances in Neural Information Processing Systems. 2017.\n\n[2] Bernstein, Jeremy, et al. \"signSGD with majority vote is communication efficient and fault tolerant.\" arXiv preprint arXiv:1810.05291 (2018).\n\n[3] Wu, Shuang, et al. \"Training and inference with integers in deep neural networks.\" arXiv preprint arXiv:1802.04680 (2018).\n\n*************\nI want to thank the authors for their detailed comments and explanations. I agree with (3) that a global floating representation is not required, but different settings of bitwidth may require additional data paths designed in the hardware (therefore a unified bitwidth for exponent and mantissa is recommended). Given revised related work and additional practical implications, I would like to champion this paper. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work makes the observation that gradients in neural network training are approximately distributed according to a log-normal distribution. This observation is then used to compress and sparsify the gradients, which can be useful in distributed optimization of neural nets. The reviewers indicate that this contribution is novel and useful and they do not find any major issues with the presented work. I recommend accepting the paper for a poster presentation.
Decision: Accept (Poster)