Low-Precision Stochastic Gradient Langevin Dynamics
Authors: Ruqi Zhang
Andrew Gordon Wilson
Christopher De Sa
Venue: ICLR 2022 Submitted (2022)

Abstract: Low-precision optimization is widely used to accelerate large-scale deep learning. Despite providing better uncertainty estimation and generalization, sampling methods remain mostly unexplored in this space. In this paper, we provide the first study of low-precision Stochastic Gradient Langevin Dynamics (SGLD), arguing that it is particularly suited to low-bit arithmetic due to its intrinsic ability to handle system noise. We prove the convergence of low-precision SGLD on strongly log-concave distributions, showing that with full-precision gradient accumulators, SGLD is more robust to quantization error than SGD; however, with low-precision gradient accumulators, SGLD can diverge arbitrarily far from the target distribution with small stepsizes. To remedy this issue, we develop a new quantization function that preserves the correct variance in each update step. We demonstrate that the resulting low-precision SGLD algorithm is comparable to full-precision SGLD and outperforms low-precision SGD on deep learning tasks.

Under review as a conference paper at ICLR 2022
LOW-PRECISION STOCHASTIC GRADIENT LANGEVIN DYNAMICS
Anonymous authors Paper under double-blind review
ABSTRACT
Low-precision optimization is widely used to accelerate large-scale deep learn- ing. Despite providing better uncertainty estimation and generalization, sampling methods remain mostly unexplored in this space. this paper, we provide the first study of low- preciiieiion Stochastic Gradient Langevin Dynamics (SGLD), arguing that particularly suited low-bit arithmetic due its intrinsic ability to han- dle system noise. We prove the convergence of low-pree low-precision SGLD on strongly log-concave distributions, showing that with full-precision gradient accumulators, SGLD is more robust to quantization error than SGD; however, with low-precision gradient accumulators, SGLD can diverge arbitrarily far from the target distribu- tion with small stepsizes. To remedy this issue, we develop new quantization function that preserves the correct variance in each update step. We demonstrate that the resulting low-precision SGLD algorithm is comparable to full- precision SGLD and outperforms low-precision SGD on deep learning tasks.
INTRODUCTION
Low-precision optimization has become increasingly popular in reducing compute and memory costs of training deep neural networks (DNNs). It uses fewer bits to represent numbers in model parameters, activations and gradients and thus can drastically lower resource demands (Gupta al., 2015; Zhou et al., 2016; De Sa et al., 2017; Li et al., 2017). Prior work has shown that using 8-bit numbers in training DNNs achieves about 4x latency speed ups and memory reduction compared to 32-bit numbers on a variety of deep learning tasks (Sun et al., 2019; Yang et al., 2019; Wang et al., 20; Banner et al., 2018). datasets and architectures grow rapidly, performing low- precision optimization enables training large-scale DNNs efficiently and enables many applications on different hardware and platforms.
Despite the impressive progress in low-precision optimization, low-precision sampling remains largely unexplored. However, we believe stochastic gradient Markov chain Monte Carlo (SGM- CMC) methods (Welling & Teh, 2011; Chen et al., 2014; Ma et al., 2015) particularly suited for low-precision arithmetic because of their intrinsic robustness to system noise. In particular: (1) single point, thus it should require precise weights or gradients: (2) SGMCMC even adds noise to the system to encourage exploration and is naturally more tolerant to quantization noise; (3) SGMCMC performs Bayesian model averaging during testing using an ensemble of models, which enables coarse representations of in- dividual models to be compensated by the overall model average (Zhu et al., 2019).
SGMCMC explores weight space instead of converging
SGMCMC is particularly compelling in Bayesian deep learning due to its ability to characterize complex and multimodal DNN posteriors, providing state-of-the-an generalization accuracy and calibration (BNNs) (Zhang et al., 2020; Li et al., 2016; Gan et al., 2016). Moreover, low-precision approaches are especially appealing this setting, where at test time we must store samples from a posterior over millions of parameters, and perform multiple forward passes through the correspond- ing models, which incurs significant memory and computational expenses.
In this paper, provide the first comprehensive study of low-precision Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011). We start by analyzing the theoretical con- vergence of low-precision SGLD on strongly log-concave distributions, proving that SGLD with full preciiree gradient accumulators (SGLDLP-F) converges to the target distribution within a dis- tance that is asymptotically smaller than the distance between the SGD estimation and the optimum.

---

Under review as a conference paper at ICLR 2022
2.40
0.35
SOLDUML
9.3
SOUDEPF
1:30
2925
232
1.20
035
2.3
0.05
(a) SGLD with = 0.001
Tree
Trus
0.00-
Trae
0.35-
SOLDENL
SCLDUU
SCLOUPT
0.0
2025
2025
020
2220
035
1.1
030
130
066
010
100
(b) SGLD with 0.0001
Figure 1: Low-precision SGLD with varying stepsizes on a Gaussian distribution. Variance- corrected SGLD with low-precision gradient accumulators (VC SGLDLP-L) and SGLD with full-precision gradient accumulators (SGLDLP-F) converge to the true distribution whereas naive SGLDLP-L diverges and the divergence increases as the stepsize decreases.
Surprisingly, we find that SGLD with low-precision gradient accumulators (SGLDLP-L) can di- verge arbitrarily far away from the target distribution with small stepsizes. We solve this issue by developing a new quantization function that preserves the correct variance in each update step. We prove that when using this new quantization function, SGLD converges to the target distribution within a bounded level of accuracy even with small stepsizes. We illustrate the behavior of different low-precision SGLD variants in Figure 1.
Empirically we demonstrate low-precision SGLD deep learning tasks, including CIFAR-10. CIFAR- 100, and ImageNet with a ResNet-18, and the IMDB dataset with LSTM. We show that variance-correctedd quantizer significantly improves the performance of low-precision SGLD. Moreover, the improvement of SGLD over SGD is larger in low-precision than in full-precision, demonstrating the promise of low-precision stochastic sampling.
2 RELATED WORK
Most work accelerating SGLD training has focused on distributed learning with synchronous or asynchronous communication (Ahn et al., 2014; Chen et al., 2016; Li et al., 2019). Another direction to reduce training costs is shorten training time by accelerating the convergence using either variance reduction techniques (Dubey et al., 2016; Baker et al., 2019) or a cyclical learning rate schedule (Zhang al., 2020). To speed up testing, distillation techniques are often used to save both compute and memory by transferring the knowledge of an ensemble of models to a single model (Korattikara et al.. 2015; Wang et al., 2018a).
Low-precision computation has become one of the most common approaches to reduce latency and memory consumption in deep learning and is widely supported on new emerging chips including CPUs, GPUs and TPUs (Micikevicius et al., 2017; Krishnamoorthi, 2018; Esser et al., 2019). Two main directions to improve low-precision training include developing new number formats (Sun et al., 2019; 2020) and studying mixed precision schemes (Courbariaux et al., 2015; Zhou et al., 2016; Banner et al., 2018). Recently, one line of work studies applying Bayesian framework to learn deterministic quantized neural network (Soudry et al., 2014; Cheng et al., 2015; Achterhold al., 2018; van Baalen et al., 2020; Meng et al., 2020).
Despite impressive progress of low-precision deep learning, few work takes advantage of it for Bayesian neural networks (BNNs). Su et al. (2019) proposes a method to train binarized variational BNNs and Cai et al. (2018) develops BNNs. The only work on low- www-precision MCMC known to us is Ferianc et al. (2021), which directly applies post-training quantization in optimization (Jacob et al., 2018) quantize BNNs trained by
efficient hardware for training low-precision variational
2

---

Under review as a conference paper at ICLR 2022
Stochastic Gradient Hamiltonian Monte Carlo (Chen et al., 2014). In contrast, study training low-precision models by SGLD from scratch, which can reduce both training and testing costs.
PRELIMINARIES
3.1 STOCHASTIC GRADIENT LANGEVIN DYNAMICS
In the Bayesian setting, given some dataset D, a model with parameters 0, and prior p(0), we are interested in sampling from the posterior p(0 x exp(-U(0)) where the energy function is
U = Σ logp(= log)(0).
ZED
When the dataset is large, the cost of computing sum over the entire dataset is expensive. Stochastic Gradient Langevin Dynamics (SGLD) reduces the cost by using a stochastic gradient estimation VŪ, an unbiased estimator of VU usually based on a subset the dataset D. Specifically, SGLD updates the parameter 0 in the (I: + 1 step following the rule
0 = 0₂ (B.)) + V205=+1,
where is the stepsize and is standard Gaussian noise. Compared to the SGD update, the only difference is the additional Gaussian noise. This close connection makes SGLD convenient to implement existing deep learning tasks for which SGD is the typical learning algorithm.
3.2 SOW-PRECISION TRAINING
We study training a low-precision model by SGLD from scratch, to reduce both training and testing costs. We fixed point in our theoretical analysis and empirical demonstration, which is common number type for reducing DNNs costs (Gupta et al., 2015; Lin et al., 2016; Li et al., 2017; Yang al., 2019). We additionally use block floating point, another common number representation, for empirical evaluations on deep learning tasks (Song et al., 2018).
In order train a network using low low-precision numbers, we need a quantization function Q to convert real-valued number into a rounded number. Such functions include deterministic rounding and stochastic rounding. Suppose that we use W' bits to represent numbers with those bits to represent the fractional part. Then the quantization gap A 2- is the distance between consecutive representable numbers. The lower and upper bounds of the representable numbers are I - F-1 and 2W-F-1 - 2 2-r, respectively. The deterministic rounding quantizes
number to its nearest representable neighbour:
A
|
1
(0) = sign(0) clip
+
.1,4
A
2
where clip(a,,,,) max u), u) Stochastic rounding quantizes number with probability:
[clip A
8
1. w.p. ,1,u). w.p.
[
(*) clip A
Ø
(1 4
Q" is unbiased, which means [[)](0)) 0. It is more favorable than Q" in practice since it can preserve gradient information when the gradient update is smaller than the quantization gap (Gupta et al., 2015; Courbariaux et al., 2016). In what follows, we use Q and Aw to denote the weights' quantizer and quantization gap, Qg and A, to denote gradients' quantizer and quantization gap- Please note that Qg means quantizing the error and gradient in each layer in backward propaga- tion (Wu et al., 2018: Yang et al., 2019).
Besides the quantization function, also need to decide what and where to quantize. There are two common choices depending on whether we store an additional copy of full-precisio weights. Full- precision gradient accumulators use full-precision weight buffer to accumulate gradient updates and only quantize weights before computing gradients. SGD with full-precision gradient accumula- tors (SGDLP-F) updates the weights
0x+1 = 02 aaq (vū(Qu (On)))
3

---

Under review as a conference paper at ICLR 2022
Gradient accumulators frequently updated during training, therefore it will be ideal to also repre- sent in low-precision to further reduce the costs. The update of SGD with low-precision gradient accumulators (SGDLP-L)
0k+1 = Qu (0e (0c-00,, aQq (
Both full- and low-precision gradient accumulators have been widely used in low-precision training. Low-precision gradient accumulators are cheaper and full-precision gradient accumulators generally have better performance because of more precisely accumulating gradient updates (Courbariaux et al., 2015; Li et al., 2017).
4 LOW-PRECISION SGLD
In this section, we first study the convergence bound of lowpprccision SGLD with full-precision gradient accumulators (SGLDLP-F) on strongly log-concave distributions and show that SGLDLP-F converges to the target distribution within a distance that is asymptotically smaller than that between the SGD estimation and the optimum. Next we analyze low-precision SGLD with low-precision gradient accumulators (SGLDLP-L) under the same setup and prove that SGLDLP-L can diverge arbitrarily far away from the target distribution with a small stepsize, which typically required by SGLD to reduce asymptotic bias. Finally, we solve this problem by developing a variance-correctéd quantization function and further prove that our SGLDLP-L converges with small stepsizes.
4.1 FULL-PRECISIA GRADIENT ACCUMULATORS
Due the similarity between SGLD and SGD, applying low-precision SGLD is straightfor- ward. Similar to SGDLP-F, we can do low-precision SGLD with full-precision gradient accumula- tors (SGLDLP-F) the following.
Dx+1 = 02 aQq (vū(Qu. (0x)) + 2205k+1-
(1)
We prove that SGLDLP-F will converge to the target distribution given small stepsizes and large number of iterations.
Our convergence analysis of low-precision SGLD based on (Dalalyan & Karagulyan, 2019), where the target distribution is assumed smooth and strongly log-concave. We additionally as- sume the energy function has Lipschitz Hessian following recent work in low-precision optimiza- tion (Yang et al., 2019). Specifically, the energy function U satisfies the following,
U(0) - (6') (U((') - 8¹) 2 (m/2) l|0 01112-
(0) (U(0')), M ||0 6' 12
v0,0 0' R
(V)(()) (2U((")) S y | 0'
for positive constants m. M and V. We further assume the variance of the stochastic gradient bounded by (() VU (0) < x² for some constant For simplicity, consider SGLD with a constant stepsize a. We measure the convergence of SGLD in terms 2- 2-sasserstein distance. We use stochastic rounding for both weight and gradient quantizers it generally better than deterministic rounding and has also been used in previous low-precision theoretical analysis (Li al., 2017; Yang et al., 2019).
Theorem 1. We SGLDLP-F under the above assumptions and with constant stepsize S 2/(m + M). Let π be the target distribution, p.g be the distribution obtained after K iterations and No be the initial distribution, then
M MA.. /d
WZ(UK, x) S am.) (M000///)))5000000M00000203333// + min
4m
2m
2 M ad d 4ak
4m
4

---

Under review as a conference paper at ICLR 2022
There are three main takeways of this theorem: First, it shows that SGLDLP-F converges to the curacy floor min VALA
MA
which depends on the quantization gap A. given large enough 2m
K and small enough a. Second, our convergence bound is O(A which is better than the distance between the SGD estimation and the optimum O(Aw) (Yang et al., 2019) (see Appendix for de- tails). Third, if we further assume the energy function is quadratic, that is V O. then SGLDLP-F converges to the target distribution asymptotically. This is similar SGDLP-F on quadratic function which converges to the optimum asymptotically (Li et al., 2017).
4.2 NOW-PRECISION GRADIENT ACCUMULATORS
To further reduce the costs, we apply low- low-precision gradient accumulators to SGLD which we denote SGLDLP-L. Mimicking the update of SGDLP-L, the update rule of SGLDLP-L is
Bc+1 = nu (0g aQ, (Vū(O)) + v20fx+1)
(2)
Surprisingly, while we can prove convergence result for SGLDLP-L, our theory suggests that can diverge arbitrarily from the target distribution with small stepsizes.
Theorem 2. We run SGLDLP-L under the same assumptions in Theorem 1. Then
V M
M
W2(UK; T) S (1 crm) (2((o,= x) 1.66MM1/m))( + min
4m
2m
(a
1 4ax²
+ - am)) + +1) V
4m
This theorem suggests that the stepsize decreases, W2 distance between the stainary distri- bution of SGLDLP-L and the target distribution increases. When a decreases, either our bound becomes loose SGLDLP-L diverges from the target distribution. We empirically test SGLDLP-L on standard Gaussian distribution in Figure 1, showing that the reason is the later. We use 8-bit fixed point and assign 2 of them to represent the integer part. We can see that SGLDLP-F always converges to the target distribution with different stepsizes whereas SGLDLP-L diverges from the target distribution and the divergence increases when the stepsize decreases.
One may choose stepsize that minimizes the above W2 distance to avoid divergence, however, getting this optimal stepsize is generally difficult as the constants are unknown practice. More- over, enabling small stepsize in SGLD is desirable, since SGLD needs small stepsize to reduce asymptotic bias of the posterior approximation (Welling & Tch, 2011).
4.3 VARIANCE-CORRFENE QUANTIZATION
To approach correcting the problems with naive SGLDLP-L, we first need to identify the source of the issue. We start by showing that the reason is the variance of each dimension of 6x+1 becomes larger due to using stochastic rounding the low-precision gradient accumulators. Specifically, given the stochastic gradient VÜ, the update of full-precision SGLD can be viewed as sampling from a Gaussian distribution for each dimension i
0K+1,i ~ N (0v. aVi 20x for = 1,....,d dd.
Using stochastic rounding as the weight quantizer and the gradient quantizer in SGLDLP-L gives us
E [0 = E Q" (z. aQ" (vū(O))) v20Eu+1s) 0z.i vV((()))
which keeps the same mean of 0k+1 as in full-precision. However the variance of 0x+1, is larger than needed. If we ignore the variance from Q2 and the stochastic gradient, since they are present and have been shown to work well in SGLDLP-F, the variance of 0k+1, is
Var (0.-1.8) [.]]
E Var QQ
(OL vOx())) + V20 (u+1..])
Q² (Dr. (V)) + v20 [c+1.d]]
+ Var E
A
1,+ + 2a.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Variance-Correcter Low-Precision SGLD (VC SGLDLP-L).
given: Stepsize a, number of training iterations K, gradient quantizer Qg deterministic rounding Q4, stochastic rounding Q" and quantization gap of weights Aw.
for k 1: K do
update 0x+1 < Qu (0x- aQ, end for output: samples }
(002((vÜ0) (Vū(Ow) 2a, Aw. Q", Q*)
where X4+1, E [0, 1] depends on the distance of OR+1. i to the discrete grid. Empirically, from Figure 1, we can see that naive SGLDLP-L gives the correct mean estimate but the wrong variance estimate, while both are estimated correctly in SGLD which adds the proper amount noise in each update. This validates our intuition that using stochastic rounding in low-precision gradient accumulators adds more variance than needed leading to an inaccurate variance estimate.
To address this issue, propose new quantization function Q in Algorithm 2. QT always guarantees the correct mean, E [0x+1,1] 0z a Vū(())) and guarantees the correct variance Var [0k+1.5] = 2a most of the time except when v < Us.. However that case rarely happens in prac- tice, because the stepsize has be extremely small. The main idea of Q is to directly sample from the discrete weight space instead of quantizing a real-valued Gaussian sample. It does so considering two cases: when the Gaussian noise is larger than the largest possible stochastic rounding variance /²/4, Q"e first adds a portion of the Gaussian noise and uses a sample from the weight grid to make up the remaining; in the other situation, Q directly samples from the weight grid achieve the target variance. Although Q does not provide a sample following a Gaussian distribution (as it is discrete), we show that this does not affect the performance in both theory and practice.
We now prove that SGLDLP-L using Q"c, denoting VC SGLDLP-L, converges to the target distri- bution up certain accuracy level with small stepsizes.
Theorem 3. We ran VC SGLDLP-L as in Algoritlum 1. Besides the same assumptions in Theorem 1, we further assume the gradient is bounded ( S G. Let up = 2 /4. Then
VA MVA m
W2(uK, S arm) W2(Ho, x) 1.665MM/m)(ad)) min
m
aa dd 4ax²
+ am
+ ((1 am) +1 VA.
4m
[5ddd,
2f >
where A max (2 aG, 4ad), otherwise
This theorem shows that when the stepsize tribution up to an error instead of diverging. Our bound is O(VA.)) which is equivalent to the distance between SGD with low-precision gradient accumulators and the optimum (Li et al., 2017: Yang et al., 2019). However, we show empirically that VC SGLDLP-L has better dependency on the quantization gap than SGD. We leave the improvement of the theoretical bound for future work.
O. VC SGLDLP-L converges to the target dis-
We empirically demonstrate VC SGLDLP-L on the standard Gaussian distribution under the same setting in the previous section in Figure 1. Regardless of the stepsize, VC SGLDLP-L converges to the target distribution and approximates the target distribution as well as SGLDLP-F, showing that preserving the correct variance is the key to ensure correct convergence.
EXPERIMENTS
We demonstrate low-precision SGLD with full-precision gradient accumulators (SGLDLP-F) and with variance-correcn low- precision gradient accumulators (VC SGLDLP-L) on a logistic regres- sion and multilayer perceptron MNIST dataset (Section 5.1), ResNet- 18 on CIFAR datasets and LSTM on IMDB dataset (Section 5.2), and ResNet-18 18 on Imagenet dataset (Section 5.3). We use qtorch (Zhang et al., 2019) to simulate low-precision training these experiments.

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Variance-Corrected Quantization Function Q"
input: (1, v, A, 09.27
// Qe returns variable with mean and variance v 11 A? /4 is the largest possible variance that stochastic rounding can cause
Vo A²/4
if v > 0 then
x x μ +++ Vu VV- vof. where (0,
<- (xx)
for all do
Nutri + InilA
A
w.p. 24
Ci
w.p. 22
outt--lt In/A
// When e > vo. we add portion of Gaussian noise and sample from the weight grid to make up the remaining
0,
otherwise
end for
0 + Q7
+ sign(r) cc
else
rt rtμ- 1-----1 (m)
for all do Us f (1
? +
Iral A
(-re + sign(r;)
if Us then
A,
w.p. 2
G
A.
セーリ w.p. 24
// When S to, we sample from the weight grid // to achieve the target variance
0,
otherwise
0 - ('(N)
else
0, + (u); end if end for end if
clip 0 if outside representable range return 0
5.1 LOGISTIC REGRESSION AND MULTILAYER PERCEPTRON
We first empirically verify our theorems on a logistic regression on MNIST dataset. We use (O, 1/6) the prior distribution following (Yang et al., 2019) and the resulting posterior dis- tribution satisfies the assumptions in Section 4. We use fixed point numbers with 2 integer bits and vary the number of fractional bits which is corresponding varying the quantization gap A.
From Figure 2a, SGLDLP-F more robust to the decay of bits than SGDLP-F since SGLDLP- F outperforms SGDLP-F on all number of bits and recovers the full-precision result with 6 bits whereas SGDLP-F needs 10 bits. This verifies Theorem that SGL.LP--F converges to the target distribution up to an eπror and more robust the quantiazation gap than SGDLP-F. With low- precision gradient accumulators, we can see that VC SGLDLP-L is significantly better than naïve SGLDLP-L, indicating the effectiveness of using the variance-corrected quantization function for quantizing gradient accumulators. These results verify Theorem 2 and Theorem 3. Besides, VC SGLDLP-L outperforms SGDLP on all number of bits and even outperforms SGDLP-F when using 2 fractional bits. VC SGLDLP-L. matches full-precision SGLD results with 6 bits whereas SGDLP. PP- bits. These observations demonstrate that SGLD still more favorable than SGD when using low wwwprreciionnn gradient accumulators.
needs
To test whether these results apply to non-log-concave distributions, we replace the logistic regres- sion model by two-layer multilayer perceptron (MLP). The MLP has 100 hidden units and RELU nonlinearilities. From Figure 2b, we observe similar results as on the logistic regression, suggesting that empirically analysis still stands likeliiddoood (NLL) comparisons in terms of number bits on the logistic regression and MLP in Figure in Appendix.
con-log-concave distributions. We also provide negative
7

---

Under review as a conference paper at ICLR 2022
16
SGDLP-L
14
SGDLP-L
Naive SGLDLP-L
Naive SGLDLP-L
Test Error (96)
14
VC SGLDLP.L
Test Enoo (96
VC SGLDIPL
SGDLP-F
10
SGDLP-F
12
SGLDLP-F
SGLDLP-F
SGDFP (7.68%)
SGDFP (2.03%)
SGLDFP (7.63%)
SGLDFP (1.9%)
4
2
2
5
6
7
9
10
2
3
7
10
Number of Fractional Bits
Number of Fractional Bits
(a) Logistic regression
(b) MLP
Figure Test errors on MNIST dataset in terms of different precision.
5.2 RESNET AND LSTM
We consider image and sentiment classification tasks: CIFAR datasets (Krizhevsky al., 2009) on ResNet-18 (He et al., 2016), and IMDB dataset (Maas et al., 2011) on LSTM (Hochreiter & Schmidhuber, 1997). We use 8-bit number representation since becomes increasingly popular in training deep models and is powered by new generation of chips (Sun et al., 2019; Banner et al., 2018; Wang et al., 2018b). We report test errors averaged over 3 runs with the standard error in Table 1.
Fixed Point We use 8-bit fixed point for weights and gradients but full-precision for activations since we find low-precision activations significantly harm the performance. Similar to the results previous sections, SGI.DL--F is better than SGDLP-- and VC SGLDLP-L significantly outper- forms naïve SGLDLP-L and SGDLP-L across datasets and architectures. Notably, the improvement of SGLD over SGD becomes larger when using more low-precision arithmetic. For example, on CIFAR- 100, VC SGLDLP-L outperforms SGDLP-L by 2.24%, SGLDLP-F outperforms SGDLP-F by 0.54% and SGLD outperforms SGDLP by 0.06%. This demonstrates that SGLD particularly compatible with low-precins oww-precision deep learning because of its natural ability to handle system noise.
Block Floating Point We also consider block floating point (BFP) which another common num- ber type and is often preferred over fixed point deep models due to less quantization error caused by overflow and underflow (Song et al., 2018). Following the block design in Yang et al. (2019), small-block for ResNet and big-block for LSTM. The Q"e function naturally generalizes to BFP and only needs a small modification (see Appendix E for the algorithm of Q with BFP). By using BFP, the results of all low-precision methods improve the performance of SGLDFP with all numbers quantized to 8-bit except gradient accumulators. VC SGLDLP-L still outperforms naive SGLDLP-L indicating the effectiveness of Qee with BFP. Again, SGLDFP-F and VC SGLDLP-L outperform their SGD counterparts on all tasks, suggesting the general applicability of low- eww-precision SGLD with different number types.
fixed point. SGLDLP-F
match
Cyclical SGLD We further apply low-precision to a recent variant of SGLD, cSGLD, which uti- lizes cyclical learning rate schedule speed up convergence (Zhang al., 2020). We observe that the results of cSGLD-F are very close to those of cSGLDFP, and VC cSGLDLP-L can match or even outperforms full- full-precision SGD with all numbers quantized 8 bits! These results indi- cate that diverse samples from different modes can countereffect the quantization error by providing complementary predictions.
5.3 IMAGENET
Finally, test low-precision SGLD on large scale image classification dataset, ImageNet, with ResNet- 18. We train SGD for 90 epochs and train SGLD for 10 epochs using the trained SGD model the initialization. The improvement of SGLDD over SGD is larger in low-precision (0. (0.76% top-1
8

---

Under review as a conference paper at ICLR 2022
Table 1: Test errors (%) CIFAR-10, CIFAR-100 with ResNet-18 and IMDB with LSTM.
CIFAR-10
CIFAR-100
IMDB
32-BIT FLOATING POINT
SGLDFP
4.65 +0.06
22.58 10.18
13.43
#0.21
SGDFP
4.71 #0.02
22.64
10.13
13.88
+0.29
cSGLDFP
4.34 #0.05
2I.63
#0.04
13.25
#0.18
8-BIT FIXED POINT
NAïVE SGLDLP-L
7.82 10.13
27.25 10.13
16.63
1.0.8
VC SGLDLP-L
7.13
26.62
10.16
15.38
10.27
SGLDLP-F
5.12 #0.06
23.30
#0.09
15,40
#0.36
SGDLP-L
8.53 +0.08
28.86 0.10
19.28
+0.63
SGDLP-F
5.20 10.14
23.84
10.12
15.74
10.79
8-BIT BLOCK FLOATING POINT
NAïVE SGLDLP-L
5.85 10.04
26.38
10.13
14.64
10.08
VC SGLDLP-L
5.51 +0.01
25.22
20.18
13.99
10.24
SGLDLP-F
4.58 +0.07
22.59
20.18
14.05 #0.33
SGDLP-L
5.86 10.18
26.19 #0.11
16.06 11.81
SGDLP-F
4.75 +0.05
22.9 +0.13
14.28
+0.17
VC CSGLDLPL
4.97 20.10
22.61
I3.09
#0.27
CSGLD-F
4.32 #0.07
21.50 #0.14
13.13 #0.37
Table Test errors (%) on ImageNet with ResNet-
Top-1
Top-5
32-BIT FLOATING POINT
SGLD
30.39 30.56
10.76
SGD
10.97
8-BIT BLOCK FLOATING POINT
VC SGLDLP-L
31.47 32.23
11.77
SGDLP-I.
12.09
error) than in full null-precision (0.17% top-1 error), showing the advantages of low-precision SGLDD on large-scale deep learning tasks.
CONCLUSION
We provide the first comprehensive study of low-precision SGLD. In theory, we prove that with full- precision gradient accumulators. SGLD converge to the target distribution within distance that is asymptotically smaller than the distance between the SGD estimation and the optimum; with low- precision gradient accumulators, SGLD can diverge arbitrarily far away from the target distribution with small stepsizes. We find that the issue is caused by the wrong variance in each update and thus develop a new varaince-correced quantization function that preserves the correct variance. that SGLD with this quantization function converges to the target distribution up to certain level depending on the quantization gap. practice, we verify our thcoretical results and demonstrate SGLDLP-F and VC SGLDLP-L are comparable full-precision SGLD with 8-bit image and sentiment classification tasks.
prove
In the future, it will be interesting extend low wwwecciiioonn computation to other stochastic MCMC methods and improve thcoretical bounds to better reflect empirical performance. We hope this work shed lights accelerating MCMC methods for Bayesian deep learning.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jan Achterhold, Jan Mathias Kochler, Anke Schmeink, and Tim Genewein. Variational network quantization. In International Conference on Learning Representations, 2018
Sungjin Ahn, Babak Shahbaba, and Max Welling. Distributed stochastic gradient meme. In Inter- national conference machine learning, pp. 1044-1052. PMLR, 2014.
Jack Baker, Paul Fearnhead, Emily Fox, and Christopher Nemeth. Control variates for stochastic gradient memc. Statistics and Computing, 29(3)5999-615, 2019.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. arXiv preprint 1805 1046, 2018.
Ruizhe Cai, Ao Ren, Ning Liu, Caiwen Ding, Luhao Wang, Xuehaí Qian, Massoud Pedram, and Yanzhi Wang. Vibnn: Hardware acceleration of bayesian neural networks. ACM SIGPLAN No- tices, 53(2):476-48,, 2018.
Changyou Chen, Nan Ding, Chunyuan Li, Yizhe Zhang, and Lawrence Carin. Stochastic gradient meme with stale gradients. arXiv preprint arXiv: 1600006610 2016.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference machine learning, pp. 1683-1691, PMLR, 2014.
Zhiyong Cheng, Daniel Soudry, Zexi Mao, and Zhenzhong Lan. Training binary multilayer neural networks for image classification using expectation backpropagation. arXiv preprint arXiv:1503.03562. 2015.
Matthieu Courbariaux, Yoshua Bengio, and Jean Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv,,v, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained or-1. arXiv preprint 1602..:600..22 2016.
Arnak S Dalalyan and Avetik Karagulyan. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. Stochastic Processes and their Applications. 129(12):5278-55 5311, 2019.
Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun. Understanding and optimizing asynchronous low-precision stochastic gradient descent. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 561-574, 2017.
Kumar Avinava Dubey, Sashank Reddi, Sinead A Williamson, Barnabas Poczos, Alexander J Smola, and Eric P Xing. Variance reduction in stochastic gradient langevin dynamics. Advances in neural information processing systems, 29: 154-1 2016.
Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint 190.v01902.08153 2019.
Martin Ferianc, Partha Maji, Matthew Mattina, and Miguel Rodrigues. On the effects of quantisation on model uncertainty in bayesian neural networks. arXiv preprint arXiv:2 /002, 2021.
Zhe Gan, Chunyuan Li, Changyou Chen, Yunchen Pu, Qinliang Su, and Lawrence Carin. Scal- able bayesian learning of recurrent neural networks for language modeling. arXiv preprint arXiv: 1.00034 2016.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. International conference on machine learning, pp. 1737-1746.
PMLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2022
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient sneger--ritthmeti-onnly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Anoop Korattikara, Vivek Rathod, Kevin Murphy, and Max Welling. Bayesian dark knowledge. arXiv preprint arXiv:/506.044 04416, 2015.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Chunyuan Li, Andrew Stevens, Changyou Chen, Yunchen Pu, Zhe Gan, and Lawrence Carin. Learn- ing weight uncertainty with stochastic gradient meme for shape classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5666-5675, 2016.
Chunyuan Li, Changyou Chen, Yunchen Pu, Ricardo Henao, and Lawrence Carin. Communication- efficient stochastic gradient meme for neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4173-4180, 2019.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. Neural Information Processing Systems, 2017.
Zheng Li and Christopher M De Sa. Dimension-free bounds for low-precision training. Advances in Neural Information Processing Systems, 2019.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep con- volutional networks. In International conference on machine learning, pp. 2849-2858. PMLR, 2016.
Yi-An Ma, Tianqi Chen, and Emily B Fox. A complete recipe for stochastic gradient meme. arXiv preprint arXiv:1506. 150.0.96.6.0 2015.
Yi-An Ma, Yuansi Chen, Chi Jin, Nicolas Flammarion, and Michael I Jordan. Sampling can be faster than optimization. Proceedings of the National Academy of Sciences, 116642)208811-00855 2019.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 152-150, 2011.
Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan. Training binary neural net- works using the bayesian learning rule. In International Conference on Machine Learning, pp. 6852 6861. PMLR, 2020.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 2017.
Zhourui Song, Zhenyu Liu, and Dongsheng Wang. Computation error analysis of block floating point arithmetic oriented convolution neural network accelerator design. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In NIPS, volume pp. 2, 2014.
Jiahao Su, Milan Cvitkovic, and Furong Huang. Sampling-frie free learning of bayesian quantized neural networks. arXiv preprint arXiv: /9/2,02992..2 2019.

---

Under review as a conference paper at ICLR 2022
Xiao Sun, Jungwook Choi, Chia-Yo Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit float- ing point (hfp8) training and inference for deep neural networks. Advances in neural information processing systems, 32:4900-4909, 2019.
Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrish- nan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Kunal Talwar. Computational separations between sampling and optimization. arXiv preprint 1710001911.02074, 2019.
Mart Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. arXiv preprint arXiv:2005.07093, 2020.
Kuan-Chieh Wang. Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adversar- ial distillation of bayesian neural network posteriors. In International Conference on Machine Learning, pp. 5190-5199. PMLR, 2018a.
Naigang Wang. Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, pp. 7686-7695, 2018b.
Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pp. 681-688. Citeseer, 2011.
Shuang Wa, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. arXiv preprint arXiv: 046.80002.466, 2018.
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christo- pher De Sa. Swalp: Stochastic weight averaging in low-precision training. International Confer- ence on Machine Learning, 2019.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cycli- cal stochastic gradient meme for bayesian deep learning. International Conference on Learning Representations, 2020.
Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa. Qpytorch: A low-precision arithmetic simulation framework. arXiv preprint arXiv:1910.04550, 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1.X6v:660.06610 2016.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4923-4932, 2019.
12

---

Under review as a conference paper at ICLR 2022
A
PROOF OF THEOREM
Our proof follows Theorem Dalalyan & Karagulyan (2019), which provides a convergence bound of Langevin dynamics with noisy gradient. We state the result of Theorem 4 in Dalalyan & Karagulyan (2019) below.
We consider Langevin dynamics whose update rule is
0x+1 a (VU(OL) (x) + V20
The noise in the gradient Ça has the following three assumptions:
E
E [Gx [0 S ddd,
E |Ic - E [Cx S oddd Ex+1 is independent of (Co, ... Ck)
where 8 > O and σ O are some constants. Under the same assumptions in Section 4, we have the convergence bound for the above Langevin dynamics.
Theorem 4. We run the above Langevin dynamics with a S 2/(m + M). Let i be the target distribution, J.K be the distribution obtained after K iterations and pn be the initial distribution. Then
8vd
(ad)
W2(uK. S (1- am)) W2(uo: T) + LL5(M//))ad))
1.65M +oym
Proof. We write the SGLDLP-F update as the following
O2+1 = 02 (V + V20ER+1
-QU (k) V205k+1
where
(.((V((( -U(())
= (0:) ((.))(()))
+ +VV(Qu. -U
Since E[Vū(x)] (z)x) and B[Q(=)) x, we have
ElGa] = E (2 (0x))) - v((Q
+ Vū(Q.. - vU(Q + (TV(QQQ))))))) vU(A.))
=
By the assumption, we know that
E (IV) VU( -U(
S M (Oxa 0x11
< 2 2
It follows that
B[(a]11 = | [E[VU(Q (0.) vU
E - vU
< 2 A
Let f: - Rd denote the function
((a) (U a (Q.))(0k) 0x))-
13

---

Under review as a conference paper at ICLR 2022
By the mean value theorem, there will exist randomness) such that
[0, 1] (a function of the weight quantization
f(1) =
So,
ElcE] E (V)(
= E VQU(0) + ((k)((s) - (x))(Qu((0)) - 0x)]
E (O,) - -0a)
(O
+ aQ (0s) 0s)) (²U(0w)) (w - 0s)
( -bx)) ((T((+)) (Qw( 0k)]
Now, by the assumption [v)(x)) vrU(ille S 2/ - v/l. we get
|[(c]]
E (0x+ - 0x)) (UU(O)) (Ox) 0x)]
+
E + - 0x)) V²U(ON) (Qn(Ox) 0x)l]
E
0a)) 112 - -A0II]
E
V (a(Qu.(0)) - 0x)l (Ow) - 0w)ll
VE
(0x) 1
This combined with the previous result gives us
[VALVd
MA
E B = min
4
2
Now considering the variance of Sk.
[[Cx
E[(x]]
[
||CR 1
< [l0 27
((() - (V) (00)
v0(0))1
Add
A
+
x²
+ 2
4
We set and Theorem Dalalyan Karagulyan (2019) as the following
2 Vd
MA
A d
A
(A
M² ) + 4x²
-min
ddd
+ x²
+
M
4
2
4
4
4
14

---

Under review as a conference paper at ICLR 2022
The assumptions in Theorem 4 in Dalalyan & Karagulyan (2019) satisfied, thus can apply the result Theorem and get
8vd
('(d)//2
W2(UK, R) S (1 - am)) (46
1/2
+
m
1.65M + +oym
åvd
ooad
(1 am) W2(u0, L,6,
m
m
V MA m Vd
(1 am) (om))>W2((m)) W2(Ho: x) 1 min
4m
2m
A + M Jad 4ax²
4m
B PROOF OF THEOREM 2
Proof. The update of SGLDLP-L is
0x+1 Qu (0x aO (VÚ((())) <22kk+1)
Let 1/++1 0 a aQg((v (Vū(())) + V20 so that 0% (wk)))). Then,
Wk+1 0/: - aQ, (VÜ().)) V20Ek+1
= -Qw(w)) (V( + 22 t/h (VVU(())) + (x) V20Ek+1
where
Sk t/s - 02 + QVū((() vU(()))
a
Uh 0z + Q₂ (Vū(O))
-V()))) vU(0.) VU(UK)
Similar to the previous proof, we know that
E[Ga] = E (0,) vU(()) (VV((Q.,))))
V MA vd
[[G]]] < min
and it suffices set above. On the other hand, the variance will be bounded by
E |k5 Eic
= E
[ (V(((O)) vū((k)
2 |]]
+ +E Vū(O)) -VU()))
th 0x + (U(())- -U(
27
+ E
2
2 E [VF(1))) - VF(O))
4
Observe that since U m-strongly convex and M-smooth, and a > M/2, must be smooth, and so
where F(0)- 1 |1011 (0).
2r
115 EiCH11?] <
ad
+
122
+
1
|
1x
a²
A
A
9
2 +
2
15

---

Under review as a conference paper at ICLR 2022
This is essentially replacing the M² in the previous analysis with -, so, supposing the distribution of xx+1 VK, it will give us the rate of
MA vdl
M
W2(UK, x) $ (1 - am) W2(Ps, T) + 1.33MM/M)((aM)))
min
4m
2m
(a 1----0 -1 + 4a
4m
We also have
W2(44, ux)
inf / ||x ||e- y11 y1l aJ(x.w) (EJ(()))
1/2
S
|10t1
Awvd 2
Combining these two results, we W2(UK; T) S W2(UK,PR) + (2(KK.
M A.w
vd
< (1 - am) W2(U0, T) 1.65( LG3(M min
4m
2m
(a +a 4ax²
Aw
4m
2
VA
MA vd
< (1 am) 2 W2(Ho, x) 1.65((Mm))((]))) + min
4m
2m
aa + a 4ax²
Aw
+ ((1-amm am) +11
4m
2
C PROOF OF THEOREM 3
Proof. The update VC SGLDLP-L
0x+1 = Que (0x - aQ, ((0((((Ú(())) 2a, Aw. Q Q²,Q²)
We ignore the variance of Q2 since it is relatively small compared to weight quantization variance in practice. Q"e is defined in Algorithm and we have E[0x+1] 02 (V.)....
Let t/++1 02 aQg (V((((( + V205x+1 then it follows that
Wh+1 0ut1 = 0 - aQg V20ck+1 - Qk+1
and
Uk+1 = 1/ - aa(U((b)) + (b) + V20
where
Wh 0n Qz QV(((()))) VU(Wk)
Sa =
dh 0x +Q,(vū(0.)) vū(O,) VŪ VU((K) + VU((k) - (U())..
Note that E la
0z] = 0. Similar to previous proof, we know that
[|[] [[VU (vx))]1 ' [1 0x
2 have that
When 2x 00
Elllyh
11
||| [|[ (0--1 -a (VŪ((.-)))))) + V20cs Qd -Q. /2a voEx)) sign(r)e
E
16

---

Under review as a conference paper at ICLR 2022
Let
b = (0c-1 (0--1-aa + +/2 v
(0s-1 - aQ, + /2a mobk)
then
bl - Aw and
E
11 12
E
( (0p- aQo( (V(((())) + V2axx
(0-1-aQ0((V(())) + 22a -wOEs) - --iggn(r)c
E
20fx V2a U0ok 1 sign(r)ell
Vaax V2a -Uo5k- - b1( + E ||sig(())c|
E
Aw
| vyd
E
V2ak V2a 105x +
2
$ (V2ax V2ax- - vo) 2 1121 + (v2a- - /2a - 0o) EI + 2vnd
< (V2a- 2a - /2a - 22 + (v2a- - /2a - 20) Aw + 2v0 d
Since 2ry S x² + +3² we get
(V2a 220 - Uo Aw 5 (V2a V2a - 2 A2
(VV2x
/2a vo) ++0 00
follows
Elllibx - 0& 11-] S (2(V2o /2a 100 + +300 d.
Note that
V2a- /2a - 00 22x V2a- 110
20 (2ax - v0)
20
100
/2x + / 2nx 10
/2
The expectation becomes
Elllbh - 0u.111 < ( v1
+ 320 d.
Since 2a vo it follows
E Bll 0ul1 < (2ng + -30 d = 50gd.
Let A 5mnd. Then
[[[[]]] S M A.
and also
||[]]]| & V A.
Therefore,
8 min (AA.VA
The variance will be bounded by
E |[Ge
E[
E
(vū((O)) vū(O,) ||] +E -
vO(
1b 0
+
E
+ vU((A) VU(())))))
A3d
2
E
|| 8]]1
4
a
2
A
4
17

---

Under review as a conference paper at ICLR 2022
Supposing the distribution of vk+1 is Vx, it will give us the rate of
MVA
W2(ux, T) $ (1 (1- am)) W2(un; m) + 1.65( (.66(MM min
m
m
aA3d + 4o
A
4m
am
We also have
inf W2(HK, (eJ(a.)
/ |1x- ull (
vi)
1/2
S ||0k+1 -
ex+il?? S SVA.
Combining these two results, we get
W2(HK, T) S W2(ux, vx) + W2(UK,
V A MVA
< (1 am) K W2(Pu, x) L.5(MM min
m
m
La + 4w02
A
+ VA
4m
V A
M A
< (1 am) W2(Ho, T) 1.65 (M/ (m min
m
m
a yd +
4an?
A
+ a1l-am)) + 1) VA.
4m
am
When 2ax
A2 since assume that the gradient is bounded by E
Q (
sa
Ox + V2ac 19]
E Elllbb 0 2112
(0a-1 - aQ,((VivV00. (Q(((((())1)))
00/1 0a11]] +E V2
-0(00----00 Q" (B alg (Vū (0(((00))) 1)) 4ad
< max
2E
Using the bound equation (6) in Li & De Sa (2019),
[1 (Ok - aQ, (Vū(( Q" (0 aQ, (Q((((((
s a (Ve
s a
It follows
Elll% 0x < max (2 4od)
Let A = max a2, 4ad). The rest is that same the case 2a >
D COMPARISON TO SGD BOUNDS
Following previous work in comparing sampling and optimization methods (Ma al., 2019; Tal- war, 2019), we also compare our 2- 2-Wasserstein distance bound with previosu SGD bounds. Previous low-precision SGD convergence bounds are shown in terms of the squared distance to the optimum 0xx 8" I| (Yang et al., 2019). In order to compare our bounds with theirs, we consider a 2- Wasserstein distance between two point distributions. Let μuk be the point distribution assigns
18

---

Under review as a conference paper at ICLR 2022
probability everywhere except 0x and x be the point distribution assigns zero probability every- where except d" Then we get
W2(KK, = (Jejnt inf (J)()))))
/ || vll dJ(x.))
1/2
SE |10 8" ppplps
||0x 0 0-11]]= is proportional to A.. Therefore, our
From (Yang et al., 2019), we know that E 2- Wasserstein distance is ( whereas SGD's 2- Wasserstein distance is O(Aw), which shows SGLD is more robust to the quantization error.
E
ALGORITHMS WITH (BLOCK) FLOATING POINT NUMBERS
The qunatization gap floating point and block floating point is computed as
2 F where Elp] clip( (max [ę4) u)
block floating point floating point
(1(
2 where ETu] = clip( [log2 (1x() ,4,
Then deterministic rounding and stochastic rounding are defined with Aw. VC SGLDLP-L with (block) floating point is outlined in Algorithm 3
Algorithm VC SGLDLP-L with (Block) Floating Point.
given: Stepsize number of training iterations K, gradient quantizer Qu- deterministic rounding with (block) floating point Q", stochastic rounding with (block) floating point Q", F bits to represent the shared exponent (block floating point) or the exponent (floating point). W bits to represent each number in the block (block floating point) or the mantissa (floating point).
let + -2 u + 2
for k 1: K do
compute μ + 0 (VŪ(On-1)
compute (m) following Equation (3) update Oc+1 + Que (n, 20T, Au (w), Q4, Q*) end for
output: samples )
Algorithm 4 Variance-Correcaa Quantization Function Q"e with (Block) Floating Point.
input: (n, A, Q4. Q*)
Co + 2
if v > vo then
x < μ + VU UU- 1u--oo Vos where & ~ (O
+ (x)
recompute - (z) following Equation (3)
w.p. 2
c
w.p. No FIA
22
0,
otherwise
return (x) + (rgn
else
the same as in fixed point numbers end
The Q"e function with (block) floating point in Algorithm 4 is the same Algorithm except the line in red. We recompute the quantization gap after adding Gaussian noise to make sure it aligns with the quantization gap of
19

---

Under review as a conference paper at ICLR 2022
SGDLP-L
SGDLPAL
1.75
Naive SGLDLP-L
1.0
Naive SGLDLP-L
1.50
VC SGLDIP-L
VC SGLDIP-L
0.8
1.25
SGDLP-F
Test NLL
SGDLP-F
SGLDLP-F
SGLDLP-F
Test
1.00
SGDFP (0.2715)
SGDFP (0.068)
0.75
SGLDFP (0.2710)
0.4
SGLDFP (0.073)
0.2
0.25
2
3
4
5
6
7
8
9
10
2
3
4
5
10
Number of Fractional Bits
Number of Fractional Bits
Figure 3: Test NLL on MNIST dataset in terms of different precision.
F
EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
F.1 SAMPLING METHODS
For both SGLD and low-precision SGLD, we collected samples model's weight, and obtained the prediction test data {z", y² by Bayesian model averaging
from the posterior of the
1 J Σ ply xx, (.O,))
ply"|=, D)
F.2 MNIST
We train all methods on logistic regression and MLP for epochs with learning rate 0.1 and batch size 64. We additionally report negative log- liglliil (NLL) comparisons in Figure 3.
F.3 CIFAR AND IMDB
For CIFAR datasets, we use batch size 128, learning rate 0.5 and weight decay 5e - 4. We train the model for 245 epochs and used the same decay learning rate schedule in Yang et al. (2019). We collect 50 samples for SGLD. For cyclical learning rate schedule, we use 7 cycles and collect models per cycle (35 models total).
For IMDB dataset, we use batch size 80, learning rate 0.3 and weight decay 5e 4. We use a two- layer LSTM. The embedding dimension and the hidden dimension are 100 and 256 respectively. We train the model for 50 epochs and used the same decay learning rate schedule as on CIFAR datasets. We collect samples for SGLD. For cyclical learning rate schedule, we use cycles and collect 20 models.
F.4 IMAGENET
We use batch size 256, learning rate 0.2 and weight decay le - 4. We use the same decay learning rate schedule in He al. (2016) and collect 20 models for SGLD.
20


Reviews:
[{"review_id": "iYsfnM1vrWH", "reviewer": "Reviewer_dB5g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nPros: the authors conducted some bias analysis for low-precision SGLD based on [Dalalyan and Karagulyan 19]'s result.\n\nCons: \n\n1. **Limited novelty in methodology**: the novelty in terms of methodology is limited and is not very interesting. Simply extending quantization results from SGD to SGLD with further indications is not that interesting. \n\n2. **Limited depth in theory**: the theoretical novelty is limited and has limited depth; although I am very familiar with Dalalyan's work, the authors don't even bother to write down Theorem 4 and the relevant background in  [Dalalyan and Karagulyan 19].\n\n3. **Inconsistencies between experiments and theories**: The experiments are mainly running in non-convex settings, which don't match the theoretical claim in strongly convex scenarios. If I were the authors, I would only run MNIST experiments based on logistic regression; unless I have theoretical results that extend to non-convex settings. Because the theory only claims these contributions in strongly-convex settings. \n\n4. **Incomplete experiments**: Running SGLD for reproducing purely optimization results seems to be not interesting and uncertainty estimations would also be appreciated.", "summary_of_the_paper": "The author proposed a comprehensive study of low-precision Stochastic Gradient Langevin Dynamics (SGLD). They proposed the convergence analysis of low-precision Stochastic Gradient Langevin Dynamics on strongly log-concave distributions.  Motivated by the diverging fact of SGLDLP-L with small stepsizes, they proposed a variance-corrected quantization function to ensure a bounded bias. Empirical experiments based on large-scale DNN examples are evaluated.", "summary_of_the_review": "The paper proposed some analysis for low-precision SGLD but the novelties or depths in terms of methodology and theory are both limited. In addition, the experiments don't match the theoretical analysis and don't have the results for uncertainty estimations, which leads me to suspect why to bother to use SGLD (I believe SGD with fine-tuning parameters can also achieve the claimed results). Based on these evaluations, I tend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N3YoCsMCVSQ", "reviewer": "Reviewer_gLYz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the idea of applying low-precision optimization to SGLD is a direction worth exploring, but the overall presentation of the paper seems to be vague to me. While SGLD is viewed as an optimization algorithm particularly for nonconvex optimization problem which aims to minimize an objective function $U$ (see e.g., Raginsky et al., 2017), Langevin Monte Carlo (LMC), which takes exact the same algorithmic form as SGLD, is intrinsically a sampling algorithm which aims to sample the stationary distribution $\\exp(-U)$. The authors has been mixing up the tasks of optimization and sampling in the theoretical and experimental results, making it hard to understand what the task the proposed algorithm wants to solve. In the theoretical results of this paper, the authors leverage results in Dalalyan and Karagulyan (2019) to provide a (sampling) bound on the Wasserstein distance between the distribution generated by the low-precision SGLD and the target distribution, but compare this with an optimization bound in Yang et al. (2019), which is the distance between the SGD estimation and the optimum. This seems very weird to me since I think these are not comparable. \n\nThen, in the experiments, I think the authors are applying the proposed low-precision SGLD as a training (i.e., optimization) algorithm instead of a sampling algorithm. The metric used (test error) also does not seem to measure the accuracy of the sampled distribution using low-precision SGLD. Thus, because of this ambiguity between applying SGLD as a sampling algorithm versus an optimization (training) algorithm, the current results in this work are quite questionable to me. \n\nSome mathematical statements and notation in this paper are also unclear. In the assumption, the authors did not mention that the third condition is that the function $U$ needs to have a Lipschitz Hessian. Representation of the stochastic gradient of $U$ by $\\nabla\\tilde{U}$ is also weird since you actually did not change the function $U$ to $\\tilde{U}$, but the gradient to a stochastic gradient. A more commonly used notation should be $\\nabla\\widetilde{U}$. Also $\\mu_0$ is not defined in Theorem 1 (and typed as $\\nu_0$ in Theorem 2 which must be a typo). The derivation in Section 4.3 must be wrong as well, since $\\theta_{k+1}$ is a vector-valued random variable, its variance must be matrix-valued. Thus, the bottom part of page 5 is wrong. Also, using $\\theta_{k+1}$ to represent both the updates with and without quantization have been leading to much confusion during my review. \n\n\n---\nRaginsky, Maxim, Alexander Rakhlin, and Matus Telgarsky. \"Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis.\" Conference on Learning Theory. PMLR, 2017.\n", "summary_of_the_paper": "This paper proposes a low-precision version of SGLD which makes use of low-bit arithmetic. In particular, since low-precision gradient accumulators might lead to divergence of SGLD, the authors propose a new quantization function to preserve the correct variance in each update step. ", "summary_of_the_review": "The idea of  applying low-precision optimization to SGLD is a direction worth exploring, but the ambiguity between applying SGLD as a sampling algorithm versus an optimization (training) algorithm in this paper has made the current results in this work quite questionable. Some mathematical statements and notation in this paper are also unclear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9ObRIKCBmY0", "reviewer": "Reviewer_yqrt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The narrative is fluent and flow of statement is smooth and natural. In general, the writing of the manuscript is well done.\n- The authors have tried their proposed algorithms in multiple classic application scenarios.\n\nWeaknesses:\n- The submission does not appear to be a major contribution to the field, as the main novel point is the analysis of the variance of low-precision gradient estimator.\n\nSpecifics:\n- page 3, section 3.2, paragraph 3, line 3: $l = -2^{W-F-2}$, why the power is $-2$ instead of $-1$ when computing the upper bound of presentable number?\n\n- page 5, section 4.2, the line below the statement of theorem 2: I think theorem 2 does not suggest \"SGLDLP-L diverges from the target distribution\", but instead, it shows the bound becomes loose when stepsize $\\alpha$ decreases. \n\n- page 4 and 5: schemes SGLDLP-F shown in equation (1) and SGLDLP-L shown in equation (2) essentially only differs in whether the noise $\\xi_{k}$ is low-precision thresholded, doesn't it? \n\n- The main theorems are proved with the results from Dalalyan and Karagulyan's work. The author should explicitly cite the specific theorem in [DK19] and explain why the argued variance of low-precision gradient estimator renders the conditions of the cited theorem satisfied and thus the theorem applicable.", "summary_of_the_paper": "This paper discusses using the stochastic gradient Langevin dynamics with low-precision implementation. The authors present convergence results in Wasserstein distance norm for three implementation cases, which distinguish specific low-precision thresholding schemes and different quantization techniques. The authors also report simulation outcome of proposed low-precision SGLD algorithms and show their convergence in actual application.", "summary_of_the_review": "In light of the specifics in the main review, the discussion of low-precision SGLD in this submission is not significant enough and there are some places in the argument that are not clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "39GB87-aYj", "reviewer": "Reviewer_bvZD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Posterior sampling of quantized neural networks is an important problem to study. While the optimization of quantized neural networks are relatively well studied, this paper is the first to consider the sampling of such networks. The technical quality of the is nice. It presents reasonable convergence results for the proposed SGLDLP algorithms. The idea of utilizing the quantization noise for sampling and variance correction makes sense.\n\nIn terms of weakness, I think the assumptions in the paper are worth considering:\n1. the gradient quantizer assumes that we first have a full-precision gradient, and then quantize it, maybe for reducing the communication cost for distributed training. The formulation does not support applications such as mixed precision training, where the gradient itself is computed in an approximate way. Is distributed training the case the authors want to address?\n2. While the convergence result presented in Theorem 1 is good, it relies on a non-converging term relates to Delta_w. It is curious to compare with the following post-training quantization algorithm:\na' generate full precision samples {theta_k} with full-precision SGLD\nb' use the quantized version {Q(theta_k)} as the low-precision samples;\nWe can imagine that the posterior distribution given by this algorithm is also close enough to the true posterior, where the distance is controlled by Delta_w. It would be better if we can show the proposed SGLDLP is better than the post-training quantization algorithm.\n\nPost rebuttal\n====\n\nThanks for addressing my concerns. I am tending to keep my score since I think it is an interesting paper with some novelties, but the novelty is limited, as commented by other reviewers. ", "summary_of_the_paper": "This paper investigate posterior sampling of weight-quantized neural networks with SGLD. The paper gives the convergence of SGLD under Wasserstein distance with quantized weights and gradients. It also considers the situation when the accumulator is also in low precision, and proposes a variance correction strategy to deal with the additional quantization noise. ", "summary_of_the_review": "A novel paper with reasonable techniques. The results might be improved by tightening the assumptions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the performance of low-precision Stochastic Gradient Langevin Dynamics (SGLD). While similar low-precision techniques have been widely used in optimization, much less is known for Markov Chain Monte Carlo (MCMC) methods. The paper develops a new quantization function to make SGLD suitable for low-precision setups and argues for its use in deep learning. 

The main concerns among the reviewers were related to the paper presentation (separation and comparison between optimization and sampling), comparison to Dalalyan-Karagulyan'19 and overview of this work, technical depth, and numerical experiments. The authors have adequately responded to the reviewers' comments and addressed them to the extent possible. However, there was ultimately not enough support to lead this paper to acceptance.

I find low-precision sampling a worthy topic of study and the contributions of the paper are interesting. The authors are encouraged to revise the paper based on the reviewers' comments, more clearly highlight the contributions, and resubmit.
Decision: Reject