Dimension-Free Bounds for Low-Precision Training

1 INTRODUCTION . As machine learning models continue to scale to target larger problems on bigger data , the task of training these models quickly and efficiently becomes an ever-more-important problem . One promising technique for doing this is low-precision computation , which replaces the 32-bit or 64-bit floating point numbers that are usually used in ML computations with smaller numbers , often 8-bit or 16-bit fixed point numbers . Low-precision computation is a broadly applicable technique that has received a lot of attention , especially for deep learning , and specialized hardware accelerators have been developed to support it ( Jouppi et al. , 2017 ; Burger , 2017 ; Caulfield et al. , 2017 ) . A major application for low-precision computation is the training of ML models using empirical risk minimization . This training is usually done using stochastic gradient descent ( SGD ) , and most research in low-precision training has focused on low-precision versions of SGD . While most of this work is empirical ( Wu et al. , 2018 ; Das et al. , 2018 ; Zhu et al. , 2016 ; Köster et al. , 2017 ; Lee et al. , 2017 ; Hubara et al. , 2016 ; Rastegari et al. , 2016 ; Zhou et al. , 2016 ; Gupta et al. , 2015 ; Courbariaux et al. , 2014 ; 2015 ; De Sa et al. , 2017 ) , significant research has also been done in the theoretical analysis of low-precision training . This theoretical work has succeeded in proving bounds on the convergence rate of low-precision SGD and related low-precision methods in various settings , including for convex ( De Sa et al. , 2018 ; Zhang et al. , 2017 ) and non-convex objectives ( De Sa et al. , 2015 ; Li et al. , 2017 ; Alistarh et al. , 2017 ) . One common characteristic of these results is that the bounds tend to depend on the dimension d of the model being learned ( equivalently , d is the number of parameters ) . For example , ( Li et al. , 2017 ) gives the convergence bound E [ f ( w̄T ) − f ( w∗ ) ] ≤ ( 1 + log ( T + 1 ) ) σ2max 2µT + σmaxδ √ d 2 , ( 1 ) where the objective f is strongly convex with parameter µ , low-precision SGD outputs w̄T after T iterations , w∗ is the true global minimizer of the objective , σ2max is an upper bound on the second moment of the stochastic gradient samples E [ ‖f̃ ( w ) ‖22 ] ≤ σ2max , and δ is the quantization step , the difference between adjacent numbers in the low-precision format . Notice that , as T → ∞ , this bound shows convergence down to a level of error that increases with the dimension d. Equivalently , in order to achieve the same level of error as d increases , we would need to use more bits of quantization to make δ smaller . Similar dimension-dependent results , where either the error or the number of bits needed increases with d , can also be seen in other work on low-precision training algorithms ( Alistarh et al. , 2017 ; Zhang et al. , 2017 ; De Sa et al. , 2018 ) . This dependence on d is unsatisfying because the motivation for low-precision training is to tackle large-scale problems on big data , where d can range up to 108 or more for commonly used models ( Simonyan and Zisserman , Table 1 : Summary of our dimension-free results compared with prior work . The values report the number of bits needed , according to the theoretical bound , for the LP-SGD ( Li et al. , 2017 ) algorithm to achieve an expected objective gap ( f ( w ) − f ( w∗ ) ) of when we let step size α → 0 , epoch length T →∞ . Here we let R denote the radius of the range of numbers representable in the low-precision format and assume ‖w∗‖2 = Θ ( R ) . The rest of the parameters can be found in the assumptions to be introduced later . NUMBER OF BITS NEEDED FOR E [ f ( w ) − f ( w∗ ) ] ≤ PRIOR DIMENSION-DEPENDENT BOUND log2O ( Rσmax √ d/ε ) OUR DIMENSION-FREE BOUND log2O ( Rσ1/ε ) DIMENSION-FREE WITH LOGARITHMIC QUANTIZATION log2O ( ( Rσ/ε ) · log ( 1 + σ1/σ ) ) 2014 ) . For example , to compensate for a factor of d = 108 in ( 1 ) , we could add bits to decrease the quantization step δ by a factor of √ d , but this would require adding log2 ( 10 4 ) ≈ 13 bits , which is significant compared to the 8 or 16 bits that are commonly used in low-precision training . In this paper , we address this problem by proving dimension-free bounds on the convergence of LP-SGD Li et al . ( 2017 ) . Our main technique for doing so is a tight dimension-independent bound on the expected quantization error of the low-precision stochastic gradients in terms of the ` 1-norm . Our results are summarized in Table 1 , and we make the following contributions : • We describe conditions under which we can prove a dimension-free bound on the convergence of SGD with fixed-point , quantized iterates on strongly convex problems . • We study non-linear quantization schemes , in which the representable low-precision numbers are distributed non-uniformly . We prove dimension-free convergence bounds for SGD using logarithmic quantization ( Lee et al. , 2017 ) , and we show that using logarithmic quantization can reduce the number of bits needed for LP-SPG to provably converge . • We study quantization using low-precision floating-point numbers , and we present theoretical analyis that suggests how to assign a given number of bits to exponent and mantissa to optimize the accuracy of training algorithms . We validate our results experimentally . 2 RELATED WORK . Motivated by the practical implications of faster machine learning , much work has been done on low-precision training . This work can be roughly divided into two groups . The first focuses on training deep models with low-precision weights , to be later used for faster inference . For some applications , methods of this type have achieved good results with very low-precision models : for example , binarized ( Courbariaux et al. , 2015 ; Hubara et al. , 2016 ; Rastegari et al. , 2016 ) and ternary networks ( Zhu et al. , 2016 ) have been observed to be effective ( although as is usual for deep learning they lack theoretical convergence results ) . However , these approaches are still typically trained with full-precision iterates : the goal is faster inference , not faster training ( although faster training is often achieved as a bonus side-effect ) . A second line of work on low-precision training , which is applied to both DNN training and nondeep-learning tasks , focuses on making various aspects of SGD low-precision , while still trying to solve the same optimization problem as the full-precision version . The most common way to do this is to make the iterates of SGD ( the wt in the SGD update step wt+1 = wt − αt∇ft ( wt ) ) stored and computed in low-precision arithmetic ( Courbariaux et al. , 2014 ; Gupta et al. , 2015 ; De Sa et al. , 2018 ; 2015 ; Li et al. , 2017 ) . This is the setting we will focus on most in this paper , because it has substantial theoretical prior work which exhibits the dimension-dependence we set out to study ( Li et al. , 2017 ; Zhang et al. , 2017 ; Alistarh et al. , 2017 ; De Sa et al. , 2018 ) . The only paper we found with a bound that was not dimension-dependent was De Sa et al . ( 2015 ) , but in that paper the authors required that the gradient samples be 1-sparse ( have only one nonzero entry ) , which is not a realistic assumption for most ML training tasks . In addition to quantizing the iterates , other work has studied quantizing the training set ( Zhang et al. , 2017 ) and numbers used to communicate among parallel workers ( Alistarh et al. , 2017 ) . We expect that our results on dimension-free bounds will be complementary with these existing theoretical approaches , and we hope that they can help to explain the success of the exciting empirical work in this area . 3 DIMENSION-FREE BOUNDS FOR SGD . In this section , we analyze the performance of stochastic gradient descent ( SGD ) using low-precision training . Though there are numerous variants of this algorithm , SGD remains the de facto algorithm used most for machine learning . We will start by describing SGD and how it can be made lowprecision . Suppose we are trying to solve the problem minimize : f ( w ) = 1 n n∑ i=1 f̃i ( w ) over : w ∈ Rd . ( 2 ) SGD solves this problem iteratively by repeatedly running the update step wt+1 = wt − α∇f̃it ( wt ) ( 3 ) where α is the step size1 or learning rate , and it is the index of a component function chosen randomly and uniformly at each iteration from { 1 , . . . , n } . To make this algorithm low-precision , we quantize the iterates ( the vectors wt ) and store them in a low-precision format . The standard format to use lets us represent numbers in a set dom ( δ , b ) = { −δ · 2b−1 , · · · , −δ , 0 , δ , · · · , δ · ( 2b−1 − 1 ) } with δ > 0 being the quantization gap , the distance between adjacent representable numbers , and b ∈ N being the number of bits we use ( De Sa et al. , 2018 ) . Usually , δ is a power of 2 , and this scheme is called fixed-point arithmetic . It is straightforward to encode numbers in this set as b-bit signed integers , by just multiplying or dividing by δ to convert to or from the encoded format— and we can even do many arithmetic computations on these numbers directly as integers . This is sometimes called linear quantization because the representable points are distributed uniformly throughout their range . However , as the gradient samples will produce numbers outside this set during iteration , we need some way to map these numbers to the set of numbers that we can represent . The standard way to do this is with a quantization function Q ( x ) : R → dom ( δ , b ) . While many quantization functions have been proposed , the one typically used in theoretical analysis ( which we will continue to use here ) is randomized rounding . Randomized rounding , also known as unbiased rounding or stochastic rounding , rounds up or down at random such that E [ Q ( x ) ] = x whenever x is within the range of representable numbers ( i.e . when −δ · 2b−1 ≤ x ≤ δ · ( 2b−1 − 1 ) ) . When x is outside that range , we quantize it to the closest representable point . When we apply Q ( δ , b ) to a vector argument , it quantizes each of its components independently . Using this quantization function , we can write the update step for low-precision SGD ( LP-SGD ) , which is a simple quantization of ( 3 ) , wt+1 = Q ( wt − α∇f̃it ( wt ) ) ( 4 ) As mentioned before , one common feature of prior bounds on the convergence of LP-SGD is that they depend on the number of dimensions d , whereas bounds on full precision SGD under the same conditions do not do so . This difference is due to the fact that , when we use a quantization function Q to quantize a number w , it increases its variance by E [ ( Q ( w ) − w ) 2 ] ≤ δ2/4 . Observe that this inequality is tight since it holds as an equality when w is in the middle of two quantization points , e.g . w = δ/2 , as illustrated in Figure 1 ( a ) . When quantizing a vector w ∈ Rd , the squared error can be increased by E [ ‖Q ( w ) − w‖22 ] = d∑ k=1 E [ ( Q ( wk ) − wk ) 2 ] ≤ δ 2d 4 , ( 5 ) and this bound is again tight . This variance inequality is the source of the d term in analyses of LP-SGD , and the tightness of the bound leads to the natural belief that the d term is inherent , and that low-precision results are inevitably dimension-dependent . However , we propose that if we can instead bound the variance in ( 5 ) with some properties of the problem itself that do not change as d changes , we can achieve a result that is dimensionindependent . One way to do this is to look at the variance graphically . Figure 3 plots the quantization error as a function of w along with the bound in ( 5 ) . Notice that the squared error looks like a series of parabolas , and the bound in ( 5 ) is tight at the top of those parabolas , but loose elsewhere . Instead , suppose we want to do the opposite and produce a bound that is tight when the error is zero ( at points in dom ( δ , b ) ) . To do this , we observe that E [ ( Q ( w ) − w ) 2 ] ≤ δ|w−z| for any z ∈ dom ( δ , b ) . This bound is also tight when z is adjacent to w , and we plot it in Figure 3 as well . The natural vector analog of this is E [ ‖Q ( w ) − w‖22 ] ≤ ∑d k=1 δ|wk − zk| = δ ‖w − z‖1 , ∀z ∈ dom ( δ , b ) d ( 6 ) 1Usually in SGD the step size is decreased over time , but here for simplicity we consider a constant learning rate schedule . where ‖·‖1 denotes the L1 norm . This is a dimension-independent bound we can use to replace ( 5 ) to bound the convergence of LP-SGD and other algorithms . However , this replacement is nontrivial as our bound is now non-constant : it depends on w , which is a variable updated each iteration . Also , in order to bound this new L1 norm term , we will need some new assumptions about the problem . Next , we will state these assumptions , along with the standard assumptions used in the analysis of SGD for convex objectives , and then we will use them to present our dimension-free bound on the convergence of SGD . Assumption 1 . All the loss functions f̃i are differentiable , and their gradients are L-Lipschitz continuous in the sense of 2-norm , that is , ∀i ∈ { 1 , 2 , · · · , n } , ∀x , y ∈ Rd , ‖∇f̃i ( x ) −∇f̃i ( y ) ‖2 ≤ L ‖x− y‖2 Assumption 2 . All the gradients of the loss functions f̃i are L1-Lipschitz continuous in the sense of 1-norm to 2-norm , that is , ∀i ∈ { 1 , 2 , · · · , n } , ∀x , y ∈ Rd , ‖∇f̃i ( x ) −∇f̃i ( y ) ‖1 ≤ L1 ‖x− y‖2 These two assumptions are simply expressing of Lipschitz continuity in different norms . Assumption 1 is a standard assumption in the analysis of SGD on convex objectives , and has been applied in the low-precision case as well in prior work ( De Sa et al. , 2018 ) . Assumption 2 is analogous to 1 , except we are bounding the L1 norm instead of the L2 norm . This holds naturally ( with a reasonable value of L1 ) for many problems , in particular problems for which the gradient samples are sparse . Assumption 3 . The gradient of the total loss function f is µ-strongly convex for some µ > 0 : ∀w , v , f ( w ) − f ( v ) − µ 2 ‖w − v‖22 ≥ ( w − v ) T∇f ( v ) Assumption 3 is a standard assumption that bounds the curvature of the loss function f , and is satisfied for many classes of convex objectives . For example , any convex loss with L2 regularization will always be strongly convex . When an objective is strongly convex and Lipschitz continuous , it is standard to say it has condition number κ = L/µ , and here we extend this to say it has L1 condition number κ1 = L1/µ . Assumption 4 . The gradient of each loss function is bounded by some constant σ near the optimal point in the sense of l1 and l2 norm , that is , E [ ∥∥∥∇f̃i ( w∗ ) ∥∥∥2 2 ] ≤ σ2 , E [ ∥∥∥∇f̃i ( w∗ ) ∥∥∥ 1 ] ≤ σ1 This assumption constrains the gradient for each loss function at the optimal point . We know ∇f ( w∗ ) = 1n ∑ i ∇̃fi ( w∗ ) = 0 , so it is intuitive that each ∇f̃i ( w∗ ) can be bounded by some value . Therefore this is a natural assumption to make and it has been used in a lot of other work in this area . Note that this assumption only needs to hold under the expectation over all f̃i . With these assumptions , we proved the following theorem for low-precision SGD : Theorem 1 . Suppose that we run LP-SGD on an objective that satisfies Assumptions 1–4 , and with step size α < 1/ ( 2κ2µ ) . After T LP-SGD update steps ( 4 ) , select w̄T uniformly at random from { w0 , w1 , . . . , wT−1 } . Then , the expected objective gap of w̄T is bounded by E [ f ( w̄T ) − f ( w∗ ) ] ≤ 1 2αT ‖w0 − w∗‖22 + ασ2 + δσ1 2 + δ2κ21µ 4 This theorem shows a bound of the expected distance between the result we get at K-th iteration and the optimal value . By choosing an appropriate step size we can achieve convergence at a 1/T rate , while the limit we converge to is only dependent on dimension-free factors . Meanwhile , as mentioned in the first section , previous work gives a dimension-dependent bound ( 1 ) for the problem , which also converges at a 1/T rate.2 Therefore our result guarantees a dimension-independent convergence limit without weakening the convergence rate . It is important to note that , because the dimension-dependent bound in ( 5 ) was tight , we should not expect our new result to improve upon the previous theory in all cases . In the worst case , κ1 = √ d·κ and similarly σ1 = √ d · σ ; this follows from the fact that for vectors in Rd , the norms are related by the inequality ‖x‖1 ≤ √ d · ‖x‖2 . Substituting this into our result produces a dimension-dependent bound again . This illustrates the importance of introducing the new parameters κ1 and σ1 and requiring that they be bounded ; if we could not express our bound in terms of these parameters , the best we could do here is recover a dimension-dependent bound . Experiments Next , we validate our theoretical results experimentally . To do this , we analyzed how the size of the noise floor of convergence of SGD and LP-SGD varies as the dimension is changed for a class of synthetic problems . Importantly , we needed to pick a class of problems for which the parameters L , L1 , µ , σ , and σ1 , did not change as we changed the dimension d. To do this , we chose a class of synthetic linear regression models with loss components sampled independently and identically as f̃i ( w ) = 1 2 ( x̃Tw − ỹ ) 2 where x̃ is a sparse vector sampled to have s nonzero entries each of which is sampled uniformly from { −1 , 1 } , and ỹ is sampled fromN ( x̃Tw∗ , β2 ) for some variance parameter β . Importantly , the nonzero entries of x̃ were chosen non-uniformly such that Pr [ x̃i 6= 0 ] = pi for some probabilities pi which decrease as i increases ; this lets us ensure that µ remains constant as d is increased . For simplicity , we sampled a fresh loss component of this form at each SGD iteration , which is sometimes called the online setting . It is straightforward to derive that for this problem µ = pd L = s L1 = s √ s σ2 = β2s σ1 = √ 2s/πσ . We set α = 0.01 , β = 0.2 , p1 = 0.9 , pd = 0.001 , and s = 16 , we chose each entry of w∗ uniformly from [ −1/2 , 1/2 ] , and we set δ such that the low-precision numbers would range from −1 to 1 . Figure 1 ( b ) shows the convergence of SGD and LP-SGD as the dimension d is changed , for both 8-bit and 6-bit quantization . Notice that while changing d has an effect on the initial convergence rate for both SGD and LP-SGD , it has no effect on the noise ball size , the eventual loss gap that the algorithm converges to . Figure 2 ( a ) measures this noise ball size more explicitly as the dimension is changed : it reports the loss gap averaged across the second half of the iterates . Notice that as the dimension d is changed , the average loss gap is almost unchanged , even for very low-precision methods for which the precision does significantly affect the size of the noise ball . This validates our dimension-free bounds , and shows that they can describe the actual dependence on d in at least one case . Figure 2 ( b ) validates our results in the opposite way : it looks at how this gap changes as our new parameters σ1 and L1 change while d , µ , and σ are kept fixed . To do this , we fixed d = 1024 and changed s across a range , setting β = 0.8/ √ s , which keeps σ2 constant as s is changed : this has the effect of changing σ1 ( and , as a side effect , L1 and L ) . We can see from figure 2 ( b ) that changing σ1 in this way has a much greater effect on LP-SGD than it does on SGD . This validates our theoretical results , and suggests that σ1 and L1 can effectively determine the effect of low-precision compute on SGD .

Summary: The paper considers the problem of low precision stochastic gradient descent. Specifically, they study updates of the form x_{t + 1} = Q (x_t - alpha * g_t), where g_t is a stochastic gradient, and Q is a quantization function. The goal is to produce quantization functions that simultaneously increase the convergence rate as little as possible, while also requiring few bits to represent. This is motivated by the desire to perform SGD on low precision machines.