Dimension-Free Bounds for Low-Precision Training

1 INTRODUCTION . As machine learning models continue to scale to target larger problems on bigger data , the task of training these models quickly and efficiently becomes an ever-more-important problem . One promising technique for doing this is low-precision computation , which replaces the 32-bit or 64-bit floating point numbers that are usually used in ML computations with smaller numbers , often 8-bit or 16-bit fixed point numbers . Low-precision computation is a broadly applicable technique that has received a lot of attention , especially for deep learning , and specialized hardware accelerators have been developed to support it ( Jouppi et al. , 2017 ; Burger , 2017 ; Caulfield et al. , 2017 ) . A major application for low-precision computation is the training of ML models using empirical risk minimization . This training is usually done using stochastic gradient descent ( SGD ) , and most research in low-precision training has focused on low-precision versions of SGD . While most of this work is empirical ( Wu et al. , 2018 ; Das et al. , 2018 ; Zhu et al. , 2016 ; Köster et al. , 2017 ; Lee et al. , 2017 ; Hubara et al. , 2016 ; Rastegari et al. , 2016 ; Zhou et al. , 2016 ; Gupta et al. , 2015 ; Courbariaux et al. , 2014 ; 2015 ; De Sa et al. , 2017 ) , significant research has also been done in the theoretical analysis of low-precision training . This theoretical work has succeeded in proving bounds on the convergence rate of low-precision SGD and related low-precision methods in various settings , including for convex ( De Sa et al. , 2018 ; Zhang et al. , 2017 ) and non-convex objectives ( De Sa et al. , 2015 ; Li et al. , 2017 ; Alistarh et al. , 2017 ) . One common characteristic of these results is that the bounds tend to depend on the dimension d of the model being learned ( equivalently , d is the number of parameters ) . For example , ( Li et al. , 2017 ) gives the convergence bound E [ f ( w̄T ) − f ( w∗ ) ] ≤ ( 1 + log ( T + 1 ) ) σ2max 2µT + σmaxδ √ d 2 , ( 1 ) where the objective f is strongly convex with parameter µ , low-precision SGD outputs w̄T after T iterations , w∗ is the true global minimizer of the objective , σ2max is an upper bound on the second moment of the stochastic gradient samples E [ ‖f̃ ( w ) ‖22 ] ≤ σ2max , and δ is the quantization step , the difference between adjacent numbers in the low-precision format . Notice that , as T → ∞ , this bound shows convergence down to a level of error that increases with the dimension d. Equivalently , in order to achieve the same level of error as d increases , we would need to use more bits of quantization to make δ smaller . Similar dimension-dependent results , where either the error or the number of bits needed increases with d , can also be seen in other work on low-precision training algorithms ( Alistarh et al. , 2017 ; Zhang et al. , 2017 ; De Sa et al. , 2018 ) . This dependence on d is unsatisfying because the motivation for low-precision training is to tackle large-scale problems on big data , where d can range up to 108 or more for commonly used models ( Simonyan and Zisserman , Table 1 : Summary of our dimension-free results compared with prior work . The values report the number of bits needed , according to the theoretical bound , for the LP-SGD ( Li et al. , 2017 ) algorithm to achieve an expected objective gap ( f ( w ) − f ( w∗ ) ) of when we let step size α → 0 , epoch length T →∞ . Here we let R denote the radius of the range of numbers representable in the low-precision format and assume ‖w∗‖2 = Θ ( R ) . The rest of the parameters can be found in the assumptions to be introduced later . NUMBER OF BITS NEEDED FOR E [ f ( w ) − f ( w∗ ) ] ≤ PRIOR DIMENSION-DEPENDENT BOUND log2O ( Rσmax √ d/ε ) OUR DIMENSION-FREE BOUND log2O ( Rσ1/ε ) DIMENSION-FREE WITH LOGARITHMIC QUANTIZATION log2O ( ( Rσ/ε ) · log ( 1 + σ1/σ ) ) 2014 ) . For example , to compensate for a factor of d = 108 in ( 1 ) , we could add bits to decrease the quantization step δ by a factor of √ d , but this would require adding log2 ( 10 4 ) ≈ 13 bits , which is significant compared to the 8 or 16 bits that are commonly used in low-precision training . In this paper , we address this problem by proving dimension-free bounds on the convergence of LP-SGD Li et al . ( 2017 ) . Our main technique for doing so is a tight dimension-independent bound on the expected quantization error of the low-precision stochastic gradients in terms of the ` 1-norm . Our results are summarized in Table 1 , and we make the following contributions : • We describe conditions under which we can prove a dimension-free bound on the convergence of SGD with fixed-point , quantized iterates on strongly convex problems . • We study non-linear quantization schemes , in which the representable low-precision numbers are distributed non-uniformly . We prove dimension-free convergence bounds for SGD using logarithmic quantization ( Lee et al. , 2017 ) , and we show that using logarithmic quantization can reduce the number of bits needed for LP-SPG to provably converge . • We study quantization using low-precision floating-point numbers , and we present theoretical analyis that suggests how to assign a given number of bits to exponent and mantissa to optimize the accuracy of training algorithms . We validate our results experimentally . 2 RELATED WORK . Motivated by the practical implications of faster machine learning , much work has been done on low-precision training . This work can be roughly divided into two groups . The first focuses on training deep models with low-precision weights , to be later used for faster inference . For some applications , methods of this type have achieved good results with very low-precision models : for example , binarized ( Courbariaux et al. , 2015 ; Hubara et al. , 2016 ; Rastegari et al. , 2016 ) and ternary networks ( Zhu et al. , 2016 ) have been observed to be effective ( although as is usual for deep learning they lack theoretical convergence results ) . However , these approaches are still typically trained with full-precision iterates : the goal is faster inference , not faster training ( although faster training is often achieved as a bonus side-effect ) . A second line of work on low-precision training , which is applied to both DNN training and nondeep-learning tasks , focuses on making various aspects of SGD low-precision , while still trying to solve the same optimization problem as the full-precision version . The most common way to do this is to make the iterates of SGD ( the wt in the SGD update step wt+1 = wt − αt∇ft ( wt ) ) stored and computed in low-precision arithmetic ( Courbariaux et al. , 2014 ; Gupta et al. , 2015 ; De Sa et al. , 2018 ; 2015 ; Li et al. , 2017 ) . This is the setting we will focus on most in this paper , because it has substantial theoretical prior work which exhibits the dimension-dependence we set out to study ( Li et al. , 2017 ; Zhang et al. , 2017 ; Alistarh et al. , 2017 ; De Sa et al. , 2018 ) . The only paper we found with a bound that was not dimension-dependent was De Sa et al . ( 2015 ) , but in that paper the authors required that the gradient samples be 1-sparse ( have only one nonzero entry ) , which is not a realistic assumption for most ML training tasks . In addition to quantizing the iterates , other work has studied quantizing the training set ( Zhang et al. , 2017 ) and numbers used to communicate among parallel workers ( Alistarh et al. , 2017 ) . We expect that our results on dimension-free bounds will be complementary with these existing theoretical approaches , and we hope that they can help to explain the success of the exciting empirical work in this area . 3 DIMENSION-FREE BOUNDS FOR SGD . In this section , we analyze the performance of stochastic gradient descent ( SGD ) using low-precision training . Though there are numerous variants of this algorithm , SGD remains the de facto algorithm used most for machine learning . We will start by describing SGD and how it can be made lowprecision . Suppose we are trying to solve the problem minimize : f ( w ) = 1 n n∑ i=1 f̃i ( w ) over : w ∈ Rd . ( 2 ) SGD solves this problem iteratively by repeatedly running the update step wt+1 = wt − α∇f̃it ( wt ) ( 3 ) where α is the step size1 or learning rate , and it is the index of a component function chosen randomly and uniformly at each iteration from { 1 , . . . , n } . To make this algorithm low-precision , we quantize the iterates ( the vectors wt ) and store them in a low-precision format . The standard format to use lets us represent numbers in a set dom ( δ , b ) = { −δ · 2b−1 , · · · , −δ , 0 , δ , · · · , δ · ( 2b−1 − 1 ) } with δ > 0 being the quantization gap , the distance between adjacent representable numbers , and b ∈ N being the number of bits we use ( De Sa et al. , 2018 ) . Usually , δ is a power of 2 , and this scheme is called fixed-point arithmetic . It is straightforward to encode numbers in this set as b-bit signed integers , by just multiplying or dividing by δ to convert to or from the encoded format— and we can even do many arithmetic computations on these numbers directly as integers . This is sometimes called linear quantization because the representable points are distributed uniformly throughout their range . However , as the gradient samples will produce numbers outside this set during iteration , we need some way to map these numbers to the set of numbers that we can represent . The standard way to do this is with a quantization function Q ( x ) : R → dom ( δ , b ) . While many quantization functions have been proposed , the one typically used in theoretical analysis ( which we will continue to use here ) is randomized rounding . Randomized rounding , also known as unbiased rounding or stochastic rounding , rounds up or down at random such that E [ Q ( x ) ] = x whenever x is within the range of representable numbers ( i.e . when −δ · 2b−1 ≤ x ≤ δ · ( 2b−1 − 1 ) ) . When x is outside that range , we quantize it to the closest representable point . When we apply Q ( δ , b ) to a vector argument , it quantizes each of its components independently . Using this quantization function , we can write the update step for low-precision SGD ( LP-SGD ) , which is a simple quantization of ( 3 ) , wt+1 = Q ( wt − α∇f̃it ( wt ) ) ( 4 ) As mentioned before , one common feature of prior bounds on the convergence of LP-SGD is that they depend on the number of dimensions d , whereas bounds on full precision SGD under the same conditions do not do so . This difference is due to the fact that , when we use a quantization function Q to quantize a number w , it increases its variance by E [ ( Q ( w ) − w ) 2 ] ≤ δ2/4 . Observe that this inequality is tight since it holds as an equality when w is in the middle of two quantization points , e.g . w = δ/2 , as illustrated in Figure 1 ( a ) . When quantizing a vector w ∈ Rd , the squared error can be increased by E [ ‖Q ( w ) − w‖22 ] = d∑ k=1 E [ ( Q ( wk ) − wk ) 2 ] ≤ δ 2d 4 , ( 5 ) and this bound is again tight . This variance inequality is the source of the d term in analyses of LP-SGD , and the tightness of the bound leads to the natural belief that the d term is inherent , and that low-precision results are inevitably dimension-dependent . However , we propose that if we can instead bound the variance in ( 5 ) with some properties of the problem itself that do not change as d changes , we can achieve a result that is dimensionindependent . One way to do this is to look at the variance graphically . Figure 3 plots the quantization error as a function of w along with the bound in ( 5 ) . Notice that the squared error looks like a series of parabolas , and the bound in ( 5 ) is tight at the top of those parabolas , but loose elsewhere . Instead , suppose we want to do the opposite and produce a bound that is tight when the error is zero ( at points in dom ( δ , b ) ) . To do this , we observe that E [ ( Q ( w ) − w ) 2 ] ≤ δ|w−z| for any z ∈ dom ( δ , b ) . This bound is also tight when z is adjacent to w , and we plot it in Figure 3 as well . The natural vector analog of this is E [ ‖Q ( w ) − w‖22 ] ≤ ∑d k=1 δ|wk − zk| = δ ‖w − z‖1 , ∀z ∈ dom ( δ , b ) d ( 6 ) 1Usually in SGD the step size is decreased over time , but here for simplicity we consider a constant learning rate schedule . where ‖·‖1 denotes the L1 norm . This is a dimension-independent bound we can use to replace ( 5 ) to bound the convergence of LP-SGD and other algorithms . However , this replacement is nontrivial as our bound is now non-constant : it depends on w , which is a variable updated each iteration . Also , in order to bound this new L1 norm term , we will need some new assumptions about the problem . Next , we will state these assumptions , along with the standard assumptions used in the analysis of SGD for convex objectives , and then we will use them to present our dimension-free bound on the convergence of SGD . Assumption 1 . All the loss functions f̃i are differentiable , and their gradients are L-Lipschitz continuous in the sense of 2-norm , that is , ∀i ∈ { 1 , 2 , · · · , n } , ∀x , y ∈ Rd , ‖∇f̃i ( x ) −∇f̃i ( y ) ‖2 ≤ L ‖x− y‖2 Assumption 2 . All the gradients of the loss functions f̃i are L1-Lipschitz continuous in the sense of 1-norm to 2-norm , that is , ∀i ∈ { 1 , 2 , · · · , n } , ∀x , y ∈ Rd , ‖∇f̃i ( x ) −∇f̃i ( y ) ‖1 ≤ L1 ‖x− y‖2 These two assumptions are simply expressing of Lipschitz continuity in different norms . Assumption 1 is a standard assumption in the analysis of SGD on convex objectives , and has been applied in the low-precision case as well in prior work ( De Sa et al. , 2018 ) . Assumption 2 is analogous to 1 , except we are bounding the L1 norm instead of the L2 norm . This holds naturally ( with a reasonable value of L1 ) for many problems , in particular problems for which the gradient samples are sparse . Assumption 3 . The gradient of the total loss function f is µ-strongly convex for some µ > 0 : ∀w , v , f ( w ) − f ( v ) − µ 2 ‖w − v‖22 ≥ ( w − v ) T∇f ( v ) Assumption 3 is a standard assumption that bounds the curvature of the loss function f , and is satisfied for many classes of convex objectives . For example , any convex loss with L2 regularization will always be strongly convex . When an objective is strongly convex and Lipschitz continuous , it is standard to say it has condition number κ = L/µ , and here we extend this to say it has L1 condition number κ1 = L1/µ . Assumption 4 . The gradient of each loss function is bounded by some constant σ near the optimal point in the sense of l1 and l2 norm , that is , E [ ∥∥∥∇f̃i ( w∗ ) ∥∥∥2 2 ] ≤ σ2 , E [ ∥∥∥∇f̃i ( w∗ ) ∥∥∥ 1 ] ≤ σ1 This assumption constrains the gradient for each loss function at the optimal point . We know ∇f ( w∗ ) = 1n ∑ i ∇̃fi ( w∗ ) = 0 , so it is intuitive that each ∇f̃i ( w∗ ) can be bounded by some value . Therefore this is a natural assumption to make and it has been used in a lot of other work in this area . Note that this assumption only needs to hold under the expectation over all f̃i . With these assumptions , we proved the following theorem for low-precision SGD : Theorem 1 . Suppose that we run LP-SGD on an objective that satisfies Assumptions 1–4 , and with step size α < 1/ ( 2κ2µ ) . After T LP-SGD update steps ( 4 ) , select w̄T uniformly at random from { w0 , w1 , . . . , wT−1 } . Then , the expected objective gap of w̄T is bounded by E [ f ( w̄T ) − f ( w∗ ) ] ≤ 1 2αT ‖w0 − w∗‖22 + ασ2 + δσ1 2 + δ2κ21µ 4 This theorem shows a bound of the expected distance between the result we get at K-th iteration and the optimal value . By choosing an appropriate step size we can achieve convergence at a 1/T rate , while the limit we converge to is only dependent on dimension-free factors . Meanwhile , as mentioned in the first section , previous work gives a dimension-dependent bound ( 1 ) for the problem , which also converges at a 1/T rate.2 Therefore our result guarantees a dimension-independent convergence limit without weakening the convergence rate . It is important to note that , because the dimension-dependent bound in ( 5 ) was tight , we should not expect our new result to improve upon the previous theory in all cases . In the worst case , κ1 = √ d·κ and similarly σ1 = √ d · σ ; this follows from the fact that for vectors in Rd , the norms are related by the inequality ‖x‖1 ≤ √ d · ‖x‖2 . Substituting this into our result produces a dimension-dependent bound again . This illustrates the importance of introducing the new parameters κ1 and σ1 and requiring that they be bounded ; if we could not express our bound in terms of these parameters , the best we could do here is recover a dimension-dependent bound . Experiments Next , we validate our theoretical results experimentally . To do this , we analyzed how the size of the noise floor of convergence of SGD and LP-SGD varies as the dimension is changed for a class of synthetic problems . Importantly , we needed to pick a class of problems for which the parameters L , L1 , µ , σ , and σ1 , did not change as we changed the dimension d. To do this , we chose a class of synthetic linear regression models with loss components sampled independently and identically as f̃i ( w ) = 1 2 ( x̃Tw − ỹ ) 2 where x̃ is a sparse vector sampled to have s nonzero entries each of which is sampled uniformly from { −1 , 1 } , and ỹ is sampled fromN ( x̃Tw∗ , β2 ) for some variance parameter β . Importantly , the nonzero entries of x̃ were chosen non-uniformly such that Pr [ x̃i 6= 0 ] = pi for some probabilities pi which decrease as i increases ; this lets us ensure that µ remains constant as d is increased . For simplicity , we sampled a fresh loss component of this form at each SGD iteration , which is sometimes called the online setting . It is straightforward to derive that for this problem µ = pd L = s L1 = s √ s σ2 = β2s σ1 = √ 2s/πσ . We set α = 0.01 , β = 0.2 , p1 = 0.9 , pd = 0.001 , and s = 16 , we chose each entry of w∗ uniformly from [ −1/2 , 1/2 ] , and we set δ such that the low-precision numbers would range from −1 to 1 . Figure 1 ( b ) shows the convergence of SGD and LP-SGD as the dimension d is changed , for both 8-bit and 6-bit quantization . Notice that while changing d has an effect on the initial convergence rate for both SGD and LP-SGD , it has no effect on the noise ball size , the eventual loss gap that the algorithm converges to . Figure 2 ( a ) measures this noise ball size more explicitly as the dimension is changed : it reports the loss gap averaged across the second half of the iterates . Notice that as the dimension d is changed , the average loss gap is almost unchanged , even for very low-precision methods for which the precision does significantly affect the size of the noise ball . This validates our dimension-free bounds , and shows that they can describe the actual dependence on d in at least one case . Figure 2 ( b ) validates our results in the opposite way : it looks at how this gap changes as our new parameters σ1 and L1 change while d , µ , and σ are kept fixed . To do this , we fixed d = 1024 and changed s across a range , setting β = 0.8/ √ s , which keeps σ2 constant as s is changed : this has the effect of changing σ1 ( and , as a side effect , L1 and L ) . We can see from figure 2 ( b ) that changing σ1 in this way has a much greater effect on LP-SGD than it does on SGD . This validates our theoretical results , and suggests that σ1 and L1 can effectively determine the effect of low-precision compute on SGD .

Summary: This paper discusses conditions under which  the convergence of training models with low-precision weights do not rely on model dimension. Extensions to two kinds of non-linear quantization methods are also provided. The dimension-free bound of the this paper is achieved through a tighter bound on the variance of the quantized gradients.  Experiments are performed on synthetic sparse data and small-scale image classification dataset MNIST.