On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks

Compression is a key step to deploy large neural networks on resource-constrained platforms . As a popular compression technique , quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight . In this paper , we study the representation power of quantized neural networks . First , we prove the universal approximability of quantized ReLU networks on a wide class of functions . Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and functiondependent structures . Our results reveal that , to attain an approximation error bound of , the number of weights needed by a quantized network is no more than O ( log5 ( 1/ ) ) times that of an unquantized network . This overhead is of much lower order than the lower bound of the number of weights needed for the error bound , supporting the empirical success of various quantization techniques . To the best of our knowledge , this is the first in-depth study on the complexity bounds of quantized neural networks . 1 INTRODUCTION . Various deep neural networks deliver state-of-the-art performance on many tasks such as object recognition and natural language processing using new learning strategies and architectures ( Chen et al. , 2017 ; He et al. , 2016 ; Kumar et al. , 2016 ; Ioffe & Szegedy , 2015 ; Vaswani et al. , 2017 ) . Their prevalence has extended to embedded or mobile devices for edge intelligence , where security , reliability or latency constraints refrain the networks from running on servers or in clouds . However , large network sizes with the associated expensive computation and memory consumption make edge intelligence even more challenging ( Cheng et al. , 2018 ; Sandler et al. , 2018 ) . In response , as will be more detailed in Section 2 , substantial effort has been made to reduce the memory consumption of neural networks while minimizing the accuracy loss . The memory consumption of neural networks can be reduced by either directly reducing the number of weights or decreasing the number of bits ( bit-width ) needed to represent and store each weight , which can be employed on top of each other ( Choi et al. , 2016 ) . The number of weights can be reduced by pruning ( Han et al. , 2015b ) , weight sparsifying ( Liu et al. , 2015 ) , structured sparsity learning ( Wen et al. , 2016 ) and low rank approximation ( Denton et al. , 2014 ) . The bit-width is reduced by quantization that maps data to a smaller set of distinct levels ( Sze et al. , 2017 ) . Note that while quantization may stand for linear quantization only ( Li et al. , 2017 ; Gysel et al. , 2016 ) or nonlinear quantization only ( Han et al. , 2015a ; Choi et al. , 2016 ) in different works , our discussion will cover both cases . However , as of today quantization is still only empirically shown to be robust and effective to compress various neural network architectures ( Hubara et al. , 2016 ; Zhou et al. , 2017b ; Zhuang et al. , 2017 ) . Its theoretical foundation still remains mostly missing . Specifically , many important questions remain unanswered . For example : • Why even binarized networks , those most extremely quantized with bit-width down to one , still work well in some cases ? • To what extent will quantization decrease the expressive power of a network ? Alternatively , what is the overhead induced by weight quantization in order to maintain the same accuracy ? In this paper , we provide some insights into these questions from a theoretical perspective . We focus on ReLU networks , which is among the most widely used in deep neural networks ( Xu et al. , 2015 ) . We follow the idea from Yarotsky ( 2017 ) to prove the complexity bound by constructing a network , but with new and additional construction components essential for quantized networks . Specifically , given the number of distinct weight values λ and a target function f , we construct a network that can approximate f with an arbitrarily small error bound to prove the universal approximability . The memory size of this network then naturally serves as an upper bound for the minimal network size . The high-level idea of our approach is to replace basic units in an unquantized network with quantized sub-networks 1 that approximate these basic units . For example , we can approximate a connection with any weight in an unquantized network by a quantized sub-network that only uses a finite number of given weight values . Even though the approximation of a single unit can be made arbitrarily accurate in principle with unlimited resources ( such as increased network depth ) , in practice , there exists some inevitable residual error at every approximation , all of which could propagate throughout the entire network . The challenge becomes , however , how to mathematically prove that we can still achieve the end-to-end arbitrary small error bound even if these unavoidable residual errors caused by quantization can be propagated throughout the entire network . This paper finds a solution to solve the above challenge . In doing so , we have to propose a number of new ideas to solve related challenges , including judiciously choosing the proper finite weight values , constructing the approximation sub-networks as efficient as possible ( to have a tight upper bound ) , and striking a good balance among the complexities of different approximation steps . Based on the bounds derived , we compare them with the available results on unquantized neural networks and discuss its implications . In particular , the main contributions of this paper include : • We prove that even the most extremely quantized ReLU networks using two distinct weight values are capable of representing a wide class of functions with arbitrary accuracy . • Given the number of distinct weights and the desired approximation error bound , we provide upper bounds on the number of weights and the memory size . We further show that our upper bounds have good tightness by comparing them with the lower bound of unquantized ReLU networks established in the literature . • We show that , to attain the same approximation error bound , the number of weights needed by a quantized network is no more thanO ( log5 ( 1/ ) ) times that of an unquantized network . This overhead is of much lower order compared with even the lower bound of the number of weights needed for the error bound . This partially explains why many state-ofthe-art quantization schemes work well in practice . • We demonstrate how a theoretical complexity bound can be used to estimate an optimal bit-width , which in turn enables the best cost-effectiveness for a given task . The remainder of the paper is organized as follows . Section 2 reviews related works . Section 3 lays down the models and assumptions of our analysis . We prove the universal approximability and the upper bounds with function-independent structure in Section 4 and extend it to function-dependent structure in Section 5 . We analyze the bound-based optimal bit-width in Section 6 . Finally , Section 7 discusses the results and gets back to the questions raised above . 2 RELATED WORKS . Quantized Neural Networks : There are rich literatures on how to obtain quantized networks , either by linear quantization or nonlinear quantization ( Zhou et al. , 2017a ; Leng et al. , 2017 ; Shayar et al. , 1Throughout this paper , we use “ sub-network ” to denote a network that is used as a part of the final network that approximates a target function . 2017 ) . Linear quantization does mapping with a same distance between contiguous quantization levels and is usually implemented by storing weights as fixed-point numbers with reduced bit-width ( Li et al. , 2017 ; Gysel et al. , 2016 ) . Nonlinear quantization maps the data to quantization levels that are not uniformly distributed and can be either preselected or learned from training . Then the weights are stored using lossless binary coding ( the index to a lookup table ) instead of the actual values ( Han et al. , 2015a ; Choi et al. , 2016 ) . It is reported that a pruned AlexNet can be quantized to eight bits and five bits in convolutional layers and fully connected layers , respectively , without any loss of accuracy . Similar results are also observed in LENET-300-100 , LENET-5 , and VGG-16 ( Han et al. , 2015a ) . One may argue that some of these benchmark networks are known to have redundancy . However , recent works show that quantization works well even on networks that are designed to be extremely small and compact . SqueezeNet , which is a state-of-the-art compact network , can be quantized to 8-bit while preserving the original accuracy ( Gysel et al. , 2016 ; Iandola et al. , 2016 ) . There are some representative works that can achieve little accuracy loss on ImageNet classification even using binary or ternary weights ( Courbariaux et al. , 2015 ; Rastegari et al. , 2016 ; Li et al. , 2016 ; Zhu et al. , 2016 ) . More aggressively , some works also reduce the precision of activations , e.g . ( Hubara et al. , 2016 ; Rastegari et al. , 2016 ; Faraone et al. , 2018 ) . Although the classification accuracy loss can be minimized , the universal approximation property is apparently lost , as with limited output precision the network can not achieve arbitrary accuracy . Accordingly , we do not include them in the discussion of this paper . The limit of quantization is still unknown while the state-of-the-art keeps getting updated . For example , VGG-16 is quantized to 3-bit while maintaining the original accuracy ( Leng et al. , 2017 ) . Motivated by the great empirical success , the training of quantized neural networks has been analyzed theoretically , but not the network capacity ( Li et al. , 2017 ; Choi et al. , 2016 ) . Universal Approximability and Complexity Bounds : The universal approximability of ReLU networks is proved in Mhaskar & Micchelli ( 1992 ) and revisited in ( Sonoda & Murata , 2017 ) . Recently , Hanin ( 2017 ) discusses the expressive power of ReLU networks with bounded width and proves that a ReLU network with width d+1 can approximate any continuous convex function of d variables arbitrarily well . Shaham et al . ( 2016 ) construct a sparsely-connected depth-4 ReLU network and prove its error bound . Liang & Srikant ( 2016 ) prove that , for a large class of piecewise smooth functions , a network with ReLU and binary step units can provide an error bound with O ( 1/ ) layers and O ( poly log ( 1/ ) ) neurons . The universal approximation property of low displacement rank ( LDR ) neural networks has been proved by Zhao et al . ( 2017 ) under a mild condition on the displacement operator , which is the result of another effective technique of neural network compression . 3 MODELS AND ASSUMPTIONS . Throughout this paper , we define ReLU networks as feedforward neural networks with the ReLU activation function σ ( x ) = max ( 0 , x ) . The ReLU network considered includes multiple input units , a number of hidden units , and one output unit . Without loss of generality , each unit can only connect to units in the next layer . Our conclusions on ReLU networks can be extended to any other networks that use piecewise linear activation functions with finite breakpoints such as leaky ReLU and ReLU6 immediately , as one can replace a ReLU network by an equivalent one using these activation functions while only increasing the number of units and weights by constant factors ( Yarotsky , 2017 ) . We denote the finite number of distinct weight values as λ ( λ ∈ Z+ and λ ≥ 2 ) , for both linear quantization and nonlinear quantization . For linear quantization , without loss of generality , we assume the finite number of distinct weight values are given as { −1 , 1λ , 2 λ , . . . , λ−1 λ } , where { 1λ , 2 λ , . . . , λ−1 λ } are uniformly spaced ( hence called “ linear ” ) in ( 0 , 1 ) and −1 is used to obtain the negative weight values . For nonlinear quantization , we assume the finite number of distinct weight values are not constrained to any specific values , i.e. , they can take any values as needed . To store each weight , we only need log ( λ ) 2 bits to encode the index , i.e . the bit-width is log ( λ ) . The overhead to store sparse structures can be ignored because it varies depending on the implementation and can be easily reduced to the same order as the weight storage using techniques such as compressed sparse row ( CSR ) for nonlinear quantization . The number of bits needed to store the codebook can also be ignored because it has lower order of complexity . 2Throughout this paper , we omit base 2 for clarity of presentation We consider any function f in the Sobolev space : f ∈Wn , ∞ ( [ 0 , 1 ] d ) and max n : |n|≤n ess sup x∈ [ 0,1 ] d |Dnf ( x ) | ≤ 1 . ( 1 ) The space Wn , ∞ consists of all locally integrable function f : Ω → R such that Dnf ∈ L∞ ( Ω ) , where |n| ≤ n and Ω is an open set in Rd . We denote this function space as Fd , n in this paper . Note that we only assume weak derivatives up to order n exist where n can be as small as 1 where the function is non-differentiable . We also only assume the Lipschitz constant to be no greater than 1 for the simplicity of the derivation . When the Lipschitz constant is bigger than 1 , as long as it is bounded , the whole flow of the proof remains the same though the bound expression will scale accordingly . When constructing the network to approximate any target function f , we consider two scenarios for deriving the bounds . The first scenario is called function-dependent structure , where the constructed network topology and their associated weights are all affected by the choice of the target function . In contrast , the second scenario is called function-independent structure , where the constructed network topology is independent of the choice of the target function in f ∈ Fd , n with a given . The principle behind these design choices ( the network topology constructions and the choice of weights ) is to achieve a tight upper bound as much as possible . One might consider that we can transform an unquantized network within the error bound to a quantized one in a straightforward way by approximating every continuous-value weight with a combination of discrete weights with arbitrary accuracy . However , the complexity of such approximation ( number of discrete weights needed ) depends on the distribution of those continuous-value weights ( e.g. , their min and max ) , which may vary depending on the training data and network structure and a closed-form expression for the upper bounds is not possible . As such , a more elegant approach is needed . Below we will establish a constructive approach which allows us to bound the approximation analytically .

Summary: This paper studies the expressive power of quantized ReLU networks from a theoretical point of view. This is well-motivated by the recent success of using quantized neural networks as a compression technique. This paper considers both linear quantization and non-linear quantization, both function independent network structures and function dependent network structures. The obtained results show that the number of weights need by a quantized network is no more than polylog factors times that of a unquantized network. This justifies the use of quantized neural networks as a compression technique.