Low Entropy Deep Networks

1 INTRODUCTION . The Importance of Data Movement Costs . Although there has been a significant amount of attention exploring both algorithmic ( Sze et al. , 2020 ) and hardware-based ( Chen et al. , 2020 ) approaches to reducing the energy costs of deep learning inference , there is often a noted disconnect between the two ( Sze et al. , 2017 ) . The most expensive energy costs lie in memory reads ( Horowitz , 2014 ; Gao et al. , 2017 ) . For every off-chip DRAM data read , you pay the equivalent of over two hundred 32-bit multiplications in energy costs1 ( Horowitz , 2014 ) . Algorithmic techniques that hope to reduce energy consumption have focussed predominantly on metrics like floating-point operations ( FLOPs ) ; intuitively , fewer FLOPs should translate into smaller energy costs from the reduced number of multiplications and potentially , fewer parameters to read from memory . However , it has been observed that this is a weak proxy for the energy consumed in running a deep learning model for inference ( Sze et al. , 2017 ) . Data movement costs can still be high if we do not consider the re-use of weights , the delay between their re-use , and filter shape effects . Simply reducing the FLOPs alone does not guarantee energy cost reductions – particularly those incurred through data read and write activity . Accelerator Design Considerations . These data movement costs are difficult to reduce in von Neumann architectures ( Li et al. , 2015 ; Sebastian et al. , 2020 ) , and so there is a move towards co-design of algorithm-hardware inference accelerators . A core consideration of accelerator design lies in exploring dataflow mappings . These mappings determine how data used in computation is distributed across memory components and optimised to take advantage of the re-use of weights , matrix multiplication partial-sums ( psums ) , and data inputs ( Han et al. , 2016 ; Chen et al. , 2017 ; 2015 ; 2020 ) . Modern deep learning accelerators also make use of hundreds of processing elements ( Chen et al. , 2020 ) ( PE ’ s ) for computation . Each PE contains a small amount of cheap and fast access memory to store a few pieces of information ( weights , psums , inputs , etc. ) . A dataflow mapping cuts down data movement costs by making the best use of each PE and ensuring that the information they contain is recycled as much as possible . A weight/input/psum that can not be used quickly after its latest use will be written to storage and require a re-read later . Less re-use leads to increased data movement costs . Consider , for example , weight-stationary dataflow , commonly used in SOTA accelerator designs ( Farshchi et al. , 2019 ; Jouppi et al. , 2017 ) . Here the items stored 145nm process statically in the PE are the model weights . Input data is then fed into the relevant PE ’ s where multiplication is conducted locally . The ideal situation for a dataflow mapping would be to pay a single data movement cost for each unique weight value and then reference these weight values using indexing . The indexing costs can be kept low using an appropriate encoding scheme such as Huffman coding and approximated by the entropy of the weight space . Thus , the indexing access costs plus unique value access costs can be much smaller than the unquantised network ( Mao & Dally , 2016 ; Wu et al. , 2018 ) . Objectives . So we ask ourselves what we could do algorithmically to maximise the benefit of accelerator dataflows ? We think an easy win is to reduce the number of unique weights a network uses . Fewer unique weights whilst fixing the network topology and the total number of parameters will mean that more weights are re-used more often . This additional re-use gives more opportunity to dataflows to maintain often-used weights in PE ’ s . To further enhance the compressibility , it is desirable for the distribution of the unique weights to be concentrated around a handful of values . The high probability density weights would then be used more often and could then be reliably stored inside PE ’ s , saving both the cost of overwriting these weight values and re-fetching them when needed later . Finally , we ask what the ideal values of these weights would be . From a computational perspective , not all multiplications are created equal . Integer powers-of-two , for example , can be implemented as simple bit-shifts . Mapping the weights used most to these values offers potential further energy reductions . Putting these three requirements together : few unique weights ; a low-entropy encoding with a distribution of weights highly concentrated around a tiny subset of values ; and a focus on powers-of-two values for weights — all motivated to reduce computation costs in accelerator designs — we present our contribution . Weight Fixing Networks . Our work ’ s overarching objective is to transform a network comprising many weights of any value ( limited only by value precision ) to one with the same number of weights but just a few unique values . Rather than selecting the unique weights a priori , we let the optimisation guide the process in an iterative cluster-then-train approach . In each iteration , we cluster an ever-increasing subset of weights to one of a few cluster centroids . We map the pre-trained network weights to these cluster centroids , which are the pool of unique weights . The training stage follows standard gradient descent optimisation to minimise performance loss with two key additions . Firstly , only an ever decreasing subset of the weights are free to be updated . And secondly , we use a new regularisation term to penalise weights in proportion to their nearest clusters ’ relative distance . We iteratively cluster subsets of weights to their nearest cluster centre , as has been demonstrated successfully previously ( Zhou et al. , 2017 ) . The way we determine which subset to move is a core component of our contribution which leads to the superior compression results we achieve . Small Relative Distance Change . Rather than selecting subsets with small Euclidean distances to cluster centres , or those that have small magnitude ( Zhou et al. , 2017 ) , we make the simple observation that the relative – as opposed to absolute – weight change matters . The distance a weight is moved when quantised is dependent on the distance between the weight wi and its new value wi + δwi . When the new value is zero — as is the case for pruning methods — then the magnitude of the weight is the distance . Yet , this is not the case more generally . We demonstrate the importance of optimising quantisation for small relative changes with simple empirical observations . Using a pre-trained ResNet-18 model , we test adding relative vs absolute noise to the layers ’ weights and measure the accuracy change . For relative noise we set a noise level β and adjust all weights wi in a layer l as : wli ← wli+N ( 0 , β|wli| ) . The standard deviation of noise added is determined by the original value wli . We contrast this with absolute noise experiments , where we instead set wli ← wli +N ( 0 , β|wl| ) where |wl| corresponds to the mean absolute weight value in layer l. We run each layer-β combination experiment multiple times – to account for fluctuation in the randomised noise – and present the results in Figure 1 . Even though the mean variation of noise added is the same , noise relative to the original weight value ( multiplicative noise ) is much better tolerated than absolute ( additive noise ) . Since moving weights to quantisation centres is analogous to adding noise , we translate these results into our approach and prioritise weights that have small relative distances to be clustered first . We find avoiding significant quantisation errors requires ensuring that |δwi||wi| is small for all weights . If this is not possible , then performance could suffer . In this case , we create an additional cluster centroid in the vicinity of an existing cluster to reduce this relative distance . Our work also challenges the almost universal trend in the literature ( Yuhang Li , Xin Dong , 2020 ; Jung et al. , 2019 ; Zhang et al. , 2018a ; Zhou et al. , 2016 ; Yamamoto , 2021 ; Oh et al. , 2021 ) of leaving the first and last layers either at full precision or 8-bit ; we attempt a full network quantisation . The cost of not quantising the first layer – which typically requires the most re-use of weights due to the larger resolution of input maps – and the final linear layer – which often contains the largest number of unique weight values – is too significant to ignore . With multiple stages of training and clustering , we finish with a significantly reduced set of unique weights . The regularisation term encourages high probability regions in the weight distribution and a lower-entropy weight-space . The initial choice of cluster centroids as powers-of-two helps us meet our third objective – energy-saving multiplication . Overall we find four distinct advantages over the works reviewed : • We assign a cluster value to all weights — including the first and last layers . • We emphasise a low entropy encoding with a regularisation term , achieving entropies smaller than even those seen using 3-bit quantisation approaches – over which we report superior performance . • We require no additional layerwise scaling ; the unique weights are shared across all layers . • WFN substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches . 2 RELATED WORK . Clip and Scale Quantisation . Typical quantisation approaches reduce the number of bits used to represent components of the network . Quantisation has been applied to all parts of the network with varying success ; the weights , gradients , and activations have all been attempted ( Hubara et al. , 2016 ; Lee et al. , 2017 ; Jung et al. , 2019 ; Yang et al. , 2019 ; Shkolnik et al . ; Zhou et al. , 2016 ) . Primarily , these approaches are motivated by the need to reduce the energy costs of the multiplication of 32-bit floating-point numbers . This form of quantisation uses a rounding function and scaling factor ( sf ) in making value adjustments where a weight wi is mapped to a new value w′i = sf ∗ round ( wi ) for some predetermined rounding function . The scaling factor ( determined by a clipping range ) can be learned channel-wise ( Jacob et al. , 2018 ; Zhang et al. , 2018c ) , or more commonly , layerwise with separate formulations . This results in different channels/layers having a diverse pool of mapping values for the network weights/activations/gradients . Quantisation can be performed without training — known as post-training quantisation , or with added training steps – called quantisation-aware training ( QAT ) . Retraining incurs higher initial costs but results in superior performance . A clipping+scaling quantisation example relevant to our own is the work of ( Zhou et al. , 2017 ) , where the authors restrict the layerwise rounding of weights to powers-of-two . The use of powers-of-two has the additional benefit of energy-cheap bit-shift multiplication . A follow-up piece of work ( Yuhang Li , Xin Dong , 2020 ) suggests additive powers-of-two ( APoT ) instead to capture the pre-quantised distribution of weights better . Weight Sharing Quantisation . Other formulations of quantisation do not use clipping and scaling factors ( Stock et al. , 2020 ; Tartaglione et al. , 2021 ; Wu et al. , 2018 ) . Instead , they adopt clustering techniques to cluster the weights and fix the weight values to their assigned group cluster centroid . These weights are stored as codebook indices , allowing for compressed representation methods such as Huffman encoding to squeeze the network further . Unlike clipping+scaling quantisation techniques , and like ours , these methods share the pool of weights across the entire network . The work by ( Wu et al. , 2018 ) is of particular interest since both the motivation and approach are related to ours . Here the authors use a spectrally relaxed k-means regularisation term to encourage the network weights to be more amenable to clustering . In their case , they focus on a filter-row codebook inspired by the row-stationary dataflow used in some accelerator designs ( Chen et al. , 2017 ) . However , their formulation is explored only for convolution , and they restrict clustering to groups of weights ( filter rows ) rather than individual weights due to computational limitations as recalibrating the k-means regularisation term is expensive during training . Similarly ( Stock et al. , 2020 ; Fan et al. , 2021 ) , focus on quantising groups of weights into single codewords rather than the individual weights themselves . Weight-sharing approaches similar to ours include ( Ullrich et al. , 2017 ) . The authors use the distance from an evolving Gaussian mixture as a regularisation term to prepare the clustering weights . Although it is successful with small dataset-model combinations , the complex optimisation — particularly the additional requirement for Inverse-Gamma priors to lower-bound the mixture variance to prevent mode collapse— limits the method ’ s practical applicability due to the high computational costs of training . In our formulation , the weights already fixed no longer contribute to the regularisation prior , reducing the computational overhead .

Summary: This work presents WFN, an ensemble of techniques to reduce data movement costs by reducing the number of unique weights in a network.  Overall the authors find some distinct advantages in their approach: They emphasise a low entropy encoding with a regularisation term, achieving entropies smaller than even those seen using 3-bit quantisation approaches. They require no additional layerwise scaling; the unique weights are shared across all layers. They substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches.