Low Entropy Deep Networks

1 INTRODUCTION . The Importance of Data Movement Costs . Although there has been a significant amount of attention exploring both algorithmic ( Sze et al. , 2020 ) and hardware-based ( Chen et al. , 2020 ) approaches to reducing the energy costs of deep learning inference , there is often a noted disconnect between the two ( Sze et al. , 2017 ) . The most expensive energy costs lie in memory reads ( Horowitz , 2014 ; Gao et al. , 2017 ) . For every off-chip DRAM data read , you pay the equivalent of over two hundred 32-bit multiplications in energy costs1 ( Horowitz , 2014 ) . Algorithmic techniques that hope to reduce energy consumption have focussed predominantly on metrics like floating-point operations ( FLOPs ) ; intuitively , fewer FLOPs should translate into smaller energy costs from the reduced number of multiplications and potentially , fewer parameters to read from memory . However , it has been observed that this is a weak proxy for the energy consumed in running a deep learning model for inference ( Sze et al. , 2017 ) . Data movement costs can still be high if we do not consider the re-use of weights , the delay between their re-use , and filter shape effects . Simply reducing the FLOPs alone does not guarantee energy cost reductions – particularly those incurred through data read and write activity . Accelerator Design Considerations . These data movement costs are difficult to reduce in von Neumann architectures ( Li et al. , 2015 ; Sebastian et al. , 2020 ) , and so there is a move towards co-design of algorithm-hardware inference accelerators . A core consideration of accelerator design lies in exploring dataflow mappings . These mappings determine how data used in computation is distributed across memory components and optimised to take advantage of the re-use of weights , matrix multiplication partial-sums ( psums ) , and data inputs ( Han et al. , 2016 ; Chen et al. , 2017 ; 2015 ; 2020 ) . Modern deep learning accelerators also make use of hundreds of processing elements ( Chen et al. , 2020 ) ( PE ’ s ) for computation . Each PE contains a small amount of cheap and fast access memory to store a few pieces of information ( weights , psums , inputs , etc. ) . A dataflow mapping cuts down data movement costs by making the best use of each PE and ensuring that the information they contain is recycled as much as possible . A weight/input/psum that can not be used quickly after its latest use will be written to storage and require a re-read later . Less re-use leads to increased data movement costs . Consider , for example , weight-stationary dataflow , commonly used in SOTA accelerator designs ( Farshchi et al. , 2019 ; Jouppi et al. , 2017 ) . Here the items stored 145nm process statically in the PE are the model weights . Input data is then fed into the relevant PE ’ s where multiplication is conducted locally . The ideal situation for a dataflow mapping would be to pay a single data movement cost for each unique weight value and then reference these weight values using indexing . The indexing costs can be kept low using an appropriate encoding scheme such as Huffman coding and approximated by the entropy of the weight space . Thus , the indexing access costs plus unique value access costs can be much smaller than the unquantised network ( Mao & Dally , 2016 ; Wu et al. , 2018 ) . Objectives . So we ask ourselves what we could do algorithmically to maximise the benefit of accelerator dataflows ? We think an easy win is to reduce the number of unique weights a network uses . Fewer unique weights whilst fixing the network topology and the total number of parameters will mean that more weights are re-used more often . This additional re-use gives more opportunity to dataflows to maintain often-used weights in PE ’ s . To further enhance the compressibility , it is desirable for the distribution of the unique weights to be concentrated around a handful of values . The high probability density weights would then be used more often and could then be reliably stored inside PE ’ s , saving both the cost of overwriting these weight values and re-fetching them when needed later . Finally , we ask what the ideal values of these weights would be . From a computational perspective , not all multiplications are created equal . Integer powers-of-two , for example , can be implemented as simple bit-shifts . Mapping the weights used most to these values offers potential further energy reductions . Putting these three requirements together : few unique weights ; a low-entropy encoding with a distribution of weights highly concentrated around a tiny subset of values ; and a focus on powers-of-two values for weights — all motivated to reduce computation costs in accelerator designs — we present our contribution . Weight Fixing Networks . Our work ’ s overarching objective is to transform a network comprising many weights of any value ( limited only by value precision ) to one with the same number of weights but just a few unique values . Rather than selecting the unique weights a priori , we let the optimisation guide the process in an iterative cluster-then-train approach . In each iteration , we cluster an ever-increasing subset of weights to one of a few cluster centroids . We map the pre-trained network weights to these cluster centroids , which are the pool of unique weights . The training stage follows standard gradient descent optimisation to minimise performance loss with two key additions . Firstly , only an ever decreasing subset of the weights are free to be updated . And secondly , we use a new regularisation term to penalise weights in proportion to their nearest clusters ’ relative distance . We iteratively cluster subsets of weights to their nearest cluster centre , as has been demonstrated successfully previously ( Zhou et al. , 2017 ) . The way we determine which subset to move is a core component of our contribution which leads to the superior compression results we achieve . Small Relative Distance Change . Rather than selecting subsets with small Euclidean distances to cluster centres , or those that have small magnitude ( Zhou et al. , 2017 ) , we make the simple observation that the relative – as opposed to absolute – weight change matters . The distance a weight is moved when quantised is dependent on the distance between the weight wi and its new value wi + δwi . When the new value is zero — as is the case for pruning methods — then the magnitude of the weight is the distance . Yet , this is not the case more generally . We demonstrate the importance of optimising quantisation for small relative changes with simple empirical observations . Using a pre-trained ResNet-18 model , we test adding relative vs absolute noise to the layers ’ weights and measure the accuracy change . For relative noise we set a noise level β and adjust all weights wi in a layer l as : wli ← wli+N ( 0 , β|wli| ) . The standard deviation of noise added is determined by the original value wli . We contrast this with absolute noise experiments , where we instead set wli ← wli +N ( 0 , β|wl| ) where |wl| corresponds to the mean absolute weight value in layer l. We run each layer-β combination experiment multiple times – to account for fluctuation in the randomised noise – and present the results in Figure 1 . Even though the mean variation of noise added is the same , noise relative to the original weight value ( multiplicative noise ) is much better tolerated than absolute ( additive noise ) . Since moving weights to quantisation centres is analogous to adding noise , we translate these results into our approach and prioritise weights that have small relative distances to be clustered first . We find avoiding significant quantisation errors requires ensuring that |δwi||wi| is small for all weights . If this is not possible , then performance could suffer . In this case , we create an additional cluster centroid in the vicinity of an existing cluster to reduce this relative distance . Our work also challenges the almost universal trend in the literature ( Yuhang Li , Xin Dong , 2020 ; Jung et al. , 2019 ; Zhang et al. , 2018a ; Zhou et al. , 2016 ; Yamamoto , 2021 ; Oh et al. , 2021 ) of leaving the first and last layers either at full precision or 8-bit ; we attempt a full network quantisation . The cost of not quantising the first layer – which typically requires the most re-use of weights due to the larger resolution of input maps – and the final linear layer – which often contains the largest number of unique weight values – is too significant to ignore . With multiple stages of training and clustering , we finish with a significantly reduced set of unique weights . The regularisation term encourages high probability regions in the weight distribution and a lower-entropy weight-space . The initial choice of cluster centroids as powers-of-two helps us meet our third objective – energy-saving multiplication . Overall we find four distinct advantages over the works reviewed : • We assign a cluster value to all weights — including the first and last layers . • We emphasise a low entropy encoding with a regularisation term , achieving entropies smaller than even those seen using 3-bit quantisation approaches – over which we report superior performance . • We require no additional layerwise scaling ; the unique weights are shared across all layers . • WFN substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches . 2 RELATED WORK . Clip and Scale Quantisation . Typical quantisation approaches reduce the number of bits used to represent components of the network . Quantisation has been applied to all parts of the network with varying success ; the weights , gradients , and activations have all been attempted ( Hubara et al. , 2016 ; Lee et al. , 2017 ; Jung et al. , 2019 ; Yang et al. , 2019 ; Shkolnik et al . ; Zhou et al. , 2016 ) . Primarily , these approaches are motivated by the need to reduce the energy costs of the multiplication of 32-bit floating-point numbers . This form of quantisation uses a rounding function and scaling factor ( sf ) in making value adjustments where a weight wi is mapped to a new value w′i = sf ∗ round ( wi ) for some predetermined rounding function . The scaling factor ( determined by a clipping range ) can be learned channel-wise ( Jacob et al. , 2018 ; Zhang et al. , 2018c ) , or more commonly , layerwise with separate formulations . This results in different channels/layers having a diverse pool of mapping values for the network weights/activations/gradients . Quantisation can be performed without training — known as post-training quantisation , or with added training steps – called quantisation-aware training ( QAT ) . Retraining incurs higher initial costs but results in superior performance . A clipping+scaling quantisation example relevant to our own is the work of ( Zhou et al. , 2017 ) , where the authors restrict the layerwise rounding of weights to powers-of-two . The use of powers-of-two has the additional benefit of energy-cheap bit-shift multiplication . A follow-up piece of work ( Yuhang Li , Xin Dong , 2020 ) suggests additive powers-of-two ( APoT ) instead to capture the pre-quantised distribution of weights better . Weight Sharing Quantisation . Other formulations of quantisation do not use clipping and scaling factors ( Stock et al. , 2020 ; Tartaglione et al. , 2021 ; Wu et al. , 2018 ) . Instead , they adopt clustering techniques to cluster the weights and fix the weight values to their assigned group cluster centroid . These weights are stored as codebook indices , allowing for compressed representation methods such as Huffman encoding to squeeze the network further . Unlike clipping+scaling quantisation techniques , and like ours , these methods share the pool of weights across the entire network . The work by ( Wu et al. , 2018 ) is of particular interest since both the motivation and approach are related to ours . Here the authors use a spectrally relaxed k-means regularisation term to encourage the network weights to be more amenable to clustering . In their case , they focus on a filter-row codebook inspired by the row-stationary dataflow used in some accelerator designs ( Chen et al. , 2017 ) . However , their formulation is explored only for convolution , and they restrict clustering to groups of weights ( filter rows ) rather than individual weights due to computational limitations as recalibrating the k-means regularisation term is expensive during training . Similarly ( Stock et al. , 2020 ; Fan et al. , 2021 ) , focus on quantising groups of weights into single codewords rather than the individual weights themselves . Weight-sharing approaches similar to ours include ( Ullrich et al. , 2017 ) . The authors use the distance from an evolving Gaussian mixture as a regularisation term to prepare the clustering weights . Although it is successful with small dataset-model combinations , the complex optimisation — particularly the additional requirement for Inverse-Gamma priors to lower-bound the mixture variance to prevent mode collapse— limits the method ’ s practical applicability due to the high computational costs of training . In our formulation , the weights already fixed no longer contribute to the regularisation prior , reducing the computational overhead .

Summary: The paper proposes a method called Weight Fixing Networks (WFN), which is designed to minimize the data movement in deep learning inference, thus minimizing the energy cost. WFN aims to maintain very few unique weights, low-entropy weight encodings, unique weight values which are amenable to energy-saving versions of hardware multiplication, while keeping the same performance. A new clustering algorithm for weight generation is shown and analyzed, and experimental evaluation shows the merits of the new scheme.