Low Entropy Deep Networks
Authors: Chris Subia-Waud
Srinandan Dasmahapatra
Venue: ICLR 2022 Submitted (2022)
Keywords: Quantisation
Compression
AI Accelerators

Abstract: The movement of data between processes and memory,  not arithmetic operations, dominate the energy costs of deep learning inference calculations.  This work focuses on reducing these data movement costs by reducing the number of unique weights in a network. The thinking goes that if the number of unique weights is kept small enough, then the entire network can be distributed and stored on processing elements (PEs) within accelerator designs,  and the data movement costs for weight reads substantially reduced.  To this end, we investigate the merits of a method, which we call Weight Fixing Networks (WFN). We design the approach to realise four model outcome objectives: i) very few unique weights, ii) low-entropy weight encodings,  iii) unique weight values which are amenable to energy-saving versions of hardware multiplication, and iv)  lossless task-performance.  Some of these goals are conflicting. To best balance these conflicts, we combine a few novel (and some well-trodden) tricks; a novel regularisation term, (i, ii) a view of clustering cost as relative distance change (i, ii, iv), and a focus on whole-network re-use of weights (i, iii).  Our Imagenet experiments demonstrate lossless compression using 56x fewer unique weights and a 1.9x  lower weight-space entropy than SOTA quantisation approaches.  

Under review as a conference paper at ICLR 2022
Low ENTROPY DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The movement of data between processor and memory, not arithmetic operations, dominates the energy cost of inference computations deep networks. Network compression offers opportunities for hardware design bring weights in memory closer to processor reducing these data movement costs. To this end, we investi- gate the merits of method, which call Weight Fixing Networks (WFN). We design the approach to realise four model outcome objectives: i) very few unique weights, ii) low-entropy weight encodings, iii) unique weight values which are amenable to energy-saving versions of hardware multiplication, and iv) lossless park-pefformance. Some of these goals are conflicting. To best balance these conflicts, we combine few novel (and some well- well-trodden) tricks; novel regu- larisation term, (i, ii) view of clustering cost relative distance change (i. ii. iv), and focus whole-network re-use of weights (i, iii). Our Imagenet exper- iments demonstrate lossless compression using 56x fewer unique weights and a 1.9x lower weight-space entropy than SOTA quantisation approaches.
INTRODUCTION
The Importance of Data Movement Costs. Although there has been a significant amount of attention exploring both algorithmic (Sze et al. 2020) and hardware-hased (Chen et al. 2020) approaches to reducing the energy costs of deep learning inference, there often noted disconnect between the two (Sze et al. 2017). The most expensive energy costs lie in memory reads (Horowitz) 2014 Gao al., 2017). For every off-chip DRAM data read. you pay the equivalent of over two hundred 32-bit multiplications in energy costs] (Horowitz) 2014). Algorithmic techniques that hope to reduce energy consumption have focussed predominantly on metries like floating-point operations (FLOPs); intuitively, fewer FLOPs should translate into smaller energy costs from the reduced number of multiplications and potentially, fewer parameters to read from memory. However, it has been observed that this is weak proxy for the energy consumed in running deep learning model for inference (Sze et al. 2017). Data movement costs can still be high if we do not consider the re-use of weights, the delay between their re-use, and filter shape effects. Simply reducing the FLOPs alone does not guarantee energy cost reductions particularly those incurred through data read and write activity.
Accelerator Design Considerations. These data movement costs are difficult reduce in von Neumann architectures (Li et al.) [2015] Sebastian et al. 2020). and so there is a move towards co-design of seoortth-hhadwwarr inference accelerators. A core consideration of accelerator design lies in exploring dataflow mappings. These mappings determine how data used in computation is distributed across memory components and optimised take advantage of the re-use of weights, matrix multiplication partial-sums (psums), and data inputs (Han et al.) 2016) Chen et al., 2017] 2015 (020). (020). Modern deep learning accelerators also make use of hundreds of processing elements [Chen et al.) 2020) (PE's) for computation. Each PE contains small amount of cheap and fast access memory to store a few pieces information (weights, psums, inputs, etc.). A dataflow mapping cuts down data movement costs by making the best use of each PE and ensuring that the information they contain is recycled as much as possible. weight/inpt//ssum that cannot be used quickly after its latest will be written to storage and require a re-read later. Less re-use leads to increased data movement costs. Consider, for example, weight-stations dataflow, commonly used in SOTA accelerator designs (Farshchi et al. 2019 Jouppi al. 2017). Here the items stored
45mm process

---

Under review as a conference paper at ICLR 2022
Baseline
APoT (3-bit)
WFN
Total Unique Param Count (log)
10t
Weight-S Spa08 Entropy
"o" tn"
71x
Fewer
Fewer
1.75x Smaller
Smaller
14"
ResNet-18
ResNet34
ResNet-18
ResNet-34
Figure 1: WFN has more weight re-use opportunities than existing quantisation approaches which can be used to reduce data movement costs. Left: The total number of unique parameters left after quantisation 56x fewer than APoT for ResNet-18 trained on the Imagenet dataset and 71x for the 34 model. Right: The entropy of the parameters across the network is 1.9x and 1.65x smaller when using the WFN approach over APoT.
statically the PE are the model weights. Input data is then fed into the relevant PE's where multiplication is conducted locally. The ideal situation for dataflow mapping would be to pay single data movement cost for each unique weight value and then reference these weight values using indexing. The indexing costs can be kept low using appropriate encoding scheme such as Huffman coding and approximated by the entropy of the weight space. Thus, the indexing access costs plus unique value access costs can be much smaller than the unquantised network (Mao & Dally 2016 Wu et al. 2018).
Objectives, So we ask ourselves what we could do algorithmically to maximise the benefit of accelerator dataflows? We think an easy win to reduce the number of unique weights network uses. Fewer unique weights whilst fixing the network topology and the total number of parameters will mean that more weights are re-used more often. This additional re-use gives more opportunity to dataflows to maintain often-used weights in PE's. To further enhance the compressibility, it is desirable for the distribution of the unique weights to be concentrated around a handful of values. The high probability density weights would then be used more often and could then be reliably stored inside PE's, saving both the cost overwriting these weight values and re-fetching them when needed later. Finally, we ask what the ideal values of these weights would be. From computational perspective, not all multiplications are created equal. Integer powers-of-two, for example, can be implemented simple bit-shifts. Mapping the weights used most these values offers potential further energy reductions. Putting these three requirements together: few unique weights; low-entropy encoding with a distribution of weights highly concentrated around tiny subset of values; and focus on powers-of-twe values for weights - all motivated to reduce computation costs in accelerator designs we present our contribution.
Weight Fixing Networks. Our work's overarching objective transform a network comprising many weights of any value (limited only by value precision) one with the same number of weights but just few unique values. Rather than selecting the unique weights a priori, we let the optimisation guide the process in iterative tluster-then-train approach. In each iteration, we cluster an ever-increasing subset of weights to one of few cluster centroids. We map the pre-trained network weights to these cluster centroids, which are the pool of unique weights. The training stage follows standard gradient descent optimisation to minimise performance loss with two key additions. Firstly, only And secondly, we use a new regularisation term to penalise weights in proportion to their nearest clusters' relative distance. We iteratively cluster subsets of weights to their nearest cluster centre, as has been demonstrated successfully previously (Zhou et al. (017). The way we determine which subset to move core component of our contribution which leads to the superior compression results we achieve.
decreasing subset of the weights
free to be updated.
Small Relative Distance Change. Rather than selecting subsets with small Euclidean distances cluster centres, or those that have small magnitude (Zhou et al. 2017). we make the simple
2

---

Under review as a conference paper at ICLR 2022
8 0.05
8 0.1
B 0.2
1.0
0.8
0.4
0.2
Noise Type
0.3
0.4
0.5
Relative Absolute
1.0
0.8
0..6
0.4
0.2
6.00
Laye-thder
Figure 2: We explore adding relative absolute noise to each of the layers (x-axis). The layer index indicates which layer was selected to have noise added. Each layer index is a separate experiment with the 95% confidence intervals shaded.
observation that the relative as opposed to absolute weight change matters. The distance weight is moved when quantised is dependent the distance between the weight w; and its new value Wi + dw,, When the new value is zero - as is the case for pruning methods then the magnitude of the weight the distance. Yet, this is the case more generally. We demonstrate the importance of optimising quantisation for small relative changes with simple empirical observations. Using pre-trained ResNet- 18 model, we test adding relative vs absolute noise to the layers' weights and measure the accuracy change. For relative noise we set noise level and adjust all weights w; in a layer as: w +- wt + +N (w The standard deviation of noise added is determined by the original value w. We contrast this with absolute noise experiments, where we instead set w + w + /0 3 w) where |w'] corresponds to the mean absolute weight value in layer I. We run each layer-3 combination experiment multiple times to account for fluctuation the randomised noise and present the results Figure [ Even though the mean variation of noise added is the same, noise relative the original weight value (multiplicative noise) is much better tolerated than absolute (additive noise). Since moving weights to quantisation centres analogous adding noise, translate these results into our approach and prioritise weights that have small relative distances to be clustered first We find avoiding significant quantisation errors requires ensuring that jówsl
|w:] is small for all weights. If this is not possible, then performance could suffer. In this case, we create an additional cluster centroid in the vicinity of an existing cluster to reduce this relative distance. Our work also challenges the almost universal trend in the literature (Yuhang Li. Xin Dong 2020; Jung al. 2019 Zhang et al., 2018a Zhou et al. [016] Yamamoto, [021] Oh et al. (021) of leaving the first and last layers either at full precision or 8-bit; we attempt full network quantisation. The cost of not quantising the first layer which typically requires the most re-use of weights due to the larger resolution of input maps and the final linear layer - which often contains the largest number of unique weight values - too significant to ignore.
With multiple stages of training and clustering, we finish with a significantly reduced set unique weights. The regularisation term encourages high probability regions in the weight distribution and a lower-entropy weight-space. The initial choice of cluster centroids as powers-of-two helps us meet our third objective- energy-saving multiplication. Overall we find four distinet advantages over the works reviewed:
We assign cluster value to all weights - including the first and last layers.
We emphasise a low entropy encoding with regularisation term, achieving entropies smaller than even those seen using 3-bit quantisation approaches over which we report superior performance.
We require additional layerwise scaling: the unique weights shared across all layers.
3

---

Under review as a conference paper at ICLR 2022
WFN substantially reduces the number of unique parameters in network when compared to existing SOTA quantisation approaches.
2 RELATED WORK
Clip and Scale Quantisation. Typical quantisation approaches reduce the number of bits used to represent components of the network. Quantisation has been applied to all parts of the network with varying success; the weights, gradients, and activations have all been attempted (Hubara et al. 2016 Lee et al., 2017; Jung et al.) 2019] Yang et al.) 2019; [Shkolnik et al. Zhhou et al. 2016 Primarly, these approaches are motivated by the need to reduce the energy costs of the multiplication of 32-bit floating-poit numbers. This form of quantisation uses rounding function and scaling factor (sf) making value adjustments where a weight wi is mapped to new value w sf round(w:, for some predetermined rounding function. The scaling factor (determined by a clipping range) can be learned channel- wise (Jacob et al.) 2018: Zhang et al. 2018c), or more commonly, layerwise with separate formulations. This results in different channels/layers having Quantisation can
a diverse pool of mapping values for the network weightt/activaaiion//rradd be performed without training - known as post-training quantisation, or with added training steps called quantisation-awan superior performance. A clipping+scaling quantisation example relevant our own is the work of (Zhou et al.) 2017). where the authors restrict the layerwise rounding of weights to powers-of-twer The use of powers-of-two has the additional benefit of energy-cheap bit-shift multiplication. A follow-up piece of work (Yuhang Li, Xin Dong. (020) suggests additive powers-ofttwo (APoT) instead to capture the pre-quantised distribution of weights better.
training (QAT). Retraining incurs higher initial costs but results in
Weight Sharing Quantisation. Other formulations quantisation do not use clipping and scaling factors (Stock al. 2020; Tartaglione et al. [021 Wu et 2018). Instead, they adopt clustering techniques to cluster the weights and fix the weight values to their assigned group cluster centroid. These weights are stored codebook indices, allowing for compressed representation methods such as Huffman encoding to squeeze the network further. Unlike clipping+scaling quantisation techniques, and like ours, these methods share the pool of weights across the entire network. The work by (Wu et al., 2018) is of particular interest since both the motivation and approach are related ours. Here the authors use a spectrally relaxed k-means regularisation term encourage the network weights to be more amenable to clustering. In their case, they focus on a filter-row code- book inspired by the -tatsotiionay dataflow used in some accelerator designs (Chen et al. 2017). However, their formulation is explored only for convolution, and they restrict clustering to groups of weights (filter rows) rather than individual weights due to computational limitations as recalibrat- ing the k-means regularisation term is expensive during training. Similarly (Stock et al. 2020 Fan et al. 202T), ) focus on quantising groups of weights into single codewords rather than the individual weights themselves. Weight-sharing approaches similar to ours include (Ullrich et al., 2017). The authors use the distance from an evolving Gaussian mixture as a regularisation term to prepare the clustering weights. Although it is successful with small dataset-model combinations, the complex optimisation - particularly the additional requirement for Inverse-Gamde the mixture variance prevent mode collapse- limits the method's practical applicability due to the high computational costs of training. In our formulation, the weights already fixed no longer contribute the regularisation prior, reducing the computational overhead.
priors to lower-bound
3 METHOD
Quantisation. Consider network N parameterised by N weights W = (wi, w. Quantising a network is the process of reformulating A' +- '' where the new network A contains weights which all take values from reduced pool of k cluster centres C {cl, ... } where k & N. After quantisation, each of the connection weights in the original network is replaced by one of the cluster centres w; Cj. W = fw/lw; E C, i 1, .N), W k, where W is the set of weights of the new network A", which has the same topology as the original N.
4

---

Under review as a conference paper at ICLR 2022
Method Outline. WFN is comprised of T fixing iterations where each iteration E T has training and clustering stage, The clustering stage is tasked with partitioning the weights into two subsets W W fixeed U W free W fixed the set of weights fixed at of the cluster centre values C E C. The weights w; E Wiired are not updated by gradient decent in this, nor any subsequent training stages, these weights are fixed. In contrast, the free-weights denoted by Wreee remain trainable during the next training stage. With each subsequent iteration we increase the proportion of weights that take on fixed cluster centre values Pr such that po P1 P1.... < PT where Pt
[W] W
WI By iteration T, all weights will be fixed to one of the cluster centres making pr = 1. The training stage combines gradient descent based enror correction along with tight-cluster regularisation term and is tasked with maintaining lossless performance as we fix more of the weights cluster centres.
Clustering Stage. In the clustering stage, we work backwards from our goal of minimising the rel- ative distance travelled for each of the weights to determine which values cluster centres G E C should take. Given a weight wi E W and cluster centre Cj E C' we use a distance function D(w,, 9j) = |wy eyl and the original weight value w; to define a relative distance measure Dre((w,, Cy) ((w) Normalising by w; ensures that distance is measured proportionally to Jws,
the original weight value, rather than in absolute terms. To use this in determining the cluster cen- tres, we enforce threshold 5 on this relative distance, D (W, Gy) S 5 for small ö. We can then define the cluster centres Gj E C' which make this possible using a simple recurrence relation, As- sume we have starting cluster centre value cj. we want the neighbouring cluster value Cj+1 to be such that if a network weight we between these clusters wt E [ej, 5-1+() then Drat (We C2) S 6.
Plugging in S+1te and ej into Drat and setting it equal to ō we have:
Cyl
(1)
F++1 -
this leads to a recurrence relation:
9y-1 9-1= 1+6 (- 0<8<L
(2)
that provides the next cluster centre value given the previous With this, we can generate all the cluster centres given some boundary conditions. Starting with some value co, we generate er which is then used generate 02 and so on. We introduce the lower-bound cluster threshold cp = og. which we use to generate the cluster centres and any absolute weight value that is less than 6o is assigned the cluster value O. This lower bound serves two purposes: firstly, reduces the number of proposal cluster centres which would otherwise increase exponentially in density around zero, and additionally, the zero-valued weights will allow sparsity-leveraging hardware to avoid operations that use these weights, reducing the computational overhead. As an upper-bound, we stop the recurrence once a cluster centre larger than the maximum weight the network, maxy lcyl < max; [wi), w; W. Cj C.
Generating the Proposed Cluster centres. Putting this together, we have starting point co do- a recurrence relation to produce cluster centres given co that maintains a relative distance change when weights are moved to their nearest cluster centre, and centre generation stopping condition Cj & maxiew |wx]. cj E C. We use the &n value as our first proposed cluster centre co with the recurrence relation generating proposed cluster set of size s. Since all these values will contain only positive values, we join this set with its negated version along with a zero value to create a proposal cluster set C {a( 1+6 ) p j 0, 1....... 8; +1,0, -1} of size 2s + 1.
5

---

Under review as a conference paper at ICLR 2022
76
ResNet-50
0,0015
74
0.01
Accuracy
6:615
ResNet-34
72-
GoogLeNet
ResNet-19
Baseline Sizes
Resliet 36: 8746
Res-lad-50: s0M4
Model Size (MB)
Figure 3:
The accuracy vs model size trade-off can be controlled by the å parameter. All experiments shown are using the ImageNet dataset, accuracy refers to top-1.
To account for the zero threshold 60 and for ease of notation as we advance, we make a slight amendment the definition of the relative distance function cr))
D(wGed) lwgl
if | 2 50
(tr cy)
(3)
0,
otherwise
Reducing i with Additive Powers-of-two Approximations. in C as centres to cluster the network weights would meet the requirement for the relative movement of weights their closest cluster to be less than 6, it would also require large number of k |C*|| clusters. In addition, the values in C are also of full 16-bit precision, and we would prefer many of the weights to be powers-of-two for ease of multiplication in hardware. With the motivation of reducing k and encouraging powers-of-two clusters whilst maintaining the relative distance movement where possible, we look a many-to-one mapping of the values of C to further cluster the cluster centres. Building the work of others (Zhou et al. 2017, Yuhang Li, Xin Dong 2020), we map each of the values oj C to their nearest power-of-two, round (s) sgn(c; G, 2 [ogz and, for flexibility, we further allow for additive powers- of--too rounding. With additive powers-of-two rounding, each cluster value can also come from the sum of powers-of-two values (b-bit) up to order w where the order represents the number powers-of-two that contribute to the approximation As example, given Cj 0.45, C. C an approximation would round e to its nearest power-of-two c = 전 = 0.5 whereas an order w approximation is cx शτ श 0.4375. A higher-order grants a better approximation, but at the cost of more cluster centroids overall and requires wa bit-shifts additions in hardware. We give full details of the algorithm to map the proposal set C to the w-order approximation Cw in the Appendix.
Although using all of the values
Minimalist Clustering. We are now ready to present the clustering procedure for particular it- eration which we give the pseudo-code for Algorithm [ We start the iteration with w = and a set of weights not yet fixed WT For the set of cluster centres Cw of order w, let (1) m.. (wt c) be the closest to weight ws ny Ie ci) counts the number of weights assigned to cluster centre of E C where the indicator function [[x]] is I if x is true and 0 otherwise. Let k. arg max; ni so that ct- is the modal cluster. For the cluster ke let permutation π of [1, (1,...,N) Nj that maps w; + "Z()) be such that the sequence (wi (k*), wng (k=)).... ...., w'y (k*)) is arranged in ascending order of relative distance from the clus- ter of.. In other words, D. (w"(k*) (,r,)) s Di (1a(w/41( (k"), eg. ) for b,....(N. (N- 1). We choose n to be the largest integer such that:
n+1
D Di (w((", S nd, and
Σ pt (w((()) (n 1)6,
(4)
Thh notion of an order has heen identified recently in work parallel ours Oh et al. 2021 where the term word is used instead, i.e. nwo-word directly translates into

---

Under review as a conference paper at ICLR 2022
and define {wi, w2. w....... w/} be the set of weights to be fixed at this stage of the iteration. These are the weights that can be moved to the cluster centre o without exceeding the average relative distance 8 of the weights from the centre. The corresponding weight indices from the original network N are in (1),... (n)), and called fixednew the algorithm. If there no such weights that can be found, i.e., for some cluster centre 1", the minimum relative dis- lance D (w1 (I*), cx-) > 8, the corresponding set firednew is empty. In this case, there are no weights that can move to this cluster centre without breaking the å constraint and we increase order < w + 1 to compute new ck. repeating the process until fizednew > O. Once the condition Ifixedzew
satisfied, we fix the identified weights {wi, w/y.... ... w/ } to their corresponding fixed- We continue the process of identifying cluster cluster centre value ex- and move them into W
centres and fixing weights to these centres until |W 2 Npt, at which point the iteration is complete and the training stage of iteration t + 1 begins. Our experiments found that larger has less impact on task performance during early iterations and so we reduce linearly with each increasing & E T (full details in the Appendix). We will show later that, with small 5, over 75% of the weights can be fixed with w = 1 and over 95% of weights with w S 2.
Training Stage. Despite the steps taken to minimise the impact of the clustering stage, without retraining, performance would suffer. To negate this, we perform gradient descent to adjust the remaining free weights W free This allows the weights to correct for any loss increase incurred after clustering where training aims select values W free that minimise the usual loss (cross-entropy in our case) whilst W Wized remain unchanged.
Cosying up to Clusters. Having the remaining free weights closer to the cluster centroids C post-training makes clustering less damaging to performance. We coerce this situation by adding regularisation term given in Equation
(wgie))
to the retraining where p(cs/ux)) > e (mpo())))))
The idea is to penalise the free-weightt Wivee proportionally to their distance away from the clusters to which they are most likely to be assigned - i.e. the closest. Clusters that are unlikely to be weight weys s nearest - and therefore final fixed value - do not contribute much to the penalisation term. We balance this regularisation term against the cross-entropy training loss with the y hyper-parameter, which we fix to be a proportion of the other loss term y Cerare Leeg
We note that the 7 term is detached from the computational graph and treated as constant (otherwise the Lreg term would cancel).
N
Greg = ?
Σ
Σ
Dt
rag (wi, (x)p((/)p)))
(5)
iEW
4 EXPERIMENT DETAILS
We apply WFN fully converged models trained on the CIFAR-10 and ImageNet datasets. Our pre-trained models are all publicly available with strong baseline accuracie Resnet-(18.34.50 (Wu et al., 2017). MobileNetV2 (Sandler al., 2018) and, GoogLeNet (Chollet) 2017). We run ten weight-fxxing iterations for three epochs, increasing the percentage of weights fixed until all weights are fixed to cluster. In total, we train for 30 epochs per experiment using the Adam optimiser KKngmma & Ba 2015) with learning rate 2 x 10-5, We use grid-search to explore param-err combinations using ResNet-18 and MobileNetV2 models with the CIFAR-10 dataset and find that the regularisation weighting
0.4 works well across all experiments reducing the need for further hyper-parameter tuning as we advance. The distance threshold 8 gives the practitioner trade-off (see Figure 33). and so we report range of
control over the camppression-perfrmmaee values. We give full details of the experiments along with the results a hyper-parameter ablation study using CIFAR-IO the Appendix.
CIIAR-11 models: : http:///iitahcccmmmmnnniiiiiiahh ImageNet models: torchvision
7

---

Under review as a conference paper at ICLR 2022
1.0
200
Fived Weights
0.8
0.6
100
75
0.4
0.2
0.0
2
Rxing Iteration
Rxing Ieration
Fixing
Reration
Weight Values
Figure 4: Far left: We increase the number of weights in the network that are fixed to cluster centres with each fixing iteration. Middle left: Here, we show how decreasing the ò threshold increases the number of cluster centres, but only towards the last few fixing iterations, which helps keep the weight-space entropy down. Middle right: The majority of all weights are order (powers-of- two), the increase order only needed for outlier weights in the final few fixing iterations. Far right: The weight distribution (top-15 most used show) is concentrated around just four values. All of these charts are produced with ResNet-1 18 models trained on the ImageNet dataset.
RESULTS
Accuracy (%)
Accuracy (%)
Model
Method
Top-1
Top-5
CR
Model
Method
Tup-1
Top-5
CR
ResNet-18
Baseline
68-9
38.9
ResNos-34
Biseline
73.3
9019
1.0
LQ-Net
68.2
LQu.Net
21,9
90.2
APOT
69.9
89.2
10.2
APST
73.4
91.1
10.6
15Q
70.2"
80.4*
9.0"
91.4"
9.2"
0.015)
67.3
87.6
WPNG8 0.015)
72.2
90.9
WTN (8 = 0.01)
69.7
89.2
12.3
WFN (6 0.01>
72,6
91.0
11.1
WFN 0.0075)
70.3
89.1
10.2
WFN 0.0075)
73.0
91.2
10.3
ReiNet-30
Baseline
928
GoogLeNi
Beseline
09.7
89.6
1.0
LQO-Nee
74.2
91.6
3.9
Deep M-Manns
09.4
89.7
3.0
APoT
75.8
92.7
9.0
Grelidee
67.3
88.9
4.5
150
75.8"
91.7*
8.1"
KQ WFN 0.01.5)
69.2
588
WFN (6 0.015)
75.1
92.1
10.3
70.5
89.9
9.0
WFN (8- 0.01)
75.4
92.5
9.8
WFN (4 0.01)
70.5
90.0
8.4
WFN 0.0075)
WFN 0.0075)
Table 1: A comparison of WFN against other quantisation and weight clustering ap- proaches. The WFN pipeline is able to achieve higher compression ratios than the works compared whilst matching or improving upon baseline accuracies.
Estimated from the LSQ puper model comparison graph, eximate be as fair possible. Open-source implementations have been unable replicuted the reported hitps//ggitth..coooooi Quummiztii..t.iiiiinin
We begin by comparing WFN for a range of values against a diverse set of quantisation ap- proaches that have comparable compression ratios (CR) in Table The 3-bit quantisation methods we compare include: LSQ (Fisser et al. 2020). LQ-Net (Zhang et al. 2018b) and APoT (Yuhang methods using the GoogLeNet model: Deep-k-Mean (Wu et al. 2018) whose method is similar to ours, KQ (Yu et al.) 2020). and GreBdec (Yu et al.) (017). Whilst the results demonstrate WFN's 's lossless performance with SOTA CR, this is not the main motivation for the method. Instead, we are interested in how WFN can reduce the number of unique parameters in a network and corresponding weight-space entropy which together give the potential benefit of future accelerator designs to reduce data move- ment costs. We note the interested reader there are additional theoretical reasons for pursuing low weight-space entropy networks (Hinton & van Camp 1993, Grünwald, 20000 Hansen & Yu (001), but our primary focus is from the perspective of reducing accelerator data movement costs. Captur- ing the benefit of algorithmic developments to reduce data movement costs in accelerator designs non-trivin non-trivial due to the significant dependency on architecture memory hierarchy, communication orchestration costs and the dataflow mapping used (Sze et al.] 2020 Kwon et al., 2019 Sakr et al.) 2017]. Some active works attempt to provide robust estimations bridging the gap between esti- mation and actual energy costs. This estimates energy efficiencies for various accelerator mappings Kwon et al., 2019). or fixes the mapping and estimates the energy costs on a particular accelera- for (Yang et al. 2017). However, this support is limited to specifie fixed architecture designs and does not account for the referenced weight-sharing that we introduce. Instead, we rely on analytical model metrics such as the proposed by (Wu et al. 2018), itself subset of measurements con-
Li, Xin Dong] 2020). We additionally compare with the clustering-quantiss
8

---

Under review as a conference paper at ICLR 2022
Model
Method
Top-1
Entropy
Param Coant
Rep(A")
Model Size
ResNet-18
Baseline
68.9
23.3
10756029
1,000
46A
APoT (3bin
69.9
5.77
9237
WFN (8 0.015)
67.3
2.72
90
0.005
3.5MB
WFN (6 0.013
69.7
3,01
164
0.807
5AMB
0.0075)
34
Baseline
23.3
24.1
19014E31
1,000
S7,NN
APoT Obio
73.4
6.77
16748
0.296
8 23MD
WIN (5
72.2
0,012
WIN
72.6
WFN = 0.0075)
73.0
3.87
233
0.004
B.MBB
ResNet-50
Basaline
76.1
24.2
19915744
1,000
97.5M SMB
25.1
WFN ( 0.013
75.4
4,00
199
0.002
10.0MB
WFN (6 = 0.0075)
76.0
4.11
261
0.003
10.2MB
Table 2: A full metric comparison of WFN Vs. APoT. For readiblity Rep(N") is given relative to baseline. Note, APoT did not release the ResNet-50 model save which is why it is not compared here.
ducted by (Sakr et al. 2017). in terms of the number No of operations that each of the |W"| unique weights of bit-width B. is involved in. The low-entropy encoding that we aspire is motivated by the accelerator designs (Moons & Verhelst 2016 Han et al. (016) that seeks to exploit the Huff- man encoding possible for the network index set. Hence, the index for each weight w; E W" can be represented with bit-width of Bas and we can account for the number No:; of times they are used in an inference computation. Thus, instead of the representational cost Nw. W of (Sakr et al, 2017 for the final network N", we use metric:
Rep(N") = Σ
Nw, Bu,
(6)
MI, w.EW!
The authors of the APoT have released the quantised model weights and code, and we use the released model-savee" of this SOTA model to compare the entropy, Rep(N"). unique parameter count, model size and accuracy in Table 2 Our work outperforms APoT in weight-space entropy, unique parameter count and weight representational cost by large margin. Taking the ResNet- 18 experiments as an example, the reduction to just 164 weights compared with APoT's 9237 demonstrates the effectiveness of WFN. This in part due our full-network quantisation (APoT, as aforementioned, does not quantise the first, last and batch-norm parameters), but even when we discount these advantages and look at weight subsets ignoring the first, last and batch-norm layers WFN uses many times fewer parameters and half the weight-space entropy. We provide a further breakdown of these results in the Appendix (Table 4.. Finally, let us examine how WFN achieves the substantial reduction in weight-space entropy. In Figure we see that not only do the resultant WFN networks have very few unique weights, but the weight distribution is such that the vast majority of all of the weights are small handful of powers-of-two values (order 1) and the other unique weights (outside of the top 4) are of low frequency and added only in the final fixing iterations.
CONCLUSION
We have presented WFN, a compression pipeline that can successfully compress whole neural networks. A single network codebook, focusing on reducing entropy of the weight-space coupled with the number of unique weights in the network. The WFN process produces highly compressible and potentially hardware-friendl. representations of networks using just a few unique weights without performance degradation. Although the results demonstrate strong compression ability, few unique parameters, and low weight-space entropy, the true potential of the method that will gives accelerator designers more scope for weight re-use, keeping most/all weights close to computation, reducing the mergy-hungry data movement costs.
"hatp://githuhh.om//hhhhi// Quantization
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yiran Chen, Yuan Xie, Linghao Song, Fan Chen, and Tianqi Tang. A Survey of Accelerator Ar-
chitectures for Deep Neural Networks. Engineering, 6(3):264 274, 2020. ISSN 20958099, doi: 101111..1111111 wtwp:///.............. 01 .1.00..0.00. 007
10.1016/j.eng.2222... 2020.01.07. URL
Yu Hsin Chen, Tushar Krishna, Joel Emer, and Vivienne Sze. Eyeriss: An Energy-Efficien Re- configurable Accelerator for Deep Convolutional Neural Networks, IEEE J. Solid-State Circuits, 52(1):127-138, jan 2017. ISSN 00189200. doi: 10. 00.11099SSCC.226626665553
Yunji Chen, Tao Luo, Shaoli Liu, Shijin Zhang, Ligiang He, Jia Wang, Ling Li, Tianshi Chen, Zhiwei Xu, Ninghui Sun, and Olivier Temam. DaDianNao: Machine-Learning Supercomputer. In Proc. Annu. Int. Symp. Microarchitecture, MICRO, 2015. ISBN 9781479969982. doi: 10.1 109/ NHCRO.2014.55
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proc. 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, 2017. ISBN 9781538604571. doi: 10.1 N0.100/CVP..201..199
Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned Step Size Quantization. 8th Int. Conf. Learn. Represent. ICLR 2020, 2020. URL
wwtpa//////...... 08153]
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. Training with Quantization Noise for Extreme Model Compression. 9th Int. Conf. Learn. Represent. ICLR 2021 Conf. Track Proc., apr 2021. URL 2004.07320
Farzad Farshchi, Qijing Huang, and Heechul Yun. Integrating NVIDIA Deep Learning Accel- erator (NVDLA) with RISC-V SoC on FireSim. In Proc. 2019 2nd Work. Energy Effic. Mach. Learn. Cogn. Comput. Embed. Appl. EMC2 2019, 2019. ISBN 9781728167633. doi: 10. 1.11199EMM2249883.11990090
Mingyu Gao, Jing Pu, Xuan Yang, Mark Horowitz, and Christos Kozyrakis. TETRIS: Scalable and efficient neural network acceleration with 3D memory. ACM SIGPLAN Not., 2017. ISSN 15232867. doi: 10.1 /45/3037697.30377002
P. Grünwald. Model selection based on minimum description length. J. Math. Psychol., 44(1): 133-152, mar 2000. ISSN 00222496. doi: 10.1006/jmps. 199...1....99.1999.. 1999.1280.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. EIE: Efficient Inference Engine on Compressed Deep Neural Network. feb 2016. URL
http://arxivv org/ hbtp//a.//11/00000 01528
Mark H. Hansen and Bin Yu. Model selection and the principle of minimum description length. J. Am. Stat. Assoc., 96(454);746-774, 2001. ISSN 1537274X. doi: 10. 10.119801121155015336663333
URL https::/ www. jstor. http://rr...rtttaaeeeeeeeeeeee/e
Geoffrey E Hinton and Drew van Camp. Keeping neural networks simple by minimizing the de-
scription length of the weights. pp. 5-13, 1993. ISBN 0897916115. doi: 10.1 155/11163334..1630 168306.
Mark Horowitz. 1.1 Computing's energy problem (and what we can do about it). In Dig. Tech. Pap. IEEE Int. Solid-State Circuits Conf., 2014. ISBN 9781479909186. doi: 10.1 109110//1SC.. 6757323.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In Adv. Neural Inf. Process. Syst., 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and Training of Neural Networks for Efficient antege--Arihmmtta Only Inference. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Par- tern Recognit., pp. 2704-2713, 2018. ISBN 9781 538664220. doi: 10.
10

---

Under review as a conference paper at ICLR 2022
Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Ba- jwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre Luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipatt, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le. Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Frick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit. In Proc. Int. Symp. Comput. Archit. 2017. ISBN 978 9781450348928. doi: 10.1145/3079856.0000222
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, 2019.
ISBN 978172817 1722132938,
doi: 10. N...10//CVRR.211900022
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd Int. Conf. Learn. Represent. ICLR 2015 Conf. Track Proc., 2015.
Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer, Angshuman Parashar, Vivek Sarkar, and Tushar Krishna. Understanding reuse, performance, and hardware cost of DNN dataflows: A data-centric approach. In Proc. Annu. Int. Symp. Microarchitecture, MICRO, 2019. ISBN 9781450369381. doi: 10.1 00.1145/3352400.333222
Edward H. Lee, Daisuke Miyashita, Elaina Chai, Boris Murmann, and S. Simon Wong. LogNet: Energy-efficient neural networks using logarithmic computation. In ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process. Proc., 2017. ISBN 9781509041176. doi: 10.1109/ICASS. 2017.7953288.
H. Li, B. Gao, Z. Chen, Y. Zhao, P. Huang, H. Ye, L. Liu, X. Liu, and J. Kang. A learnable parallel processing architecture towards unity of memory and computing. Sci. Rep., 2015. ISSN 20452322. doi: 00.1008//rep13330.
Huizi Mao and William Dally, Deep Compression: Compressing Deep Neural. Iclr 2016, pp. 1-14, 2016. URLhttts://aaxxx www.aawwwww....a..a..aroaaaa.eoa wttpg//b//1111...0 1510000141
Bert Moons and Marian Verhelst. 0.3-2.6 TOPS/W precision-scalaie scaaaable processor for real-time large-scale ConvNets. IEEE Symp. VLSI Circuits, Dig. Tech. Pap., 2016-Septe:1-2, 2016. doi:
10.1109/VLSCC22216.557773
Sangyun Oh, Hyeonuk Sim, Sugil Lee, and Jongeun Lee. Automated Log-Scale Quantization for Low-Cost Deep Neural Networks. Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 2021.
Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. Analytical guarantees on numerical precision of deep neural networks. 34th Int. Conf. Mach. Learn. ICML 2017, 144660346155 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bileNetV2: Inverted Residuals and Linear Bottlenecks. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 4510-4520, 2018. ISSN 10636919.
doi: 10.1109/CVPR.22010000
Abu Sebastian, Manuel Le Gallo, Riduan Khaddam-Aljameh, and Evangelos Eleftheriou. Memory devices and applications for in-memory computing, 2020. ISSN 17483395.
Moran Shkolnik, Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, and Uri Weiser. Robust Quantization: One Model Rule Them All. Technical report.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks, 2020. ISSN 23318422.

---

Under review as a conference paper at ICLR 2022
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient Processing of Deep Neural Net- works: Tutorial and Survey. mar 2017. RRLhttp::2
Vivienne Sze. Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. Efficient Processing of Deep Neural Networks. Synth. Lect. Comput. Archit, 2020. ISSN 1935-3235. doi: 10.2200/
s01004ed1 50110200000122
Enzo Tartaglione, Stéphane Lathuilière, Attilio Fiandrotti, Marco Cagnazzo, and Marco Grangetto.
HEMP: High-order Entropy Minimization for neural network comPression. 2021. doi: 10.10 neucom.2021 007.022. URL httpt:::///axxi..or////tate
http//bb//210110 05298 /htttp:///x
doi. org.orr/110 10. 1016//-....... neucom. 07.21.77.02
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compres- sion. arXiv, pp. 1-16, 2017.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. 35th Int. Conf. Mach. Learn. ICML 2018, 12:8523- 8532, 2018.
Songtao Wu, Shenghua Zhong, and Yan Liu. ResNet. Multimed. Tools Appl., 2017. ISSN 15737721.
doi: 10.1007/s 0444-017-444-4-4
Kohei Yamamoto. Learnable Companding Quantization for Accurate Low-bit Neural Networks. pp-
5029-5038, 2021. URL [tttt://aaxxiiii org/ 2103. 07156
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian Sheng Hua. Quantization networks. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2019. ISBN 97817281388
doi: 10.110///PR.20199
Tien Ju Yang. Yu Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural net- works using energy-aware pruning. In Proc. 30th IEEE Conf. Comput. Vis. Puttern Recognition, CVPR 2017, 2017. ISBN 9781538604571. doi: 10. 00.1109/VVP..2017.643
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. Proc. 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, 2017-Jamua:67-766 2017. doi: 10. 10.1109/CVR..2017.155
Zhongzhi Yu, Yemin Shi. Tiejun Huang, and Yizhou Yu. Kernel Quantization for Efficient Network
Compression. 2020. URL http: /ttp://///rii..///a wwwxivv.rg//e. wwt.abs////aeo.o.oee....a/a.aeaa http:/abbb2003.514.4/
Wei Wang Yuhang Li, Xin Dong. Additive Powers-of- Two Quantization: an Efficient Non- Uniform Discretization for Neural Networks. Iclr, (2014):201620)) 2019, 2020. URL https:
2id=BkgxT24EDS
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quanti- zation for highly accurate and compact deep neural networks. In Lect. Notes Comput. Sei. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), volume 11212 LNCS, pp. 373-390, 2018a. ISBN 9783030012366. doi: 10.1007/978-3-030-01237--- htttps://githue htt.:aaabbccommocccooooo//o//--.
URL
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), 2018b. ISBN 9783030012366. doi:
Dongging Zhang. Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks, 2018c. ISSN 23318422. URI. https:
7/ //github. com http://icroooooo crosoft
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Nett Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. jun 2016.
URL hatt:://arxa org/ http//abs11166.5555 06160
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1
THE WEN OVERVIEW ALGORITHM
Algorithm 1: Clustering Np: weights at the 1 iteration.
while W W Ny: do
fixednew [1
while fizednew is empry do Increase the order w f- w
List the cluster centres in C with smallest distances to each wy W Set the cluster centre with the most weights assigned of (i) Sort the weights w; W free by their distance to (1)
free
do
Append to fizednew the w, W free with next smallest distance
10
while The mean ( fizednew ct (1) S 8
12
Fix the all the weights in fizednew to cluster centre e
A.2 DETAILS OF THE POWERS-OF-TWO APPROXIMATION ALGORITHM
We map our proposal C to w-order approximation where each of the clusters O E CS are written as powers-of-two (Eq 7). We do so using Algorithm Figure 5 demonstrates how the values of C are rounded given different orders.
W
Ck
T ry E E(
26
2/+1' 25 O, 25 20+1
26
(7)
1=1
Algorithm 2: Determining possible clusters
Input:
The full precision proposal set: C allowable relative distance: 8, pow2 rounding function: round(1)) sgn(x))2llogx(a))
Output: An order w precision cluster set: T
C +-1
for C 8 do ch round((s.) w do 8op t C2 cj
for i
if 8eg 2 dck then
c t c + round(6e,)
10
end
11
end
CH + 1H0 Uf6}
12
13
end
A.3 REDUCING
To account for the observation that smaller relative distances well tolerated in the early stages but less so as the final weights are quantised (high t), linearly. For any given E T we use
5th 5(T 1. 1++1 1) our threshold.
13

---

Under review as a conference paper at ICLR 2022
c
0.0
0.1
0.3
0.4
0.5
Potential Cluster Center Value
Figure 5: Approximating clusters in C with different orders
A.4 EXPERIMENT DETAILS
We give full breakdown of the parameters used across all experiments ran in Table
Opt
Epochs per
ResSect 18
ImageNett
Adam
2e-4
10
128
[000, dAE5, 0,015, 1011, 00075, duts
(0.2. 0:4)
ResNet 34
ImageNet
Adam
2e-4
10
64
(0.05, 0,025.. 0,015, 0.01, 0.075, 0.05
{0.4)
ResNee-50
InsageNet
{0.05 0.025. 0.015, D.01, 0.075, 0.03
[0.4)
GoogLaNet
ImageNet
Adam
26-4
10
64
{0.01, 0.015, 0.015) (0.01, 0,02, 0.03, 0.01, 0.05) (0.01, 0.03, 0.04, 0.05)
[0.4]
18
CIFAR 10
Adam
3e-4
10
(3.5, 10)
512
(0.0, 0.1. 0.4
MobileNet
CIEAR-10
26-4
[3.5,
0.4,
Table 3: Full set of hyper-parameters explored for each model-dataset combination.
A.5 HYPER HYPER-PARAMETER PARAMETER EXPLORATION
We conducted parammer search on both the ResNet-18 and MobileNet (Figures 6 7. We find similar results across models when selecting byper-paraameeess and additionally, observe that as we increase the regularisation term weighting, a weight-space entropy reduction. Further, found that negating this term altogether resultant entropy values and causes an accuracy performance drop, demonstrating the utility of the regularisation term
O has the surprising effect of increasing the
A.6 LAYERWISE BREAKDOWN
In Figure examine how the parameter count and layer-parameter entropy change with each layer for both the WFN and APoT approaches. find both gains over the unquantised layers of APoT, but also that the entropy and parameter count in the convolutional layers (those quantised by APoT) are similar.
A.7 A FULL METRIC COMPARISON
In Table 4 we give the full metric breakdown comparing WFN to the state-of-the-art APoT work. We calculate the unique parameter count and entropy values for subsets the weights. No BN corresponds to all weights other than those in the batch-norm layers, and No BN-FL is the set of weights not including the first-last and batch-norm layers. It's clear here that WFN outperforms APoT even when we discount the advantage gained of taking on the challenge of quantising all layers.
A.8 PRUNING EXPERIMENTS
To explore how WFN interacts with pruning we conduct a simple set of experiments. Instead of starting the WFN process with all weights un-fixed randomly select p% of the weights to be pruned in each layer. We then run WEN as before starting with pt = p. reducing the number of T' iterations. The results, shown in Figure conducted with ResNet-18 18 and CIFAR-10
14

---

Under review as a conference paper at ICLR 2022
MobileNet Cifar-10
0.94
0.92
Accuracy
0.88
0.86
125
100
50
1.4
0.0
0.1
0.2
0.4
0.9
0.01
0.02
0.03
0.04
0.05
False
True
Inc BN
Figure 6: Experiments exploring the hyper-paramene space with MobileNet model trained on the CIFAR-10 dataset. Columns: Left: varying the regularisation ratio a, middle: varying the distance change value 6, Right: whether we fix the batch-norm variables. Rows; top: top-1 accuracy test-set CIFAR-10, Middle: total number of network weights, bottom: entropy of the weights.
Full Network
No BN
BN-FL
Model
Method
Top-1
Entropy
Parum Count
Entrupy
Parum Count
Entropy
Param Count
Model Size
ResNes18
Roscline
23.3
10756079
23.3
10744788
10276360
46R
(Sba)
9237
(6 0.015)
67.1
2.72
90
2.71
81
2.5
81
3.MMB
WFN (8 0.01)
69.7
3.01
164
3.00
153
2.75
142
3 SMB
0.0075)
4.13
176
34
Hascline
23.3
24.1
19014310
24.1
189960200
24.10
18551024
STAMB
APoT (3bit)
23.4
6.77
16748
6.75
16474
6.62
389
82MB
WFN
100
2.68
6-MS
3.47
WIN (8 0.0075)
73.0
3.87
233
3.85
191
3.24
187
8.5MB
ResNor-0
Basalime
76.
24.2
19915744
24.2
19872598
24.20
18255490
3MB
105
W (8 0.01)
75.4
3.97
169
3.88
163
10.0MB
WFN (8 0.0075)
76.0
4.11
261
4.09
223
4.00
217
10.2MB
Table 4: A full metric comparison of WFN Vs. APoT. We compare the unique parameter count and entropy of all parameters in the full network, as well as the same measures but not including the batch-norm layers (No BN) and the parameters not including the batch-norm and first and last layers (No BN-FL..
combination, painting a mixed picture. On the one hand, WFN and pruning at lower levels (< 50%) well tolerated and provide two benefits, lower weight-space entropy and fewer weight-fixing iterations. On the other hand, full-precision networks can tolerate much higher ranges of pruning so there would seem that a certain amount of synergy between the two approaches is present but this is tempered compared to full precision networks.
It's important to note that WFN already has a form of pruning built-in with the dg value balancing the emphasis on pruning over quantisation.
15

---

Under review as a conference paper at ICLR 2022
R
0.93
0.92
0.91
0.90
0.89
0.88
250
200
150
50
0.625
0.600
HWW
0.575
0.550
0.01
0.05
True
Figure 7: Experiments exploring the hyper-parameter space with ResNet18 model trained the CIFAR-10 dataset. Columns; Left: varying the regularisation ratio a. Middle: varying the distance change value 8, Right: whether fix the batch-norm variables. Rows; Top: top- accuracy test-set CIFAR- 10. middle: total number of network weights, bottom: entropy of the weights.
16

---

Under review as a conference paper at ICLR 2022
WP% (4-08) (5-08)
1020
-18)
800
Layer Unique Param Count
620
400
owbw owiloo
th to to cwobi partis
laycr2
leyard
layse
Mayos
laycre
layor?
layert lyyo0
loye
laper13
loyoee
layera lasert mpees7 laya
10
Layer Entropy
barth bar barth
tac de terbi ho cwitis cartro betwitt beitwit twe by bar
layari
layard
layors
layard
byeer
Figure 8: We compare WFN with a traditional quantisation set-up (APoT) with varying bit-widths applied ResNet18 model trained on the ImageNet dataset. The top chart shows the layerwise unique parameter count where WFN has consistently fewer unique parameters per layer.
1.0
0.00
0.8
0.19
ACC
0.6
Entropy
0.0
0.7
0.6
615
014
83
822
0.3
04
0.5
0.5
Prunod at Initiation
96 Prunod Initiation
Figure 9: How does WFN and random pruning at initialisation interact?
17


Reviews:
[{"review_id": "h5JpRUV6hq-", "reviewer": "Reviewer_geXX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The most expensive energy costs lie in memory reads. Thus, data read and movement operations, not arithmetic operations, dominate the energy costs of deep learning inference. This work focuses on reducing data movement costs by reducing the number of unique weights in a neural network. \n\nWFN is an interative scheme, where at each iteration there is a training and a clustering stage. The clustering stage carefully choses the closest fixed centers for some of the weights. A method is given to produce and increasingly larger set of cluster centers given some boundary conditions. Further representational reduction is realized by using power of two approximations, which lend themselves to very efficient multiplication by shifting. After each clustering stage, a retraining is performed in order to maintain good performance. \n\nExperimental evaluation on CIFAR-10 and ImageNet datasets shows the compression power of WFN. WFN networks have very few unique weights, and the weight distribution is a small handful of powers-of-two values, while the other unique weights are of low frequency and added only in the final fixing iterations.\n", "summary_of_the_paper": "The paper proposes a method called Weight Fixing Networks (WFN), which is designed to minimize the data movement in deep learning inference, thus minimizing the energy cost. WFN aims to maintain very few unique weights, low-entropy weight encodings, unique weight values which are amenable to energy-saving versions of hardware multiplication, while keeping the same performance. A new clustering algorithm for weight generation is shown and analyzed, and experimental evaluation shows the merits of the new scheme.", "summary_of_the_review": "The paper introduces WFN, weight fixing networks, an algorithm that transforms a neural network into a representation with very few unique weights, the ones most frequently used being powers of two, all while maintaining good inference performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RFTSr9q6ahr", "reviewer": "Reviewer_vtcG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenths: This paper discusses from the viewpoint of FLOPs do not accurately reflect the complexity of a model, but the data movement costs do. This viewpoint is quite novel to the reviewer and may shed lights into many related works. The technique part is clear and easy to follow. Empirical studies support that this method is able to reduce the parameter counts significantly.\n\nweakness:  The empirical studies seem to be a little bit strange because parameter counts are also a type of \"weak proxy\" for the energy consumption. How about the reduction of the energy consumption by using WFN? Despite of the parameter counts, what will WFN do to memery and latency? These might be more intereting and important to see than the parameter counts. Because parameter count is not linearly proportion to the latency and memery.", "summary_of_the_paper": "This paper focuses on reducing the data movement costs by reducing the number of unique weights in a network, i.e., resuing weights. Based on this consideration, a new compression pipeline that can successfully compress whole neural networks has been proposed, called WFN. WFN is based on traditional quantisation manner but aims to cluster the unique weights and reduce the weight-space entropy of the whole network. By doing so, the empirical studies show that the proposed WFN is efficient in terms of reducing parameter counts.", "summary_of_the_review": "The idea of this paper is novel and may inspire broadly future works. The technique design is clear and correct. The empirical design can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H9BYoI9kk9D", "reviewer": "Reviewer_kvqy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is interesting and appears to achieve its goals. Nevertheless, the impact of the work is currently difficult to assess. \n\nA comparison is made between: \n\na) A scheme that minimises the number of unique weights a network uses:\n    - enabling these to be stored locally in each PE\n    - the weights are shared across all layers of the network \n      (and require no layerwise scaling)\n    - weights are then stored as indexes into this (limited) set of weight values \n\nb) Quantizing and storing narrow weights:\n     - Quantization is performed and the weight values are stored, e.g. as 3-bit values\n     - To support such a scheme additional quantization parameters must be stored\n       on a per layer/filter basis and as such increases the number of unique parameters\n       in the network.\n\nThe argument is made that the approach presented would reduce data movement. What is a realistic expectation here? If we look at the \"No BN-FL\" param counts it would suggest we would require 81-162 parameters vs. 274. Compared to data movement in general, updating a small numbers of parameters within a PE's memory would seem insignificant? Could a stronger argument be made regarding the reductions in data movement that might result from this work? Infrequent data movements, between periods of significant computation, are of less concern perhaps? \n\nAn explanation of precisely what values need to be stored (and how the unique parameter counts are generated) would also make the paper easier to read. \n\nIn the case of WFN, how are weights stored precisely? i.e. to determine the model size. \n\nAre the weights in your scheme compressible with a simple scheme (i.e. suitable to be placed on-chip in each PE) in a way that would further reduce data movement? Again, perhaps the potential gains in the reduction of data movement on-chip could be made clearer?\n", "summary_of_the_paper": "A technique is developed to reduce the number of unique weights used in a network. The ultimate goal is to exploit this to reduce data movement in hardware. The unique weights are discovered multiple stages of training and clustering. An additional goal is to produce compressible weights. Results demonstrate that good compression ratios are possible, comparable to SoA quantization techniques, but with far fewer numbers of unique parameters. \n\n", "summary_of_the_review": "The work appears interesting but I am unsure about the potential impact on data movement. I found the paper quite difficult to follow in general. For this reason I am currently recommended that the paper is not accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Au2gCizG6zK", "reviewer": "Reviewer_wcJb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written (with some minor grammatical errors at some places) and the results shown by the author are comprehensive. \nOne major weakness I found in the paper is that the analysis is not exactly correct. The authors have shown all experiemntal benefits with parameter counts. But reduction in parameters does not necessarily mean energy efficiency. There is a lot of communication related latency costs that add to energy. The type of router/NoC, their data width etc. play a role in characterizing this. Also, just reducing parameters or unique weights does not mean you will have to access the memory less. Depending upon the PE array size, FIFO width etc. , your data access costs may vary.\n\n\nMinor comments for the authors are:\n1) Is their approach only advantageous for accelerators where weights have to be moved from main memory to PEs? Can the authors comment on how their approach is amenable to many emerging accelerators with weight stationary like flow [1, 2], such as compute-in memory or crossbar based where all weights are already programmed in the hardware? [2] talks about a compute in memory architecture for quantized neural networks which is relevant to the author's proposition. \n[1]Shafiee, Ali, et al. \"ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars.\" ACM SIGARCH Computer Architecture News 44.3 (2016): 14-26.\n[2] Vasquez, Karina, et al. \"Activation Density based Mixed-Precision Quantization for Energy Efficient Neural Networks.\" arXiv preprint arXiv:2101.04354 (2021).\n2) The authors have shown that their work performs better than many clip and scale quantization works. Can the authors comment if their work has any resemblance to pruning methods? For e.g. if we start with a pruning at initialization based network and then try to perform the WFN technique), will it have any repurcussions in the clustering and encoding of unique weights.\n3) Further, there has been recent works [3, 4] that have tied adversarial attacks with weight sharing/quantization together. Can the authors comment on impliction of WFN and how introducing more unique weights can impact adversarial robustness?\n[3]Lin, Ji, Chuang Gan, and Song Han. \"Defensive quantization: When efficiency meets robustness.\" arXiv preprint arXiv:1904.08444 (2019).\n[4]Panda, Priyadarshini. \"QUANOS: adversarial noise sensitivity driven hybrid quantization of neural networks.\" Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design. 2020.\n\n", "summary_of_the_paper": "This work presents WFN, an ensemble of techniques to reduce data movement costs by reducing the number of unique\nweights in a network.  Overall the authors find some distinct advantages in their approach:\nThey emphasise a low entropy encoding with a regularisation term, achieving entropies smaller than even those seen using 3-bit quantisation approaches. They require no additional layerwise scaling; the unique weights are shared across all layers. They substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches.\n", "summary_of_the_review": "I think the authors have come up with a very interesting technique of methods and are targeting a novel issue of data storage for efficient implementation of neural networks. This paper falls under the same category as pruning or quantization, but provides a very fresh perspective. However, their metrics of evaluation using parameter reduction is weak and does not really support the energy efficiency claim that the authors make. Thus, I have given a rating of 5 for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method to reduce the number of unique weights in a network. The motivation is that this reduces energy consumption, and can lead to speed-ups.
The reviewers agree that the paper is generally well-written. They also agree that the method presented is interesting and useful to reduce the number of unique weights of the network.
However, reviewers challenge the authors claim that fewer unique weights necessarily lead to lower energy consumption. Operations involved with this method may even lead to more memory transfers. The experimental section supports well the claim that the number of unique weights is decreased at very little cost in terms of accuracy, but does not provide much data in terms of actual energy consumption.
Generally, the reviewers agree that the fundamental idea of this paper is good and should be presented, but since this paper motivation is entirely focused on energy consumption, I suggest that the authors revise the paper and submit it in a future venue. I can see two possible ways to revise the paper: 1) Add actual energy consumption data to the experimental section; or 2) change the motivation of the paper to solely focus on reducing the number of unique weights. It may be useful for some niche applications, or as a starting point for future works.
Decision: Reject