Year: 2021

Abstract:
The movement of data between processor and memory, not arithmetic operations, dominates the energy cost of inference computations in deep networks. Network compression offers opportunities for hardware design to bring weights in memory closer to processor reducing these data movement costs. To this end, we investigate the merits of a method, which we call Weight Fixing Networks (WFN). We design the approach to realise four model outcome objectives: i) very few unique weights, ii) low-entropy weight encodings, iii) unique weight values which are amenable to energy-saving versions of hardware multiplication, and iv) lossless task-performance. Some of these goals are conflicting. To best balance these conflicts, we combine a few novel (and some well-trodden) tricks; a novel regularisation term, (i, ii) a view of clustering cost as relative distance change (i, ii, iv), and a focus on whole-network re-use of weights (i, iii). Our Imagenet experiments demonstrate lossless compression using 56x fewer unique weights and a 1.9x lower weight-space entropy than SOTA quantisation approaches.

heading: 1 INTRODUCTION
text: The Importance of Data Movement Costs. Although there has been a significant amount of attention exploring both algorithmic (Sze et al., 2020) and hardware-based (Chen et al., 2020) approaches to reducing the energy costs of deep learning inference, there is often a noted disconnect between the two (Sze et al., 2017). The most expensive energy costs lie in memory reads (Horowitz, 2014; Gao et al., 2017). For every off-chip DRAM data read, you pay the equivalent of over two hundred 32-bit multiplications in energy costs1 (Horowitz, 2014). Algorithmic techniques that hope to reduce energy consumption have focussed predominantly on metrics like floating-point operations (FLOPs); intuitively, fewer FLOPs should translate into smaller energy costs from the reduced number of multiplications and potentially, fewer parameters to read from memory. However, it has been observed that this is a weak proxy for the energy consumed in running a deep learning model for inference (Sze et al., 2017). Data movement costs can still be high if we do not consider the re-use of weights, the delay between their re-use, and filter shape effects. Simply reducing the FLOPs alone does not guarantee energy cost reductions – particularly those incurred through data read and write activity.
Accelerator Design Considerations. These data movement costs are difficult to reduce in von Neumann architectures (Li et al., 2015; Sebastian et al., 2020), and so there is a move towards co-design of algorithm-hardware inference accelerators. A core consideration of accelerator design lies in exploring dataflow mappings. These mappings determine how data used in computation is distributed across memory components and optimised to take advantage of the re-use of weights, matrix multiplication partial-sums (psums), and data inputs (Han et al., 2016; Chen et al., 2017; 2015; 2020). Modern deep learning accelerators also make use of hundreds of processing elements (Chen et al., 2020) (PE’s) for computation. Each PE contains a small amount of cheap and fast access memory to store a few pieces of information (weights, psums, inputs, etc.). A dataflow mapping cuts down data movement costs by making the best use of each PE and ensuring that the information they contain is recycled as much as possible. A weight/input/psum that cannot be used quickly after its latest use will be written to storage and require a re-read later. Less re-use leads to increased data movement costs. Consider, for example, weight-stationary dataflow, commonly used in SOTA accelerator designs (Farshchi et al., 2019; Jouppi et al., 2017). Here the items stored
145nm process
statically in the PE are the model weights. Input data is then fed into the relevant PE’s where multiplication is conducted locally. The ideal situation for a dataflow mapping would be to pay a single data movement cost for each unique weight value and then reference these weight values using indexing. The indexing costs can be kept low using an appropriate encoding scheme such as Huffman coding and approximated by the entropy of the weight space. Thus, the indexing access costs plus unique value access costs can be much smaller than the unquantised network (Mao & Dally, 2016; Wu et al., 2018).
Objectives. So we ask ourselves what we could do algorithmically to maximise the benefit of accelerator dataflows? We think an easy win is to reduce the number of unique weights a network uses. Fewer unique weights whilst fixing the network topology and the total number of parameters will mean that more weights are re-used more often. This additional re-use gives more opportunity to dataflows to maintain often-used weights in PE’s. To further enhance the compressibility, it is desirable for the distribution of the unique weights to be concentrated around a handful of values. The high probability density weights would then be used more often and could then be reliably stored inside PE’s, saving both the cost of overwriting these weight values and re-fetching them when needed later. Finally, we ask what the ideal values of these weights would be. From a computational perspective, not all multiplications are created equal. Integer powers-of-two, for example, can be implemented as simple bit-shifts. Mapping the weights used most to these values offers potential further energy reductions. Putting these three requirements together: few unique weights; a low-entropy encoding with a distribution of weights highly concentrated around a tiny subset of values; and a focus on powers-of-two values for weights — all motivated to reduce computation costs in accelerator designs — we present our contribution.
Weight Fixing Networks. Our work’s overarching objective is to transform a network comprising many weights of any value (limited only by value precision) to one with the same number of weights but just a few unique values. Rather than selecting the unique weights a priori, we let the optimisation guide the process in an iterative cluster-then-train approach. In each iteration, we cluster an ever-increasing subset of weights to one of a few cluster centroids. We map the pre-trained network weights to these cluster centroids, which are the pool of unique weights. The training stage follows standard gradient descent optimisation to minimise performance loss with two key additions. Firstly, only an ever decreasing subset of the weights are free to be updated. And secondly, we use a new regularisation term to penalise weights in proportion to their nearest clusters’ relative distance. We iteratively cluster subsets of weights to their nearest cluster centre, as has been demonstrated successfully previously (Zhou et al., 2017). The way we determine which subset to move is a core component of our contribution which leads to the superior compression results we achieve.
Small Relative Distance Change. Rather than selecting subsets with small Euclidean distances to cluster centres, or those that have small magnitude (Zhou et al., 2017), we make the simple
observation that the relative – as opposed to absolute – weight change matters. The distance a weight is moved when quantised is dependent on the distance between the weight wi and its new value wi + δwi. When the new value is zero — as is the case for pruning methods — then the magnitude of the weight is the distance. Yet, this is not the case more generally. We demonstrate the importance of optimising quantisation for small relative changes with simple empirical observations. Using a pre-trained ResNet-18 model, we test adding relative vs absolute noise to the layers’ weights and measure the accuracy change. For relative noise we set a noise level β and adjust all weights wi in a layer l as: wli ← wli+N (0, β|wli|). The standard deviation of noise added is determined by the original value wli. We contrast this with absolute noise experiments, where we instead set wli ← wli +N (0, β|wl|) where |wl| corresponds to the mean absolute weight value in layer l. We run each layer-β combination experiment multiple times – to account for fluctuation in the randomised noise – and present the results in Figure 1. Even though the mean variation of noise added is the same, noise relative to the original weight value (multiplicative noise) is much better tolerated than absolute (additive noise). Since moving weights to quantisation centres is analogous to adding noise, we translate these results into our approach and prioritise weights that have small relative distances to be clustered first. We find avoiding significant quantisation errors requires ensuring that |δwi||wi| is small for all weights. If this is not possible, then performance could suffer. In this case, we create an additional cluster centroid in the vicinity of an existing cluster to reduce this relative distance. Our work also challenges the almost universal trend in the literature (Yuhang Li, Xin Dong, 2020; Jung et al., 2019; Zhang et al., 2018a; Zhou et al., 2016; Yamamoto, 2021; Oh et al., 2021) of leaving the first and last layers either at full precision or 8-bit; we attempt a full network quantisation. The cost of not quantising the first layer – which typically requires the most re-use of weights due to the larger resolution of input maps – and the final linear layer – which often contains the largest number of unique weight values – is too significant to ignore.
With multiple stages of training and clustering, we finish with a significantly reduced set of unique weights. The regularisation term encourages high probability regions in the weight distribution and a lower-entropy weight-space. The initial choice of cluster centroids as powers-of-two helps us meet our third objective – energy-saving multiplication. Overall we find four distinct advantages over the works reviewed:
• We assign a cluster value to all weights — including the first and last layers.
• We emphasise a low entropy encoding with a regularisation term, achieving entropies smaller than even those seen using 3-bit quantisation approaches – over which we report superior performance.
• We require no additional layerwise scaling; the unique weights are shared across all layers.
• WFN substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches.

heading: 2 RELATED WORK
text: Clip and Scale Quantisation. Typical quantisation approaches reduce the number of bits used to represent components of the network. Quantisation has been applied to all parts of the network with varying success; the weights, gradients, and activations have all been attempted (Hubara et al., 2016; Lee et al., 2017; Jung et al., 2019; Yang et al., 2019; Shkolnik et al.; Zhou et al., 2016). Primarily, these approaches are motivated by the need to reduce the energy costs of the multiplication of 32-bit floating-point numbers. This form of quantisation uses a rounding function and scaling factor (sf ) in making value adjustments where a weight wi is mapped to a new value w′i = sf ∗ round(wi) for some predetermined rounding function. The scaling factor (determined by a clipping range) can be learned channel-wise (Jacob et al., 2018; Zhang et al., 2018c), or more commonly, layerwise with separate formulations. This results in different channels/layers having a diverse pool of mapping values for the network weights/activations/gradients. Quantisation can be performed without training — known as post-training quantisation, or with added training steps – called quantisation-aware training (QAT). Retraining incurs higher initial costs but results in superior performance. A clipping+scaling quantisation example relevant to our own is the work of (Zhou et al., 2017), where the authors restrict the layerwise rounding of weights to powers-of-two. The use of powers-of-two has the additional benefit of energy-cheap bit-shift multiplication. A follow-up piece of work (Yuhang Li, Xin Dong, 2020) suggests additive powers-of-two (APoT) instead to capture the pre-quantised distribution of weights better.
Weight Sharing Quantisation. Other formulations of quantisation do not use clipping and scaling factors (Stock et al., 2020; Tartaglione et al., 2021; Wu et al., 2018). Instead, they adopt clustering techniques to cluster the weights and fix the weight values to their assigned group cluster centroid. These weights are stored as codebook indices, allowing for compressed representation methods such as Huffman encoding to squeeze the network further. Unlike clipping+scaling quantisation techniques, and like ours, these methods share the pool of weights across the entire network. The work by (Wu et al., 2018) is of particular interest since both the motivation and approach are related to ours. Here the authors use a spectrally relaxed k-means regularisation term to encourage the network weights to be more amenable to clustering. In their case, they focus on a filter-row codebook inspired by the row-stationary dataflow used in some accelerator designs (Chen et al., 2017). However, their formulation is explored only for convolution, and they restrict clustering to groups of weights (filter rows) rather than individual weights due to computational limitations as recalibrating the k-means regularisation term is expensive during training. Similarly (Stock et al., 2020; Fan et al., 2021), focus on quantising groups of weights into single codewords rather than the individual weights themselves. Weight-sharing approaches similar to ours include (Ullrich et al., 2017). The authors use the distance from an evolving Gaussian mixture as a regularisation term to prepare the clustering weights. Although it is successful with small dataset-model combinations, the complex optimisation — particularly the additional requirement for Inverse-Gamma priors to lower-bound the mixture variance to prevent mode collapse— limits the method’s practical applicability due to the high computational costs of training. In our formulation, the weights already fixed no longer contribute to the regularisation prior, reducing the computational overhead.

heading: 3 METHOD
text: Quantisation. Consider a network N parameterised by N weights W = {w1, ..., wN}. Quantising a network is the process of reformulating N ′ ← N where the new network N ′ contains weights which all take values from a reduced pool of k cluster centres C = {c1, ..., ck} where k N . After quantisation, each of the connection weights in the original network is replaced by one of the cluster centres wi ← cj , W ′ = {w′i|w′i ∈ C, i = 1, · · · , N}, |W ′| = k, where W ′ is the set of weights of the new network N ′, which has the same topology as the original N .
Method Outline. WFN is comprised of T fixing iterations where each iteration t ∈ T has a training and a clustering stage. The clustering stage is tasked with partitioning the weights into two subsets W = W tfixed ∪ W tfree. W tfixed is the set of weights fixed at one of the cluster centre values ck ∈ C. The weights wi ∈ W tf ixed are not updated by gradient decent in this, nor any subsequent training stages, these weights are fixed. In contrast, the free-weights denoted by W tfree remain trainable during the next training stage. With each subsequent iteration t we increase the proportion of weights that take on fixed cluster centre values pt such that p0 < p1 . . . < pT , where pt = |W tfixed| |W | . By iteration T , all weights will be fixed to one of the cluster centres making pT = 1. The training stage combines gradient descent based error correction along with a tight-cluster regularisation term and is tasked with maintaining lossless performance as we fix more of the weights to cluster centres.
Clustering Stage. In the clustering stage, we work backwards from our goal of minimising the relative distance travelled for each of the weights to determine which values cluster centres ci ∈ C should take. Given a weight wi ∈ W and cluster centre cj ∈ C we use a distance function D(wi, cj) = |wi − cj | and the original weight value wi to define a relative distance measure Drel(wi, cj) =
D(wi,cj) |wi| . Normalising by wi ensures that distance is measured proportionally to
the original weight value, rather than in absolute terms. To use this in determining the cluster centres, we enforce a threshold δ on this relative distance, Drel(wi, cj) ≤ δ for small δ. We can then define the cluster centres cj ∈ C which make this possible using a simple recurrence relation. Assume we have a starting cluster centre value cj , we want the neighbouring cluster value cj+1 to be such that if a network weight wi is between these clusters wi ∈ [cj , cj+1+cj2 ] then Drel(wi, cj) ≤ δ. Plugging in cj+1+cj2 and cj into Drel and setting it equal to δ we have:
| cj+1+cj 2 − cj | cj+1+cj
2
= δ, (1)
this leads to a recurrence relation:
cj+1 = cj( 1 + δ
1− δ ), 0 < δ < 1, (2)
that provides the next cluster centre value given the previous one. With this, we can generate all the cluster centres given some boundary conditions. Starting with some value c0, we generate c1 which is then used to generate c2 and so on. We introduce the lower-bound cluster threshold c0 = δ0, which we use to generate the cluster centres and any absolute weight value that is less than δ0 is assigned the cluster value 0. This lower bound serves two purposes: firstly, it reduces the number of proposal cluster centres which would otherwise increase exponentially in density around zero, and additionally, the zero-valued weights will allow sparsity-leveraging hardware to avoid operations that use these weights, reducing the computational overhead. As an upper-bound, we stop the recurrence once a cluster centre is larger than the maximum weight in the network, maxj |cj | ≤ maxi |wi|, wi ∈W, cj ∈ C.
Generating the Proposed Cluster centres. Putting this together, we have a starting point c0 = δ0, a recurrence relation to produce cluster centres given c0 that maintains a relative distance change when weights are moved to their nearest cluster centre, and a centre generation stopping condition cj ≤ maxi∈W |wi|, cj ∈ C. We use the δ0 value as our first proposed cluster centre c0 with the recurrence relation generating a proposed cluster set of size s. Since all these values will contain only positive values, we join this set with its negated version along with a zero value to create a proposal cluster set CS = {a( 1+δ1−δ ) jδ0 | j = 0, 1 · · · s; a = +1, 0,−1} of size 2s+ 1.
To account for the zero threshold δ0 and for ease of notation as we advance, we make a slight amendment to the definition of the relative distance function Drel(wi, cj) :
D+rel(wi, cj) =
{ D(wi,cj)
|wi| , if |wi| ≥ δ0
0, otherwise (3)
Reducing k with Additive Powers-of-two Approximations. Although using all of the values in CS as centres to cluster the network weights would meet the requirement for the relative movement of weights to their closest cluster to be less than δ, it would also require a large number of k = |CS | clusters. In addition, the values in CS are also of full 16-bit precision, and we would prefer many of the weights to be powers-of-two for ease of multiplication in hardware. With the motivation of reducing k and encouraging powers-of-two clusters whilst maintaining the relative distance movement where possible, we look to a many-to-one mapping of the values of CS to further cluster the cluster centres. Building on the work of others (Zhou et al., 2017; Yuhang Li, Xin Dong, 2020), we map each of the values ci ∈ CS to their nearest power-of-two, round(ci) = sgn(ci)2
blog2(ci)e and, for flexibility, we further allow for additive powers-of-two rounding. With additive powers-of-two rounding, each cluster value can also come from the sum of powers-of-two values (b-bit) up to order ω where the order represents the number of powers-of-two that can contribute to the approximation2. As an example, given ck = 0.45, ck ∈ CS , an ω = 1 approximation would round ck to its nearest power-of-two c1k = 1 21 = 0.5 whereas an order ω = 2 approximation is c2k = 1 21 − 1 24 = 0.4375. A higher-order grants a better approximation, but at the cost of more cluster centroids overall and requires ω bit-shifts + additions in hardware. We give full details of the algorithm to map the proposal set CS to the ω-order approximation C̃ω in the Appendix.
Minimalist Clustering. We are now ready to present the clustering procedure for a particular iteration t, which we give the pseudo-code for in Algorithm 1. We start the iteration with ω = 1 and a set of weights not yet fixed W tfree. For the set of cluster centres C̃
ω of order ω, let cω∗ (i) = minc∈C̃ω D + rel(wi, c) be the one closest to weight wi. n ω k = ∑ i I[cωk = cω∗ (i)] counts the number of weights assigned to cluster centre cωk ∈ C̃ω , where the indicator function I[x] is 1 if x is true and 0 otherwise. Let k∗ = argmaxk nωk so that c ω k∗ is the modal cluster. For the cluster k∗ let permutation π of {1, . . . , N} that maps wi 7→ w′π(i), be such that the sequence (w′1(k ∗), w′2(k ∗), . . . , w′N (k
∗)) is arranged in ascending order of relative distance from the cluster cωk∗ . In other words, D + rel(w ′ i(k ∗), cωk∗) ≤ D + rel(w ′ i+1(k
∗), cωk∗), for i = 1, . . . , (N − 1). We choose n to be the largest integer such that:
n∑ i=1 D+rel(w ′ i(k ∗), cωk∗) ≤ nδ, and n+1∑ i=1 D+rel(w ′ i(k ∗), cωk∗) > (n+ 1)δ, (4)
2The notion of an order has been identified recently in work parallel to ours (Oh et al., 2021), where the term word is used instead, i.e. two-word directly translates into an ω = 2.
and define {w′1, w′2, . . . , w′n} to be the set of weights to be fixed at this stage of the iteration. These are the weights that can be moved to the cluster centre cωk∗ without exceeding the average relative distance δ of the weights from the centre. The corresponding weight indices from the original network N are in {π−1(1), . . . , π−1(n)}, and called fixednewin the algorithm. If there are no such weights that can be found, i.e., for some cluster centre l∗, the minimum relative distance D+rel(w ′ 1(l ∗), cl∗) > δ, the corresponding set fixednew is empty. In this case, there are no weights that can move to this cluster centre without breaking the δ constraint and we increase order ω ← ω + 1 to compute a new cωk∗ , repeating the process until |fixednew| > 0. Once the condition |fixednew| > 0 is satisfied, we fix the identified weights {w′1, w′2, . . . , w′n} to their corresponding cluster centre value cωk∗ and move them into W t+1 fixed. We continue the process of identifying cluster centres and fixing weights to these centres until |W t+1fix | ≥ Npt, at which point the iteration t is complete and the training stage of iteration t + 1 begins. Our experiments found that a larger δ has less impact on task performance during early t iterations and so we reduce δ linearly with each increasing t ∈ T (full details in the Appendix). We will show later that, with a small δ, over 75% of the weights can be fixed with ω = 1 and over 95% of weights with ω ≤ 2. Training Stage. Despite the steps taken to minimise the impact of the clustering stage, without retraining, performance would suffer. To negate this, we perform gradient descent to adjust the remaining free weightsW tfree. This allows the weights to correct for any loss increase incurred after clustering where training aims to select values W tfree that minimise the usual loss (cross-entropy in our case) whilst Wfixed remain unchanged.
Cosying up to Clusters. Having the remaining W tfree weights closer to the cluster centroids C post-training makes clustering less damaging to performance. We coerce this situation by adding a regularisation term — given in Equation 5 — to the retraining where p(cj |wi) = e −D+reg(wi,cj)∑k l e −D+reg(wi,cl) . The idea is to penalise the free-weights W tfree proportionally to their distance away from the clusters to which they are most likely to be assigned — i.e. the closest. Clusters that are unlikely to be weight wi’s nearest — and therefore final fixed value — do not contribute much to the penalisation term. We balance this regularisation term against the cross-entropy training loss with the γ hyper-parameter, which we fix to be a α proportion of the other loss term γ = αLcross entropyLreg . We note that the γ term is detached from the computational graph and treated as a constant (otherwise the Lreg term would cancel).
Lreg = γ N∑
i∈Wfree
k∑ j D+reg(wi, cj)p(cj |wi) (5)

heading: 4 EXPERIMENT DETAILS
text: We apply WFN to fully converged models trained on the CIFAR-10 and ImageNet datasets. Our pre-trained models are all publicly available with strong baseline accuracies3: Resnet-(18,34,50) (Wu et al., 2017), MobileNetV2 (Sandler et al., 2018) and, GoogLeNet (Chollet, 2017). We run ten weight-fixing iterations for three epochs, increasing the percentage of weights fixed until all weights are fixed to a cluster. In total, we train for 30 epochs per experiment using the Adam optimiser (Kingma & Ba, 2015) with a learning rate 2× 10−5. We use grid-search to explore hyper-parameter combinations using ResNet-18 and MobileNetV2 models with the CIFAR-10 dataset and find that the regularisation weighting α = 0.4 works well across all experiments reducing the need for further hyper-parameter tuning as we advance. The distance threshold δ gives the practitioner control over the compression-performance trade-off (see Figure 3), and so we report a range of values. We give full details of the experiments along with the results of a hyper-parameter ablation study using CIFAR-10 in the Appendix.
3CIFAR-10 models : https://github.com/kuangliu/pytorch-cifar, ImageNet models: torchvision
5 RESULTS
* Estimated from the LSQ paper model size comparison graph, we over-estimate to be as fair as possible. + Open-source implementations have so far been unable to replicated the reported results: https://github.com/hustzxd/LSQuantization.
We begin by comparing WFN for a range of δ values against a diverse set of quantisation approaches that have comparable compression ratios (CR) in Table 5. The 3-bit quantisation methods we compare include: LSQ (Esser et al., 2020), LQ-Net (Zhang et al., 2018b) and APoT (Yuhang Li, Xin Dong, 2020). We additionally compare with the clustering-quantisation methods using the GoogLeNet model: Deep-k-Means (Wu et al., 2018) whose method is similar to ours, KQ (Yu et al., 2020), and GreBdec (Yu et al., 2017). Whilst the results demonstrate WFN’s lossless performance with SOTA CR, this is not the main motivation for the method. Instead, we are interested in how WFN can reduce the number of unique parameters in a network and corresponding weight-space entropy which together give the potential benefit of future accelerator designs to reduce data movement costs. We note to the interested reader there are additional theoretical reasons for pursuing low weight-space entropy networks (Hinton & van Camp, 1993; Grünwald, 2000; Hansen & Yu, 2001), but our primary focus is from the perspective of reducing accelerator data movement costs. Capturing the benefit of algorithmic developments to reduce data movement costs in accelerator designs is non-trivial due to the significant dependency on architecture memory hierarchy, communication orchestration costs and the dataflow mapping used (Sze et al., 2020; Kwon et al., 2019; Sakr et al., 2017). Some active works attempt to provide robust estimations in bridging the gap between estimation and actual energy costs. This estimates energy efficiencies for various accelerator mappings (Kwon et al., 2019), or fixes the mapping and estimates the energy costs on a particular accelerator (Yang et al., 2017). However, this support is limited to specific fixed architecture designs and does not account for the referenced weight-sharing that we introduce. Instead, we rely on analytical model metrics such as the one proposed by (Wu et al., 2018), itself a subset of measurements con-
ducted by (Sakr et al., 2017), in terms of the number Nw of operations that each of the |W ′| unique weights of bit-width Bw is involved in. The low-entropy encoding that we aspire to is motivated by the accelerator designs (Moons & Verhelst, 2016; Han et al., 2016) that seeks to exploit the Huffman encoding possible for the network index set. Hence, the index for each weight wi ∈W ′ can be represented with a bit-width of Bwi and we can account for the number Nwi of times they are used in an inference computation. Thus, instead of the representational cost Nw|W ′|Bw of (Sakr et al., 2017) for the final network N ′, we use as metric:
Rep(N ′) = ∑
wi∈W ′ NwiBwi (6)
The authors of the APoT have released the quantised model weights and code, and we use the released model-saves4 of this SOTA model to compare the entropy, Rep(N ′), unique parameter count, model size and accuracy in Table 2. Our work outperforms APoT in weight-space entropy, unique parameter count and weight representational cost by a large margin. Taking the ResNet-18 experiments as an example, the reduction to just 164 weights compared with APoT’s 9237 demonstrates the effectiveness of WFN. This is in part due to our full-network quantisation (APoT, as aforementioned, does not quantise the first, last and batch-norm parameters), but even when we discount these advantages and look at weight subsets ignoring the first, last and batch-norm layers WFN uses many times fewer parameters and half the weight-space entropy. We provide a further breakdown of these results in the Appendix (Table 4). Finally, let us examine how WFN achieves the substantial reduction in weight-space entropy. In Figure 4 we see that not only do the resultant WFN networks have very few unique weights, but the weight distribution is such that the vast majority of all of the weights are a small handful of powers-of-two values (order 1) and the other unique weights (outside of the top 4) are of low frequency and added only in the final fixing iterations.

heading: 6 CONCLUSION
text: We have presented WFN, a compression pipeline that can successfully compress whole neural networks. A single network codebook, focusing on reducing entropy of the weight-space coupled with the number of unique weights in the network. The WFN process produces highly compressible and potentially hardware-friendly representations of networks using just a few unique weights without performance degradation. Although the results demonstrate strong compression ability, few unique parameters, and low weight-space entropy, the true potential of the method is that it will gives accelerator designers more scope for weight re-use, keeping most/all weights close to computation, reducing the energy-hungry data movement costs.
4https://github.com/yhhhli/APoT Quantization

heading: A APPENDIX

heading: A.1 THE WFN OVERVIEW ALGORITHM
text: Algorithm 1: Clustering Npt weights at the tth iteration. 1 while |W t+1fix | ≤ Npt do 2 ω ← 0 3 fixednew ← [ ] 4 while fixednew is empty do 5 Increase the order ω ← ω + 1 6 List the cluster centres in C̃ω with smallest distances to each wi ∈W t+1free 7 Set the cluster centre with the most weights assigned as cω∗ (i) 8 Sort the weights wi ∈W t+1free by their distance to cω∗ (i) 9 do
10 Append to fixednew the wi ∈W t+1free with next smallest distance 11 while The mean D+rel(w ∈ fixednew, cω∗ (i)) ≤ δ 12 Fix the all the weights in fixednew to cluster centre cω∗ (i)

heading: A.2 DETAILS OF THE POWERS-OF-TWO APPROXIMATION ALGORITHM
text: We map our proposal set CS to a ω-order approximation where each of the clusters ck ∈ CS are written as ω powers-of-two (Eq 7). We do so using Algorithm 2. Figure 5 demonstrates how the values of CS are rounded given different orders.
ck = ω∑ j=1 rj , rj ∈ {− 1 2b , . . . ,− 1 2j+1 ,− 1 2j , 0, 1 2j , 1 2j+1 , . . . 1 2b } (7)
Algorithm 2: Determining possible clusters 1 Input: The full precision proposal set: CS , allowable relative distance: δ, pow2 rounding
function: round(x) = sgn(x)2blog2(x)e
2 Output: An order ω precision cluster set: C̃ω
3 C̃ω ← [ ] 4 for ck ∈ CS do 5 c′k = round(ck) 6 for i = 0→ ω do 7 δck ← ck − c′k 8 if |δck | ≥ δck then 9 c′k ← c′k + round(δck)
10 end 11 end 12 C̃ω ← C̃ω ∪ {c′k} 13 end
A.3 REDUCING δ
To account for the observation that smaller relative distances are well tolerated in the early stages but less so as the final weights are quantised (high t), δ linearly. For any given t ∈ T we use δt = δ(T − t+ 1) as our threshold.

heading: A.4 EXPERIMENT DETAILS
text: We give a full breakdown of the parameters used across all experiments ran in Table 3.

heading: A.5 HYPER-PARAMETER EXPLORATION
text: We conducted a hyper-parameter search on both the ResNet-18 and MobileNet (Figures 6, 7). We find similar results across models when selecting hyper-parameters, and additionally, we observe that as we increase the regularisation term weighting, a weight-space entropy reduction. Further, we found that negating this term altogether α = 0 has the surprising effect of increasing the resultant entropy values and causes an accuracy performance drop, demonstrating the utility of the regularisation term.

heading: A.6 LAYERWISE BREAKDOWN
text: In Figure 8 we examine how the parameter count and layer-parameter entropy change with each layer for both the WFN and APoT approaches. We find both gains over the unquantised layers of APoT, but also that the entropy and parameter count in the convolutional layers (those quantised by APoT) are similar.

heading: A.7 A FULL METRIC COMPARISON
text: In Table 4 we give the full metric breakdown comparing WFN to the state-of-the-art APoT work. We calculate the unique parameter count and entropy values for subsets of the weights. No BN corresponds to all weights other than those in the batch-norm layers, and No BN-FL is the set of weights not including the first-last and batch-norm layers. It’s clear here that WFN outperforms APoT even when we discount the advantage gained of taking on the challenge of quantising all layers.

heading: A.8 PRUNING EXPERIMENTS
text: To explore how WFN interacts with pruning we conduct a simple set of experiments. Instead of starting the WFN process with all weights un-fixed we randomly select p% of the weights to be pruned in each layer. We then run WFN as before starting with pt = p, reducing the number of T iterations. The results, shown in Figure 9, are conducted with a ResNet-18 and CIFAR-10
combination, painting a mixed picture. On the one hand, WFN and pruning at lower levels (< 50%) is well tolerated and provide two benefits, a lower weight-space entropy and fewer weight-fixing iterations. On the other hand, full-precision networks can tolerate much higher ranges of pruning so there it would seem that a certain amount of synergy between the two approaches is present but this is tempered compared to full precision networks.
It’s important to note that WFN already has a form of pruning built-in with the δ0 value balancing the emphasis on pruning over quantisation.

Review summaries:
A technique is developed to reduce the number of unique weights used in a network. The ultimate goal is to exploit this to reduce data movement in hardware. The unique weights are discovered multiple stages of training and clustering. An additional goal is to produce compressible weights. Results demonstrate that good compression ratios are possible, comparable to SoA quantization techniques, but with far fewer numbers of unique parameters.   

This work presents WFN, an ensemble of techniques to reduce data movement costs by reducing the number of unique weights in a network.  Overall the authors find some distinct advantages in their approach: They emphasise a low entropy encoding with a regularisation term, achieving entropies smaller than even those seen using 3-bit quantisation approaches. They require no additional layerwise scaling; the unique weights are shared across all layers. They substantially reduces the number of unique parameters in a network when compared to existing SOTA quantisation approaches. 

The paper proposes a method called Weight Fixing Networks (WFN), which is designed to minimize the data movement in deep learning inference, thus minimizing the energy cost. WFN aims to maintain very few unique weights, low-entropy weight encodings, unique weight values which are amenable to energy-saving versions of hardware multiplication, while keeping the same performance. A new clustering algorithm for weight generation is shown and analyzed, and experimental evaluation shows the merits of the new scheme.

This paper focuses on reducing the data movement costs by reducing the number of unique weights in a network, i.e., resuing weights. Based on this consideration, a new compression pipeline that can successfully compress whole neural networks has been proposed, called WFN. WFN is based on traditional quantisation manner but aims to cluster the unique weights and reduce the weight-space entropy of the whole network. By doing so, the empirical studies show that the proposed WFN is efficient in terms of reducing parameter counts.