---
abstract: 'Network compression for deep neural networks has become an important part of deep learning research, because of increased demand for deep learning models in practical resource-constrained environments.  In this paper, we observe that the weights in adjacent convolution layers share strong similarity in shapes and values, i.e., the weights tend to vary smoothly along the layers.  We call this phenomenon *Smoothly Varying Weight Hypothesis* (SVWH).  Based on SVWH and an inter-frame prediction method in conventional video coding schemes, we propose a new *Inter-Layer Weight Prediction* (ILWP) and quantization method which quantize the predicted residuals of the weights.  Since the predicted weight residuals tend to follow Laplacian distributions with very low variance, the weight quantization can more effectively be applied, thus producing more zero weights and enhancing weight compression ratio.  In addition, we propose a new loss for eliminating non-texture bits, which enabled us to more effectively store only texture bits.  That is, the proposed loss regularizes the weights such that the collocated weights between the adjacent two layers have the same values.  Our comprehensive experiments show that the proposed method achieved much higher weight compression rate at the same accuracy level compared with the previous quantization-based compression methods in deep neural networks.' 
author:
- |
    Kang-Ho Lee, JoonHyun Jeong, and Sung-Ho Bae[^1]\
    Department of Computer Science and Engineering\
    Kyung Hee University\
    `{ho7719,doublejtoh,shbae}@khu.ac.kr`
title: 'An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis'
---

Introduction
============

Deep neural networks have demonstrated great performance for various tasks in many fields, such as image classification (@lecun1990handwritten; @krizhevsky2012imagenet; @he2016deep), object detection (@ren2015faster; @he2017mask; @redmon2018yolov3), image captioning [@jia2015guiding], and speech recognition (@hinton2012deep; @xiong2018microsoft).  Heavy and deep neural networks achieved great accuracy with the aid of the enormous number of weight parameters and high computational cost (@simonyan2014very; @he2016deep; @huang2017densely).  Recently, however, as demands toward constructing the neural networks in the resource-constrained environments have been increasing, making the resource-efficient neural network while maintaining its accuracy performance becomes an important research area of deep neural networks. 

Many studies have aimed to solve this problem.  In @lecun1990optimal, @hassibi1993second, @han2015learning and @li2016pruning, network pruning methods were proposed for memory-efficient architecture, where unimportant weights were forced to have zero values in terms of accuracy performance.  In @fiesler1990weight, @gong2014compressing and @han2015deep, weights were quantized and stored in memory, enabling less memory usage of deep neural networks.  On the other hand, some literature decomposed convolution operations into sub operations (e.g., depth-wise separable convolution) requiring less computation costs at similar accuracy levels (@howard2017mobilenets; @zhang2018shufflenet; @sandler2018mobilenetv2; @ma2018shufflenet). 

In this paper, we found out that the weights between the adjacent two convolution layers tend to share high similarity in shapes and values.  We call this phenomenon *Smoothly Varying Weight Hypothesis* (SVWH).  This paper explores an efficient neural network method that fully takes advantage of SVWH. 

Specifically, inspired by the prediction techniques widely used in video compression field (@wiegand2003overview; @sullivan2012overview), we propose a new weight compression scheme based on an inter-layer weight prediction technique used in the conventional video compression frameworks, which can be successfully incorporated into the depth-wise separable convolutional blocks (@howard2017mobilenets; @zhang2018shufflenet; @sandler2018mobilenetv2; @ma2018shufflenet). 

**Contributions:** This paper delivers the following major contributions:

-   From comprehensive experiments, we found out that the weights between the adjacent layers tend to share strong similarities, which lead us to establishing SVWH. 

-   based on SVWH, we propose a simple and effective *Inter-Layer Weight Prediction* and quantization framework enabling more compressed neural networks than only applying quantization on the weights of the neural networks. 

-   To further enhance the effectiveness of the proposed *Inter-Layer Weight Prediction*, we devise a new regularization function that aims to minimize the difference between collocated weight values in the adjacent layers, resulting in significant bit saving for non-texture bits (i.e., bits for indices of prediction). 

-   Our comprehensive experiments showed that, the proposed scheme achieved about 53% compression ratio on average in 8-bit quantization at the same accuracy level compared to the traditional quantization method (without prediction) in both MobileNetV1 [@howard2017mobilenets] and MobileNetV2 [@sandler2018mobilenetv2] neural networks. 

Related Work {#related_work}
============

**Network pruning:** Network pruning methods prune the unimportant weight parameters, enabling to reduce the redundancy of weight parameters inherent in neural networks.  @lecun1990optimal and @hassibi1993second reduced the number of weight connections implicitly through setting an proper objective function for training.  @han2015learning successfully removed the unimportant weight connections through certain thresholds for the weight values, showing no harm of accuracy in the state-of-the-art convolutional neural network models.  Recently, structured (filter/channel/layer-wise) pruning methods have been proposed in @liu2017learning and @li2016pruning, where a set weights is pruned based on certain criteria (e.g., the sum of absolute values in the set of weights), demonstrating significantly reduced number of weight parameters and computational costs.  This paper is linked to the pruning methods in perspective of inducing more zero weights for weight compression. 

**Quantization:** Quantization reduces the representation bits of original weights in neural networks.  @fiesler1990weight proposed a weight quantization using weight discretization in neural networks.  @han2015learning incorporated a vector quantization into the pruning, proving that quantization and pruning can jointly work for weight compression without accuracy degradation.  This pruning-quantization framework, i.e., called Deep Compression, became a milestone in model compression research of deep neural networks.  @lin2016fixed proposed a fixed-point quantization using a linear scale factor for weight values where bit-widths for quantization are adaptively found for each layer, thus enabling 20% reduction of the weight size in memory without any loss in accuracy compared to the baseline fixed-point quantization method. 

For 1-bit quantization which is named as Binary Neural Networks, several studies (@BinaryConnect; @BNN; @QNN) binarized the weights and/or activations in the process of back-propagation, enjoying considerably reduced usage of memory space and computation overhead.  Compared to the aforementioned quantization techniques, our work applies quantization in the combination of the residuals for the weights in inter layers rather than the weight values themselves.  Empirically, we found that the residuals tend to produce much larger portion of zero values when quantized, since they often follow very narrow Laplacian distributions.  Therefore, our proposed method can more significantly reduce usage in memory space of neural networks as shown in Section \[experiments\]. 

**Prediction in conventional video coding:** Prediction technique is considered one of the most crucial parts in video compression, aiming at minimizing the magnitudes of signals to be encoded by subtracting the input signals to the most similar encoded signals in a set of prediction candidates (@wiegand2003overview; @rijkse1996h; @sullivan2012overview).  The prediction methods can produce the residuals of signals with low magnitudes as well as a large number of non/near zero signals.  Therefore, they have effectively been incorporated into transforms and quantization for concentrating powers in low frequency regions and reducing the entropy, respectively.  There are two prediction techniques: inter- and intra- predictions.  The inter-prediction searches the best prediction signals from the encoded neighbor frames out of the current frame, while the intra-prediction generates a set of prediction signals from the input signals and determine the best prediction (@wiegand2003overview; @sullivan2012overview).  This is because intra frames that use only intra-prediction for compression are used as reference frames for subsequent frames to be predicted. 

Note that a few studies explored to apply the transform techniques of video and/or image coding to the weight compression problem in neural networks.  @wang2016cnnpack and @ko2017adaptive applied DCT (Discrete Cosine Transform) used in the JPEG (Joint Picture Encoding Group) algorithm to the model compression problem of deep neural networks such that the energy of weight values became concentrated in low frequency regions, thus producing more non/near-zero DCT coefficients for the weights.  Compared to the aforementioned papers, our work does not adopt transform techniques to reduce model sizes, because the transforms introduce much computation in inference, decreasing the effectiveness of the weight compression in practical applications. 

In this paper, we found out that the inter-prediction technique can play a crucial role for weight compression under the SVWH condition (i.e., the weights between adjacent layers tend to be similar).  The proposed regularization function, denoted as *inter-layer loss*, reinforces SVWH for reducing the non-texture bits in the training process.  As a result, the proposed inter-prediction and quantization framework for weight compression yields impressive compression performance enhancement at the similar accuracy level compared to the baseline models. 

Method
======

Basic Inter-Layer Weight Prediction {#basic_interlayer_weight_prediction}
-----------------------------------

![ Our Inter-Layer Weight Prediction scheme consists of (1) finding the most similar kernel $K_{u,v}$ at all previous layer ($u<i$), using $L_{1}$ distance; (2) calculating the residuals between two kernels; (3) saving the residuals with the index of the reference kernel, i.e., the best prediction, into memory space.[]{data-label="fig:inter-layer_weight_prediction"}](images/overview.png)

Figure \[fig:inter-layer\_weight\_prediction\] shows an simple example of the proposed *Inter-Layer Weight Prediction* (ILWP) method with a full search strategy.  The full search strategy indicates that, for the $j$-th weight in the kernel of the $i$-th convolution layer ($K_{i,j}$), it searches the most similar weight kernel (i.e., $K_{u,v}$ in Figure \[fig:inter-layer\_weight\_prediction\]) in the range of \[1, $i-1$\]-th layers given a pre-trained neural network model.  We then compute the residuals ($R_{i,j}$) between the current weight and the best prediction, which is finally quantized in a certain bit representation and stored in memory space with the index of the best prediction (i.e., $(u,v)$ in Figure \[fig:inter-layer\_weight\_prediction\]). 

The proposed ILWP is performed from the second layer to the last layer in a sequential manner.  It is noted that, because of the large portion of non-texture bits (indices of the best prediction), ILWP with the full search strategy tends to produce more bits (both texture (residual) and non-texture bits) than the standard weight values without ILWP, which makes ILWP worthless in compressing the weights.  In the next subsection, we describe how SVWH solves this problem effectively. 

Smoothly Varying Weight Hypothesis {#smoothly_varying_weight_hypothesis}
----------------------------------

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in the pre-trained MobileNet and MobileNetV2. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_baseline"}](images/heatmap_MobileNet-CIFAR10.png "fig:")

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in the pre-trained MobileNet and MobileNetV2. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_baseline"}](images/heatmap_MobileNet-CIFAR100.png "fig:")

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in the pre-trained MobileNet and MobileNetV2. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_baseline"}](images/heatmap_MobileNetV2-CIFAR10.png "fig:")

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in the pre-trained MobileNet and MobileNetV2. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_baseline"}](images/heatmap_MobileNetV2-CIFAR100.png "fig:")

Figure \[fig:heatmap\_baseline\] shows the heatmaps for the percentage in the number of the best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in the pre-trained neural networks i.e., MobileNet, MobileNetV2 in the first and second columns, respectively.  As shown in Figure \[fig:heatmap\_baseline\]-(a), -(b), -(c) and -(d), the dominant portions of the best predictions in the current ($i$-th layer) layer are obtained from its previous ($i-1$-th layer) layers, being consistently observed in all the layers of the neural networks. 

From these observations, we claim that the adjacent layers tend to consists of the similar weight values, leading us to proposing *Smoothly Varying Weight Hypothesis* (SVWH).  The proposed SVWH can be mathematically expressed as

$$\mathrm{P}[L(K_{i,j},K_{i-1,v}) >  L(K_{i,j},K_{u,v})] \: > \: \mathrm{P}[L(K_{i,j},K_{i-1,v}) \leq L(K_{i,j},K_{u,v})],
\label{eq:svwh}$$

where $\mathrm{P}[\cdot]$ is a probability, $L(\cdot,\cdot)$ is a distance between two kernels, and $(u, v)$ are the indices of the layer and kernel, where $1 \leq u < i-1$ (See Figure \[fig:inter-layer\_weight\_prediction\]). 

Inspired by SVWH, we propose an enhanced version of ILWP i.e., ILWP with a local search strategy that finds the best prediction only from the the previous layer.  This local search strategy can effectively reduce the non-texture bits, because the non-texture bits for the layer index are not required.  Moreover, this local search strategy allows the network to keep the weights only in the previous one layer in inference, thus reducing the memory buffer for keeping the weights.  Our experimental results in Section \[experiments\] showed that, the local search strategy enhances the compression performance compared to the full search strategy in the ILWP framework of deep neural networks.  In this paper, to further increase the effectiveness of the proposed ILWP, we devise a new regularization loss, namely inter-layer loss, by which non-texture bits are not needed to be stored anymore. 

Inter-layer Loss {#inter-layer_loss}
----------------

To further exploit SVWH, We propose a new inter-layer loss which makes the collocated filter weights between the adjacent layers have almost the same values.  We found out that our ILWP can more effectively be applied to the depth-wise convolution ($3\times3$ spatial convolution) layer in the depth-wise separable convolution block [@howard2017mobilenets], compared to the traditional 3D convolutions ($3\times 3 \times C$ convolution).  This is because high dimensionality of the weights in the traditional 3D convolution filter tends to hinder finding out the best prediction sharing strong similarity.  This can introduce a longer tail and wider shape of the Laplacian distribution for the residuals of the weights, thus decreasing the compression efficiency.  Moreover, it is not possible to predict the weight kernels having different channel dimensions from the current weight kernel.  This can limit the usability of the proposed ILWP. 

On the other hand, the depth-wise convolution consists only of nine elements, and all the depth-wise convolutions have canonical kernel shapes with $3\times3$ size in whole networks.  Also, since the point-wise convolution ($1\times 1$ convolution) learns channel correlations for the output features of the depth-wise convolution layer, forcing the collocated weights to be similar does hardly affect the accuracy performance of neural networks.  These characteristics of depth-wise convolution enhance the usability of the proposed ILWP in the weight compression of neural networks. 

Moreover, it becomes more popular to use depth-wise separable convolutions in the recent neural network architectures, due to its high efficiency (@kaiser2017depthwise; [@sandler2018mobilenetv2]; @zhang2018shufflenet; @ma2018shufflenet).  Therefore, we apply the proposed method into the spatial convolution in the depth-wise separable convolution block.  Our proposed inter-layer loss can dramatically eliminate the non-texture bits and is defined as follows:

$$\mathcal{L}_{\text{inter-layer}} = \frac {1} {Z} \sum_{i=2}^{N} \sum_{j=0}^{c_{i-1}} \left\vert K_{i,j} - K_{i-1,v} \right\vert,
\label{eq:inter-layer-loss}$$

where $Z$ is the number of weights to be predicted, and $N$ is the number of depth-wise convolution layers in a whole neural network.  In Eq. ( \[eq:inter-layer-loss\]), $v$ is the index in the previous layer which is equal to $j \bmod c_{i-1}$ in the inter layer loss.  The proposed loss function in Eq. ( \[eq:inter-layer-loss\]) not only regularizes the weights but also allows us to eliminate all the non-texture bits for indices of the best predictions, since the network can always predict the weight values of the current layer from the collocated weights in its previous layer.  For training the neural networks with our proposed loss, our total loss is formulated as follow:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{cls}} + \lambda \mathcal{L}_{\text{inter-layer}},
\label{eq:total-loss}$$

where $\mathcal{L}_{\text{cls}}$ is the conventional classification loss using the cross-entropy loss [@he2016deep].  $\lambda$ in Eq. ( \[eq:total-loss\]) is the control parameter to the proposed regularizer and is set to 1 over all experiments.  From our comprehensive experiments, we found that setting $\lambda$ in Eq. ( \[eq:total-loss\]) to 1 is suitable to match the trade-off between the performance of neural networks and the parameter size of neural networks (See Figure \[fig:lambda\_study\] in Section \[optimal\_lambda\]). 

Through our new *inter-layer loss*, SVWH is explicitly controlled in a more elaborate manner as shown in Figure \[fig:heatmap\_interlayer\_loss\].  Figure \[fig:heatmap\_interlayer\_loss\] shows the Heatmaps for the percentage in the number of best predictions with respect to minimizing $L_1$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in MobileNet and MobileNetV2 trained with the proposed inter-layer loss.  Finally, the reconstruction of the current weight kernel is performed as follow:

$$\hat{K}_{i,j} = \tilde{R}_{i,j} + \hat{K}_{i-1,v},
\label{eq:calc-residual-inter-layer-loss}$$

where $\tilde{R}_{i,j}$ is the quantized residuals at the $i$-th layer and $j$-th filter position.  $\hat{K}$ in Eq. ( 3) is the final reconstruction of $K$. Note that, to reconstruct the current weights, the weights only in the previous layer (i.e., $\hat{K}_{i-1,v}$ in Eq. ( \[eq:calc-residual-inter-layer-loss\])) are required.  The residuals of the weights are quantized through a linear quantization technique, and then saved using Huffman coding for the purpose of efficient model capacity [@han2015learning].  It is noticed that, thanks to the prediction scheme, our method usually remains more high-valued non-zero weights than the traditional quantization methods.  Since high weight values importantly contribute to the prediction performance [@han2015learning], our method can retain both high accuracy and weight compression performance. 

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in MobileNet and MobileNetV2 trained with the proposed inter-layer loss. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_interlayer_loss"}](images/heatmap_new_MobileNet-CIFAR10.png "fig:")

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in MobileNet and MobileNetV2 trained with the proposed inter-layer loss. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_interlayer_loss"}](images/heatmap_new_MobileNet-CIFAR100.png "fig:")

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in MobileNet and MobileNetV2 trained with the proposed inter-layer loss. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_interlayer_loss"}](images/heatmap_new_MobileNetV2-CIFAR10.png "fig:")

[0.48]{} ![ Heatmaps that show the percentage in the number of best predictions with respect to minimizing $L_{1}$ distance between the source layer ($x$-axis) and the target layer ($y$-axis) in MobileNet and MobileNetV2 trained with the proposed inter-layer loss. ( a) MobileNet trained on CIFAR-10; (b) MobileNet trained on CIFAR-100; (c) MobileNetV2 trained on CIFAR-10; (d) MobileNetV2 trained on CIFAR-100.[]{data-label="fig:heatmap_interlayer_loss"}](images/heatmap_new_MobileNetV2-CIFAR100.png "fig:")

Note that the weight kernels in the first layer are not quantized since it is as important as Intra-frames (i.e., reference frames) in a group of pictures of the video coding scheme [@wiegand2003overview].  If the weight kernels in the first layer are quantized, the weight kernels in subsequent layers which are predicted based on the weight kernels in the first layer are negatively affected, leading to accuracy drop in neural networks. 

Experiments
===========

In this section, we evaluate our proposed method on image classification tasks, specifically for CIFAR-10 and CIFAR-100 datasets.  For securing generality of our proposed IWLP, we applied our method in two convolutional neural networks, MobileNet and MobileNetV2.  Environmental settings for the experiments are that we use four Nvidia 2080-Ti Graphic Processing Unit (GPU) with the Intel i9-7900X CPU.  For the hyper-parameter setting in the training process, we set the initial learning rate as 0.1, which is multiplied by 0.94 per 2 epochs.  We used Stochastic Gradient Descent (SGD) optimizer with Nesterov momentum [@sutskever2013importance] factor 0.9.  We set the batch size to 256 and trained all the neural networks for 200 epochs.  In the following experiments, every network is trained for 6 times and the average validation accuracy over 6 times is recorded in all the following figures and table. 

Optimal $\lambda$ selection {#optimal_lambda}
---------------------------

[0.48]{} ![ Comparison of parameter size in kilobytes (KB) and top-1 test accuracy in MobileNet trained on CIFAR-10 dataset for different $\lambda$ in Eq. ( \[eq:total-loss\]).[]{data-label="fig:lambda_study"}](images/lambda-param_MobileNet-CIFAR10.png "fig:")

[0.48]{} ![ Comparison of parameter size in kilobytes (KB) and top-1 test accuracy in MobileNet trained on CIFAR-10 dataset for different $\lambda$ in Eq. ( \[eq:total-loss\]).[]{data-label="fig:lambda_study"}](images/lambda-Acc_MobileNet-CIFAR10.png "fig:")

First, we found the optimal $\lambda$ in Eq. ( \[eq:total-loss\]) that is suitable to match the trade-off between the accuracy performance and the parameter size of neural networks.  In Figure \[fig:lambda\_study\]-(a) shows the parameter sizes in MobileNet trained on the CIFAR-10 dataset for different $\lambda$ in Eq. ( \[eq:total-loss\]) after quantization and Huffman coded.  We can see that the case of $\lambda = 1$ has the smallest parameter size in neural networks, and slightly lower top-1 test accuracy in the CIFAR-10 dataset.  So, we set the $\lambda$ in Eq. ( \[eq:total-loss\]) to 1. 

Experimental results {#experimental_results}
--------------------

Figure \[fig:acc\_param\] shows the parameter size of our proposed methods for different quantization bits in $\{2, 3, 4, ..., 8\}$ on CIFAR-10 and CIFAR-100 datasets.  In Figure \[fig:acc\_param\]: (a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].  It shows our proposed ILWP-ILL yields higher compression ratio in the weight parameters.  On the other hand, ILWP-FSS and ILWP-LSS have larger parameter sizes than the baseline.  This is because both IWLP-FSS and IWLP-LSS contain non-texture bits (bits for indices of the best predictions), resulting in bit overheads. 

For more analysis on the texture and non-teuxtre bits in the proposed ILWP, Table \[table:parameter\_size\] shows the performance comparison of our proposed methods in terms of the parameter sizes for the depth-wise convolution kernels and top-1 test accuracy.  Due to the inter-layer prediction scheme, all the proposed methods show less amount of texture bits compared to the baseline.  However, the total amounts of bits of ILWP-FSS and ILWP-LSS are higher than the baseline model, which is due to the presence of the non-texture bits ($u$ and $v$ in Figure \[fig:inter-layer\_weight\_prediction\]).  However, ILWP-ILL demonstrated much reduced amount of total bits compared to the baseline, which is due to the property that this method does not require saving any indices of reference kernels while maintaining good accuracy under the SVWH condition. 

\[ht\]

[0.49]{} ![ Comparison of Parameter size in kilobytes (KB) for different quantization bits in $\{2, 3, 4, ..., 8\}$ on CIFAR-10 and CIFAR-100 datasets.  Top: MobileNet, Bottom: MobileNetV2; (a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].[]{data-label="fig:acc_param"}](images/Qb-Param_MobileNet-CIFAR10.png "fig:")

[0.49]{} ![ Comparison of Parameter size in kilobytes (KB) for different quantization bits in $\{2, 3, 4, ..., 8\}$ on CIFAR-10 and CIFAR-100 datasets.  Top: MobileNet, Bottom: MobileNetV2; (a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].[]{data-label="fig:acc_param"}](images/Qb-Param_MobileNet-CIFAR100.png "fig:")

[0.49]{} ![ Comparison of Parameter size in kilobytes (KB) for different quantization bits in $\{2, 3, 4, ..., 8\}$ on CIFAR-10 and CIFAR-100 datasets.  Top: MobileNet, Bottom: MobileNetV2; (a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].[]{data-label="fig:acc_param"}](images/Qb-Param_MobileNetV2-CIFAR10.png "fig:")

[0.49]{} ![ Comparison of Parameter size in kilobytes (KB) for different quantization bits in $\{2, 3, 4, ..., 8\}$ on CIFAR-10 and CIFAR-100 datasets.  Top: MobileNet, Bottom: MobileNetV2; (a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].[]{data-label="fig:acc_param"}](images/Qb-Param_MobileNetV2-CIFAR100.png "fig:")

              **texture bits**   **non-texture bits**   **total bits**   **Test accuracy**
  ---------- ------------------ ---------------------- ---------------- -------------------
  Baseline         70.192                 -                 70.192            92.125
  ILWP-FSS         67.716               11.177              78.893            92.225
  ILWP-LSS         67.415               8.540               75.955            92.235
  ILWP-ILL       **33.373**               -               **33.373**           91.48

  : Parameter size (texture bits, non-texture bits, total bits) in kilobytes (KB) and top-1 test accuracy of depth-wise convolution kernels after 8 bit quantization and Huffman coding, in MobileNet trained on CIFAR-10.  We compared the performance of the three variations of ILWP: ILWP-FSS, ILWP-LSS, and ILWP-ILL.[]{data-label="table:parameter_size"}

Figure \[fig:ablation\_study\] shows the results of our proposed methods on CIFAR-10 and CIFAR-100 datasets for the parameter size in kilobytes (KB) and test accuracy.  As shown in Figure \[fig:ablation\_study\], the ILWP-ILL outperforms the other compared ones in terms of the trade-off between the amounts of compressed bits and accuracy.  This is because ILWP-ILL significantly saves the weight bits by eliminating the non-texture bits as well as increasing the effectiveness of the quantization process as the residuals tend to follow much narrower Laplacian distributions than the original weight values. 

[0.49]{} ![ Performance results of our proposed methods on CIFAR-10 and CIFAR-100 datasets for the parameter size in kilobytes (KB) and test accuracy.  Top: MobileNet, Bottom: MobileNetV2.  $x$-axis: the total size of the quantized and Huffman coded depth-wise convolutional kernels in storage; $y$-axis: Top-1 test accuracy. ( a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].  In all the figures, each dot represents the average test accuracy over 6 training times in each quantization bit in $ \{ 2, 3, 4, ..., 8 \} $.[]{data-label="fig:ablation_study"}](images/Param-Acc_MobileNet-CIFAR10.png "fig:")

[0.49]{} ![ Performance results of our proposed methods on CIFAR-10 and CIFAR-100 datasets for the parameter size in kilobytes (KB) and test accuracy.  Top: MobileNet, Bottom: MobileNetV2.  $x$-axis: the total size of the quantized and Huffman coded depth-wise convolutional kernels in storage; $y$-axis: Top-1 test accuracy. ( a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].  In all the figures, each dot represents the average test accuracy over 6 training times in each quantization bit in $ \{ 2, 3, 4, ..., 8 \} $.[]{data-label="fig:ablation_study"}](images/Param-Acc_MobileNet-CIFAR100.png "fig:")

[0.49]{} ![ Performance results of our proposed methods on CIFAR-10 and CIFAR-100 datasets for the parameter size in kilobytes (KB) and test accuracy.  Top: MobileNet, Bottom: MobileNetV2.  $x$-axis: the total size of the quantized and Huffman coded depth-wise convolutional kernels in storage; $y$-axis: Top-1 test accuracy. ( a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].  In all the figures, each dot represents the average test accuracy over 6 training times in each quantization bit in $ \{ 2, 3, 4, ..., 8 \} $.[]{data-label="fig:ablation_study"}](images/Param-Acc_MobileNetV2-CIFAR10.png "fig:")

[0.49]{} ![ Performance results of our proposed methods on CIFAR-10 and CIFAR-100 datasets for the parameter size in kilobytes (KB) and test accuracy.  Top: MobileNet, Bottom: MobileNetV2.  $x$-axis: the total size of the quantized and Huffman coded depth-wise convolutional kernels in storage; $y$-axis: Top-1 test accuracy. ( a) ILWP with the full search strategy (ILWP-FSS); (b) ILWP with the local search strategy (ILWP-LSS); (c) ILWP trained with inter-layer loss and without searching (ILWP-ILL) described in Section \[inter-layer\_loss\].  In all the figures, each dot represents the average test accuracy over 6 training times in each quantization bit in $ \{ 2, 3, 4, ..., 8 \} $.[]{data-label="fig:ablation_study"}](images/Param-Acc_MobileNetV2-CIFAR100.png "fig:")

Compared to ILWP-FSS, ILWP-LSS shows very slight improvement in trade-off between the size of parameters and accuracy as ILWP-LSS does not store the layer index of the best prediction ($u$ in Figure 1).  This is due to the fact that the portion of the bits for the layer indices is much smaller than the portion of the bits for the kernel indices.  Compared to the baseline models, it is worth noting that both ILWP-FSS and ILWP-LSS have often worse compression efficiency compared to the baseline.  This is because they introduce non-texture bits consisting of a large portion in the total bits for weight parameters.  Therefore, it can be concluded that our foundation of SVWH and the proposed inter-layer loss allow the network to make fully use of advantages in the inter-layer prediction scheme of the conventional video coding frameworks. 

Analysis
========

[0.32]{} ![ Comparison of the distributions for the weights and residuals in all the depth-wise convolution kernels in MobileNet trained on CIFAR-100. ( a) Distribution of the weight kernels in baseline model; (b) Distribution of residuals in ILWP-FSS; (c) Distribution of the residuals in ILWP-ILL.[]{data-label="fig:comparison_distribution"}](images/original_weights.png "fig:")

[0.32]{} ![ Comparison of the distributions for the weights and residuals in all the depth-wise convolution kernels in MobileNet trained on CIFAR-100. ( a) Distribution of the weight kernels in baseline model; (b) Distribution of residuals in ILWP-FSS; (c) Distribution of the residuals in ILWP-ILL.[]{data-label="fig:comparison_distribution"}](images/original_residuals.png "fig:")

[0.32]{} ![ Comparison of the distributions for the weights and residuals in all the depth-wise convolution kernels in MobileNet trained on CIFAR-100. ( a) Distribution of the weight kernels in baseline model; (b) Distribution of residuals in ILWP-FSS; (c) Distribution of the residuals in ILWP-ILL.[]{data-label="fig:comparison_distribution"}](images/new_residuals.png "fig:")

Figure \[fig:comparison\_distribution\] compares the distributions for the weights and residuals in all the depth-wise convolution kernels in MobileNet trained on CIFAR-100.  In Figure \[fig:comparison\_distribution\], (a), (b), and (c) visualize the distribution of the weight kernels in baseline model, the distribution of residuals in ILWP-FSS and the distribution of the residuals in ILWP-ILL, respectively.  As shown in Figure \[fig:comparison\_distribution\], ILWP-FSS produces a much narrower and single modal Laplacian distribution compared to the baseline.  This nice property contributes to high compressed capacity for storing the weights in neural networks when these residuals are quantized and Huffman coded.  However, this method requires saving the additional indices, leading to consuming more of bit amounts. 

In Figure \[fig:comparison\_distribution\]-(c), it is observed that for ILWP-ILL, the residuals are located in near-zero positions (about 46% of the weights) , following a very sharp Laplacian distribution.  This indicates that ILWP-ILL with quantization allows neural networks to archive remarkable weight compression performance by generating a large amount of zero coefficients after quantization. 

Conclusion
==========

We propose a new inter-layer weight prediction with inter-layer loss for efficient deep neural networks.  Motivated by our observation that the weights in the adjacent layers tend to vary smoothly, we successfully build a new weight compression framework combining the inter-layer weight prediction scheme, the inter-layer loss, quantization and Huffman coding under SVWH condition.  Intuitively, our prediction scheme significantly decreases the entropy of the weights by making them much narrower Laplacian distributions, thus leading remarkable compression ratio of the weight parameters in neural networks.  Also, the proposed inter-layer loss effectively eliminates the non-texture bits for the best predictions.  To the best of our knowledge, this work is the fist to report the phenomenon of the weight similarities between the neighbor layers and to build a prediction-based weight compression scheme in modern deep neural network architectures. 

[34]{} \[1\][\#1]{} \[1\][`#1`]{} urlstyle \[1\][doi: \#1]{}

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.  Binaryconnect: Training deep neural networks with binary weights during propagations.  In *Advances in neural information processing systems*, pp.  3123–3131, 2015. 

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. * arXiv preprint arXiv:1602.02830*, 2016. 

Emile Fiesler, Amar Choudry, and H John Caulfield.  Weight discretization paradigm for optical neural networks.  In *Optical interconnections and networks*, volume 1281, pp.  164–174.  International Society for Optics and Photonics, 1990. 

Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev.  Compressing deep convolutional networks using vector quantization. * arXiv preprint arXiv:1412.6115*, 2014. 

Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. * arXiv preprint arXiv:1510.00149*, 2015. 

Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  In *Advances in neural information processing systems*, pp.  1135–1143, 2015. 

Babak Hassibi and David G Stork.  Second order derivatives for network pruning: Optimal brain surgeon.  In *Advances in neural information processing systems*, pp.  164–171, 1993. 

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp.   770–778, 2016. 

Kaiming He, Georgia Gkioxari, Piotr Doll[á]{}r, and Ross Girshick.  Mask r-cnn.  In *Proceedings of the IEEE international conference on computer vision*, pp.   2961–2969, 2017. 

Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, et al.  Deep neural networks for acoustic modeling in speech recognition. * IEEE Signal processing magazine*, 29, 2012. 

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications. * arXiv preprint arXiv:1704.04861*, 2017. 

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp.   4700–4708, 2017. 

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Quantized neural networks: Training neural networks with low precision weights and activations. * The Journal of Machine Learning Research*, 180 (1):0 6869–6898, 2017. 

Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars.  Guiding the long-short term memory model for image caption generation.  In *Proceedings of the IEEE International Conference on Computer Vision*, pp.   2407–2415, 2015. 

Lukasz Kaiser, Aidan N Gomez, and Francois Chollet.  Depthwise separable convolutions for neural machine translation. * arXiv preprint arXiv:1706.03059*, 2017. 

Jong Hwan Ko, Duckhwan Kim, Taesik Na, Jaeha Kung, and Saibal Mukhopadhyay.  Adaptive weight compression for memory-efficient neural networks.  In *Proceedings of the Conference on Design, Automation & Test in Europe*, pp.   199–204.  European Design and Automation Association, 2017. 

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In *Advances in neural information processing systems*, pp.  1097–1105, 2012. 

Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel.  Handwritten digit recognition with a back-propagation network.  In *Advances in neural information processing systems*, pp.  396–404, 1990. 

Yann LeCun, John S Denker, and Sara A Solla.  Optimal brain damage.  In *Advances in neural information processing systems*, pp.  598–605, 1990. 

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.  Pruning filters for efficient convnets. * arXiv preprint arXiv:1608.08710*, 2016. 

Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy.  Fixed point quantization of deep convolutional networks.  In *International Conference on Machine Learning*, pp.  2849–2858, 2016. 

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang.  Learning efficient convolutional networks through network slimming.  In *Proceedings of the IEEE International Conference on Computer Vision*, pp.   2736–2744, 2017. 

Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In *Proceedings of the European Conference on Computer Vision (ECCV)*, pp.   116–131, 2018. 

Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement. * arXiv preprint arXiv:1804.02767*, 2018. 

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In *Advances in neural information processing systems*, pp.  91–99, 2015. 

Karel Rijkse.  H. 263: video coding for low-bit-rate communication. * IEEE Communications magazine*, 340 (12):0 42–45, 1996. 

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pp.   4510–4520, 2018. 

Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition. * arXiv preprint arXiv:1409.1556*, 2014. 

Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand.  Overview of the high efficiency video coding (hevc) standard. * IEEE Transactions on circuits and systems for video technology*, 220 (12):0 1649–1668, 2012. 

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.  On the importance of initialization and momentum in deep learning.  In *International conference on machine learning*, pp.  1139–1147, 2013. 

Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu.  Cnnpack: Packing convolutional neural networks in the frequency domain.  In *Advances in neural information processing systems*, pp.  253–261, 2016. 

Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra.  Overview of the h. 264/avc video coding standard. * IEEE Transactions on circuits and systems for video technology*, 130 (7):0 560–576, 2003. 

Wayne Xiong, Lingfeng Wu, Fil Alleva, Jasha Droppo, Xuedong Huang, and Andreas Stolcke.  The microsoft 2017 conversational speech recognition system.  In *2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp.   5934–5938.  IEEE, 2018. 

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pp.   6848–6856, 2018. 

\[appendix\]

[^1]: Corresponding auther
