Year: 2020

Abstract:
Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped. In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures. From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match weight shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy. Moreover, taking EfficientNet-B0 as baseline, our NES yields an improvement of 1.2% but has 10% less MAdd. In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings (<350M MAdd). Code can be found at https://github.com/zhoudaquan/NES.

text: Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped. In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures. From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match weight shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy. Moreover, taking EfficientNet-B0 as baseline, our NES yields an improvement of 1.2% but has 10% less MAdd. In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings (<350M MAdd). Code can be found at https://github.com/zhoudaquan/NES.

heading: 1 INTRODUCTION
text: Despite the remarkable performance achieved in many applications, powerful deep convolutional neural networks (CNNs) typically suffer from high complexity (Han et al., 2015a). The large model size and computation cost hinders their deployment on resource limited devices, such as mobile phones. Very recently, huge efforts have been made to compress powerful CNNs. Existing compression techniques can be generally categorized into four categories: network pruning (Han et al., 2015a; Collins & Kohli, 2014; Han et al., 2015b), low rank factorization (Jaderberg et al., 2014), quantization (Jacob et al., 2018; Hubara et al., 2017; Rastegari et al., 2016), and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016). Network pruning targets on removing unimportant connections or weights to reduce the number of parameters and multiply-adds (MAdd). Low rank factorization decomposes an existing layer into lower-rank and smaller layers to reduce the computation cost. Weights quantization aims to use less number of bits to store the weights and activation maps. Knowledge distillation uses a well trained teacher network to train a lightweight student network. All of those compression methods assume that the model parameters (weight tensors) must have one-to-one correspondence to the architectures. As a result, they suffer from performance drop since changing architectures will inevitably lead to loss of informative parameters.
∗correspondence
In this paper, we consider the network compression problem from a new perspective where the shape of the weight tensors and the architecture are designed independently. The key insight is that the network parameters can be disentangled from the architecture and can be compactly represented by a small-sized parameter set (called epitome), inspired by success of epitome methods in image/video modeling and data sparse coding (Jojic et al., 2003; Cheung et al., 2008; Aharon & Elad, 2008). As shown in Figure 1, unlike conventional convolutional layers that use the architecture tied weight tensors to convolve with the input feature map, our proposed neural epitome search (NES) approach first learns a compact yet expressive epitome along with an adaptive transformation function to expand the epitomes. The transformation function is able to generate a variety of parameters from epitomes via a novel learnable transformation function, which also guarantees the representation capacity of the resulting weight tensors to be large. Our transformation function is differentiable and hence enables the NES approach to search for optimal epitome end-to-end, achieving a good trade-off between required model size and performance. In addition, we propose a novel routing map to record the index mapping used for the transformation between the epitome and the convolution kernel. During inference, this routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.
Benefiting from the learned epitomic network parameters and transformation method, compared to existing compression approaches, NES has less performance drop. To the best of our knowledge, this is the first work that automatically learns compact epitomes of network parameters and the corresponding transformation function for network compression. To sum up, our work offers the following attractive properties:
• Our method is flexible. It allows the weight tensors to be independent of the architecture design. We can easily control the model size by defining the size of the epitomes given a specified network architecture. This is especially beneficial in the context of edge devices. • Our method is effective. The learning-based transformation method empowers the epitomes
with highly expressive capability and hence incurs less performance drop even with large compression ratios. • Our method is easy to use. It can be encapsulated as a drop in replacement to the current
convolutional operator. There is no dependence on specialized platforms/frameworks for NES to conduct compression.
To demonstrate the efficacy of the proposed approach, We conduct extensive experiments on CIFAR10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009). On CIFAR-10 dataset, our method outperforms the baseline model by 1.3%. On ImageNet, our method outperforms MobileNetV2 full model by 1.47% in top-1 accuracy with 25% MAdd reduction, and MobileNetV2-0.35 baseline by 3.78%. Regarding MobileNetV2-0.7 backbone, our method improves AMC (He et al., 2018) by 2.47%. Additionally, when taking EfficientNet-b0 (Tan & Le, 2019) as baseline, we have an improvement of 1.2% top-1 accuracy with 10% MAdd reduction.

heading: 2 RELATED WORK
text: Traditional model compression methods include network pruning (Collins & Kohli, 2014; Han et al., 2015b), low rank factorization (Jaderberg et al., 2014), quantization (Jacob et al., 2018; Hubara et al.,
2017; Rastegari et al., 2016) and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016). For all of those methods, as mentioned in Section 1 extensive expert knowledge and manual efforts are needed and the process might need to be done iteratively and hence is time consuming.
Recently, AutoML based methods have been proposed to reduce the experts efforts for model compression (He et al., 2018; Zoph et al., 2018; Noy et al., 2019; Li et al., 2019) and efficient convolution architecture design (Liu et al., 2018; Wu et al., 2018; Tan et al., 2018). As proposed in AutoML for model compression (AMC (He et al., 2018)), reinforcement learning can be used as an agent to remove redundant layers by adding resource constraints into the rewards function which however is highly time consuming. Later, gradient based search method such as DARTS (Liu et al., 2018) is developed for higher search efficiency over basic building blocks. There are also methods that use AutoML based method to search for efficient architectures directly (Wu et al., 2018; He et al., 2018). All of those methods are searching for optimized network architecture with an implicit assumption that the weights and the model architecture have one-to-one correspondence. Different from all of the above mentioned methods, our method provides a new search space by separating the model weights from the architecture. The model size can thus be controlled precisely by nature.
Our method is also related to the group-theory based network transformation. Based on the group theory proposed in Cohen & Welling (2016), recent methods try to design a compact filter to reduce the convolution layer computation cost such as WSNet (Jin et al., 2017) and CReLU (Shang et al., 2016). WSNet tries to reduces the model parameters and computations by allowing overlapping between adjacent filters of 1D convolution kernel. This can be seen as a simplified version of our method as the overlapping can be regarded as a fixed rule transformation. CReLu tries to learn diversified features by concatenating ReLU output of original and negated inputs. However, as the rule is fixed, the design of those schemes are application specific and time consuming. Besides, the performance typically suffers since the scheme is not optimized during the training. In contrast, our method requires negligible human efforts and the transformation rule is learned end-to-end.

heading: 3 METHOD

heading: 3.1 OVERVIEW
text: A convolutional layer is composed of a set of learnable weights (or kernels) that are used for feature transformation. The observation of this paper is that the learnable weights in CNNs can be disentangled from the architecture. Inspired by this fact, we provide a new perspective on the network compression problem, i.e., finding a compact yet expressive parameter set, called epitome, along with a proper transformation function to fully represent the whole network, as illustrated in Figure 1.
Formally, consider a convolutional network with a fixed architecture consisting of a stack of convolutional layers, each of which is associated with a weight tensor θi (i is layer index). Further, let L(X,Y ; θ) be the loss function used to train the network, where X and Y are the input data and label respectively and θ = {θi} denotes the parameter set in the network. Our goal is to learn epitomes E = {Ei} which have smaller sizes than θ and the transformation function τ = {τi} to represent the network parameter θ with the compact epitome as τ(E). In this way, network compression is achieved. The objective function of neural epitome search (NES) for a given architecture is to learn optimal epitomes E∗ and transformation functions τ∗:
{E∗, τ∗} = argmin E,τ L(X,Y ; τ(E)), s.t. |E| < |θ|, (1)
where | · | calculates the number of all elements. The above NES approach provides a flexible way to achieve network compression since the epitomes can be defined to be of any size. By learning a proper transformation function, the epitomes of predefined sizes can be adaptively transformed to match and instantiate the specified network architecture. In the following sections, we will elaborate on how to learn the transformation functions τ(·), the epitome (E ∈ RWE×HE×CEin×CEout ) and how to compress models via NES in an end-to-end manner. In this paper, we use “a sub-tensor in the epitome” to describe a patch of the epitome that will be selected to construct the convolution weight tensor. The sub-tensor, Es, is represented with the starting index and the length along each dimension in the epitome as shown below:
Es = E[p : p+ w, q : q + h, cin : cin + β1, cout : cout + β2], (2)
where (p, q, cin, cout) denote the starting index of the sub-tensor and (w, h, β1, β2) denotes the length of the sub-tensor along each dimension.

heading: 3.2 DIFFERENTIABLE SEARCH FOR EPITOME TRANSFORMATION
text: As aforementioned, NES generates convolution weight tensors from the epitome E via a transformation function τ . In this section, we explain how the transformation τ is designed and optimized. We start with the formulation of a conventional convolution operation. Then we introduce how the transformed epitome is deployed to conduct the convolution operations, with a reduced number of parameters and calculations.
A conventional 2D convolutional layer transforms an input feature tensor F ∈ RW×H×Cin to an output feature tensor G ∈ RW×H×Cout through convolutional kernels with weight tensor θ ∈ Rw×h×Cin×Cout . Here (W,H,Cin, Cout) denote the width, height, input and output channel numbers of the feature tensor; w and h denote width and height of the convolution kernel. The convolution operation can be formulated as:
Gtw,th,c = w−1∑ i=0 h−1∑ j=0 Cin−1∑ m=0 Ftw+i,th+j,mθi,j,m,c,∀ tw ∈ [0,W ), th ∈ [0, H), c ∈ [0, Cout). (3)
Instead of maintaining the full weight tensor θ, NES maintains a much smaller epitome E that can generate the weight tensor and thus achieves model compression. To make sure the generated weights τ(E) can conduct the above convolution operation without incurring performance drop, we carefully design the transformation function τ with following three novel components: (1) a learnable indexing function η to determine starting indices of the sub-tensor within the epitome E to sample the weight tensor; (2) a routing mapM that records the location mapping from the sampled sub-tensors epitome to the generated weight tensor; (3) an interpolation-based sampler to perform the sampling from E even when the indices are fractional. We now explain their details.
Indexing function The indexing function η is used to localize the sub-tensor within the epitome that is used to generate the weight tensor, as illustrated in Figure 2. Concretely, given the input feature tensor F ∈ RW×H×Cin , the function generates indices as follows,
(p,q, cin, cout) = S(η(F )), (4)
where p,q, cin, cout are vectors of learned starting indices along the spatial, input channel and filter dimensions respectively to sample the sub-tensor within epitome to generate the model weight tensors.
Each vector contains a set of starting indices and the element number inside each vector is equal to the number of transformations that will be applied along each dimension. Note they are all non-negative real numbers. η is the index learner and it outputs the normalized indices (p′,q′, c′in, c ′ out) through a sigmoid function, each ranging from 0 to 1. These outputs are further up-scaled by a scaling function S(·) to the corresponding dimension of the epitome by S(·)
S(p′,q′, c′in, c ′ out) = [W E , HE , CEin, C E out]⊗ [p′,q′, c′in, c′out], (5)
whereWE , HE , CEin, C E out are dimensions of the epitome and⊗ denotes element-wise multiplication. The learned indices are then fed into the following interpolation based sampler to generate the weight tensor. We implement the indexing learner by a two-layer convolution module that can be jointly optimized with the backbone network end-to-end. In particular, we use separate indexing learners and epitomes for each layer of the network. More implementation details are given in Appendix A.3.
Routing map The routing map is constructed to record the position correspondence between the convolution weight tensor and the sub-tensor in the epitome. It takes a position within the weight tensor as input and returns the corresponding starting index of the sub-tensor in the epitome. The mapped starting index of the sub-tensor in the epitome can thus be retrieved from the routing map fast. More importantly, the indexing learner can be removed during inference with the help of the routing map. The routing map is built as a look-up table during the training phase by recording the moving average of the output index from the index learner η. For example, the starting index pair as shown in Figure 2 can be fetched via (pt, qt) =M(i, j,m) where (i, j,m) is the spatial location in the weight tensor and (pt, qt) is the starting index of the selected sub-tensor in the epitome at training epoch t. The routing mapM is constructed via Eqn. (6) as shown below with momentum µ during the training phase. µ is treated as a hyper-parameter and is decided empirically1:
M(i, j,m) = (pt, qt) = (pt−1, qt−1) + µ · η(x). (6) Interpolation based sampler The learned starting index and the pre-defined dimension (w, h, β1, β2) of the sub-tensor in the epitome is then fed into the sampler function. The sampler function samples the sub-tensor within the epitome to generate the weight tensor. To simplify the illustration on the sampler function, we use the transformation along the spatial dimension as an example as shown in Figure 2. The weight tensor is generated via the equation as shown below:
θ(:,:,m) = τ(E|(p, q)) = WE−1∑ nw=0 HE−1∑ nh=0 G(nw, p)G(nh, q)E(nw:nw+w,nh:nh+h), (7)
where G(a, b) = max(0, 1−|a−b|); nw and nh enumerate over all integral spatial locations withinE. Following Eqn. (7), we first find all sub-tensors in the epitome whose starting indices (nw, nh) along
1We set µ to be 0.97 in our experiments.
spatial dimensions satisfy: G(nw, pt)G(nh, qt) > 0. Then a weighted summation (or interpolation) over the involved sub-tensors is computed according to Eqn. (7). In the case of applying the sampling along input channel dimension, (p, q) in the equation is replaced with the learned starting index cin and the weight tensor is generated by iterating along the input channel dimension as shown below:
θ(:,:,m:m+β1) = τ(E|cin) = Rcin−1∑ nc=0 G(nw, cin)E(:,:,cin:cin+β1), (8)
Where Rcin = dCin/β1e is the number of samplings applied along the input channel dimension.An example of the generation process with Eqn. (8) along the channel dimension can be found in Figure 3.
The transformation function can be applied in any dimension of the weight tensor. Figure 2 illustrates the transformation along the spatial dimension and Figure 3 shows the transformation along the input channel dimension. The transformation along the filter dimension is the same as the transformation along the input channel dimension. However, transformation along the filter dimension is easier for the computation reuse. We will show this in details in section 3.4.

heading: 3.3 LEARNING TO SEARCH EPITOMES END-TO-END
text: Benefiting from the differentiable Eqn. (7), the elements in E and the transformation learner η can be updated together with the convolutional layers through back propagation in an end-to-end manner. For each element in the epitome E, as its transformed weight parameter can be used in multiple positions in weight tensor, the gradients of the epitome are thus the summation of all the positions where the weight parameters are used.
Here, for clarity, we use {τ−1(p, q)} to denote the set of the indices in the convolution kernel that are mapped from the same position (p, q) in E. Note that here we abuse the notion of (p, q) to denote the integer spatial position in the epitome. The gradients of E(p,q) can, thus, be calculated as:
∇E(p,q)L = ∑
z∈{τ−1(p,q)}
αz∇θzL, (9)
where θz is the kernel parameters that are transformed from E(p,q), and αz are the fractions that are assigned to E(p,q) during the transformation. The epitome can thus be updated via Eqn. (10):
Et(pt,qt) = E t−1 (pt,qt) − ∇Et−1 (pt,qt) L, (10)
where denotes the learning rate and subscript t denotes the training epoch. Eqn. (9) and (10) use the parameter updating rule along the spatial dimension as an example. The above equations can be applied on any dimension by replacing the index mapping. The indexing learner η can be simply updated according to the chain rule.

heading: 3.4 COMPRESSION EFFICIENCY
text: Parameter reduction By using the routing map which records location mappings from the sub-tensor in the epitome to the convolution weight tensor, the indexing learner can be removed during the inference phase. Thus, the total number of parameters during inference is decided by the size of the epitome and the routing map. Recall that the epitome E is a four dimensional tensor with shape (WE , HE , CEin, C E out). The size of an sub-tensor in the epitome is denoted as (w, h, β1, β2)
2 where β1 ≤ CEin and β2 ≤ CEout. The size of the epitome can be calculated as WE ×HE × CEin × CEout. The size of the routing mapM is calculated as 3×Rcin +Rcout where Rcout = dCout/β2e is the number of starting indices learned along the output channel dimension, and 3×Rcin = 3×dCin/β1e is the number of starting index learned along the spatial and input channel dimension. Note that we can enlarge the size of the sub-tensor in the epitome to reduce the size of the routing map. Here, the size is referring to the number of parameters. Detailed explanations of how Rcin is calculated can be found in the Figure 3. The parameter compression ratio r can thus be calculated via Eqn. (11):
r = w × h× Cin × Cout WE ×HE × CEin × CEout + 3×Rcin +Rcout ≈ Cout × Cin × w × h CEout × CEin ×WE ×HE , (11)
2We set β1 to CEin and β2 to C E out in this paper.
From Eqn. (11), it can be seen that the compression ratio is nearly proportional to the ratio between the size of the epitome and the generated weight tensor. Detailed proof can be found in Appendix E. The above analysis demonstrates that NES provides a precise control of the model size via the proposed transformation function.
Computation reduction As the weight tensor θ is generated from the epitome E, the computation in convolution can be reused when different elements in θ are from the same portion of elements in E.
Concretely, we propose two novel schemes to reuse the computation along the input channel dimension and the filter dimension respectively.
Channel wrapping. During the inference, the computation along the input channel dimension is reduced with channel wrapping as illustrated in Figure 4. For the elements in the input feature map that are multiplied with the same element in the epitome, we group the feature map elements first and then multiplied with the weight tensor in the epitome as follows:
F̃ (i, j,m) = Rcin−1∑ c′=0 F (p, q,m+ c′ × CEin + cin), (12)
where Rcin = dCin/CEine is the number of samplings (Eqn. (8)) applied along the input channel dimension and m + cin + c′ × CEin is the learned position with (p, q, cin) = M(i, j,m) and cin ∈ [0, CEin). This process is also illustrated in Figure 4. Product map and integral map. For the reuse along the filter dimension, given the routing mapM for the transformation, we first calculate the convolution results between the epitome and the input feature map once and then save the results as a product map P .
During inference, given P , the multiplication in convolution can be reused in a lookup table manner with O(1) complexity:
Gtw,th,c = W−1∑ i=0 H−1∑ j=0 Cin−1∑ m=0 Ftw+i,th+j,mθM(i,j,m,c) = W−1∑ i=0 H−1∑ j=0 Cin−1∑ m=0 PM(i,j,m,c). (13)
The additions in Eqn. (13) can also be reused via an integral map I (Crow, 1984) as done in Viola et al. (2001). With the product map and the integral map, the MAdd can be calculated as:
Reduced MAdd = (2CinWEHE − 1)WHCEout +WHWEHECEout +2RcinWHβ1 +2Rcoutβ2. (14)
Hence, the computation cost reduction ratio can be written as:
MAdd Reduction Ratio = CoutHW (2Cinwh− 1)
CEoutHW (W EHE + 2CEinW EHE − 1) + 2RcinWHβ1 + 2β2Rcout (15)
See more details and analysis in Appendix E. Discussion. We make a few remarks on the advantages of our proposed method as follows. The proposed NES method disentangles the weight tensors from the architecture by using a learnable transformation function. This provides a new research direction for model compression by bringing in better design flexibility against the traditional compression methods on both sides of software and hardware. On the software side, NES does not require re-implementation of acceleration algorithms. All the operations employed by NES are compatible with popular neural network libraries and can be encapsulated as a drop in operator. On the hardware side, the memory allocation of NES is more flexible by allowing easily adjust the epitome size. This is especially helpful for hardware platform where the off chip memory access is the main power consumption as demonstrated in Han et al. (2016). NES provides a way to balance the computation/memory-access ratio in hardware: a smaller epitome with a complex transformation function results in a computation intense model while a large epitome with simple transformation function results in a memory intensive model. Such ratio is an important hardware optimization criteria which however is not covered by most previous compression methods.

heading: 4 EXPERIMENTS
text: We first evaluate the efficacy of our method in 1D convolutional model compression on the sound dataset ESC-50 (Piczak, 2015) for the comparison with WSNet. We then test our method with MobileNetV2 and EfficientNet as the backbone on 2D convolutions on ImageNet dataset (Deng et al., 2009) and CIFAR-10 dataset (Krizhevsky & Hinton, 2009). Detailed experiments settings can be found in Appendix A. For all experiments, we do not use additional training tricks including the squeeze-and-excitation module (Hu et al., 2018) and the Swish activation function (Ramachandran et al., 2017) which can further improve the results unless those are used in the bachbone model
originally. The calculation of MAdd is performed for all convolution blocks. We evaluate our methods in terms of three criteria: model size, multiply-adds(MAdd) and the classification performance.

heading: 4.1 1D CNN COMPRESSION
text: For 1D convolution compression, we compare with WSNet. Similar to WSNet (Jin et al., 2017), we use the same 8-layer CNN model for a fair comparison. The compression ratio in WSNet is decided by the stride (S) and the repetition times along the channel dimension (C), as shown in Table 1. From Table 1, one can see that with the same compression ratio, our method outperforms WSNet by 6.5% in classification accuracy. This is because our method is able to learn proper weights and learn a transformation rules that are adaptive to the dataset of interest and thus overcome the limitation of WSNet where the sampling stride is fixed. More results can be found in Appendix B.

heading: 4.2 2D CNN COMPRESSION
text: Implementation details. We use both MobilenetV2 (Sandler et al., 2018) and EfficientNet (Tan & Le, 2019) as our backbones to evaluate our approach on 2D convolutions. Both models are the most representative mobile networks very recently and have achieved great performance on ImageNet and CIFAR-10 datasets with much fewer parameters and MAdd than ResNet (He et al., 2016) and VGGNet (Simonyan & Zisserman, 2014). For a fair comparison, we follow the experiment settings as in the original papers.
Results on ImageNet. We first conduct experiments on the ImageNet dataset to investigate the effectiveness of our method. We use the same width multiplier as in Sandler et al. (2018) as our baseline. We choose four common width multiplier values, i.e., 0.75, 0.5, 0.35 and 0.18 (Sandler et al., 2018; Zhang et al., 2018; Yu et al., 2018) for a fair comparison with other compression approaches.
The performance of our method and the baseline is summarized in Table 2. For all the width multiplier values, our method outperforms the baseline by a significant margin. It can be observed that under higher compression ratio the performance gain is also larger. Moreover, the performance of MobileNetV2 drops significantly when the compression ratio is larger than 3×. However, our NES method increases the performance by 3.78% at a large compression ratio. This is because when the
compression ratio is high, each layer in the baseline model does not have enough capacity to learn good representations. Our method is able to generate more expressive weights from the epitome with a learned transformation function.
We also compare our method with the state-of-the-art compression methods in Table 3. Since an optimized architecture tends to allocate more channels to upper layers (He et al., 2018; Yu & Huang, 2019a), we also run experiments with larger size of the epitome for upper layers. The results is denoted with suffix ‘-A’ in Table 3. Comparison with more models are shown in Figure 5. As shown, NES outperforms the current SOTA mobile model (less than 400M MAdd model) EfficientNet-b0 by 1.2% with 40M less MAdd. Obviously, our method performs even better than some NAS-based methods.Although our method does not modify the model architecture, the transformation from the epitome to the convolution kernel optimizes the parameter allocation and enriches the model capacity through the learned weight combination and sharing.
Results on CIFAR-10. We also conduct experiments on CIFAR-10 dataset to verify the efficiency of our method as shown in Table 6. Our method achieves 3.5×MAdd reduction and 5.64× model size reduction with only 1% accuracy drop, outperforming NAS-based AUTO-SLIM (Yu & Huang, 2019a). More experiments and implementation details are shown in the supplementary material.
Discussion. From the above results, one can observe significant improvements of our method over competitive baselines, even the latest architecture search based methods. The improvement of our method mainly comes from alleviating the performance degradation due to insufficient model size by learning richer and more reasonable combination of weights parameters and allocating suitable weight parameters sharing among different filters. The learned transformation from the epitome to the convolution kernel increases the weight representation capability with less increase on the memory footprint and the computation cost. This distinguishes our method from previous compression methods and improves the model performance significantly under the same compression ratio.

heading: 5 CONCLUSION
text: We present a novel neural epitome search method which can reuse the parameters efficiently to reduce the model size and MAdd with minimum classification accuracy drop or even increased accuracy in certain cases. Motivated by the observation that the parameters can be disentangled form the architecture, we propose a novel method to learn the transformation rule between the filters to make the transformation adaptive to the dataset of interest. We demonstrate the effectiveness of the method on CIFAR-10 and ImageNet dataset with extensive experimental results.
Acknowledgement Jiashi Feng was partially supported by NUS IDS R-263-000-C67-646, ECRA R-263-000-C87-133, MOE Tier-II R-263-000-D17-112 and AI.SG R-263-000-D97-490.

heading: B MORE RESULTS ON 1D CONVOLUTION COMPRESSION
text: We also conduct experiments to examine the highest compression ratio that our method can achieve without performance drop compared to WSNet (Jin et al., 2017). For a fair comparison, we choose the same 8-layer CNN model backbone as used in WSNet. Configuration details have been demonstrated in Table 1 in the formal paper. The results are shown in Table 5.

heading: C EXPERIMENT RESULTS ON CIFAR10
text: The experiment results are shown in Table 6

heading: D NES FOR FULLY-CONNECTED LAYER
text: Let D ∈ RNin×Nout denotes the parameter matrix for a fully-connected (FC) layer. We could use Eqn. (18) to sampling along the input dimension and Eqn. (23) to sample along the output dimension. We also conduct experiments on CIFAR-10 dataset to verify the efficacy of our method on FC layers as shown in Table 7. We take MobileNetV2-1.0 as our baseline.

heading: E PROOF ON PARAMETER AND COMPUTATION REDUCTION
text: With the epitome E as defined in the main text, our method introduces a novel transformation function where the convolution filter weights are transformed from E with τ(·). In this section, we start with the most general situation where the epitome is two-dimensional,E ∈ RW
E×HE , and we will show that increasing the dimension of epitome can reduce the computation memory cost aggressively. Our method can be extended to n-dimension convolution and fully-connected layers straightforwardly. The transformation process along the input channel dimension is illustrated in Figure 3 In our method, all the weight parameters θi,j,m,c are transformed from the compact epitome E. The convolution with NES is shown as below:
Gtw,th,c = ftw,th ∗ kc = W∑ i H∑ j Cin∑ m Ftw+i,th+j,mEM(i,j,m,c). (17)
Proof on computational cost reduction. Since the weight elements per convolutional layer are formed based on the same E, there is computational redundancy when two convolution kernels are selected from the same portion of E as shown in Figure 2. To reuse the multiplication in convolution, we first compute the multiplication between each element in the epitome E and the input feature map. The results are saved as a product map P such that the computations are done only once. However, given the epitome E ∈ RW
E×HE , the product map size is W ×H × Cin ×WE ×HE × Cout which consumes large computational memory. To reduce its size, we propose to increase the dimensions of the epitome to E ∈ RW E×HE×CEin×C E out in order to group the computation results in the product map. Each entry in the product map P is calculated as the dot product between the channel dimension along the input feature map and the third dimension along the compact weight matrix. We set CEin to be smaller than Cin to further boost the compression and β1 equal to C E in. As illustrated in Figure 4, the input channels of the feature map is first grouped by
F̃ (i, j,m) = Rcin−1∑ c′=0 F (p, q,m+ c′ × CEin + cin), (18)
where Rcin = dCin/CEine is the compression ratio along the input channel dimension and m+ cin + c× CEin is the learned position with (p, q, cin, n) =M(i, j, c,m), where cin ∈ [1, CEin]. The transformed input feature map F̃ has the same number of channels as CEin. Let (i, j,m) index the transformed feature map location. The product map is then calculated as Pi,j,p,q,n = F̃i,j,: · E′p,q,:,n, (19) where · denotes the dot product operation.
Now, the multiplications can be reused by replacing the convolution kernels with the product map P . Based on Eqns. (17), (18), (19), the convolution can be reduced by
Gtw,th,n = W−1∑ i=0 H−1∑ j=0 P(tw+i,th+j,p,q,n). (20)
To reuse the additions, we adopt an integral image I which is proposed in (Crow, 1984) but differently we extend the integral image dimension to make it suitable for 2D convolution based on P . Our integral image can be constructed by
I(tw, th, p, q, n) =  P0,th,p,q,n, tw = 0 Ptw,0,p,q,n, th = 0 Ptw,th,0,q,n, p = 0 Ptw,th,p,0,n, q = 0 Ptw,th,p,q,0, n = 0
I(tw − 1, th − 1, p− 1, q − 1, n− 1) + P (tw, th, p, q, n), else.
(21)
From Eqn. (21), the 2D convolution results can be retrieved in a similar way to (Jin et al., 2017) as follows:
Gtw,th,p,q,n = I(tw+w−1, th+h−1, p+w−1, q+h−1, n)−I(tw−1, th−1, p−1, q−1, n−1). (22)
As we set CEout to be smaller than the output channel dimension of the convolution kernel, we reuse the computation results from the epitome via
G̃tw,th,rout×β2:(rout+1)×β2 = Gtw,th,n:n+β2 , (23)
where rout ∈ {0, 1, ..., Rcout − 1}, Rcout = dCout/β2e is the number of samplings conducted along the output channel dimension, and n =M(rout × β2) is the learned mapping along the filter dimension. The filter length β2 ∈ {1, 2, ..., CEout} is a hyper-parameter and is decided empirically. In our experiments, we choose β2 = Cout. Thus, the MAdds can be calculated as
Reduced MAdd = (2CinWEHE − 1)WHCEout︸ ︷︷ ︸ From Eqn. (19) +WHWEHECEout︸ ︷︷ ︸ From Eqn. (21) +2RcinWHβ1︸ ︷︷ ︸ From Eqn. (18) + 2Rcoutβ2︸ ︷︷ ︸ From Eqn.(23) . (24)
Suppose we use sliding window with a stride of 1 and no bias term, the MAdds of the conventional convolution can be calculated based on Eqn. (3) as shown below:
MAdd = (2× Cin × w × h− 1)×H ×W × Cout. (25)
Therefore, the total MAdd reduction ratio is
MAdd Reduction Ratio = CoutHW (2Cinwh− 1)
CEoutHW (W EHE + 2CEinW EHE − 1) + 2RcinWHβ1 + 2β2Rcout
≈ CoutCinwh CEoutC E inW EHE
(26)
Proof on parameter reduction. The parameter compression ratio for a 2D convolution layer can be calculated as follows:
r = whCinCout
WE ×HE × CEin × CEout + 3×Rcin +Rcout , (27)
where w and h denote the width and height of the kernel of the layer. Wc and Hc denote the spatial size of the corresponding epitome. From Eqn. (26) and Eqn. (27), it can be observed that the compression ratio is mainly decided by CoutCinwh
CEoutC E inW
EHE .

Review summaries:
This paper focuses on the problem of  neural network compression, and proposes a new scheme, the neural epitome search. It learns to find compact yet expressive epitomes for weight parameters of a specified network architecture. The learned weight tensors are independent of the architecture design.  It can be encapsulated as a drop in replacement to the current