Title: NEURAL EPITOME SEARCH FOR ARCHITECTURE-AGNOSTIC NETWORK COMPRESSION

Abstract: Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped. In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures. From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match weight shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy. Moreover, taking EfficientNet-B0 as baseline, our NES yields an improvement of 1.2% but has 10% less MAdd. In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings (<350M MAdd). Code can be found at https://github.com/zhoudaquan/NES.

1. INTRODUCTION:

Despite the remarkable performance achieved in many applications, powerful deep convolutional neural networks (CNNs) typically suffer from high complexity (Han et al., 2015a) . The large model size and computation cost hinders their deployment on resource limited devices, such as mobile phones. Very recently, huge efforts have been made to compress powerful CNNs. Existing compression techniques can be generally categorized into four categories: network pruning (Han et al., 2015a; Collins & Kohli, 2014; Han et al., 2015b) , low rank factorization (Jaderberg et al., 2014) , quantization (Jacob et al., 2018; Hubara et al., 2017; Rastegari et al., 2016) , and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016) . Network pruning targets on removing unimportant connections or weights to reduce the number of parameters and multiply-adds (MAdd). Low rank factorization decomposes an existing layer into lower-rank and smaller layers to reduce the computation cost. Weights quantization aims to use less number of bits to store the weights and activation maps. Knowledge distillation uses a well trained teacher network to train a lightweight student network. All of those compression methods assume that the model parameters (weight tensors) must have one-to-one correspondence to the architectures. As a result, they suffer from performance drop since changing architectures will inevitably lead to loss of informative parameters. Epitome E has a different shape as defined by the architecture. A transformation is learned automatically to transform the E to a shape that match the architecture defined shape. By designing a smaller E, significant compression can be achieved with less performance drop. In certain cases, the performance can be increased with less computation as shown in Table 3 .

In this paper, we consider the network compression problem from a new perspective where the shape of the weight tensors and the architecture are designed independently. The key insight is that the network parameters can be disentangled from the architecture and can be compactly represented by a small-sized parameter set (called epitome), inspired by success of epitome methods in image/video modeling and data sparse coding (Jojic et al., 2003; Cheung et al., 2008; Aharon & Elad, 2008) . As shown in Figure 1 , unlike conventional convolutional layers that use the architecture tied weight tensors to convolve with the input feature map, our proposed neural epitome search (NES) approach first learns a compact yet expressive epitome along with an adaptive transformation function to expand the epitomes. The transformation function is able to generate a variety of parameters from epitomes via a novel learnable transformation function, which also guarantees the representation capacity of the resulting weight tensors to be large. Our transformation function is differentiable and hence enables the NES approach to search for optimal epitome end-to-end, achieving a good trade-off between required model size and performance. In addition, we propose a novel routing map to record the index mapping used for the transformation between the epitome and the convolution kernel.

During inference, this routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.

Benefiting from the learned epitomic network parameters and transformation method, compared to existing compression approaches, NES has less performance drop. To the best of our knowledge, this is the first work that automatically learns compact epitomes of network parameters and the corresponding transformation function for network compression. To sum up, our work offers the following attractive properties:

• Our method is flexible. It allows the weight tensors to be independent of the architecture design. We can easily control the model size by defining the size of the epitomes given a specified network architecture. This is especially beneficial in the context of edge devices. • Our method is effective. The learning-based transformation method empowers the epitomes with highly expressive capability and hence incurs less performance drop even with large compression ratios. • Our method is easy to use. It can be encapsulated as a drop in replacement to the current convolutional operator. There is no dependence on specialized platforms/frameworks for NES to conduct compression.

To demonstrate the efficacy of the proposed approach, We conduct extensive experiments on CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) . On CIFAR-10 dataset, our method outperforms the baseline model by 1.3%. On ImageNet, our method outperforms MobileNetV2 full model by 1.47% in top-1 accuracy with 25% MAdd reduction, and MobileNetV2-0.35 baseline by 3.78%. Regarding MobileNetV2-0.7 backbone, our method improves AMC (He et al., 2018) by 2.47%. Additionally, when taking EfficientNet-b0 (Tan & Le, 2019) as baseline, we have an improvement of 1.2% top-1 accuracy with 10% MAdd reduction.

2. RELATED WORK:

Traditional model compression methods include network pruning (Collins & Kohli, 2014; Han et al., 2015b) , low rank factorization (Jaderberg et al., 2014) , quantization (Jacob et al., 2018; Hubara et al., 2017; Rastegari et al., 2016) and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016) . For all of those methods, as mentioned in Section 1 extensive expert knowledge and manual efforts are needed and the process might need to be done iteratively and hence is time consuming.

Recently, AutoML based methods have been proposed to reduce the experts efforts for model compression (He et al., 2018; Zoph et al., 2018; Noy et al., 2019; Li et al., 2019) and efficient convolution architecture design (Liu et al., 2018; Wu et al., 2018; Tan et al., 2018) . As proposed in AutoML for model compression (AMC (He et al., 2018) ), reinforcement learning can be used as an agent to remove redundant layers by adding resource constraints into the rewards function which however is highly time consuming. Later, gradient based search method such as DARTS (Liu et al., 2018) is developed for higher search efficiency over basic building blocks. There are also methods that use AutoML based method to search for efficient architectures directly (Wu et al., 2018; He et al., 2018) . All of those methods are searching for optimized network architecture with an implicit assumption that the weights and the model architecture have one-to-one correspondence. Different from all of the above mentioned methods, our method provides a new search space by separating the model weights from the architecture. The model size can thus be controlled precisely by nature.

Our method is also related to the group-theory based network transformation. Based on the group theory proposed in Cohen & Welling (2016) , recent methods try to design a compact filter to reduce the convolution layer computation cost such as WSNet (Jin et al., 2017) and CReLU (Shang et al., 2016) . WSNet tries to reduces the model parameters and computations by allowing overlapping between adjacent filters of 1D convolution kernel. This can be seen as a simplified version of our method as the overlapping can be regarded as a fixed rule transformation. CReLu tries to learn diversified features by concatenating ReLU output of original and negated inputs. However, as the rule is fixed, the design of those schemes are application specific and time consuming. Besides, the performance typically suffers since the scheme is not optimized during the training. In contrast, our method requires negligible human efforts and the transformation rule is learned end-to-end.

3. METHOD:



3.1. OVERVIEW:

A convolutional layer is composed of a set of learnable weights (or kernels) that are used for feature transformation. The observation of this paper is that the learnable weights in CNNs can be disentangled from the architecture. Inspired by this fact, we provide a new perspective on the network compression problem, i.e., finding a compact yet expressive parameter set, called epitome, along with a proper transformation function to fully represent the whole network, as illustrated in Figure 1 .

Formally, consider a convolutional network with a fixed architecture consisting of a stack of convolutional layers, each of which is associated with a weight tensor θ i (i is layer index). Further, let L(X, Y ; θ) be the loss function used to train the network, where X and Y are the input data and label respectively and θ = {θ i } denotes the parameter set in the network. Our goal is to learn epitomes E = {E i } which have smaller sizes than θ and the transformation function τ = {τ i } to represent the network parameter θ with the compact epitome as τ (E). In this way, network compression is achieved. The objective function of neural epitome search (NES) for a given architecture is to learn optimal epitomes E * and transformation functions τ * :

EQUATION

where | • | calculates the number of all elements.

The above NES approach provides a flexible way to achieve network compression since the epitomes can be defined to be of any size. By learning a proper transformation function, the epitomes of predefined sizes can be adaptively transformed to match and instantiate the specified network architecture. In the following sections, we will elaborate on how to learn the transformation functions

τ (•), the epitome (E ∈ R W E ×H E ×C E in ×C E out

) and how to compress models via NES in an end-to-end manner. In this paper, we use "a sub-tensor in the epitome" to describe a patch of the epitome that will be selected to construct the convolution weight tensor. The sub-tensor, E s , is represented with the starting index and the length along each dimension in the epitome as shown below:

E s = E[p : p + w, q : q + h, c in : c in + β 1 , c out : c out + β 2 ],

(2)

Spatial index Sigmoid 3 × 3 Conv 1 × 1 Conv Epitome Weight Tensor (p 1 , q 1 ) (p 2 , q 2 ) (p 3 , q 3 )

... 7) are concatenated together to form the weight tensor. Note that we use a moving average way to update M so that the indexing learner can be removed during inference. This is shown in details in section 3.2 in the paragraph 'Routing map'. The whole training process is end-to-end trainable and hence can be used for any specified network architecture.

Here, we abuse the notion for the starting index pair (pn, qn) by using subscript n to denote the n th pair of the starting index during sampling(Eqn. ( 7)) while in the main text, we use subscript t to denote the training epoch.

The learned indices and the epitome are fed into the interpolation-based sampler (Eqn. ( 7)) and the outputs of Eqn. ( 7) are concatenated together to form the convolution weights tensor.

where (p, q, c in , c out ) denote the starting index of the sub-tensor and (w, h, β 1 , β 2 ) denotes the length of the sub-tensor along each dimension.

3.2. DIFFERENTIABLE SEARCH FOR EPITOME TRANSFORMATION:

As aforementioned, NES generates convolution weight tensors from the epitome E via a transformation function τ . In this section, we explain how the transformation τ is designed and optimized.

We start with the formulation of a conventional convolution operation. Then we introduce how the transformed epitome is deployed to conduct the convolution operations, with a reduced number of parameters and calculations.

A conventional 2D convolutional layer transforms an input feature tensor F ∈ R W ×H×Cin to an output feature tensor G ∈ R W ×H×Cout through convolutional kernels with weight tensor θ ∈ R w×h×Cin×Cout . Here (W, H, C in , C out ) denote the width, height, input and output channel numbers of the feature tensor; w and h denote width and height of the convolution kernel. The convolution operation can be formulated as:

G tw,t h ,c = w-1 i=0 h-1 j=0 Cin-1 m=0 F tw+i,t h +j,m θ i,j,m,c , ∀ t w ∈ [0, W ), t h ∈ [0, H), c ∈ [0, C out ). (3)

Instead of maintaining the full weight tensor θ, NES maintains a much smaller epitome E that can generate the weight tensor and thus achieves model compression. To make sure the generated weights τ (E) can conduct the above convolution operation without incurring performance drop, we carefully design the transformation function τ with following three novel components: (1) a learnable indexing function η to determine starting indices of the sub-tensor within the epitome E to sample the weight tensor;

(2) a routing map M that records the location mapping from the sampled sub-tensors epitome to the generated weight tensor;

(3) an interpolation-based sampler to perform the sampling from E even when the indices are fractional. We now explain their details.

Indexing function:

The indexing function η is used to localize the sub-tensor within the epitome that is used to generate the weight tensor, as illustrated in Figure 2 . Concretely, given the input feature tensor F ∈ R W ×H×Cin , the function generates indices as follows,

EQUATION

where p, q, c in , c out are vectors of learned starting indices along the spatial, input channel and filter dimensions respectively to sample the sub-tensor within epitome to generate the model weight tensors. 8) by replacing the starting index (p,q) in Eqn. ( 7) with (cin) and enumerating over the input channel dimension. In this example, β1 is set to 4.

Each vector contains a set of starting indices and the element number inside each vector is equal to the number of transformations that will be applied along each dimension. Note they are all non-negative real numbers. η is the index learner and it outputs the normalized indices (p , q , c in , c out ) through a sigmoid function, each ranging from 0 to 1. These outputs are further up-scaled by a scaling function S(•) to the corresponding dimension of the epitome by S(•)

EQUATION

where W E , H E , C E in , C E out are dimensions of the epitome and ⊗ denotes element-wise multiplication. The learned indices are then fed into the following interpolation based sampler to generate the weight tensor. We implement the indexing learner by a two-layer convolution module that can be jointly optimized with the backbone network end-to-end. In particular, we use separate indexing learners and epitomes for each layer of the network. More implementation details are given in Appendix A.3.

Routing map:

The routing map is constructed to record the position correspondence between the convolution weight tensor and the sub-tensor in the epitome. It takes a position within the weight tensor as input and returns the corresponding starting index of the sub-tensor in the epitome. The mapped starting index of the sub-tensor in the epitome can thus be retrieved from the routing map fast. More importantly, the indexing learner can be removed during inference with the help of the routing map. The routing map is built as a look-up table during the training phase by recording the moving average of the output index from the index learner η. For example, the starting index pair as shown in Figure 2 can be fetched via (p t , q t ) = M(i, j, m) where (i, j, m) is the spatial location in the weight tensor and (p t , q t ) is the starting index of the selected sub-tensor in the epitome at training epoch t. The routing map M is constructed via Eqn. ( 6) as shown below with momentum µ during the training phase. µ is treated as a hyper-parameter and is decided empirically1 :

EQUATION

)

Interpolation based sampler The learned starting index and the pre-defined dimension (w, h, β 1 , β 2 ) of the sub-tensor in the epitome is then fed into the sampler function. The sampler function samples the sub-tensor within the epitome to generate the weight tensor. To simplify the illustration on the sampler function, we use the transformation along the spatial dimension as an example as shown in Figure 2 . The weight tensor is generated via the equation as shown below:

EQUATION

where G(a, b) = max(0, 1-|a-b|); n w and n h enumerate over all integral spatial locations within E. Following Eqn. ( 7), we first find all sub-tensors in the epitome whose starting indices (n w , n h ) along spatial dimensions satisfy: G(n w , p t )G(n h , q t ) > 0. Then a weighted summation (or interpolation) over the involved sub-tensors is computed according to Eqn. ( 7). In the case of applying the sampling along input channel dimension, (p, q) in the equation is replaced with the learned starting index c in and the weight tensor is generated by iterating along the input channel dimension as shown below:

EQUATION

Where R cin = C in /β 1 is the number of samplings applied along the input channel dimension.An example of the generation process with Eqn. ( 8) along the channel dimension can be found in Figure 3 .

The transformation function can be applied in any dimension of the weight tensor. Figure 2 illustrates the transformation along the spatial dimension and Figure 3 shows the transformation along the input channel dimension. The transformation along the filter dimension is the same as the transformation along the input channel dimension. However, transformation along the filter dimension is easier for the computation reuse. We will show this in details in section 3.4.

3.3. LEARNING TO SEARCH EPITOMES END-TO-END:

Benefiting from the differentiable Eqn. ( 7), the elements in E and the transformation learner η can be updated together with the convolutional layers through back propagation in an end-to-end manner. For each element in the epitome E, as its transformed weight parameter can be used in multiple positions in weight tensor, the gradients of the epitome are thus the summation of all the positions where the weight parameters are used.

Here, for clarity, we use {τ -1 (p, q)} to denote the set of the indices in the convolution kernel that are mapped from the same position (p, q) in E. Note that here we abuse the notion of (p, q) to denote the integer spatial position in the epitome. The gradients of E (p,q) can, thus, be calculated as:

EQUATION

where θ z is the kernel parameters that are transformed from E (p,q) , and α z are the fractions that are assigned to E (p,q) during the transformation. The epitome can thus be updated via Eqn. ( 10):

EQUATION

where denotes the learning rate and subscript t denotes the training epoch. Eqn. ( 9) and (10) use the parameter updating rule along the spatial dimension as an example. The above equations can be applied on any dimension by replacing the index mapping. The indexing learner η can be simply updated according to the chain rule.

3.4. COMPRESSION EFFICIENCY:

Parameter reduction By using the routing map which records location mappings from the sub-tensor in the epitome to the convolution weight tensor, the indexing learner can be removed during the inference phase. Thus, the total number of parameters during inference is decided by the size of the epitome and the routing map. Recall that the epitome E is a four dimensional tensor with shape

(W E , H E , C E in , C E out ).

The size of an sub-tensor in the epitome is denoted as (w, h, β 1 , β 2 )2 where

β 1 ≤ C E in and β 2 ≤ C E out .

The size of the epitome can be calculated as

W E × H E × C E in × C E out .

The size of the routing map M is calculated as 3 × R cin + R cout where R cout = C out /β 2 is the number of starting indices learned along the output channel dimension, and 3 × R cin = 3 × C in /β 1 is the number of starting index learned along the spatial and input channel dimension. Note that we can enlarge the size of the sub-tensor in the epitome to reduce the size of the routing map. Here, the size is referring to the number of parameters. Detailed explanations of how R cin is calculated can be found in the Figure 3 . The parameter compression ratio r can thus be calculated via Eqn. (11): From Eqn. ( 11), it can be seen that the compression ratio is nearly proportional to the ratio between the size of the epitome and the generated weight tensor. Detailed proof can be found in Appendix E. The above analysis demonstrates that NES provides a precise control of the model size via the proposed transformation function.

EQUATION

Computation reduction As the weight tensor θ is generated from the epitome E, the computation in convolution can be reused when different elements in θ are from the same portion of elements in E.

Concretely, we propose two novel schemes to reuse the computation along the input channel dimension and the filter dimension respectively.

Channel wrapping. During the inference, the computation along the input channel dimension is reduced with channel wrapping as illustrated in Figure 4 . For the elements in the input feature map that are multiplied with the same element in the epitome, we group the feature map elements first and then multiplied with the weight tensor in the epitome as follows:

EQUATION

where R cin = C in /C E in is the number of samplings (Eqn. ( 8)) applied along the input channel dimension and m + c in + c × C E in is the learned position with (p, q, c in ) = M(i, j, m) and

c in ∈ [0, C E in )

. This process is also illustrated in Figure 4 . Product map and integral map. For the reuse along the filter dimension, given the routing map M for the transformation, we first calculate the convolution results between the epitome and the input feature map once and then save the results as a product map P .

During inference, given P , the multiplication in convolution can be reused in a lookup table manner with O(1) complexity:

EQUATION

)

The additions in Eqn. ( 13) can also be reused via an integral map I (Crow, 1984) as done in Viola et al. (2001) . With the product map and the integral map, the MAdd can be calculated as: Hence, the computation cost reduction ratio can be written as:

EQUATION

)

EQUATION

See more details and analysis in Appendix E.

Discussion. We make a few remarks on the advantages of our proposed method as follows. The proposed NES method disentangles the weight tensors from the architecture by using a learnable transformation function. This provides a new research direction for model compression by bringing in better design flexibility against the traditional compression methods on both sides of software and hardware. On the software side, NES does not require re-implementation of acceleration algorithms. All the operations employed by NES are compatible with popular neural network libraries and can be encapsulated as a drop in operator. On the hardware side, the memory allocation of NES is more flexible by allowing easily adjust the epitome size. This is especially helpful for hardware platform where the off chip memory access is the main power consumption as demonstrated in Han et al. (2016) . NES provides a way to balance the computation/memory-access ratio in hardware: a smaller epitome with a complex transformation function results in a computation intense model while a large epitome with simple transformation function results in a memory intensive model. Such ratio is an important hardware optimization criteria which however is not covered by most previous compression methods.

4. EXPERIMENTS:

We first evaluate the efficacy of our method in 1D convolutional model compression on the sound dataset ESC-50 (Piczak, 2015) for the comparison with WSNet. We then test our method with MobileNetV2 and EfficientNet as the backbone on 2D convolutions on ImageNet dataset (Deng et al., 2009) and CIFAR-10 dataset (Krizhevsky & Hinton, 2009) . Detailed experiments settings can be found in Appendix A. For all experiments, we do not use additional training tricks including the squeeze-and-excitation module (Hu et al., 2018) and the Swish activation function (Ramachandran et al., 2017) which can further improve the results unless those are used in the bachbone model originally. The calculation of MAdd is performed for all convolution blocks. We evaluate our methods in terms of three criteria: model size, multiply-adds(MAdd) and the classification performance.

4.1. 1D CNN COMPRESSION:

For 1D convolution compression, we compare with WSNet. Similar to WSNet (Jin et al., 2017) , we use the same 8-layer CNN model for a fair comparison. The compression ratio in WSNet is decided by the stride (S) and the repetition times along the channel dimension (C), as shown in Table 1 . From Table 1 , one can see that with the same compression ratio, our method outperforms WSNet by 6.5% in classification accuracy. This is because our method is able to learn proper weights and learn a transformation rules that are adaptive to the dataset of interest and thus overcome the limitation of WSNet where the sampling stride is fixed. More results can be found in Appendix B.

4.2. 2D CNN COMPRESSION:

Implementation details. We use both MobilenetV2 (Sandler et al., 2018) and EfficientNet (Tan & Le, 2019) as our backbones to evaluate our approach on 2D convolutions. Both models are the most representative mobile networks very recently and have achieved great performance on ImageNet and CIFAR-10 datasets with much fewer parameters and MAdd than ResNet (He et al., 2016) and VGGNet (Simonyan & Zisserman, 2014) . For a fair comparison, we follow the experiment settings as in the original papers.

Results on ImageNet. We first conduct experiments on the ImageNet dataset to investigate the effectiveness of our method. We use the same width multiplier as in Sandler et al. (2018) as our baseline. We choose four common width multiplier values, i.e., 0.75, 0.5, 0.35 and 0.18 (Sandler et al., 2018; Zhang et al., 2018; Yu et al., 2018) for a fair comparison with other compression approaches.

The performance of our method and the baseline is summarized in Table 2 . For all the width multiplier values, our method outperforms the baseline by a significant margin. It can be observed that under higher compression ratio the performance gain is also larger. Moreover, the performance of MobileNetV2 drops significantly when the compression ratio is larger than 3×. However, our NES method increases the performance by 3.78% at a large compression ratio. This is because when the ChamNet (Dai et al., 2018) and ChannelNet (Gao et al., 2018) . Our method outperforms all the methods within the same level of MAdd. Here, MobileNetV2 * is our implementation of baseline models and MobileNetV2 is the original model with width multiplier of 0.35,0,5,0,75 and 1. The backbone model for our results are EfficientNet b1 and b0 (Tan & Le, 2019) with multiplier 0.5 and MobileNetV2 with multiplier of 0.75, 0.5, 0.35 and 0.2, respectively (from top to bottom). Note that we do not use additional training tricks including the squeeze-and-excitation module (Hu et al., 2018) and the Swish activation function (Ramachandran et al., 2017) .

compression ratio is high, each layer in the baseline model does not have enough capacity to learn good representations. Our method is able to generate more expressive weights from the epitome with a learned transformation function.

We also compare our method with the state-of-the-art compression methods in Table 3 . Since an optimized architecture tends to allocate more channels to upper layers (He et al., 2018; Yu & Huang, 2019a) , we also run experiments with larger size of the epitome for upper layers. The results is denoted with suffix '-A' in Table 3 . Comparison with more models are shown in Figure 5 . As shown, NES outperforms the current SOTA mobile model (less than 400M MAdd model) EfficientNet-b0 by 1.2% with 40M less MAdd. Obviously, our method performs even better than some NAS-based methods.Although our method does not modify the model architecture, the transformation from the epitome to the convolution kernel optimizes the parameter allocation and enriches the model capacity through the learned weight combination and sharing.

Results on CIFAR-10. We also conduct experiments on CIFAR-10 dataset to verify the efficiency of our method as shown in Table 6 . Our method achieves 3.5× MAdd reduction and 5.64× model size reduction with only 1% accuracy drop, outperforming NAS-based AUTO-SLIM (Yu & Huang, 2019a) . More experiments and implementation details are shown in the supplementary material.

Discussion. From the above results, one can observe significant improvements of our method over competitive baselines, even the latest architecture search based methods. The improvement of our method mainly comes from alleviating the performance degradation due to insufficient model size by learning richer and more reasonable combination of weights parameters and allocating suitable weight parameters sharing among different filters. The learned transformation from the epitome to the convolution kernel increases the weight representation capability with less increase on the memory footprint and the computation cost. This distinguishes our method from previous compression methods and improves the model performance significantly under the same compression ratio.

5. CONCLUSION:

We present a novel neural epitome search method which can reuse the parameters efficiently to reduce the model size and MAdd with minimum classification accuracy drop or even increased accuracy in certain cases. Motivated by the observation that the parameters can be disentangled form the architecture, we propose a novel method to learn the transformation rule between the filters to make the transformation adaptive to the dataset of interest. We demonstrate the effectiveness of the method on CIFAR-10 and ImageNet dataset with extensive experimental results.

A IMPLEMENTATION DETAILS

A.1 MOBILENETV2 SETTINGS:

Epitome dimensions for MobileNetV2 bottleneck. With our NES method, the shape of the feature map produced by each layer can be kept the same as the ones from the original model before compression. However, the number of channels in the feature map is reduced using the width multiplier method for MobileNetV2. Hence, for a fair comparison, we only apply the width multiplier on the output dimension of the first 1 × 1 convolutional layer and the input channel dimension of the second 1 × 1 convolutional layer within the bottleneck blocks of MobileNetV2 for obtaining the same feature map shape between blocks as our method. Based on this principle, we generate weight tensor based on the epitome along the filter dimension for the first 1 × 1 convolutional layers within the bottleneck and along the input channel dimension for the second 1 × 1 convolutional layers.

Specifically, we set the epitome shape per layer as (#in channels, #out channels×expansion multiplier , 1, k) for the first 1 × 1 convolution layer and ( #in channels×expansion multiplier , #out channels, 1, k) for the second 1 × 1 layer as shown in Table 4 . Here, expansion is referring to the ratio between the input size of the bottleneck and the inner size as detailed in Figure 2 of (Sandler et al., 2018) . The shape represents the number of input channels, the number of output channels and the kernel size, respectively. The compression ratio for each layer, c, can thus be calculated as c = 1 multiplier .

Table 4 : Epitome dimensions for the inverted residual blocks of the MobileNetV2 backbone. Here w, h, k, k denotes the spatial size, input channels and output channels of the input feature map respectively. Variables wc and hc denote the spatial size of the epitome and are set to 1 for 1 × 1 convolutional layer. c is used to set the compression ratio for each layer and is similar to the concept of width multiplier as defined in MobileNet (Howard et al., 2017) . t is the expansion ratio as defined in MobileNetV2.

Input:

Operators Output Epitome Comp. ratio

h × w × k 1 × 1, conv2d, ReLU6 h × w × tk wc × hc × k × ctk wc×hc c h s × w s × tk 3 × 3, depth-wise separable, ReLU6 h s × w s × tk - 1 h s × w s × tk 1 × 1, conv2d, linear h s × w s × k wc × hc × ctk × k wc×hc c

A.2 EPITOME DIMENSION DESIGN:

The size of the epitome can be calculated precisely by the original model and the desired compression ratio r. For a CNN with C n-dimensional convolutional layers and K fully-connected layers, its number of parameters can be calculated as

C i=1 d L i d + K k=1 N k in N k out

, where L i d denotes the length of the convolution weight tensor along the d th dimension of the i th convolutional layer. N k in and N k out denote the input and output dimension of the k th fully-connected layer.

We assign an epitome E j ∈ R W E j ×H E j for each layer j, and a routing map M : (x1, x2, . . . , xn) → (p, q), i.e., the weight value of an n-d filter at location (x1, x2, . . . , xn) being equal to E(p, q). We define the dimension of the epitome to be 2D here to illustrate the general case and later, we will show that in practice, the dimension of the epitome can be increased to save the computation memory. The size of the total epitome is, thus, C+K j=1 W E j × H E j . After learning the routing map for layer j, we store the location mapping as a lookup table of size Mj. The size of all the mapping tables is C+K j=1 Mj. Hence, the compression ratio can be calculated as

EQUATION

In our analysis, we use a uniform compression ratio for all the layers. Therefore, given a compression ratio r, the size of the epitome for each layer can be calculated accordingly. This deterministic design of the epitome size is hardware friendly and can be used to control the memory allocation.

Epitome patch design:

We set the patch size along each dimension to be w, h, β1 and β2. Note that the transformation along each dimension are independent and hence can be conducted separately. The starting index of the transformation along the spatial dimension and the input channel dimension are learned in pairs. This is because the transformation along the spatial dimension will also increase the input channel dimension.

Published as a conference paper at ICLR 2020

A.3 INDEX SEARCH SPACE IN EPITOME:

The selection space of the starting index for the transformation is not all the indices available along the channel dimension. We partition the channels into groups to build a super-index with a group length lg. For example, for an epitome that has C channels along the input channel dimension, the potential channel index ranges from 0 to C -1. With our super-index scheme, adjacent lg channels are grouped as a single index Cg and thus, the selection space of the index ranges from 0 to C/lg -1. The range of the super-index is used to scale the output from the transformation learner.

B MORE RESULTS ON 1D CONVOLUTION COMPRESSION:

We also conduct experiments to examine the highest compression ratio that our method can achieve without performance drop compared to WSNet (Jin et al., 2017) . For a fair comparison, we choose the same 8-layer CNN model backbone as used in WSNet. Configuration details have been demonstrated in Table 1 in the formal paper. The results are shown in Table 5 . 

D NES FOR FULLY-CONNECTED LAYER:

Let D ∈ R N in ×N out denotes the parameter matrix for a fully-connected (FC) layer. We could use Eqn. (18) to sampling along the input dimension and Eqn. ( 23) to sample along the output dimension. We also conduct experiments on CIFAR-10 dataset to verify the efficacy of our method on FC layers as shown in Table 7 . We take MobileNetV2-1.0 as our baseline. 

E PROOF ON PARAMETER AND COMPUTATION REDUCTION:

With the epitome E as defined in the main text, our method introduces a novel transformation function where the convolution filter weights are transformed from E with τ (•). In this section, we start with the most general situation where the epitome is two-dimensional,E ∈ R W E ×H E , and we will show that increasing the dimension of epitome can reduce the computation memory cost aggressively. Our method can be extended to n-dimension convolution and fully-connected layers straightforwardly. The transformation process along the input channel dimension is illustrated in Figure 3 In our method, all the weight parameters θi,j,m,c are transformed from the compact epitome E. The convolution with NES is shown as below:

Gt w ,t h ,c = ft w ,t h * kc = (17)

Proof on computational cost reduction. Since the weight elements per convolutional layer are formed based on the same E, there is computational redundancy when two convolution kernels are selected from the same portion of E as shown in Figure 2 . To reuse the multiplication in convolution, we first compute the multiplication between each element in the epitome E and the input feature map. The results are saved as a product map P such that the computations are done only once. However, given the epitome E ∈ R W E ×H E , the product map size is W × H × Cin × W E × H E × Cout which consumes large computational memory. To reduce its size, we propose to increase the dimensions of the epitome to E ∈ R W E ×H E ×C E in ×C E out in order to group the computation results in the product map. Each entry in the product map P is calculated as the dot product between the channel dimension along the input feature map and the third dimension along the compact weight matrix. We set C E in to be smaller than Cin to further boost the compression and β1 equal to C E in . As illustrated in Figure 4 , the input channels of the feature map is first grouped by

EQUATION

where Rcin = Cin/C E in is the compression ratio along the input channel dimension and m + cin + c × C E in is the learned position with (p, q, cin, n) = M(i, j, c, m), where cin ∈ [1, C E in ]. The transformed input feature map F has the same number of channels as C E in . Let (i, j, m) index the transformed feature map location. The product map is then calculated as Pi,j,p,q,n = Fi,j,: • E p,q,:,n , (19) where • denotes the dot product operation. Now, the multiplications can be reused by replacing the convolution kernels with the product map P . Based on Eqns. ( 17), ( 18), ( 19), the convolution can be reduced by Gt w ,t h ,n = W -1 i=0 H-1 j=0 P (tw +i,t h +j,p,q,n) .

(20)

:

We set µ to be 0.97 in our experiments.

We set β1 to C E in and β2 to C E out in this paper.

Figures and Tables:

Figure 1: (a) illustrates the conventional convolution process; (b) shows the convolution with NES method.Epitome E has a different shape as defined by the architecture. A transformation is learned automatically to transform the E to a shape that match the architecture defined shape. By designing a smaller E, significant compression can be achieved with less performance drop. In certain cases, the performance can be increased with less computation as shown in Table3.

Figure2: Our proposed compression process along the spatial dimension. We only show the transformation along spatial dimensions for easy understanding and the transformation along the channel dimension can be found in Figure3. The indexing learner learns the position mapping function, M : (i, j, m) -→ (p, q), between the convolution kernel elements and the sub-tensor in the epitome Es. The learned starting indices and the epitome are fed into Eqn. (7) to sample the weight tensor. The outputs of Eqn. (7) are concatenated together to form the weight tensor. Note that we use a moving average way to update M so that the indexing learner can be removed during inference. This is shown in details in section 3.2 in the paragraph 'Routing map'. The whole training process is end-to-end trainable and hence can be used for any specified network architecture. Here, we abuse the notion for the starting index pair (pn, qn) by using subscript n to denote the n th pair of the starting index during sampling(Eqn. (7)) while in the main text, we use subscript t to denote the training epoch. The learned indices and the epitome are fed into the interpolation-based sampler (Eqn. (7)) and the outputs of Eqn. (7) are concatenated together to form the convolution weights tensor.

Figure 3: Transformation along the input channel dimension of NES. In the figure, we only show three dimensions of the epitome with E ∈ R W E ×H E ×C E in ×1 . To simplify the illustration, we set W E = w and H E = h where w and h are the size of the convolution kernel. Thus, the starting indices along the spatial dimension,(p,q), are not shown in the figure. The generated weight tensor has input channel number equal to 8. Rcin = Cin/β1 = 2 denotes the number of samplings applied along the input channel dimension. During each transformation, sub-tensor with shape w × h × β1 is selected each time based on Eqn. (8) by replacing the starting index (p,q) in Eqn. (7) with (cin) and enumerating over the input channel dimension. In this example, β1 is set to 4.

Figure 4: NES transformation along the input channel dimension with channel wrapping. To simplify the illustration, we choose an epitome with shape R w×h×3×1 and C E in = 3. In this example, the transformation is applied twice and the two learned starting indices are 0 . The input feature map F is first added based on the learned interpolation position of the kernels. Input feature map F1 and F4 are both multiplied with the first channel in the epitome since W1 and W4 are both generated with E1. To reuse the multiplication, feature map F1 and F4 are first added together before multiplying with the weights kernel E1. The figure uses integer index to simplify the illustration. When the learned indices are fractions, the feature maps are the weighted summation of the two nearest integer indexed sub-tensors in the epitome as shown in Eqn. (8). For example, if the two starting indices in this figure are 0.6 and 0.3, the calculation becomes (0.6F1 + 0.3F4 + 0.4F3 + 0.7F6)⊗E1 + (0.4F1 + 0.7F4 + 0.6F2 + 0.3F5) ⊗E2 + (0.4F2 + 0.7F5 + 0.6F3 + 0.3F6) ⊗E3. Since we group the feature map first before the convolution, the computation cost is reduced.

Figure 5: ImageNet classification accuracy of our method, EfficientNet, MobileNetV2 baselines and other NAS based methods including AMC(He et al., 2018), IGCV3(Sun et al., 2018), MNasNet(Tan et al., 2018), ChamNet(Dai et al., 2018) and ChannelNet(Gao et al., 2018). Our method outperforms all the methods within the same level of MAdd. Here, MobileNetV2 * is our implementation of baseline models and MobileNetV2 is the original model with width multiplier of 0.35,0,5,0,75 and 1. The backbone model for our results are EfficientNet b1 and b0(Tan & Le, 2019) with multiplier 0.5 and MobileNetV2 with multiplier of 0.75, 0.5, 0.35 and 0.2, respectively (from top to bottom). Note that we do not use additional training tricks including the squeeze-and-excitation module(Hu et al., 2018) and the Swish activation function(Ramachandran et al., 2017).

Figure 6: NES transformation along the input channel dimension with channel wrapping. To simplify the illustration, we choose an epitome with shape R w×h×3×1 and β1 = 3. The transformation is applied twice and the two starting indices are 0. The input feature map F is first grouped based on the learned interpolation position of the kernels. Input feature map F1 and F4 are both multiplied with the first channel in the epitome since W1 and W4 are both generated with E1. To reuse the multiplication, feature map F1 and F4 are first added together before multiplying with the weights kernel E1. The figure uses integer index to simplify the illustration. When the learned indices are fractions, the feature maps are the weighted summation of the two nearest integer indexed sub-tensors in the epitome as shown in Eqn. (7). For example, if the two starting indices in this figure are 0.6 and 0.3, the calculation becomes (0.6F1 + 0.3F4 + 0.4F3 + 0.7F6)⊗E1 + (0.4F1 + 0.7F4 + 0.6F2 + 0.3F5) ⊗E2 + (0.4F2 + 0.7F5 + 0.6F3 + 0.3F6) ⊗E3. Since we group the feature map first before the convolution, the cost is reduced.

Ft w +i,t h +j,mEM(i,j,m,c).

Comparison with WSNet on ESC-50 dataset. We use the same configuration but change the sampling stride to be learnable. 'S' denotes the stride and 'C' denotes the repetition times along the input channel dimension. We use the 'S' in WSNet as initial values and learns the offsets.

Table content: <table><tr><td>Method</td><td>Conv1</td><td>Conv2</td><td>Conv3</td><td>Conv4</td><td>Conv5</td><td>Conv{6-8}</td><td>Acc. (%)</td><td>Params</td></tr><tr><td>Config.</td><td>S C</td><td>S C</td><td>S C</td><td>S C</td><td>S C</td><td>S C</td><td/><td/></tr><tr><td>baseline</td><td>1 1</td><td>1 1</td><td>1 1</td><td>1 1</td><td>S 1</td><td>1 1</td><td>66.0</td><td>1×</td></tr><tr><td>WSNet</td><td>8 1</td><td>4 1</td><td>2 2</td><td>1 2</td><td>S 4</td><td>1 8</td><td>66.5</td><td>4×</td></tr><tr><td>Ours</td><td>8 1</td><td>4 1</td><td>2 2</td><td>1 2</td><td>S 4</td><td>1 8</td><td>73.0</td><td>4×</td></tr></table>

Results of ImageNet classification. Our method uses vanilla MobileVetV2 as backbone. For a fair comparison, we evaluate multiple width multiplier values of 0.75, 0.5, 0.35 and 0.18 and only apply it on the filter dimension of the first 1 × 1 convolution. We apply the proposed method on all the invert residual blocks equally to disentangle the architecture affects on the performance. MAdd are calculated based on all convolution blocks with an assumption that the batch normalization layers are merged. ' * ' denotes our own implementation.

Table content: <table><tr><td>Methods</td><td>MAdd(M)</td><td>Parameters</td><td>Param Compression Rate</td><td>Top-1 Accuracy(%)</td></tr><tr><td>MobilenetV2-1.0</td><td>301</td><td>3.4M</td><td>1.00×</td><td>71.8</td></tr><tr><td>MobilenetV2-0.75  *</td><td>217</td><td>2.94M</td><td>1.17×</td><td>69.14</td></tr><tr><td>MobilenetV2-0.5  *</td><td>153</td><td>2.52M</td><td>1.36×</td><td>67.22</td></tr><tr><td>MobilenetV2-0.35  *</td><td>115</td><td>2.26M</td><td>1.54×</td><td>65.18</td></tr><tr><td>MobilenetV2-0.18  *</td><td>71</td><td>1.98M</td><td>1.80×</td><td>60.70</td></tr><tr><td>Our method-0.75</td><td>220</td><td>2.94M</td><td>1.17×</td><td>71.54</td></tr><tr><td>Our method-0.5</td><td>157</td><td>2.52M</td><td>1.36×</td><td>69.42</td></tr><tr><td>Our method-0.35</td><td>120</td><td>2.26M</td><td>1.54×</td><td>67.01</td></tr><tr><td>Our method-0.18</td><td>79</td><td>1.95M</td><td>1.80×</td><td>64.48</td></tr></table>

Comparison of our method with other state-of-the-art models on ImageNet where our method shows superior performance over all other methods. MAdd are calculated based on all convolution blocks with an assumption that the batch normalization layers are merged. Suffix '-A' means we use larger compression ratio for front layers. Our method does not modify the backbone model architecture and applies a uniform compression ratio, unless specified with suffix '-A'. All experiments are using MobileNetV2 as backbone unless labeled with EfficientNet as suffix.

Table content: <table><tr><td colspan="2">GROUP Methods</td><td colspan="3">MAdd (M) Params Top-1 Acc. (%)</td></tr><tr><td/><td>MobilenetV2-0.35 (Sandler et al., 2018)</td><td>59</td><td>1.7M</td><td>60.3</td></tr><tr><td>60M MAdd</td><td>S-MobilenetV2-0.35 (Yu et al., 2018) US-MobilenetV2-0.35 (Yu &amp; Huang, 2019b) MnasNet-A1 (0.35x) (Tan et al., 2018)</td><td>59 59 63</td><td>3.6M 3.6M 1.7M</td><td>59.7 62.3 62.4</td></tr><tr><td/><td>Our method-0.18</td><td>79</td><td>2.0M</td><td>64.48</td></tr><tr><td/><td>MobilenetV2-0.5 (Sandler et al., 2018)</td><td>97</td><td>2.0 M</td><td>65.4</td></tr><tr><td>100M</td><td>S-MobilenetV2-0.5 (Yu et al., 2018)</td><td>97</td><td>3.6M</td><td>64.4</td></tr><tr><td>MAdd</td><td>US-MobilenetV2-0.5 (Yu &amp; Huang, 2019b)</td><td>97</td><td>3.6M</td><td>65.1</td></tr><tr><td/><td>Our method-0.35</td><td>120</td><td>2.2M</td><td>67.01</td></tr><tr><td/><td>MobilenetV2-0.75 (Sandler et al., 2018)</td><td>209</td><td>2.6 M</td><td>69.8</td></tr><tr><td/><td>S-MobilenetV2-0.75 (Yu &amp; Huang, 2019a)</td><td>209</td><td>3.6M</td><td>68.9</td></tr><tr><td/><td>US-MobilenetV2-0.75 (Yu &amp; Huang, 2019b)</td><td>209</td><td>3.6M</td><td>69.6</td></tr><tr><td>200M+</td><td>FBNet-A (Wu et al., 2018)</td><td>246</td><td>4.3M</td><td>73</td></tr><tr><td>MAdd</td><td>AUTO-S-MobilenetV2-0.75 (Yu &amp; Huang, 2019a)</td><td>207</td><td>4.1M</td><td>73</td></tr><tr><td/><td>Our method-0.5</td><td>157</td><td>2.5M</td><td>69.42</td></tr><tr><td/><td>Our method-0.75</td><td>220</td><td>2.9M</td><td>71.54</td></tr><tr><td/><td>Our method-0.75-A</td><td>225</td><td>3.7M</td><td>73.27</td></tr><tr><td/><td>Our method-0.5 (EfficientNet-b0)</td><td>240</td><td>3.92M</td><td>75.55</td></tr><tr><td/><td>Our method-0.5 (EfficientNet-b1)</td><td>350</td><td>5.46M</td><td>77.5</td></tr></table>

Comparison with WSNet on ESC-50 dataset. We choose the compressed model by WSNet method as our baseline. By decreasing the size of the epitome, we can achieve higher compression ratio. We apply uniform compression ratio for all layers. It is observed that the highest compression ratio we can achieve before our method's performance become smaller than WSNet is 3.16×.

Table content: <table><tr><td>Methods</td><td colspan="2">Compression Rate Accuracy (%) (top1)</td></tr><tr><td>WSNet</td><td>1.00×</td><td>66.5</td></tr><tr><td>Our method-1</td><td>1×</td><td>73.0</td></tr><tr><td>Our method-2</td><td>2.35×</td><td>69.25</td></tr><tr><td>Our method-3</td><td>3.16×</td><td>65.5</td></tr><tr><td colspan="2">C EXPERIMENT RESULTS ON CIFAR10</td><td/></tr><tr><td colspan="2">The experiment results are shown in Table 6</td><td/></tr></table>

Comparison with other state-of-the-art models in classification accuracy on CIFAR-10.

Table content: <table><tr><td>Our method</td></tr></table>

Experiment results of our method applied on fully connected layer of MobileNetV2.

Table content: <table><tr><td>Methods</td><td colspan="4">Parameters Model Comp. Rate FC Param. Comp. Rate Top-1 Acc. (%)</td></tr><tr><td>MobileNetV2-1</td><td>2.2M</td><td>1.00×</td><td>1.00×</td><td>94.06</td></tr><tr><td>MobileNetV2-0.5 FC  *</td><td>0.4 M</td><td>5.55×</td><td>2.00×</td><td>91.8</td></tr><tr><td>Our method-0.5 FC</td><td>0.39M</td><td>5.64×</td><td>2.00×</td><td>92.96</td></tr></table>

Peer review (iclr2020):
In this work, the authors describe a technique for compressing neural networks by learning a so-called Epitome (E), and a transformation function (	heta) such that the weights for each layer can be constructed using 	heta(E). The epitome and the transformation function can be learnt jointly while optimizing the network for the task specific loss. The main idea of this paper is really interesting, and the experimental results which compare against other recent techniques also validate the proposed technique. However, while I think the main idea is relatively clear, I personally found the description of the proposed techniques -- particularly 3.2 and 3.4 -- to be somewhat hard to follow, particularly given that some details only appear in the Appendix. I would suggest that the authors try to revise this section by trying to move Figures 4 and 5 from the appendix into the main text. Some additional suggestions also appear below. Overall, I would while I like the ideas in this paper, based on the current presentation I am inclined to rate the paper as a “weak reject”, though I would raise my rating if the paper was revised to improve the presentation. Main comments: 1. The authors mention that “During inference, (the) routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.” It would be nice to include some results which indicate what the savings are with the proposed routing map. 2. Section 3.2: There were a few aspects of section 3.2 that I think could be improved for clarity. A.) Personally, I found Figure 2 somewhat tricky to follow. I would suggest removing the 3x2 “Epitome” and “Generated Kernel” figures on the extreme right of the image, since I’m assuming the only goal of these is to indicate that “orange” and “blue” correspond to “Epitome” and “Kernel” respectively. I would also suggest mentioning the correspondence of the colors as the first sentence in the caption. Finally, if possible, I would suggest adding a small description alongside the (a), (b), (c) subcaptions: e.g., (a) straightforward but non-differentiable mapping, … , (c) Generated Weigh Kernel. B.) I believe that the authors use a separate E for each layer, and that Epitomes are not shared across layers. I may have missed this in the text, but it would be useful to clarify this explicitly again in the section. C.) The “parameterized transformation layer” is mentioned before Equation 3. I think it would be useful to mention that this is implemented using neural networks in your work for clarity. E.g.: “To handle the above two obstacles, ... three parts: (1) a parameterized transformation learner ? (implemented using a neural network in this work) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).” or “To handle the above two obstacles, ... three parts: (1) a parameterized transformation learner ? (See Section 3.3) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).” D.) The exact structure of the “parameterized transformation layer” wasn’t exactly clear to me. In 3.3, the authors mention that it consists of “of two convolutional layer, followed by a sigmoid function … takes the feature map of the convolutional layer as input”. Please clarify exactly what is fed in as the input to this network e.g., (input feature map: F, and the indices i,j). 3. I personally also found Section 3.4 which discusses the Computation reduction was also somewhat hard to follow. Some clarification questions: Is the memory/computation cost of the storing/creating the routing map included in the compression calculations? I think it is important to factor these costs when computing the savings achieved by the model. Also, I was unclear on what R_{cin} and R_{cout} are, and why they appear in Equation 5. Could the authors please clarify. Minor Comments: 1. Abstract: “Traditional compression methods … all assume that network architectures and parameters should be hardwired.” What does it mean for them to be “hardwired” in this context? 2. Abstract: “Experiments demonstrate that, … with 25% MAdd reduction and AutoML for Model Compression (AMC) by 2.5% with nearly the same compression ratio.” --> “Experiments demonstrate that, … with 25% MAdd reduction, and a 2.5% Madd reduction for AutoML for Model Compression (AMC) with nearly the same compression ratio.” ------- Update after Author Response -------- I would like to thank the authors for their responses and for the updates which strengthen the paper in my view. I have updated my score accordingly.