Neural Epitome Search for Architecture-Agnostic Network Compression
Authors: Daquan Zhou
Xiaojie Jin
Qibin Hou
Kaixin Wang
Jianchao Yang
Jiashi Feng
Venue:  (2020)
Keywords: Network Compression
Classification
Deep Learning
Weights Sharing

Abstract: Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters should be hardwired.  In this work, we propose a new perspective on network compression, i.e., network parameters can be disentangled from the architectures.  From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction and AutoML for Model Compression (AMC) by 2.5% with nearly the same compression ratio. Moreover, taking EfficientNet-B0 as baseline, our NES yields an improvement of 1.2% but are with 10% less MAdd.  In particular, our method achieves a new state-of-the-art results of 77.5% under mobile settings (<350M MAdd). Code will be made publicly available.

Published as a conference paper at ICLR 2020
NEURAL EPITOME SEARCH FOR ARCHITECTURE- AGNOSTIC NETWORK COMPRESSION
Daquan Zhou', Xiaojie Jin"; Qibin Hou', Kaixin Wang' .Jianchao , Jiashi Feng'
'Department of Electrical and Computer Engineering, National University Singapore *Bytedance Inc., Mountain View, USA
(e0357894, kaixin. wang) nus. edu andrewhcux@@mail. com
{jinxiaojie, yanq yawpleeaaannnnnnniiianaaaa cianchaa (awnchaao@bbttednceee. el jeffia@nus, edu..s.e...s...
ABSTRACT
Traditional compression methods including network pruning, quantization, low rank factorization and knowledge distillation all assume that network architectures and parameters one-to-one mapped. In this work, we propose on network compression, i.e., network parameters can be disentangled from the architectures. From this viewpoint, we present the Neural Epitome Search (NES), a new neural network compression approach that learns to find compact yet expres- sive epitomes for weight parameters of specified network architecture end-to-end. The complete network to compress can be generated from the learned epitome via novel transformation method that adaptively transforms the epitomes match weight shapes of the given architecture. Compared with existing compression methods, NES allows the weight tensors to be independent of the architecture design and hence can achieve good trade-off between model compression rate and performance given a specific model size constraint. Experiments demonstrate that, on ImageNet, when taking MobileNetV2 as backbone, our approach improves the full-model baseline by 1.47% in top-1 accuracy with 25% MAdd reduction, and with the same compression ratio, improves AutoML for Model Compression (AMC) by 2.5% in top-1 accuracy. Moreover, taking EfficientNet-BB as baseline, NES yields improvement of 1.2% but has 10% less MAdd. In particular, our method achieves a new state-of-the-ar results of 77.5% under mobile settings (<350M MAdd). Code can be found https:/gidhbb..om//hooaauaullul
perspective
INTRODUCTION
Despite the remarkable performance achieved in many applications, powerful deep convolutional neural networks (CNNs) typically suffer from high complexity (Han et al., 2015a). The large model size and computation cost hinders their deployment mobile phones. Very recently, huge efforts have been made to compress powerful CNNs. Existing compression techniques can be generally categorized into four categories: network pruning (Han et al., 5a; Collins & Kohli, 2014; Han et al., 2015b), low rank factorization (Jaderberg et al., 2014), quantization (Jacob et al., 2018: Hubara al., 2017: Rastegari et al., 2016), and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016). Network pruning targets on removing unimportant connections or weights reduce the number of parameters and multiply-add (MAdd). Low rank factorization decomposes existing layer into lower-rank and smaller layers reduce the computation cost. Weights quantization aims to use less number of bits to store the weights and activation maps. Knowledge distillation uses a well trained teacher network to train a lightweight student network. All of those compression methods assume that the model parameters (weight tensors) must have one-to-one correspondence to the architectures. As a result, they suffer from performance drop since changing architectures will inevitably lead to loss informative parameters.
resource limited devices, such as
"correspondence

---

Published as a conference paper at ICLR 2020
W Wiight tensor;
Compression
Epitome: E
Weight tensor
Input: F
Output: G
Input:
<(E)
Output: G
Convolution
Convolution
Figure 1: (a) illustrates the conventional convolution process; (b) shows the convolution with NES method. Epitome E has different shape defined by the architecture. A transformation is learned automatically to transform the compression can achieved with less performance drop. certain cases, the performance can be increased with less computation as shown in Table 3.
shape that match the architecture defined shape. By designing smaller E, significant
In this paper, we consider the network compression problem from a new perspective where the shape of the weight tensors and the architecture are designed independently. The key insight that the network parameters can be disentangled from the architecture and can be compactly represented by small-sized parameter
(called epitome). inspired by success of epitome methods image/video modeling and data sparse coding (Jojic et al., 2003; Cheung et al., 2008; Aharon & Elad, 2008). As shown in Figure 1, unlike conventional convolutional layers that use the architecture tied weight tensors convolve with the input feature map, our proposed neural epitome search (NES) approach first learns a compact yet expressive epitome along with an adaptive transformation function to expand the epitomes. The transformation function is able to generate variety of parameters from epitomes via a novel learnable transformation function, which also guarantees the representation capacity of the resulting weight tensors be large. Our transformation function is differentiable and hence enables the NES approach to search for optimal epitome end-to-end, achieving a good trade-off between required model size and performance. In addition, we propose novel routing map record the index mapping used for the transformation between the epitome and the convolution kernel. During inference, this routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.
Benefiting from the learned epitomic network parameters and transformation method, compared to existing compression approaches. NES has less performance drop. To the best of our knowledge, this is the first work that automatically learns compact epitomes of network parameters and the corresponding transformation function for network compression. To sum up, our work offers the following attractive properties:
Our method is flexible. allows the weight tensors to be independent of the architecture design. We can easily control the model size by defining the size of the epitomes given specified network architecture. This is especially beneficial in the context of edge devices.
Our method is effective. The learning-based transformation method empowers the epitomes with highly expressive capability and hence incurs less performance drop even with large compression ratios.
Our method is easy use. It can be encapsulated a drop in replacement to the current convolutional operator. There is no dependence on specialized platforns/framework, for NES to conduct compression.
To demonstrate the efficacy of the proposed approach, We conduct extensive experiments CIFAR- 10 (Krizhevsky Hinton, 2009) and ImageNet (Deng et al., 2009). On CIFAR- 10 dataset, our method outperforms the baseline model by 1.3%. On ImageNet, our method outperforms MobileNetV2 full model by 1.47% in top-1 accuracy with 25% MAdd reduction, and MobileNetV2-0.35 baseline by 3.78%. Regarding MobileNerV2-007 backbone, our method improves AMC (He al., 2018) by 2.47%. Additionally, when taking EfficientNet-b0 (Tan & Le, 2019) as baseline, we have an improvement of 1.2% top-1 accuracy with 10% MAdd reduction.
RELATED WORK
Traditional model compression methods include network pruning (Collins & Kohli, 2014; Han et al., 2015b), low rank factorization (Jaderberg et al., 2014), quantization (Jacob al., 2018; Hubara et al.,
2

---

Published as a conference paper at ICLR 2020
2017: Rastegari et al., 2016) and knowledge distillation (Hinton et al., 2015; Papernot et al., 2016). For all of those methods, as mentioned in Section 1 extensive expert knowledge and manual efforts needed and the process might need to be done iteratively and hence is time consuming.
Recently, AutoML based methods have been proposed to reduce the experts efforts for model compression (He al., 2018: 8; Zoph et al., 2018 8: Noy et al., 2019; Li et al., 2019) and efficient convolution architecture design (Liu et al., 2018; Wu et al., 2018; Tan et al., 2018). As proposed in AutoML for model compression (AMC (He et al., 2018)), reinforcement learning can be used as an agent to remove redundant layers by adding resource constraints into the rewards function which however is highly time consuming. Later, gradient based search method such as DARTS (Liu et al., 2018) is developed for higher search efficiency over basic building blocks. There are also methods that use AutoMI. based method to search for efficient architectures directly (Wu et al., 2018; He et al., 2018). All of those methods are searching for optimized network architecture with an implicit assumption that the weights and the model architecture have one-to-one correspondence. Different from all of the above mentioned methods, our method provides new search space by separating the model weights from the architecture. The model size can thus be controlled precisely by nature.
Our method is also related to the group-theory based network transformation. Based on the group theory proposed in Cohen & Welling (2016), recent methods try to design a compact filter to reduce the convolution layer computation cost such WSNet (Jin et al., 2017) and CReLU (Shang et al., 2016). WSNet tries to reduces the model parameters and computations by allowing overlapping between adjacent filters of 1D convolution kernel. This can be seen simplified version of our method as the overlapping can be regarded as fixed rule transformation. CReLu tries to learn diversified features by concatenating ReLU output of original and negated inputs. However, as the rule is fixed, the design of those schemes are application specific and time consuming. Besides, the performance typically suffers since the scheme is not optimized during the training. In contrast, our method requires negligible human efforts and the transformation rule is learned end-to-end.
3
METHOD
3.1
OVERVIEW
convolutional layer is composed of set of learnable weights (or kernels) that are used for feature transformation. The observation of this paper is that the learnable weights in CNNs can be disentangled from the architecture. Inspired by this fact, we provide a new perspective on the network compression problem, i.e., finding a compact yet expressive parameter set, called epitome, along with a proper transformation function to fully represent the whole network, as illustrated in Figure
Formally, consider a convolutional network with fixed architecture consisting of stack of convo- lutional layers, each of which is associated with a weight tensor 0; (i is layer index). Further, let C(X, 0) be the loss function used to train the network, where X and Y are the input data and label respectively and = { denotes the parameter set in the network. Our goal is to learn epitomes E = { which have smaller sizes than 0 and the transformation function T {T.) to represent the network parameter 0 with the compact epitome as T(F). . In this way, network compression is achieved. The objective function of neural epitome search (NES) for given architecture is to learn optimal epitomes E" and transformation functions T":
{E", arg min C(X, (E))
8.l. E < 10).
E.r
where || calculates the number of all elements.
The above NES approach provides a flexible way to achieve network compression since the epitomes can be defined to be of any size. By learning proper transformation function, the epitomes of predefined sizes can be adaptively transformed to match and instantiate the specified network architecture. In the following sections, will elaborate on how to learn the transformation functions 7()), the epitome (E' IR W H² x xcu)) and how to compress models via NES in an end-to-end manner. In this paper, we use "a sub-tensor in the epitome" to describe patch of the epitome that will be selected to construct the convolution weight tensor. The sub-tensor, Eg, is represented with the starting index and the length along each dimension in the epitome as shown below:
F, Flp: p + W. 0 h, Cin Cin 31, Cout Cout + 82),
3

---

Published as a conference paper at ICLR 2020
Spatial index
Epitome
Input
3 Cure
X Cure
Output
Sigriid
S(-)
Concat
Indexing Learner
W eight T Tensor
Convolution
Figure 2: Our proposed compression process along the spatial dimension. We only show the transformation along spatial dimensions for easy understanding and the transformation along the channel dimension can be found in Figure 3. The indexing learner learns the position mapping function, M (i, m²) (p. q). between the convolution kernel elements and the sub-tensor in the epitome Eg The learned starting indices and the epitome are fed into Equ. (7) sample
weight tensor. The outputs of Eqn. (7) are concatenated together to form the weight tensor. Note that we use moving average way to update M so that the indexing learner can be removed during inference. This is shown in details in section 3.2 in the paragraph 'Routing map'. The be used for any specified network architecture. Here, we abuse the notion for the starting index pair (Pa: (n) by using subscript to denote the nh pair of the starting index during sampling(Egn. (7)) while in the main text, we use subscript to denote the training epoch. The learned indices and the epitome are fed into the interpolation-base bbsed sampler (Eqn. (7)) and the outputs of Eqn. (7) are concatenated together to form the convolution weights tensor.
whole training process is end-to-end trainable and hence
where (p, q; Cins Cout) denote the starting index of the sub-tensor and (w, h, B1, 32) denotes the length of the sub-tensor along each dimension.
3.2 DIFFERENTIABLE SEARCH FOR EPITOME TRANSFORMATION
As aforementioned, NES generates convolution weight tensors from the epitome via a transfor- mation function In this section, explain how the transformation is designed and optimized. We start with the formulation of conventional convolution operation. Then we introduce how the transformed epitome deployed conduct the convolution operations, with reduced number of parameters and calculations.
conventional 2D convolutional layer transforms an input feature tensor F' WxHC..... to an output feature tensor G E IR XH XCuu through convolutional kernels with weight tensor B. XCaut Here (W, H, C.n. Cout) denote the width, height, input and output chan- nel numbers of the feature tensor, and h denote width and height of the convolution kernel. The convolution operation can be formulated as:
Gruithie ΣΣ
Fistist 1.mtttt.4t..mm.m. & to E [0, W),th e H), ce [0, Cout).
(3)
Instead of maintaining the full weight tensor 0, NES maintains much smaller epitome E that can generate the weight tensor and thus achieves model compression. To make sure the generated weights T(E) can conduct the above convolution operation without incurring performance drop, we carefully design the transformation function with following three novel components: (1) learnable indexing function 1) to determine starting indices of the sub-tensor within the epitome E to sample the weight tensor; (2) a routing map M that records the location mapping from the sampled sub-tensors epitome to the generated weight tensor; (3) an interpolation based sampler to perform the sampling from E even when the indices are fractional. We now explain their details.
Indexing function The indexing function 7) is used to localize the sub-tensor within the epitome that is used generate the weight tensor, as illustrated in Figure 2. Concretely, given the input feature tensor E x the function generates indices as follows.
(p. q, Cin Cout) (n)
where p. q, Cin. Cout are vectors of learned starting indices along the spatial, input channel and filter dimensions respectively sample the sub-tensor within epitome generate the model weight tensors.
4

---

Published as a conference paper at ICLR 2020
1 1x1 Coor
Sigmoid
Channel index
Eo. 1:1+4,]]
Fj., 2:2+4.]
S6-)
Cin 1.8
0.2
0.8 X
Ra=2, A 6
Concat
Weight tensor
Epitome
Cu=B
If
Cin 2.6 (1- 0.6)
0.6
starting Index
Figure 3: Transformation along the input channel dimension of NES. In the figure, we only show three dimensions
of the epitome with E' E Rw* 8 R xc x1 To simplify the illustration, we set W and H F = - where and l are the size of the convolution kernel. Thus, the starting indices along the spatial dimension..(pg)) shown in figure. The generated weight tensor has input channel number equal
Rein = [Cin /3 denotes the number of samplings applicd along the input channel dimension. During each transformation, sub- tensor with shape h x 81 is selected each time based on Eqn. (8) by replacing the starting index (p.q) in Eqn. (7) with (co.) and enumerating over the input channel dimension. In this example, 31 is set 4.
Each vector contains a set of starting indices and the element number inside each vector is equal the number of transformations that will be applied along each dimension. Note they are all non-negative real numbers. 1/ is the index learner and it outputs the normalized indices (p', q' cim= Sout) through sigmoid function, each ranging from 0 to 1. These outputs are further up-scaled by scaling function S(-) to the corresponding dimension of the epitome by S-))
S(p', q', cin cout) [W HB C C.ud [p'.q' Cin Coutl
where W H. C. C. are dimensions of the epitome and @ denotes element- -wise multiplication. The learned indices are then fed into the following interpolation based sampler to generate the weight tensor. We implement the indexing learner by two-layer convolution module that can jointly optimized with the backbone network end-to-end. In particular, we use separate indexing learners and epitomes for each layer of the network. More implementation details are given in Appendix A.3.
Routing map The routing map is constructed record the position correspondence between the convolution weight tensor and the sub-tensor in the epitome. It takes a position within the weight tensor as input and returns the corresponding starting index of the sub-tensor in the epitome. The mapped starting index of the sub-tensor in the epitome can thus be retrieved from the routing map fast. More importantly, the indexing learner can be removed during inference with the help of the routing map. The routing map is built look-up table during the training phase by recording the moving average of the output index from the index learner r)- For example, the starting index pair as shown in Figure 2 can be fetched via (Pt, q:) Mi, j, m) where (i, j. m) is the spatial location in the weight tensor and (Pt, q:)) is the starting index of the selected sub-tensor in the epitome at training epoch t. The routing map M is constructed via Eqn. (6) as shown below with momentum y during the training phase. ia is treated as a hyper-parameter
parameter and is decided empirically!:
M(i.j. m) (p., 9t) = (p:-1. 9:-1 + μ 17().
(6)
Interpolation based sampler The learned starting index and the pre-defined dimension (w, h,,,, (1,82) (2) of the sub-tensor in the epitome is then fed into the sampler function. The sam- pler function samples the sub-tensor within the epitome to generate the weight tensor. To simplify the illustration the sampler function, example as shown in Figure 2. The weight tensor is generated via the equation as shown below:
the transformation along the spatial dimension as an
W"-1H' 1H
(cnmm) = = ((F
= Σ The
Σ
((nw.P)G(na, 4) Exgrin tw
12 =
where G(a, b) = max(0, 1 1- b(): nuw and n. enumerate over all integral spatial locations within E. Following Eqn. (7), we first find all sub-tensors in the epitome whose starting indices (Now n4,) along
We set j be 0.97 in our experiments.
5

---

Published as a conference paper at ICLR 2020
spatial dimensions satisfy: ((n Pt (m) q:) > 0. Then weighted summation (or interpolation) over the involved sub-tensors is computed according to Eqn. (7). In the case of applying the sampling along input channel dimension, (p, q) in the equation is replaced with the learned starting index Cin and the weight tensor is generated by iterating along the input channel dimension as shown below:
Rew
8 (...mm+B))) = T(Elmm)
Σ
(T Cin ((siieinee)
(8)
Where Rein = [ is the number of samplings applied along the input channel dimension. example of the generation process with Eqn. (8) along the channel dimension can be found in Figure 3.
The transformation function can be applied in any dimension of the weight tensor. Figure illustrates the transformation along the spatial dimension and Figure 3 shows the transformation along the input channel dimension. The transformation along the filter dimension is the same as the transformation along the input channel dimension. However, transformation along the filter dimension is easier for the computation reuse. We will show this in details in section 3.4.
3.3 LEARNING TO SEARCH EPITOMES END-TO-END
Benefiting from the differentiable Egn. (7), the elements in E and the transformation learner can be updated together with the convolutional layers through back propagation an end-to-end manner. For each element in the epitome E, as its transformed weight parameter can be used in multiple positions weight tensor, the gradients of the epitome are thus the summation of all the positions where the weight parameters are used.
Here, for clarity, we use (p- q)} to denote the set the indices in the convolution kernel that are mapped from the same position (p,4) q) in E. Note that here we abuse the notion of (p, q) to denote the integer spatial position in the epitome. The gradients of E(p..) can, thus, be calculated as:
Eip.a) C
Σ
O2 Vo.C.
#E(T 1(m0))
where 0g is the kernel parameters that are transformed from E(p.g) and a₂ are the fractions that are assigned to Elpp.) during the transformation. The epitome can thus be updated via Eqn. (10);
F ap) =
Et-1 (p):)))
VV
f.
(10)
(pp
where e denotes the learning rate and subscript t denotes the training epoch. Eqn. (9) and (10) use the parameter updating rule along the spatial dimension as an example. The above equations can be applied on any dimension by replacing the index mapping. The indexing learner can be simply updated according to the chain rule.
3.4 COMPRESSION EFFICIENCY
Parameter reduction By using the routing map which records location mappings from the sub-tensor in the epitome to the convolution weight tensor, the indexing learner can be removed during the inference phase. Thus, the total number of parameters during inference is decided by the size of the epitome and the routing map. Recall that the epitome E is a four dimensional tensor with shape (WE, H., C. C.E in C. The size of an sub-tensor in the epitome is denoted as (w, h, 31, 82)² where S... and B2 I C... The size of the epitome can be calculated as WE x H x C x CE Cout- The size of the routing map M is calculated as 3 x Rein + Reous where Roout Couut [2 is the number of starting indices learned along the output channel dimension, and 3 x Rein = 3 x [Cin /3 is the number of starting index learned along the spatial and input channel dimension. Note that can enlarge the size of the sub-tensor in the epitome to reduce the size the routing map. Here, the size is referring to the number of parameters. Detailed explanations of how Rai is calculated can be found in the Figure 3. The parameter compression ratio r can thus be calculated via Eqn. (11):
Cin Cout
Cout X Cin w X h
r WE x HE x C.E. x C + 3 x Rein + Reout
CE x C x WE x HE
(11)
We set B1 to C and B2 to G this paper.

---

Published as a conference paper at ICLR 2020
Fi
Eg
Fg
E
Wy= Fg
Learued Channel Wrapping
Wi
E
Egilome, G
W/ E.
R
Starting judex 1 m0 Starting index -0
Wi E
F.
E
B
Figure 4: transformation along the input channel dimension with channel wrapping. simplify illustration, we choose an epitome with shape R g=xhx3x1 and C 3. In this example, the transformation applied twice and the two learned starting indices are 0. The input feature map J is first added based on the learned interpolation position of the kernels. Input feature map Fi and Fa are both multiplied with the first channel in the epitome since Wi and Wa both generated with F1. To reuse the multiplication, feature map Fi and F1 are first added together before multiplying with the weights kernel Ej. to simplify the illustration. When the learned indices are fractions, the feature maps are the weighted summation of the two nearest integer indexed sub-tensors in the epitome as shown in Eqn. (8). For example. if the two starting indices in this figure are 0.6 and 0.3, the calculation becomes (0.6F) + 0.3.F 0.4F's + 0.7 EL (0.4F; + 0.7 + 0.6.2, + 0.37)) & (0.4F2 0.7%% + 0.673 + O.3F.) @ Eg. Since we group the feature map first before the convolution, the computation cost is reduced.
figure uses integer index
From Eqn. (11), it can be seen that the compression ratio is nearly proportional the ratio between the size the epitome and the generated weight tensor. Detailed proof can be found in Appendix E. The above analysis demonstrates that NES provides a precise control of the model size via the proposed transformation function.
Computation reduction As the weight tensor 0 is generated from the epitome E, the computation in convolution can be reused when different elements in 0 are from the same portion elements in E.
Concretely, we propose two novel schemes to reuse the computation along the input channel dimen- sion and the filter dimension respectively.
Channel wrapping. During the inference, the computation along the input channel dimension is reduced with channel wrapping as illustrated in Figure 4. For the elements in the input feature map that are multiplied with the same element the epitome, we group the feature map elements first and then multiplied with the weight tensor in the epitome as follows:
Rain Σ
F(,jjm)
F(p.g, d x C + cin),
(12)
where Rein [Cin/ / is the number of samplings (Eqn. (8)) applied along the input channel dimension and Cin x C is the learned position with (p, q, Cin) MMii,j, m) and Cin [0, C... This process is also illustrated in Figure 4.
Product map and integral map. For the reuse along the filter dimension, given the routing map M for the transformation, first calculate the convolution results between the epitome and the input feature map once and then save the results as a product map P.
During inference, given P. the multiplication in convolution can be reused in lookup table manner with (1) complexity;
W-1
-11--10.-
W-1 H-10
Gta.ta.c
Σ 1= m=0
Σ
Σ
Frstittt A M(ij.m.c) = Σ Σ m=0
PM(ljjmm.)
(13)
The additions in Eqn. (13) can also be reused via integral map / (Crow, 1984) done in Viola et al. (2001). With the product map and the integral map, the MAdd can be calculated as:
Reduced MAdd = (2C,n) B E 1)W HCE out + WHWE H C
out + 2R H + 2R
(14)

---

Published as a conference paper at ICLR 2020
Table 1: Comparison with WSNet on ESC-50 dataset. We the same configuration but change the sampling stride to learnable. 'S' denotes the stride denotes repetition times along the input channel dimension. We use the 'S' in WSNet as initial values and leams the offsets.
Method
Conv1
Conv2
Conv3
Conv4
Conv5
Conv(6-8)
Acc. (%)
Params
Config.
S
S
C
C
S
C
$
C
S
C
baseline
1
1
1
1
1
S
1
1
66.0
1x
WSNet
8
1
2
2
4
8
66.5
4x
Ours
2
4
73.0
Table 2: Results of ImageNet classification. Our method vanilla MobileVetV2 backbone. For fair comparison, we evaluate multiple width multiplier values of 0.75, 0.5, 0.35 and 0.18 and only apply on filter dimension of the first 1 x convolution. We apply the proposed method on all the invert residual blocks equally to disentangle the architecture affects on the performance. MAdd are calculated based on all convolution blocks with assumption that the batch normalization layers merged. '*" denotes
implementation.
Methods
MAdd(M)
Parameters
Param Compression Rate
Top-1 Accuracy (%)
MobilenetV2-1.0
301
3.4M
1.00x
71.8
Mobilenet V2-0.775
217
2.94M
1.17x
69.14
MobilenetV2-0.55
153
2.52M
1.36x
67.22
VobilenetV2-0.35"
115
2.26M
1.54x
65.18
MobilenetV2-0.18*
1.98M
1.80x
60.70
Our method-0.75
220
2,94M
1.17x
71.54
Our method-0.5
157
2.52M
1.36x
69.42
Our method-0.35
120
2.26M
1.54x
67.01
Our method-0.18
79
1.95M
1.80x
64.48
Hence, the computation cost reduction ratio can be written as:
HW
MAdd Reduction Ratio
C HWWWE 20
2R WH3, + Roodd
(15)
See more details and analysis in Appendix E. Discussion. We make a few remarks on the advantages of our proposed method as follows. The proposed NES method disentangles the weight tensors from the architecture by using learnable transformation function. This provides a new research direction for model compression by bringing in better design flexibility against the traditional compression methods on both sides of software and hardware. On the software side. NES does not require re- immllmmmntaaionn of acceleration algorithms. All the operations employed by NES are compatible with popular neural network libraries and can be encapsulated as drop in operator. On the hardware side, the memory allocation of NES is more flexible by allowing easily adjust the epitome size. This is especially helpful for hardware platform where the off chip memory access is the main power consumption demonstrated in Han al. (2016). NES provides a way to balance the computation/memmory-eeee epitome with complex transformation function results in computation intense model while large epitome with simple transformation function results in a memory intensive model. Such ratio an important hardware optimization criteria which however is not covered by most previous compression methods.
ratio in hardware: a smaller
EXPERIMENTS
We first evaluate the efficacy of our method in 1D convolutional model compression on the sound dataset ESC-50 (Piczak, 2015) for the comparison with WSNet. We then test our method with MobileNetV2 and EfficientNet the backbone on 2D convolutions on ImageNet dataset (Deng et al., 2009) and CIFAR-10 10 dataset (Krizhevsky found in Appendix A. For all experiments, we do not additional training tricks including the squeeze-and and-exiitation module (Hu et al., 2018) and the Swish activation function (Ramachandran et al., 2017) which can further improve the results unless those used in the bachbone model
Hinton, 2009). Detailed experiments settings can be
8

---

Published as a conference paper at ICLR 2020
Table 3: Comparison of method with other state-of- mate-of-the-art models ImageNet where method shows superior performance over all other methods. MAdd are calculated based on all convolution blocks with an assumption that the batch normalization layers are merged. Suffix '-A' means we use larger compression ratio for front layers. Our method does not modify the backbone model architecture and applies uniform compression ratio, unless specified with suffix '-A', All experiments using MobileNetV2 as backbone unless labeled with EfficientNet suffx.
GROUP
Methods
MAdd (M)
Params
Top-1 Acc. (%)
MobilenetVV-00.5 (Sandler et al., 2018)
59
1.7M
60.3
S -obilenneVV2-0.35 (Yu et al., 2018)
59
3.6M
59.7
60M MAdd
US US--Mobilenet22-0.22 (Yu & Huang. 2019b)
59
3.6M
62.3
MnasNet-AI (0.35x) (Tan et al., 2018)
1.7M
62.4
Our method-0.18
2.0M
64.48
MobilenetV2-0.5 (Sandler al., 2018)
2.0M
65.4
100M
S-Moobleene12--0.5 (Yu et al., 2018)
97
3.6M
64.4
MAdd
JS-MMobllene(V2-0.. (Yu & Huang, 2019b)
97
3.6M
65.1
Our mcthod-0.35
120
2.2M
67.01
MobilenetV2-0075 (Sandler al., 2018)
209
2.6M
69.8
S-MobilenetV2--0775 (Yu Huang, 2019a)
209
3.6M
68.9
US-MobilenetV2-0.75 (Yu & Huang, 2019b)
3.6M
69.6
200M+
FBNet-A (Wu et al., 2018)
246
4.3M
73
MAdd
TTTD-S-MoblleeeV222000 (Yu & Huang, 2019a)
207
4.1M
73
Our method-0.5
157
2.5M
69.42
Our method-0.75
220
2.9M
71.54
Our method-0.755
225
3.7M
73.27
Our method-0.5 (EfficiemtNet-OO)
240
3.92M
75.55
Our method-0.5 (Efficien(Net-bl)
350
5.46M
77.5
originally. The calculation of MAdd is performed for all convolution blocks. We evaluate our methods in terms of three criteria: model size, multiply-add<(MAA Add) and the classification performance.
4.1 ID CNN COMPRESSION
For ID convolution compression, compare with WSeet. Similar to WSNet (Jin et al., 2017), use the same 8-layer CNN model for a fair comparison. The compression ratio in WSNet decided by the stride (S) and the repetition times along the channel dimension (C), shown in Table From Table one can see that with the same compression ratio, our method outperforms WSNet by 6.5% in classification accuracy. This is because transformation rules that are adaptive to the dataset of interest and thus overcome the limitation of WSNet where the sampling stride fixed. More results can be found in Appendix B.
method is able learn proper weights and learn
4.2 2D CNN COMPRESSION
Implementation details. We use both MobilenetV2 (Sandler et al., 2018) and EfficientNet (Tan & Le, 2019) as our backbones to evaluate our approach on 2D convolutions. Both models are the most representative mobile networks very recently and have achieved great performance on ImageNet and CIFAR-10 datasets with much fewer parameters and MAdd than ResNet (He et al., 2016) and VGGNet (Simonyan & Zisserman, 4). For fair comparison, we follow the experiment settings as in the original papers.
Results on ImageNet. We first conduct experiments on the ImageNet dataset to investigate the effectiveness of our method. We use the same width multiplier as in Sandler et al. (2018) as our baseline. We choose four common width multiplier values, iee., 0.75, 0.5, 0.35 and 0.18 (Sandler et al., 2018; Zhang al., 2018; Yu et al., 2018) for fair comparison with other compression approaches.
The performance of our method and the baseline is summarized in Table 2. For all the width multiplier values, our method outperforms the baseline by a significant margin. can be observed that under higher compression ratio the performance gain is also larger. Moreover, the performance of MobileNetV2 drops significantly when the compression ratio is larger than 3x. However, our NES method increases the performance by 3. 3.78% at large compression ratio. This is because when the

---

Published as a conference paper at ICLR 2020
80
77.50
78.60
75
Clessification Accuracy
75
75,30
73.27
70
09.42
Our Meched
Effiiennnns bo
65
05.08
Mobile
MilssNet
ISCV3 D:0.75)
60
AMC MOSI ENETV2-0.75
MRusteul-0-75
ChamNet
Charnelkervz-o
55 100
200 300
400
500
600
700
MAdd
Figure 5: ImageNet classification accuracy of our method, EfficientNet, MobileNetV2 baselines and other NAS based methods including AMC (He et al., 2018), IGCV3 (Sun et al., 2018), MNasNet (Tan et al., 2018), ChamNet (Dai et al., 2018) and ChannelNet (Gao al., 2018). Our method outperforms all the methods within the same level of MAdd. Here, MohileNetV2 is implementation of baseline models and MobileNetV2 the original model with width multiplier of 0.35,0.5.0,75 and The hackbone model for our results are EfficientNet bl and 60 (Tan & Lc, 2019) with multiplier 0.5 and MobileNetV2 with multiplier of 0.75, 0.5, 0.35 and 0.2, respectively (from top bottom). Note that we do not use additional training tricks including the suueeze-and-exciaeee module (Hu et al.. 2018) and the Swish activation function (Ramachandran et al., 2017).
compression ratio is high, each layer in the baseline model does not have enough capacity to learn good representations. Our method able generate more expressive weights from the epitome with a learned transformation function.
We also compare our method with the state-of-the-art compression methods in Table Since an optimized architecture tends to allocate more channels to upper layers (He et al., 2018; Yu & Huang, 2019a), we also run experiments with larger size of the epitome for upper layers. The results denoted with suffix "-A" in Table 3. Comparison with more models are shown in Figure 5, As shown, NES outperforms the current SOTA mobile model (less than 400M MAdd model) EfficientNet-b0 by 1.2% with 40M less MAdd. Obviously, our method performs even better than some NAS based methods. methods.Allhhough Although our method does not modify the model architecture, the transformation from the epitome to the convolution kernel optimizes the parameter allocation and enriches the model capacity through the learned weight combination and sharing.
Results on CIFAR-10. We also conduct experiments CIFAR-10 10 dataset verify the efficiency of our method as shown in Table 6. Our method achieves 3.5x MAdd reduction and 5.64 model size reduction with only 1 1% accuracy drop, outperforming NAS-based AUTO-SLIM (Yu & Huang, 2019a). More experiments and implementation details are shown in the supplementary material.
Discussion. From the above results,
observe significant improvements of our method over competitive baselines, even the latest architecture search based methods. The improvement of our method mainly comes from alleviating the performance degradation due insufficient model size by learning richer and more reasonable combination of weights parameters and allocating suitable weight parameters sharing among different filters. The learned transformation from the epitome the convolution kernel increases the weight representation capability with less increase on the memory footprint and the computation cost. This distinguishes our method from previous compression methods and improves the model performance significantly under the same compression ratio.
CONCLUSION
We present a novel neural epitome search method which can reuse the parameters efficiently to reduce the model size and MAdd with minimum classification accuracy drop even increased accuracy in certain cases. Motivated by the observation that the parameters can be disentangled form the architecture, we propose novel method learn the transformation rule between the filters make the transformation adaptive to the dataset of interest. We demonstrate the effectiveness of the method on CIFAR-10 and ImageNet dataset with extensive experimental results.
Acknowledgement Jiashi Feng was partially supported by NUS IDS R- R-263-000-C66-646, ECRA R- R-263-000-C87-133 MOE Tier-ll R- R-263-0000-D7--112 and AL.SG R-263-000-D99--490
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Michal Aharon and Michael Elad. Sparse and redundant modeling of image content using an image-signature- dictionary. SIAM Journal on Imaging Sciences, 1(3):228-247, 2008.
Vincent Cheung, Brendan J Frey, and Nebojsa Jojic. Video epitomes. International Journal of Computer Vision, 76(2):141-15,, 2008.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 2016.
Maxwell D Collins and Pushmeet Kohli. Memory bounded deep convolutional networks. arXiv preprint arXiv 2014.
2014.
Franklin C Crow. Summed-area tables for texture mapping. In ACM SIGGRAPH Computer Graphics, volume 18, pp. 207-212. ACM, 1984.
Xiaoliang Dai, Peizhao Zhang. Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang. Marat Dukhan, Yunging Hu, Yiming Wu, Yangqing Jia, et al. Chamnet: Towards efficient network design through platform-aww model adaptation. arXiv preprint arXiv:1812.08934, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Hongyang Gao. Zhengyang Wang, and Shuiwang Ji. Channelnets: Compact and efficient convolutional neural networks via channel-wise convolutions. In Advances in Neural Information Processing Systems, pp. 5197-5205, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510.00149, 2015a.
Song Han. Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135 1143, 2015b.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: efficient inference engine compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243-254. IEEE, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han, Ame: Automl for model compression and acceleration mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 784-800,200 2018
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 022.3033...00.,,.... 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17X 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-ex-iee computer vision and pottern recognition, pp. 7132-7141, 2018.
networks. In Proceedings of the IEEE conference on
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yani,,,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research. 18(1): 18(1);6869 98999,668888888 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient antter--rithmeetic- only inference. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2704-2713, 2018.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 1405.3866, 2014.
Xiaojie Jin, Yingzhen Yang, Ning Xu, Jianchao Yang, Nebojsa Jojic, Jiashi Feng, and Shuicheng Yan. Wsnet: Compact and efficient networks through weight sampling. arXiv preprint arXiv: 10067, 2017.
11

---

Published as a conference paper at ICLR 2020
Nebojsa Jojie, Brendan J Frey, and Anitha Kannan. Epitomic analysis of appearance and shape. In ICCV, volume 3. pp. 34, 2003.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng. Partial order pruning: for best speed/accuracy trade-off in neural architecture search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recogmition, pp. 9145-9153, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1rXiv:1806.09055 2018.
Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Mannik-Manor, Asap: Architecture search, anneal and prune. arXiv preprint arXiv: /04.://09440433,, 2019.
Nicolas Papernot, Martín Abadi, Ulfar Frlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data, arXiv preprint 161000555, 2016.
Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM
international conference on Multimedia, pp. 1015-1018. ACM, 2015.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Swish: self- gelf-gated activation function. arXiv preprint arXiv:1710.05941, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. Springer. 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen, Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510 45200, 2018
Wenling Shang. Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning, pp. 2217,225,.222 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint orXiv:1409. 1556, 2014.
Ke Sun, Mingjie Li, Dong Liu, and Jingdong Wang. 1gev3: Interleaved low-rank group convolutions for efficient deep neural networks. arXiv preprint arXiv:/806.00177, 2018.
Mingxing Tan and Quoc V Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 1905 1946, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoe Lc. Mnasnet: Platfor--awree neural architecture scarch for mobile. arXiv preprint arXiv: 1807. 1626, 2018.
Paul Viola. Michael Jones, al. Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2001.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbuet: Hardware-aware efficient convnet design via differentiable neural architecture search. arXiv preprint arXiv:1812..0344332. 12 2018.
Jiahui and Thomas Huang. Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11778; 2019a.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. arXiv preprint arXiv: 1e0e:990300533 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv 2018.
preprint a8Xiv:
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shuffienet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6848-6856, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018
12

---

Published as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
A.1 MOBILENETV2 SETTINGS
Epitome dimensions for MobileNetV2 bottleneck. With our NES method, the shape of the feature map produced by each layer can be kept the same as the ones from the original model before compression. However, the number of channels the feature map is reduced using the width multiplier method for MobileNetV2. Hence, for a fair comparison, we only apply the width multiplier on the output dimension of the first x convolutional layer and the input channel dimension of the second x convolutional layer within the bottleneck blocks of MobilcNetV2 for obtaining the same feature map shape between blocks as our method. Based on this principle, we generate weight tensor hased on the epitome along the filter dimension for the first 1 convolutional layers within the bottleneck and along the input channel dimension for the second 1 1 convolutional layers.
Specifically, we set the epitome shape per layer as (#in channels, #buu.channels x expansion 1,)) for the first multiplier
1 convolution layer and ( #in channels x expansion
1, k)) for the second x layer as shown in Table 4. Here, expansion is referring to the ratio between the input size of the bottleneck and the inner size as detailed in Figure 2 of (Sandler et al., 2018). The shape represents the number of input channels, the number of output channels and the keree size, respoctively. The compression ratio for each layer, can thus be calculated
multipller
as maltiplierr
Table 4: Epitome dimensions for the inverted residual blocks of the MobileNetV2 backbone. Here w, h, k. K' denotes the spatial size, input channels and output channels of the input feature map respectively. Variables we and he denote the spatial size of the epitome and set set the compression ratio for cach layer and similar to the concept of width multiplier as defined in MobileNet (Howard et al., 2017). is the expansion ratio as defined in MobileNetV2.
for 1 x 1 convolutional layer. is used
Input
Operators
Output
Epitome
Comp. ratio
h x u x k
X 1, conv2d, ReLU6
h x w X th
We x he x & x ctk
w.xha
X x tk
x 3, depth- wise separable, ReLU6
h x "
x tk
x 1, conv2d, linear
h x w
x K'
We x be x etk K
Me xhe
A.2 EPITOME DIMENSION DESIGN
The size of the epitome can be calculated precisely by the original model and the desired compression ratio For a CNN with C n-dimensional convolutional layers and K fully-connected layers, its number of parameters can be calculated as Σ IL I'as + ΣK N Nk where T. denotes the length of the convolution weight tensor along the t dimension of the ,th convolutional layer. Nė. and N denote the input and output dimension of the h fully-connected layer.
We assign an epitome F. E H2WFXR5 for each layer j. and routing map M : (Ir1, .2,....,,.) (p, q), i.e., the weight value of an n-d filter location (2, ... ..,) being equal to E(p, 4). We define the dimension of the epitome to be 2D here to illustrate the general case and later, we will show that in practice, the dimension of the epitome can be increased to save the computation memory. The size of the total epitome is, thus, EC+K W.F x H.F. After learning the routing map for layer j. we store the location mapping as lookup table of size My. The size of all the mapping tables is EØ* Mj. Hence, the compression ratio can be calculated as
Σ 1 + - & Nº. N Σ ECEK (W x 11 + M,)
(16)
In our analysis, we uniform compression ratio for all the layers. Therefore, given compression ratio r, the size of the epitome each layer can be calculated accordingly. This deterministic design of the epitome size is hardware friendly and can be used to control the memory allocation.
Epitome patch design We set the patch size along each dimension to be w, h 31 and B2. Note that the transformation along each dimension are independent and hence can be conducted separately. The starting index of the transformation along the spatial dimension and the input channel dimension are leamned in pairs. This is hecause the transformation along the spatial dimension will also increase the input channel dimension.
13

---

Published as a conference paper at ICLR 2020
A.3 INDEX SEARCH SPACE EPITOME
selection space of starting index for transformation not indices available along the channel dimension. We partition the channels into groups to build super-index with a group length I For example, for an epitome that has C channels along the input channel dimension, the potential channel index ranges from to C 1. With super-index scheme, adjacent Lg channels grouped as single index C. and thus, the selection space
index ranges from 0 C/ty The range the super-index used scale output from the transformation learner.
B MORE RESULTS ON ID CONVOLUTION COMPRESSION
also conduct experiments to examine highest compression ratio our method can achieve without performance drop compared to WSNet (Jin et al., 2017). For a fair comparison, we choose the same 8-layer CNN model hackbone as used in WSNet. Configuration details have been demonstrated in Table 1 in the formal paper. The results shown in Table 5.
Table Comparison with WSNet ESC-50 dataset. We choose the compressed model by WSNet method our baseline. By decreasing size the epitome, we can achieve higher compression ratio. We apply uniform compression ratio for all layers. It is observed that the highest compression ratio we can achieve before our method's performance become smaller than WSNet is 3.16x.
Methods
Compression Rate
Accuracy (%) (top1)
WSNet
1.00x
66.5
Our method-1 1
1x
73.0
Our method-2
2.35x
69.25
Our method-3
3.16x
65.5
C EXPERIMENT RESULTS ON CIFAR10
The experiment results are shown in Table 6
Table 6: Comparison with other state-of- of-the-art models in classification accuracy on CIFAR-10 10. Our method outperforms the recently proposed AUTO-S SLIM (Yu & Huang, 2019a) which is compression method using AutoML method.
Methods
Parameters
MAdd (M)
Top-1 Accuracy(56)
MobilenerV2-11
2.2M
93.52
94.06"
MobilenetV2-00
0.4 M
21.59
91.70
Auto-Slim
0.7M
59
93.00
Auto-Slim
0.3M
28
92.00
Our method
0.39M
26.8
93.22 :: 0.013
D NES FOR CULLY-CONNECTED LAYER
Let NM. x Nout denotes the parameter matrix for a fully-connected (FC) layer. We could use Eqn. (18)
to sampling along the input dimension and Eqn. (23) to sample along the output dimension. We also conduct experiments CIFAR- 10 dataset verify the efficacy of our method on FC layers as shown in Table 7. We take MobileNetV2-1.00
baseline.
Table 7: Experiment results of our method applied on fully connected layer of MobileNetV2.
Methods
Parameters
Model Comp. Rate
FC Param. Comp. Rate
Top-1 Acc. (9)
MobileNetV2-1 MobileNetV2-0.5 FC
2.2M
1.00
1.00
94,06
0.4 M
5.55
2.00x
91.8
Our method-0.5 FC
0.39M
5.64
x.00
92.96
14

---

Published as a conference paper at ICLR 2020
Fi
W;=F; Eg
Fg
E
Wy= Fg
Learned Channel Wrapping
Wi
E
Egilome, G=
W/ E.
R
Starting judex 1 =0 Starting index -0
W. E
F.
E
B
Figure 6: NES transformation along the input channel dimension with channel wrapping. simplify illustration, we choose an epitome with shape It g=xxxxxx1 and 31 3. The transformation is applied twice and the two starting indices are O. The input feature map F is first grouped based on the learned interpolation position of the kernels. Input feature map Fi and F4 are both multiplied with the first channel in the epitome since W1 and Wa both generated with F1. To reuse the multiplication, feature map Fi and F4 are first added together before multiplying with the weights kernel Fig. The figure uses integer index to simplify the illustration. When the learned indices are fractions, the feature maps are the weighted summation of the two nearest integer indexed sub-tensors in the epitome shown in Eqn. (7). For example, if the two starting indices in this figure are 0.6 and 0.3, the calculation becomes (0.6F) + 0.3.4 0.4f's 0.7 E1 (0.4F, 0.7P4 + 0.672 + O.3F)) &E2+ + (0.4/% + 0.7.% + 0.6 + 0.3Fo) @ Ea. Since we group the feature map first before the convolution, the cost is reduced.
E PROOF ON PARAMETER AND COMPUTATION REDUCTION
With the epitome E defined in the main text, method introduces novel transformation function where the convolution filter weights are transformed from E with T(.). In this section, we start with the most general situation where the epitome two-dimensional, F E TR. TRW = and we will show that increasing the dimension of epitome can reduce the computation memory cost aggressively. Our method can be extended to re-dimension convolution and fully-connected layers straightforwardll. The transformation process along the input channel dimension is illustrated in Figure In our method, all the weight parameters Or,.mm. are transformed from the compact epitome E. The convolution with NES shown as below;
W H Cin
Gew.tn. = Sew.e ki ΣV
Σ
F Poutidatioo E.iss.m..)
(17)
Proof on computational cost reduction. Since the weight elements per convolutional layer are formed based on the same E, there is computational redundancy when two convolution kernels are selected from the same portion of F as shown in Figure 2. To reuse the multiplication in convolution, multiplication between each element in the epitome E' and the input feature map. The results saved as product map P such that the computations done only once. However, given the epitome E I Bw R the product map size is x Cin x W x = x Cout which consumes large computational memory. To reduce its size, we propose to increase the dimensions of the epitome to E E R RW x xxx x x in order to group the computation results in the product map. Each entry in the product map P is calculated as the dot product between the channel dimension along the input feature map and the third dimension along the compact weight matrix. We set Cf,, to be smaller than C.. further boost the compression and B1 equal C. As illustrated in Figure 4, the input channels of the feature map first grouped by
first compute the
Ruin
f(. =
Σ F(p.g.m + + x C.. + Cin),
(18)
where Resn = [Ct.. /C. is compression ratio along the input channel dimension and m Cin + x C is the learned position with (p, g, Cin, m) = M(i, J, c, m), where can e [1, C..). The transformed input feature map f has the same number of channels as C Let (i, m) index the transformed feature map location. The product map is then calculated as
P.J.p... Fr.. P..a..
(19)
where- denotes the dot product operation.
Now, the multiplications can reused by replacing the convolution kernels with the product map P. Based on Eqns. (17), (18), (19). the convolution can be reduced by
Gtw.
Σ 1
Pltttit +im-g.))
(20)
15

---

Published as a conference paper at ICLR 2020
To reuse the additions, adopt integral image which proposed in (Crow, 1984) but differently extend the integral image dimension to make it suitable for convolution based on P. Our integral image can be constructed by
Po
t
T(t Pr q.n) n.)
p
(21)
P.w.tt...O..
9
0
[tw - 1,th 1,p 1,1-1,1111 1) P(w,, t, n.m, clse.
From Eqn. (21), 2D convolution results can be retrieved a similar way to (Jin al., 2017) as follows:
G.urtt.p.... [tw + 1,44 + h- 1, + 1, h- 1,m) - I(t - 1, 1. 9- (22)
As we set Cout to be smaller than the output channel dimension of the convolution kernel, we reuse the computation results from the epitome via
Gtwsthotoue x X2i(foo = GL A2 1g.mnm
(23)
where Tour E {0, 1,.,, Reout - 11, Reout [Cout/B2] the number of samplings conducted along the output channel dimension, and n = MMrruut x (2) is the learned mapping along the filter dimension. The filter length 32 21,2,... Couc)
hyper-parameter and decided empirically. In experiments, choose 82 Cout. Thus, the MAdds can be calculated as
Reduced MAdd (2C.. W B2 Fh 1)W NCC WHW FFE F C + 2Rein WHB + Recu B2
(24)
From Eqo. (19)
Fin. (21)
Eqp. (18)
Fige.233
Suppose we use sliding window with stride of and no bias term, the MAdds the conventional convolution can be calculated based on Eqn. (3) as shown below:
MAdd (2 Oin x x 1) H W Cout.
(25)
Therefore, the total MAdd reduction ratio is
Cout MatHW HW.HHW (2Cmn 1)
MAdd Reduction Ratio
out C HWW.WWTTTTT
WC -1) + 2R WHB: + 28ouu
(26)
Coue wh wh CE CEWEHE
Proof on parameter reduction. The parameter compression ratio for 2D convolution layer be calculated as follows:
whCin.oo.
(27)
WE x H x CE x C.E. + x Ruin + Remn!
where and h denote the width and height of the kernel of the layer. We and He denote the spatial size of the corresponding epitome. From Eqn. (26) and Eqn. (27), it observed that the compression ratio mainly Cave wh decided by 6h CR EHE
16


Reviews:
[{"review_id": "rkenIi7GqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors describe a technique for compressing neural networks by learning a so-called Epitome (E), and a transformation function (\\theta) such that the weights for each layer can be constructed using \\theta(E). The epitome and the transformation function can be learnt jointly while optimizing the network for the task specific loss. \n\nThe main idea of this paper is really interesting, and the experimental results which compare against other recent techniques also validate the proposed technique. However, while I think the main idea is relatively clear, I personally found the description of the proposed techniques -- particularly 3.2 and 3.4 -- to be somewhat hard to follow, particularly given that some details only appear in the Appendix. I would suggest that the authors try to revise this section by trying to move Figures 4 and 5 from the appendix into the main text. Some additional suggestions also appear below.\n\nOverall, I would while I like the ideas in this paper, based on the current presentation I am inclined to rate the paper as a “weak reject”, though I would raise my rating if the paper was revised to improve the presentation.\n\nMain comments:\n1. The authors mention that “During inference, (the) routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost.” It would be nice to include some results which indicate what the savings are with the proposed routing map.\n\n2. Section 3.2: There were a few aspects of section 3.2 that I think could be improved for clarity.\nA.) Personally, I found Figure 2 somewhat tricky to follow. I would suggest removing the 3x2 “Epitome” and “Generated Kernel” figures on the extreme right of the image, since I’m assuming the only goal of these is to indicate that “orange” and “blue” correspond to “Epitome” and “Kernel” respectively. I would also suggest mentioning the correspondence of the colors as the first sentence in the caption. Finally, if possible, I would suggest adding a small description alongside the (a), (b), (c) subcaptions: e.g., (a) straightforward but non-differentiable mapping, … , (c) Generated Weigh Kernel.\n\nB.) I believe that the authors use a separate E for each layer, and that Epitomes are not shared across layers. I may have missed this in the text, but it would be useful to clarify this explicitly again in the section.\n\nC.) The “parameterized transformation layer” is mentioned before Equation 3. I think it would be useful to mention that this is implemented using neural networks in your work for clarity. E.g.: “To handle the above two obstacles, ... three\nparts: (1) a parameterized transformation learner η (implemented using a neural network in this work) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”\nor\n“To handle the above two obstacles, ... three\nparts: (1) a parameterized transformation learner η (See Section 3.3) used to learn a set of starting indices for patches in epitome (i.e. all elements in the same epitome patch share identical starting indices); … and an interpolation based generator (Eqn. 3).”\n\nD.) The exact structure of the “parameterized transformation layer” wasn’t exactly clear to me. In 3.3, the authors mention that it consists of “of two convolutional layer, followed by a sigmoid function … takes the feature map of the convolutional layer as input”. Please clarify exactly what is fed in as the input to this network e.g., (input feature map: F, and the indices i,j).\n\n3. I personally also found Section 3.4 which discusses the Computation reduction was also somewhat hard to follow. Some clarification questions: Is the memory/computation cost of the storing/creating the routing map included in the compression calculations? I think it is important to factor these costs when computing the savings achieved by the model. Also, I was unclear on what R_{cin} and R_{cout} are, and why they appear in Equation 5. Could the authors please clarify. \n\nMinor Comments:\n1. Abstract: “Traditional compression methods … all assume that network architectures\nand parameters should be hardwired.” What does it mean for them to be “hardwired” in this context?\n2. Abstract: “Experiments demonstrate that, … with 25% MAdd reduction and AutoML for Model Compression (AMC) by 2.5% with nearly the same compression ratio.” --> “Experiments demonstrate that, … with 25% MAdd reduction, and a 2.5% Madd reduction for AutoML for Model Compression (AMC) with nearly the same compression ratio.”\n\n------- Update after Author Response --------\nI would like to thank the authors for their responses and for the updates which strengthen the paper in my view. I have updated my score accordingly.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1lORAAAcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the problem of  neural network compression, and proposes a new scheme, the neural epitome search. It learns to find compact yet expressive epitomes for weight parameters of a specified network architecture. The learned weight tensors are independent of the architecture design.  It can be encapsulated as a drop in replacement to the current\nconvolutional operator. It can incur less performance drop. Experiments are conducted to show the effectiveness of the proposed method. However, there are some concerns to be addressed.\n-It is not too clear how to learn the epitomes and transformation functions.\n-Authors stated that the proposed method is independent of the architecture design. From the current statements, it is not explained clearly.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gMrWSXFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors learn epitomes, which are small weight tensors which can be used with a learnt transform to produce tensors of an appropriate size (e.g. the sizes used in MobileNet v2).  This gives a reduction in the number of parameters required, and the number of MAdds in theory.\n\nThis paper is badly written, and could do with a rewrite:\n\n- Citations are used incorrectly (\\cite should be used when the citation is meant to be read as part of the sentence). \n- \"less elements\" --> \"fewer elements\"\n- \"misuse the notion\" --> \"abuse the notation\"? \n- \"for fair comparisons\" --> \"for a fair comparison\"\n\nThe method is poorly explained; I have read Section 3.2 several times, and I'm still not entirely certain of what's going on. Figure 1 is helpful, but Figure 2 is not, and could be redesigned. 2(b) makes it look like you are going from a 3x3 epitome to a 2x2 kernel, which is clearly not what is happening. I think it would be helpful to give a detailed pictoral example of an epitome mapping to a weight tensor, with arrows between relevant indices changing. \n\nOn initial reading, I thought the method allowed dynamic allocation of your epitomes to lots of different tensor sizes. From what I can tell, the network has to be trained from scratch for each possible size, so it isn't flexible in that respect.  From what I can gather, the paper is presenting an alternate approach to *downscale* networks, as opposed to say, reducing width or depth. The comparisons to different widths of MobileNet v2 make more sense under this scenario.\n\nThe main sell of the methods appears to be on the basis of MAdd reduction. This makes me nervous, as it doesn't necessarily correspond to actual speed-up or a reduction in energy (see https://arxiv.org/abs/1801.04326). EfficientNet was mainly about Madds too, but they provided some inference times. Would it be possible to add these? The method used with the integral image sounds expensive.\n\nThe results look good, but error bars,on the CIFAR experiments at the very least, would be appreciated.\n\nPros:\n-------\n- Good results\n- Method appears largely novel (although bears some resemblance to https://arxiv.org/abs/1906.04309)\n\nCons:\n--------\n- Badly written\n- The method is poorly explained\n- Uncertainty re: MAdds as a primary comparator\n\nI propose a weak reject for this paper for two primary reasons:\n1) The standard of writing, and the explanation of the all-important method are not up to scratch for a top tier conference\n2) I have concerns regarding the MAdd calculations. Perhaps you could provide some pseudo-code in the author response?\n\nI am happy to upgrade my score if the authors deal with these issues sufficiently.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a novel way to compress arbitrary networks by learning epitiomes and corresponding transformations of them to reconstruct the original weight tensors. The idea is very interesting and the paper presented good experimental validations of the proposed method on state-of-the-art models and showed good MAdd reduction. The authors also put a lot of efforts addressing the concerns of all the reviewers by improving the presentation of the paper, which although can still be further improved, and adding more explanations and validations on the proposed method. Although there's still concerns on whether the reduction of MAdd really transforms to computation reduction, all the reviewers agreed the paper is interesting and useful and further development of such work would be useful too. 
Decision: Accept (Poster)