FSNet: Compression of Deep Convolutional Neural Networks by Filter Summary
Authors: Yingzhen Yang
Jiahui Yu
Nebojsa Jojic
Jun Huan
Thomas S. Huang
Venue:  (2020)
Keywords: Compression of Convolutional Neural Networks
Filter Summary CNNs
Weight Sharing

Abstract: We present a novel method of compression of deep Convolutional Neural Networks (CNNs) by weight sharing through a new representation of convolutional filters. The proposed method reduces the number of parameters of each convolutional layer by learning a $1$D vector termed Filter Summary (FS). The convolutional filters are located in FS as overlapping $1$D segments, and nearby filters in FS share weights in their overlapping regions in a natural way. The resultant neural network based on such weight sharing scheme, termed Filter Summary CNNs or FSNet, has a FS in each convolution layer instead of a set of independent filters in the conventional convolution layer. FSNet has the same architecture as that of the baseline CNN to be compressed, and each convolution layer of FSNet has the same number of filters from FS as that of the basline CNN in the forward process. With compelling computational acceleration ratio, the parameter space of FSNet is much smaller than that of the baseline CNN. In addition, FSNet is quantization friendly. FSNet with weight quantization leads to even higher compression ratio without noticeable performance loss. We further propose Differentiable FSNet where the way filters share weights is learned in a differentiable and end-to-end manner. Experiments demonstrate the effectiveness of FSNet in compression of CNNs for computer vision tasks including image classification and object detection, and the effectiveness of DFSNet is evidenced by the task of Neural Architecture Search.

Published as a conference paper at ICLR 2020
FSNET: COMPRESSION OF DEEP CONVOLUTIONAL NEURAL NETWORKS BY FILTER SUMMARY
Yingzhen Yang' Jiahui Yu², Nebojsa Jojic Jun Huan", Thomas S. Huang?
1 School of Computing, Informatics, and Decision Systems Engineering, Arizona State University y yingzhenn.yaan yangganunuaa.
2 Beckman Institute, University of Illinois at {jyu79, t teettaagggeeeiinneeeeoooo iiiiiiniiiiiiieeeiiieiiiiiiiiiii edu
3 Microsoft Research
jojicêmi crosoft.com com
Styling Inc.
lukehuan& wwweeneeeeeeeeneen.eee..ee com
ABSTRACT
We present novel method of compression of deep Convolutional Neural Networks (CNNs) by weight sharing through new representation of convolutional filters. The proposed method reduces the number parameters of each convolutional layer by learning 1D vector termed Filter Summary (FS). The convolutional filters are located in FS as overlapping 1D segments, and nearby filters in FS share weights in their overlapping regions in natural way. The resultant neural network based on such weight sharing scheme, termed Filter Summary CNNs or FSNet, has FS in each convolution layer instead of set of independent filters in the conventional convolution layer. FSNet has the same architecture as that of the baseline CNN to be compressed, and each convolution layer of FSNet has the same number of filters from FS as that of the basline CNN in the forward process. With compelling computational acceleration ratio, the parameter space of FSNet is much smaller than that of the baseline CNN. In addition, FSNet is quantization friendly. FSNet with weight quantization leads to even higher compression ratio without noticeable performance loss. We further propose Differentiable FSNet where the way filters share weights is learned in a differentiable and end-to-end manner. Experiments demonstrate the effectiveness of FSNet in compression of CNNs for computer vision tasks including image classification and object detection, and the effectiveness of DFSNet is evidenced by the task of Neural Architecture Search.
INTRODUCTION
Deep Convolutional Neural Networks (CNNs) have achieved stunning success in various machine learning and pattern recognition tasks by learning highly semantic and discriminative representation of data (LeCun et al., 2015). Albeit the power of CNNs, they are usually over- aver-parametriiee and of large parameter space, which makes it difficult for deployment of CNNs on mobile platforms or other platforms with limited storage. Moreover, the large parameter space of CNNs encourages researchers to study regularization methods that prevent overfitting (Srivastava et al., 2014).
In the recently emerging architecture such as Residual Network (He al.. 2016) and Densely Connected Network (Huang et al., 2017), most parameters concentrate on convolution filters, which are used to learn deformation invariant features in the input volume. The deep learning community has developed several compression methods of reducing the parameter space of filters and the entire neural network, such as pruning (Luo et al., 2017: Li et al., 2017: Anwar et al., 2017), weight sharing and quantization (Han et al., 2016; Tung & Mori, 2018; Park et al., 2017) and low-rank and sparse representation of the filters (Ioannou et al., 2016; Yu et al., 2017). Quantization based methods (Tung & Mori, 2018; Park al., 2017) may not substantially improve the execution time of CNNs, and many methods (Lebedev & Lempitsky, 2016; Zhang et al., 2018) have also been proposed to improve the inference speed of CNNs.

---

Published as a conference paper at ICLR 2020
Weight sharing has been proved to be effective way of reducing the parameter space of CNNs. The success of deep compression (Han et al., 2016) and filter pruning (Luo et al., 2017) suggest that there is considerable redundancy in the parameter space of filters of CNNs. Based on this observation, goal of compression can be achieved by encouraging filters to share weights.
In this paper, we propose novel representation of filters, termed Filter Summary (FS), which enforces weight sharing across filters so as to achieve model compression. FS a 1D vector from which filters extracted overlapping ID segments. Each filter in the form of 1D segment can be viewed as an "unwrapped" version of the conventional 3D filter. Because of weight sharing across nearby filters that overlap each other, the parameter space of convolution layer with FS much smaller than its counterpart in conventional CNNs. In contrast, the model compression literature broadly adopts a two-step approach: learning large CNN first, then compressing the model by various model compression techniques such pruning, quantization and coding (Han et al., 2016; Luo et al., 2017), or low-rank and sparse representation of filters (Ioannou et al., 2016; Yu al., 2017).
Our FSNet is novel with the following contributions:
We propose Filter Summary (FS) as a compact 1D representation for convolution filters. Filters in Filter Summary are 1D segments, as opposed to the conventional representation of filters as 3D arrays. FS is quantization friendly so to achieve significant compression ratio.
We propose fast convolution algorithm for convolution with FS, named Fast Convolution by Filter Summary (FCFS), taking advantage of the 1D representation of filters in FS. Our FCFS is significantly different from the integral image method for fast convolution in Weight Sampling Network (WSNet) (Jin et al., 2018). The integral image method for WSNet is primarily designed for 1D CNNs where filters are intrinsically 1-dimensional and 1D convolution performed. Such method cannot be directly extended to handling the convolution in regular 2D CNNs where filters are 3-dimensional an efficient manner. Although one can approximate higher-dimensional convolution by 1D convolution, the approximation error is inevitable. By representing filters as overlapping 1D segments in 1D space, FCFS performs exact convolution with compelling computational acceleration ratio, which is the product of the compression ratio for convolution and the first spatial size of filter.
In order to learn optimal way filters share weights, propose Differentiable Filter Summary Network, DFSNet. DFSNet represents the location of filters in each FS as Iearnable and differentiable parameters, that the location of filters can be learned and adjusted accordingly. In this way, potential better location of filters can be obtained by training DFSNet in an end-to-end manner so as to guarantee the competitive accuracy of DFSNet.
1.1 RELATED WORKS
The idea of using overlapping patterns for compact representation of images videos is presented in Epitome (Jojic et al., 2003), which is developed for learning a condensed version of Gaussian Mixture Models (GMMs). FSNet is inspired by Epitome and uses overlapping structure to represent filters. Weight Sampling Network (WSNet) (Jin et al., 2018) also studies overlapping filters for compression of 1D CNNs. However, WSNet is not primarily designed for regular 2D CNNs, and its fast convolution method cannot be straightforwardly extended to the case of convolution in regular 2D CNNs without approximation.
1.2 NOTATIONS
Throughout this paper, we use m to indicate integers between m and n inclusively, and [n] is defined as 1 n. We use subscripts to indicate the index of an element of vector, and Vmn indicates vector consisting of elements of with indices in m n.
FORMULATION
We propose Filter Summary Convolutional Neural Networks (FSNet) in this section. The Filter Summary (FS) is firstly introduced, and then the fast convolution algorithm named Fast Convolution by Filter Summary (FCFS) is developed. We then introduce the training of FSNet, FSNet with Weight Quantization (FSNet-WO) and Differentiable FSNet (DFSNet).
2

---

Published as a conference paper at ICLR 2020
2.1 FILTER SUMMARY (FS)
We aim to reduce the parameter space of convolution layers, or convolution filters. We propose Filter Summary (FS) as a compact representation of the filters. The filters are overlapping ID segments residing in the Filter Summary. Each filter in a FS is 1D segment comprised of consecutive elements of the FS. It is widely understood that a regular convolution filter is a 3D array with two spatial dimensions and one channel dimension. In FS, each segment representing a filter can be viewed as the "unwrapped" version of the corresponding regular 3D filter by concatenating all the elements of the 3D filter into a vector in channel-major order. In the sequel, filter has a 1D representation as a segment in FS without confusion. Figure ((a) shows an example of a FS. Figure ((b) shows how filters are located in the FS.
Filer
1-th
Filter
1+(f-1xa 1-1xy I+i-DDs
File Suammay Compact Aeprossttiin Fikm
Locriss efinia in Fikez Surueary
(a)
(b)
Figure 1: (a) Illustration of Filter Summary (FS). The two filters marked in red and green two overlapping segments in the FS. The two filters share two weights, indicated as red and yellow dots, in their overlapping region. In this example, the stride for extracting filters s K 2 for illustration purpose, since every two neighboring filters share two weights. (b) Illustration of location of filters in the FS. The filter stride is
[-1 J. The i-th segment or filter in the FS has K elements with indices 1 + (i - 1)s: (i - 1)s + K - 1, for i e [Cout]. In this way, the Cou filters reside in the FS in weight sharing manner. Padding is used so that the last filter has valid elements.
Formally, let a convolution layer have Cout filters of size Cin x S1 x S2 = K where S2) is the spatial size and Cn the channel size. So each filter has K elements, and the total number of parameters of all the filters are KCout. We propose compact representation of these parameters, which is a 1D vector named Filter Summary (FS), denoted by F. Suppose the goal is to reduce the number of parameters in the convolution layer, i.e. K KCouts by r times, then the length of the FS is CC J. In the following text, is referred to as the compression ratio for convolution.
I,
For the purpose of the fast convolution algorithm introduced in Section 2.2, each filter is rep- resented as 1D segment of length K in the FS. With a defined filter stride
LCom: the i-th filter can be represented a segment of length K staring location 1 + (i 1)s in the FS, i.e. With i ranging over [1... Cout], we have Cout filters residing in the F$ in
F Fi+ (b(f--))):+((<11eee weight sharing manner. Note that padding is adopted so that the last filter, i.e. the Cout-th filter, has valid elements. We ignore such padding for reduced amount of notations in the following text.
Due to weight sharing, the size of FS is much smaller than that of independent Cout filters. It can be seen that FS can be used to compress all the variants of convolution layers in regular CNNs, including convolution layers with filters of spatial size 1 x 1. FSNet and its baseline CNN have the same architecture except that each convolution layer of FSNet has compact representation of filters, namely a FS, rather than a set of independent filters in the baseline. FS is designed such that the number of filters in it is the same as the number of filters in the corresponding convolution layer of the baseline CNN. A more concrete example is given here to describe the compact convolution layer of FSeet. Suppose that a convolution layer of the baseline CNN has 64 filters of channel size 64 and spatial size 3 x 3, and the compression ratio for convolution is r = 4. Then the corresponding
convolution layer in the FSNet has a FS of length 61x61x3x3 9216 L. The 64 filters of size
64 x 3 576 are segments located by striding along the FS by s 64
9215
143 elements.
In order to keep notations simple, we let
Please refer to more details in the appendix.
C.out
3

---

Published as a conference paper at ICLR 2020
slice of feature patch
feature patch
Inpur feature
Unwrapped input map
Figure 2: Illustration of unwrapping of the input feature map.
The equivalent 3D filter
slice of the equivalent 3D filter
slice patch
Filter Summary
1L,- 1L
Reature
Unwrapped input feature map
Figure 3: Illustration of efficient convolution using the Filter Summary (FS).
2.2 FAST CONVOLUTION FILTER SUMMARY (FCFS)
Representing filters overlapping segments in FS leads accelerated convolution, which is detailed in this subsection. Convolution computes the inner product between patches of the input feature map and all the filters of convolution layer. Feature patches usually share weights due to small convolution stride. Therefore, for the case of FS wherein filters also share weights, conventional convolution unnecessarily computes the product between elements of feature patches and elements of the filters many times when these elements of features patches or filters are shared weights.
To handle this problem, we propose Fast Convolution by Filter Summary (FCFS) which avoids the unnecessary computation. The 1D representation of filters facilitates the design of FCFS. We unwrap the input feature map of size Cin D2 D: and reshape Here (D,, D2) is the spatial size of the input feature map. The unwrapping of the feature map is in -hannel-maaor order, and element with indices (V, k) corresponds to M D Ci.t- Therefore, different slices of feature patch of size Cin X Si S2 appear in different locations of M. For example, in Figure 2, two slices, marked in green and blue, of a feature patch are two segments of where each segment is of size S1.
1D vector, denoted by M.
Let F denote the unwrapped version of equivalent conventional 3D filter in channel-major order, different slices of the equivalent 3D filter are different segments of FS, as illustrated by the upper part of Figure 3. To circumvent unnecessary multiple computation of product between weights shared in
length
the convolution layer. Each filter the FS can also be viewed as
4

---

Published as a conference paper at ICLR 2020
feature patches and filters, we compute the product between all the elements of M and F, and store the result as a matrix A with A, = M,E,, [C E [L]. Because each slice of a feature patch or filter is segment of M F, it can be verified that each line inside A which is either the principal diagonal of A or parallel to the principal diagonal contributes to the inner product between slices of feature patch and that of filter. Such lines are defined as the critical lines of A. Two examples of the critical lines are illustrated in Figure 3 by dashed lines in green and blue that cross A, where the dashed line in red is the principal diagonal of A and the dashed line in blue is parallel to the principal diagonal. For each critical line of A denoted by a vector I of length T, we compute its
1D integral image or integral line denoted by IL, where II, = Σ J, i E [T]. The integral lines are t=1
co-located with their corresponding critical lines. Figure 3 illustrates the integral line for the principal diagonal of A. It can be seen that the inner product between a slice of feature patch and slice of filter FS can computed efficiently by a subtraction along an integral line, For example, Figure illustrates case where the inner product of a filter in green and a slice of a filter can be computed as Th 11, where L in this example the integral line for the principal diagonal of A.
Therefore, one can efficiently compute the inner product between feature patch and filter by summing the inner products for their S₂ pairs of slices, where one pair of slices has a slice of the feature patch and one proper slice of the filter. By concatenating such results for all feature patches and filters, one obtain the final result of the convolution between the input feature map and all the filters in FS in efficient manner. After the above full description of FCFS, it can be verified that FCFS only needs to compute U. L' elements for each row of A. This is due to the fact that different slices of filters are separated by multiplies of CinS1 elements in F. It should be emphasized that FCFS accelerate convolution only for convolution layer with kernel size S2 1. The formal formulation of FCFS is described by Algorithm and Algorithm 2 in Section A.2.
2.3 COMPUTATIONAL ACCELERATION RATIO OF FCFS
The computational complexity of conventional convolution for an input feature map of size Cin x
D1 D2 and Cout filters, each of which has size of Ci S1 x S2 K, is Cout D1 D2K. It can be verified that the FCFS algorithm in Section 2.2 requires 20 D2L' + Cout D2S2 steps. Such complexity result can be obtained by noticing that FCFS has three stages. The first stage is the computation of A which takes Cia D2 D2L'' steps since the size of the portion of A contributing to convolution result is Cn DzLL'. The second stage is the computation of integral lines for all the critical lines of A which also takes Ci, D2 steps. The third stage is to compute the inner product between every feature patch and every filter by summing the inner products of their corresponding S2 pairs of slices, and it takes S2 steps for the inner product of a feature patch and a filter by virtue of the integral lines. Concatenating such results form the final result of the convolution. The computational complexity of the third stage is Cout Dj D2S2. If we only consider the steps involving foating-point multiplication, then the computational complexity of FCFS is Cn D2 Dz + Cout D2 D2S2 since the second stage only has adddttion/uubraation operations. Therefore, the computational acceleration ratio FCFS with respect to the conventional convolution is
Cout D2 D2K Oin DDD''' + Cou Di
Because K >> S2 in most cases, the above acceleration ratio is approximately Coa Dy D2K Ci,, D. D.D,D,D D2 = rS1, which is the product of S and the compression ratio for the convolution layer. Note that the depthwise separable convolution in MobileNets (Howard et al., 2017) has computational acceleration ratio of around Si x S2, and FCFS has even faster convolution when the compression ratio r > S2.
2.4 TRAINING FSNET
Given bascline CNN, we design FSNet by replacing each the convolution layer of the baseline CNN by convolution layer with FS. The weights of FSNet are trained using regular back procppropagation. The detailed training procedure is described in Section 3.
TThis is reasonable since the Cout D, D2K steps for the conventional convolution also ignores addition operations.
5

---

Published as a conference paper at ICLR 2020
2.5 FSNET WITH WEIGHT QUANTIZATION (FSNET- WQ)
The model size of FSNet can be further compressed by weight quantization. FSNet with Weight Quantization (FSNet- WQ) is proposed to further boost the compression ratio of FSNet, without noticeable loss of prediction accuracy shown in our experiments. The quantization process for FSNet described as follows. After a FSNet trained, one-time linear weight quantization is applied to the obtained FSNet. More concretely, 8-bit linear quantization is applied to the filter summary of each convolution layer and the fully connected layer of the FSNet. 256 levels are evenly set between the maximum and minimum values of the weights of layer to be quantized, and then each weight is set to its nearest level. In this way, a quantized weight is represented by a byte together with the original maximum and minimum values stored in each quantized layer. The resultant model is named FSNet- WQ. The number of effective parameters of FSNet- WQ is computed by considering a quantized weight as 1 1/4 parameter since the storage required for byte is 1/4 4 of that for a floating number in our experiments.
2.6 EXTENSION TO DIFFERENTIABLE ESNET (DFSNET)
It can be observed that location of filters determines the way how filters share weights in FS. So far the filters evenly located in FS. It remains an interesting question whether the way filters share weights can be learned, which potentially leads to a better way of weight sharing across the filters. To this end, we propose Differentiable FSNet (DFSNet) in this subsection, where the location of each filter is a learnable parameter and it can be fraction number rather than the integer location considered so far. Formally, the staring location of filter g of length K in FS F of length L is parameterized by a parameter through the sigmoid function as Ite- L, and g can be
represented as g (1 + [4] - f)F [J:[0 KK (1 L()) ( [ K All such specifying the location of filters in FS are learned during the training of DFSNet in an end-to-end manner. is almost impossible be integer, the derivative with respect to almost always exists. Note that
one may not use FCFS to accelerate convolution operation in DFSNet due to the fraction location of the filters.
EXPERIMENTAL RESULTS
We conduct experiments with CNNs for image classification and object detection tasks in this section, demonstrating the compression results of FSNet.
3.1 FSNET FOR CLASSIFICATION
We show the performance of FSNet in this subsection by comparative results between FSNet and its bascline CNN for classification task on the CIFAR- 10 dataset (Krizhevsky, 2009). Using ResNet (He al., 2016) or DenseNet (Huang al., 2017) as baseline CNNs, we design FSNet by replacing all the convolution layers of ResNet or DenseNet by convolution layers with FS. We train the baseline CNNs using 300 training epochs. The initial learning rate is 0.1, and it is divided by 10 at 50% and 75% of the 300 epochs. The test accuracy and the parameter number of all the models are reported in Table I with compression ratio for convolution
In the sequel, CR stands for compression ratio computed by the ratio of the number of parameters of the baseline CNN over that of the obtained FSNet. can be observed in Table that FSNet with a compact parameter space achieves accuracy
Table 1: Performance of FSNet the CIFAR-10 dataset
Before Compression
PS
PSNer-Wo
Model
Param
Accuracy
Param
Accuracy
CR
Param
Accuracy
CR
ReaNei-110
1.74M
93.91%
0.44M
93.81%
3.95
0.12M
93.81%
14.50
ResNet
ReaNes-
1.73M
94.30%
0.47M
94.60%
3.68
0.16M
94.65%
10.81
DenseNet- 100 (k:
1.25M
95.31%
0.37M
94.46%
3.38
0. 15M
94.40%
8.33
DenseNet
DenseNet- 100 (k = 24
4.83M
96.71%
133M
96.40%
3.63
0.44M
95.43%
10.38
comparable with that of different baselines including ResNet-111 and ResNet-164, DenseNet-100
with growth rate k = 12 and & 24. We use the idea of cyclical learning rates (Smith, 2015) for training FSNet. cycles are used for training FSNet, and each cycle uses the same schedule of learning rate and same number of epochs as that of the baseline. A new cycle starts with the initial

---

Published as a conference paper at ICLR 2020
learning rate of 0.1 after the previous cycle ends. The cyclical learning method is only used for FSNet on the CIFAR-10 dataset, and the training procedure of FSNet is exactly the same as that of its baseline throughout all the other experiments.
Table I also shows that FSNet with weight quantization, or FSNet-WO, boosts the compression ratio without sacrificing performance. The effective number of parameters defined in Section 2.5 is reported. FSNer-Wa -Q achieves more than 10x compression ratio for all the two types of ResNet and DenseNet- 100 with A 24, and it has less than 0.4% accuracy drop for ResNet-110 and DenseNet-100 with k = 24. It is interesting to observe that TSNet-WQ even enjoys slight better accuracy than FSNet with
24. In addition, FSNet- WQ does not hurt the accuracy of FSNet for ResNet-110. We argue that weight quantization imposes regularization on the filter summary which reduces the complexity of the filter summary thus improve its prediction performance. We further demonstrate that FSNet and FSNet- WQ achieve better accuracy and compression ratio with respect to the competing filter pruning method (Li et al., 2017) in Table The model size of FSNet-WOO WC is less than 1/10 of that of (Li et al., 2017) while the accuracy is better than that of the latter by 0.5%.
for ResNeet1164 and DenseNet-Iue
Table 2: Comparison between FSNet and the filter pruning method in (Li et al., 2017) on the CIFAR-10 dataset
Performance
Model
# Params
Accuracy
ReaNet-110 10
1.71M
93.91%
Filler Pruming (Li al., 2017)
1.16M
93.30%
FSNett
0.1-1M
93. 81.8
FSN
0.12M
93.81%
Training Less
Training
Test Logs
Error
Epech
Spach
Epach
Spock
Figure 4: From left to right: training loss, training error, test loss and test error of FSNet and the baseline ResNet-50 on the ImageNet dataset. FSNet has a compression ratio for convolution r = 2.
In order to evaluate FSNet on large-scale dataset, Table 3 shows its performance on ILSVRC-12 dataset (Russakovsky et al., 2015) using ResNet-50 as baseline. The accuracy is reported on the standard 50k validation set. We train ResNet-50 and the corresponding FSNet for 100 epochs. The initial learning rate is 0.1, and it is divided by 10 at epoch 30, 60, 90 respectively. We tested two settings with compression ratio for convolution being r = 2 and 3.7, and the corresponding two models are denoted by F and 22 respectively. The training loss/error and test loss/error of F are shown in Figure 4. We can see that the patterns of training and test of FSNet are similar to that of its baseline, ResNet-50, on the ILSVRC-12 dataset. Table 4 shows the comparative results between FSNet-1-Wo and ThiNet (Luo et al., 2017), a competing compression method by filter pruning. We can observe that the quantized FSNet-1 render smaller model size and reduced FLOPs. Here the number of GFLOPs (1 GFLPs 10 FLOPss for FSNet and FS are almost the same, and the details are introduced in Section A.3.
3.2 FSNET FOR OBJECT DETECTION
We evaluate the performance of FSNet for object detection in this subsection. The baseline neural network is the Single Shot MultiBox Detector (SSD300) (Liu et al., 2016). The baseline is adjusted by adding batch normalization (loffe & Szegedy, 2015) layers so that it can be trained from scratch. Both SSD300 and FSNet are trained on the VOC 2007 2207 training datasets, and the mean average precision (mAP) is reported on the VOC 2007 test dataset shown in Table 5. We employ two versions of FSNet with different compression ratios for convolution, denoted by FSNet-1 and FSNet-2 respectively. Again, weight quantization either slightly improves mAP (for FSNet-1), or only slightly hurts it (for FSNet-2). Compared to Tiny SSD (Wong et al., 2018), FSNet-1-WQ enjoys smaller
7

---

Published as a conference paper at ICLR 2020
Table 3: Performance of FSNet ImageNet
Model
Perforaacce
Params
Top-1
Top-5
Table 4: Comparative results of FSNet on ImageNet
ReeNet-50
25.61M
75.11%
92.61 %
Model
Performance
Params
'Top-
GFLOPs
FSNet
73.11%
91.37%
ThiNet (Laso et al., 2017)
12.38M
71.01%
3.41
FSNe1-1-Wo I-QQ
3.55M
72.59%
91. 20%
FSNet-1-WW
3.55M
72.50%
2.47
FSNet-2
8.49M
70.36%
89.79%
FSNGF2-WQ
2.20M
69.87%
89.61%
parameter space while its mAP is much better. Note that while the reported number of parameters of Tiny SSD is 1.13M, its number of effective parameters only half of this number. i.e. 0.565M, as the parameters are stored in half precision floating-poing In addition, the model size of FSNet-1-WQ 1.85MB, around 20% smaller than that of Tiny SSD, 2.3MB. is also interesting to observe that our 2 and FSNet-2-W6 2-NQ are both smaller than MobileNetV2 SSD-Lite (Sandler et al., 2018) with better MAP. Since MobileNetV2 SSD-Lite is believed to newer than MobileNetV1 SSD, the latter is not reported in this experiment.
Table 5: Performance of FSNet for object detection
Model
# Params
mAP
SSD300
26.32M
77.31%
Tiny SSD (Wong al., 2018)
0.56M
61.3%
MobileNet22 SSD-Lite (Sandler 2018)
3.46M
68.60%
FSNet-1
1.67M
67.60%
FSNet-1-We I-WQ
0.45M
67.63%
FSNa-2
2.59M
70.14%
Fe 2 2-Q
0.68M
70.00%
3.3 USING DESNET FOR NEURAL ARCHITECTURE SEARCH
We also study the performance of integrating FS into Neural Architecture Search (NAS). The goal of NAS to automatically search for relatively optimal network architecture for the sake of obtaining better performance than that of manually designed neural architecture. We adopt Differentiable Architecture Search (DARTS) (Liu et al., 2019) as our NAS method due to its effective and efficient searching scheme, where the choice for different architectures is encoded as learnable parameters which can be trained in an end-to-end manner. Since DFSNet is also a differentiable framework for model compression, architecture aiming at great performance. We design a OPSNet-DARTT model by replacing all the 1 x convolution layers, including those for the depthwise separable convolution, in the DARTS search space with FS convolution layers. We perform NAS on the CIFAR-10 dataset using the DFSNet-DARTS model following the training procedure described in DARTS (Liu et al., 2019), and report the test accuracy and model size of the obtained neural network after searching in Table 6. can be observed that the model found by DFSNet-DARTSS has 40% less parameters than that by DARTS while the accuracy loss is only 0.31%, %, clearly indicating the effectiveness of FS convolution in the task of NAS. Please refer to Section A.4 for more details about DASNE--DARTS.
combine DFSNet and DARTS so as to search for compact neural
Table 6: Performance of DFSNet-DARTS on the CIFAR-10 dataset
Model
# Params
Accuracy
DARTS
3. 13M
97. 50%
DESNet-DARTS
97.
CONCLUSION
We present a novel method for compression of CNNs through learning weight sharing by Filter Sum- mary (FS). Each convolution layer of the proposed FSNet learns FS wherein the convolution filters are overlapping 1D segments, and nearby filters share weights naturally in their overlapping regions. By virtue of the weight sharing scheme, FSNet enjoys fast convolution and much smaller parameter space than its baseline while maintaining competitive predication performance. The compression ratio is further improved by one-time weight quantization. Experimental results demonstrate the
8

---

Published as a conference paper at ICLR 2020
effectiveness of FSNet in tasks of image classification and object detection. Differentiable FSNet (DFSNet) is further proposed wherein the location of filters in FS are optimized a differentiable manner in the end-to-end training process, and the performance of DFSNet is shown by experiment with Neural Architecture Search (NAS).
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. J. Emerg. Technol. Comput. Syst., 13(3):32:1332111 February 2017.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning. trained guantization and huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1 704.04861, 2017.
Gao Huang. Zhuang Liu. Laurens der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In IEÉE Conference on Computer Vision and Puttern Recognition, CVPR, Honolulu, HI, USA, pp. 2261-2269, 2017.
Yani Ioannou, Dunean Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 I July 2015, pp. 448 2015.
Xiaojic Jin, Yingzhen Yang, Ning Xu, Jianchao Yang, Nebojsa Jojic, Jiashi Feng, and Shuicheng Yan. Wsnet: Compact and efficient networks through weight sampling. In International Conference on Machine Learning, ICML, Stockholmsmässan, Stockholm, Sweden, pp. 2357-2366. 2018.
Nebojsa Jojic. Brendan J. Frey, and Anitha Kannan. Epitomic analysis of appearance and shape. In 9th IEEE International Conference on Computer Vision ICCV. Nice, France, pp. 34 43, 2003.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Vadim Lebedev and Victor S. Lempitsky. Fast convnets using group-wise brain damage. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp.
2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 5 2015. doi: 10. 1038/nature 14539.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In Yoshua Bengio and Yann LeCun (cds.), 2nd International Conference on Learning Representations, ICLR 2014, Banjf. AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Yang Fu, and Alexan- der C. Berg. SSD: single shot multibox detector. In European Conference Computer Vision, ECCV, Amsterdam, The Netherlands, pp. 37, 2016.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision, ICCV, Venice, Italy, 2017.
E. Park, J. Ahn, and S. Yoo. Weighted-enttoop-boee quantization for deep neural networks. In 2017 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7197-7205, July 2017.

---

Published as a conference paper at ICLR 2020
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, (1(())211-255) 2015.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 4510-4520, 2018.
Leslie N. Smith. Cyclical Learning Rates for Training Neural Networks. arXiv e-prints, arXiv: 186, June 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Rustan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929 -158, January 2014.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in-parallel pruming-qunntzaation In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Alexander Wong, Mohammad Javad Shafiee, Francis Li, and Brendan Chwyl. Tiny SSD: A Tiny Single-shot Detection Deep Convolutional Neural Network for Real-time Embedded Object Detection. arXiv e-prints, art. arXiv:1802.06485, February 2018.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, HI, USA, 2017.
X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6848 6856, June 2018.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jun-Hui Zhu. Discrimination-aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems 31; Annual Conference on Neural Information Processing Systems 2018, NewrlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 883-894, 2018
A APPENDIX
A.1 ADDITIONAL EXPERIMENTAL RESULT
We demonstrate the performance of FSNet in this subsection by more comparative results between ESNet and its baseline CNN for classification task on the CIFAR-10 datasct. Using ResNet (He ct al., 2016) or DenseNet (Huang et al., 2017) as basclines originally designed for ImageNet data, we design FSNet by replacing all the convolution layers of ResNet or DenseNet by convolution layers with FS. We train FSNet and its bascline CNN following the same training procedure specified in the paper, and show the test accuracy and the parameter number of all the models in Table 7.
Table 7: Performance of FSNet the CIFAR- 10 dataset, using larger ResNet and DenseNet baselines
Before Compression
FSNet
FSNet WQ
Model
Param
Accuracy
Param
Accuracy
Faram
Accuracy
ResNet-18
I1.18M
94. 18%
93.93%
13.80
U.22M
93.91%
ResNet
KesNet 34
31.30M
94.72%
L68M
94. 24%
12.68
0.45M
94.327
47.33
K0
23.57M
96. 16%
2.51M
94.91%
9.39
0.72M
94.92%
32.
ResNet-101
42.610 61M
95. 62%
1.81MM
95. 23%
8.80
1.38M
95. 23%
30.88
DenseNer
DenseNet- 121
7.04M
96. 13%
1.24M
95.
5.68
0.44M
95. 13%
16
Because the results of FSNet in Table I are obtained by cyelic training with four eycles, we show the performance of FSNet using the same training procedure (with only one training cycle) as its baseline in Table 8. We can see that the accuracy loss of FSNet is still less than 1% compared to its baseline, and FSNet with one cycle has even higher accuracy (654 than that with four cycles (95)40%) for DenseNet-100 with growth rate of 24.
Furthermore, we show the performance of FSNet with ResNet-56 the CIFAR-10 dataset with different compression ratios, and compare FSNet to Discrimination aware Channel Pruning (DCP) method (Zhuang et al., 2018). It can be observed that with the compression ratio of 4.22 and 4.58, FSNet-1 and FSNet-2 achieve the accuracy of 94.08% and 94. 12% Both of them enjoy smaller models with higher accuracy compared to DCP (Zhuang et al., 2018). In addition, very aggressive compression ratio of 13.76 pushes the accuracy down to 92.87%, and the incurred accuracy loss is still less than 1% compared the original ResNet-56.
10

---

Published as a conference paper at ICLR 2020
Table 8: Performance of FSNet with one-cyele (the same training procedure as the haseline) on the CIFAR-10
dataset
Betore Compression
FSNet
FSNet-WQ
Model
Param
Accuracy
Param
Accuracy
Param
Accuracy
ResNet. 110
1.74M
93.91%
0.44M
92.022%
3.95
0.12M
93.00%%
14.50
ResNet
ResNet- 104
1.73M
94.39%%
0.47M
93.70%
3.68
0. 1.T
93.84%
10.81
DenseNet: 100 (k
1.25M
95.31%
94.36%
3.38
0.15M
94.31%
DenseNet
DenseNet-100 (A+ 24)
4.83M
95.71%
1.33M
95.4
3.63
0.41M
95.
10.98
Table 9: Comparison between FSNet and the Discrimination-awarr Channel Pruning (DCP) method (Zhuang et al., 2018) on the CIFAR-10 dataset, where the x shows the compression ratio (the ratio of the number of parameters of the original model to that of the compressed model).
Performance
# Params
Accuracy
Model
RexNet-56 56
0.86M
93.80%
DCP (Zhuang et al., 2018)
3.37
93.81%
FSNet-1
4.22x
94.08%
FSNet-2
4.58x
94.12%
FSNer-3
4.94
93.74%
PSNet-4
13.76x
92.87%
Table 10: Performance of FSNet- WQ and Linear Quantization on CIFAR-10
Params
Accuracy
Model
ResNet-164
1.73M
94.39%
FSNet-WQ
0.16M
9-1.6%%
Lineur Quantization
0.5UM
94.36%
Table 11: Performance of FSNet- WQ and Linear Quantization ImageNet
Model
Params
Top-1
Top-5
ResNer-50
25.61M
75.11%
92.61%
FSNet. WQ
3.55M
72.59%
91. 20%
Linear Quantization (8-bii)
8.56M
74.88%
92. 48%
Linear Quantization (4-bit)
5.71M
0.086%
0.578%
We are also interested in the visualization of the learned Filter Summary. For the visualization purpose. we change the representation of a Filter Summary from ID vector to 3D tensor. We design a 3D Filter Summary (FS) for the first convolution layer (convl) Network Network (NIN) (Lin et al., 2014) on the CIFAR-10 dataset. The conv1 of the original NIN has 192 filters of spatial size 5 x and channel size 3. The 3D Filter Summary for convl of the FSNet version of NIN is a 3D tensor of size 24 x 8 x 3, with spatial size of 24 x 8 extracted as overlapping 3D tensors from the 3D FS. The illustration of the learned 3D FS for convl of the FSNet version of NIN and the independent 192 filters learned convl the original NIN are shown Figure S. We can observe that, compared to the independent 192 filters, the filters in the 3D FS are "smoothed" by the weight sharing scheme because patterns of one filter are shared across its neighboring filters in the 3D FS.
and a channel size of 3. In the forward process, 192 filters of size 5 5 x
A.2 MORE DETAIL. ABOUT FILTER STRIDE 8 IN FAST CONVOLUTION BY FILTER SUMMARY (FCFS)
In the paper, it mentioned that the filter stride is set to s 1-11 J. In order to accelerate the convolution operation using FS for convolution layer with Sa 1. require that be the largest multiple of C,,.,, which no greater than ا-ع Namely, let $1 = Cout Couut
_-1 J. then = |T CI. The reason is that FS can be
viewed as the unwrapped version of conventional 3D filter by shaping its elements into vector in the order of channel, the first spatial dimension (with size Si, the second spatial dimension (with size S2). Therefore, different slices along the second spatial dimension are separated by Sn elements in the FS. By letting & be the multiple of SI, we only need to compute c = I,' elements for each row of matrix A.
Algorithm describes the algorithm which computes matrix for FCFS. Algorithm describes the FCFS algorithm which computes the convolution between the i-th filter in the FS of a convolution layer of FSNet and the input feature map M of size Cia Dy D2 x for that convolution layer. The output feature map for that convolution layer of FSNet is obtained by running Algorithm 2 for all E [Cour]. Throughout this paper, we let the convolution stride he 1. FCFS be he used accelerate convolution with different convolution strides.
11

---

Published as a conference paper at ICLR 2020
10
20
(a)
(b)
Figure (a) Illustration of the learned 3D Filter Summary (FS) for the first convolution layer of Network In Network (NIN) (Lin et al., 2014), where 192 filters of size 5 x 5 3 are extracted as overlapping 3D tensors from the 3D FS in the forward process. (b) The learned 192 independent filters in the first convolution layer of the original NIN.
Algorithm Compute the matrix A for FCFS
Input:
The filter summary F of size L = Cout K in a convolution layer of FSNet, the input feature map M of size Cn DD x 1.
1: for 1 < Cn do j == mod Cin
3:
while j < do
4:
A M.F,
5: j Cn.S
6: end while
end
8:
for Each critical line I of do
Compute 1D integral image
I, where IL, Σ Ic, E [T] and the length
10:
Assign L; to the co-located element of A
11: end for
Output: The matrix A
A.3 MORE DETAILS ABOUT FLOPS OF ESNET-WO WQ
It is mentioned that FSNe--WQ has almost the number of FLOPs that of FSNet. The underlying explained in detail in this section. Suppose a convolution layer or fully-connected layer of FSNet performs the following operation:
Wx + b,
(I)
where x is vector representing the input of this layer, y is the output, and the operation performed by this layer represented hy linear function parameterized by W and b. Let W, and be denote the quantized W and
12

---

Published as a conference paper at ICLR 2020
Algorithm 2 Convolution between the i-th filter in the filter summary and the input feature map for convolution layer of FSNet
Input:
The filter summary F of size L = Cout K in convolution layer of FSNet, the filter index i E [Cout], the input feature map M of size Cin D2 D2 the matrix A computed by Algorithm
1:
Build matrix O of size
D2 with zero initialization
Obtain padded input feature map of size Cin (D1 + S 1) S2 1) 1, which still denote by M
3:
for < V Dj do D2 do
4:
for
for 1 S2 do
6:
t= (n+ D + (m 1)CIn
7:
Obtain the inner product between the slice of the input feature patch Mc Ci, and the corre- sponding slice of the i-th filter, i.e. (1+(1=1)++(+
using the corresponding
critical line of A
Omn mm + Ph
9: end for
10:
end for
11: end for
Output: Return as i-th channel of output feature map of convolution layer FSNet
after performing the 256-1evel linear quantization described Section 2.5 of paper. Then recovered weights from the quantized weights are
W So - + 7bo,
where So is matrix of the same size W with all elements being so which is the smallest element of W, vector of the size will all elements being 8g- is the quantization step computed by 255 and 81 is maximum element W. can verified FSNet- WQ performs the following operation in this layer using the recovered weights W and b:
21_40
ÿ = Wx + Sox + T bu).
(3)
We can perform the operation yo = Wox bo using the same number FLOPs as that required for Wx Then each element of the output of this layer can be computed by y; = (yo): + so(s2: + 1) where sr is the sum of elements of the input Therefore, compared to FSNet, FSNet-WQ only needs additional FLOPs to compute the sum of elements of the input (plus layer. Our empirical study shows that the FLOPs of FSNet-WO WQ and FSNet are almost the same.
multiplication) for each convolution layer or fully-connected
After FSNet is trained, one-time linear weight quantization applicd to the obtained FSNet. More concrctely, 8-bit linear quantization is applied to the filter summary of each convolution layer and the fully connected layer of the FSNet. More comparative results between FSNet-WQ and linear quantization on the CIFAR-11 10 data set and the ILSVRC-12 dataset shown in Table 10 and Table 11 respectively. Because linear quantization used FSNet-Wo WQ, the compression results of FSNet- WQ are compared that linear quantization.
A.4 MORE DETAILS ABOUT DFSNET-DARTS
We use SGD with momentum 0.9 to optimize the weights of the DFSNet-DARTS model on the CIFAR- 10 dataset. The initial learning rate is 0.01, and the learning rate is gradually reduced to zero following cosine schedule. The weight decay is set to 0.0002. The DFSNe--DARTS model is trained for 600 epochs with mini-hatch size of 96. The neural architecture found by DESNet-DARTS is illustrated by Figure 6.
13

---

Published as a conference paper at ICLR 2020
can,
C
mag. DORANT
B) Nortaal
poot 30
E-l Jor Rec.
mau paol
Figure 6: The neural architecture found by DFSNet-DARTS
14


Reviews:
[{"review_id": "SkxyixFH6H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper authors propose a novel idea (called Filter Summary, FS)  how to compress convolutional neural networks with 2D convolutions (kernels are 3D tensors, it is also applicable to the 1D convolutions). Compression of the convolution operation is done with weight sharing: unwrapped kernel (channel-major) is packed into 1D vector by having intersected segments (shared weights). \n\nThe strong sides of the paper are the following:\n- Proposed algorithm for efficient computation of convolution operation with FS (less number of multiplication operations with comparison to the standard convolution).\n- Via experiments it is demonstrated that proposed approach provides compression of the model while having close to the baseline quality for image classification and object detection tasks and for small and large datasets.\n- Proposed method gives better compression factor and better quality than state-ot-the-art models for classification and object detection tasks.\n- It is experimentally showed that architecture search works for proposed convolution.\n\nThere are still ways to improve the paper:\n- In the introduction section it is written that \"FS is quantization friendly\". I would say the main concern should be \"FS quantization achieves the same quality as original FS while having higher compression factor\". Standard convolutions are also quantization friendly, no? At the same time I don't see huge gain to do quantization of the model. Yes, to store it we need less memory, however, for the inference we still use float-point computation. How will this quantization be helpful for inference on CPU/GPU (memory/speed)?\n- Did you implement CUDA kernel to have efficient FS convolution (I suspect the answer yes)? It would be interesting to see the training time per epoch/forward/backward comparison for standard CNN and FS CNN as soon as you are using the same training process (for models you trained to have an idea for practical usage). Also it is interesting to see comparison of inference time on CPU/GPU. \n- For illustrations (figs 1, 2, 3) of input and filters packing it would be very helpful to mark where are channel and spatial dimensions on the figure for simpler understanding the packing. Typos for the figures: \"in Figure 2, two slices, marked in green and blue\" -> \"in Figure 2, two slices, marked in green and red\"; \"Figure 3 by dashed lines in green and blue that cross A, where the dashed line in red\" -> \"Figure 3 by dashed lines in green and blue that cross A, where the dashed line in green\" (on Figure 3 there are blue and green lines only).\n- In the paragraph where the number of elements of matrix A is specified to be computed, it would be better to have detailed explanation why we skip C_in*S_1 elements. It is not so trivial when you read the paper the first time.\n- In section 2.3 for complexity of the three stages: seems the third stage doesn't have float-point computations (this is addition for S_2 locations in matrix A), however, the third stage is included into analysis of multiplication operations (this doesn't affect the final result of complexity)\n- For DFSNet are filters ordered for FS F (monotonically located)? Or now they can be located independently and starting point defined by its alpha?\n- Typo: page 7 \"in Table 9\" -> \"in Table 2\"\n- In Table 4 it would be better to have results for FSNet-1 too (without quantization).\n- In section 3.3 there is no details how the alpha parameter from DFSNet is used in DARTS.\n- In Tables 10 and 11 there are linear quantization, what does it mean? what is the difference in quantization procedure with FSNet-WQ (maybe I missed something in the paper).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkehCgjltr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel compact parameterization of convolution filters. Normally a filter bank is stored as a 3D array of shape C_out x C_in x H x W = K. This paper proposes to use a flat array instead, and obtain each of the C_out filters of shape C_in x H x W by indexing this array with stride. Thus, adjacent filters will share some of their weights. A fast algorithm is presented for convolving using filters parameterized in this way. An extension is presented where the location of each filter in the flat array is learned. The results show the potential of the method to reduce the number of parameters at a modest drop in accuracy, though it is not clear how the method stacks up against state of the art, or what the improvement in wall-clock runtime is. Overall, I would rate this paper \"weak accept\".\n\nOne minor weakness of the experiments on classification (3.1) is that the training procedure for FSNet and the baselines are different, with FSNet using cyclic learning rates and a larger number of epochs, making the results somewhat difficult to interpret. I suppose that the authors tried using the same training procedure but it did not work for FSNet - is that correct? If not, reporting those numbers would be preferable. If indeed the same training procedure does not work for FSNet that is not a fatal flaw, but we would want to see results for the baseline architectures trained in the same way as FSNet (to make sure the difference is not attributable to the training procedure). Also, we would want to make sure that the baseline results as reported are about as good as they can get for that architecture, e.g. by comparing to results for those architectures published by others, and by copying their training procedure. (I think the baseline results are fine, but it would be good to take away any doubt in the reader's mind).\n\nThe paper has a clear structure and is fairly well written, though it may still be beneficial to go over the text with a native speaker.\n\nOverall, I find that the experiments convincingly show that FSNet provides a way to compactly store filters with modest loss of accuracy. I am not an expert in this field though, and so I don't know how it relates to state of the art. One paper that may be good to compare to is \"Discrimination-aware Channel Pruning for Deep Neural Networks\" by Zhuang et al.\n\nIt would be nice to see wall-clock time / speed improvements reported, instead of only reporting the reduction in parameter count.\n\nTypos & minor issues\nThe paper uses \\cite{} in many places where \\citep{} would be more appropriate.\n\"fist spatial size\"\n\"each filer in a\"\n\"substraction\"\n\"epoches\"\n\"efficiency searching scheme\"\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "B1l0SlB6YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method, termed as Filter Summary (FS), for weight sharing across filters of each convolution layer. Moreover, a fast convolution algorithm is designed for the convolution layer with the FS. Some promising results demonstrate the effectiveness in CNN compression and the acceleration on the tasks of image classification, object detection and neural architecture search. I like the idea of weight sharing which seems like a reasonable choice for model compression. I expect it to be a general and standard component for model compression and acceleration. I have the following concerns:\n\n1.FSNet achieves a small model (FSNet-2-WQ in Table 5) with only 0.68M parameters and an mAP of 70.00% on the VOC2007 dataset. Can the authors include the model size of FSNet-2-WQ in MB so that the model size can be directly compared to that of the recently proposed YOLO Nano [1] (with a size of 4MB and an mAP of 69.1% on VOC 2007 dataset)? \n\n2. The backbone of FSNet-2-WQ in Table 5 is SSD300, which is a relatively large model for object detection. Please shed more insights on why weight sharing by Filter Summary can have such a high compression ratio, even with the help of quantization.\n\n3 Please explain why the simple linear quantization is compatible with Filter Summary for image classification and object detection. More concretely, why the simple linear quantization does not hurt the performance of FSNet on these tasks? Can similar results be extended to more models and tasks?\n\n4 Please show the illustration of the learned Filter Summary which can help understand the structure of the filters in this new representation.\n\n5 It could be better if the paper can reveal the correlation between the compression ratio and the accuracy of the compressed models by FSNet.\n\n[1] Wong et al. YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detection. arXiv: 1910.01271\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to compress convolutional neural networks via weight sharing across filters of each convolution layer. A fast convolution algorithm is also designed for the convolution layer with this approach. Experimental results show (i) effectiveness in CNN compression, (ii) acceleration on the tasks of image classification, object detection and neural architecture search. While the authors addressed most of reviewers' concerns, the weakness of the paper which remains is that no wall-clock runtime numbers (only FLOPS) are reported - so efficiency of the approach in practice in uncertain.

Decision: Accept (Poster)