Title: WSNET: COMPACT AND EFFICIENT NETWORKS WITH WEIGHT SAMPLING

Abstract: We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via ad hoc processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces parameter sharing throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to 180× smaller and theoretically up to 16× faster than the well-established baselines, without noticeable performance drop.

1. INTRODUCTION:

Despite remarkable successes in various applications, including e.g. audio classification, speech recognition and natural language processing, deep neural networks (DNNs) usually suffer following two problems that stem from their inherent huge parameter space. First, most of state-of-the-art deep architectures are prone to over-fitting even when trained on large datasets (Simonyan & Zisserman, 2015; Szegedy et al., 2015) . Secondly, DNNs usually consume large amount of storage memory and energy (Han et al., 2016) . Therefore these networks are difficult to embed into devices with limited memory and power (such as portable devices or chips). Most existing networks aim to reduce computational budget through network pruning (Han et al., 2015; Anwar et al., 2017; Li et al., 2017; Collins & Kohli, 2014) , filter factorization (Jaderberg et al., 2014; Lebedev et al., 2014) , low bit representation (Rastegari et al., 2016) for weights and knowledge transfering (Hinton et al., 2015) . In contrast to the above works that ignore the strong dependencies among weights and learn filters independently based on existing network architectures, this paper proposes to explicitly enforce the parameter sharing among filters to more effectively learn compact and efficient deep networks.

In this paper, we propose a Weight Sampling deep neural network (i.e. WSNet) to significantly reduce both the model size and computation cost of deep networks, achieving more than 100× smaller size and up to 16× speedup at negligible performance drop or even achieving better performance than the baseline (i.e. conventional networks that learn filters independently). Specifically, WSNet is parameterized by layer-wise condensed filters from which each filter participating in actual convolutions can be directly sampled, in both spatial and channel dimensions. Since condensed filters have significantly fewer parameters than independently trained filters as in conventional CNNs, learning by sampling from them makes WSNet a more compact model compared to conventional CNNs. In addition, to reduce the ubiquitous computational redundancy in convolving the overlapped filters and input patches, we propose an integral image based method to dramatically reduce the computation cost of WSNet in both training and inference. The integral image method is also advantageous because it enables weight sampling with different filter size and minimizes computational overhead to enhance the learning capability of WSNet.

In order to demonstrate the efficacy of WSNet, we conduct extensive experiments on the challenging acoustic scene classification and music detection tasks. On each test dataset, including MusicDet200K (a self-collected dataset, as detailed in Section 4), ESC-50 (Piczak, 2015a) , Urban-Sound8K (Salamon et al., 2014) and DCASE (Stowell et al., 2015a) , WSNet significantly reduces the model size of the baseline by 100× with comparable or even higher classification accuracy. When compressing more than 180×, WSNet is only subject to negligible accuracy drop. At the same time, WSNet significantly reduces the computation cost (up to 16×). Such results strongly establish the capability of WSNet to learn compact and efficient networks. Although we detailed experiments mostly limited to 1D CNNs in this paper, we will explore how the same approach can be naturally generalized to 2D CNNs in future work.

2. RELATED WORKS:



2.1. AUDIO CLASSIFICATION:

In this paper we considered Acoustic Scene Classification (ASC) tasks as well as music detection tasks. ASC aims to classify the surrounding environment where an audio stream is generated given the audio input (Barchiesi et al., 2015) . It can be applied in many different ways such as audio tagging (Cai et al., 2006) , audio collections management (Landone et al., 2007) , robotic navigation (Chu et al., 2006) , intelligent wearable interfaces (Xu et al., 2008) , context adaptive tasks (Schilit et al., 1994) , etc. Music detection is a related task to determine whether or not a small segment of audio is music. It is usually treated as a binary classification problem given an audio segment as input, i.e., to classify the segment into two categories: music or non-music.

As evident in many other areas, convolutional neural networks (CNN) have been widely used in audio classification tasks (Valenti et al., 2016) (Salamon & Bello, 2017) . SoundNet (Aytar et al., 2016) stands out among different CNNs for sound classification due to the following two reasons. First, it is trained from the large-scale unlabeled sound data using visual information as a bridge, while many other networks are trained with smaller datasets. Secondly, SoundNet directly takes one dimensional raw wave signals as input so that there is no need to calculate time-consuming audio specific features, e.g. MFCC (Pols et al., 1966) (Davis & Mermelstein, 1980) and spectrogram (Flanagan, 1972) . SoundNet has yielded significant performance improvements on state-of-the-art results with standard benchmarks for acoustic scene classification. In this paper, we demonstrate that the proposed WSNet achieves a comparable or even better performance than SoundNet at a significantly smaller size and faster speed.

2.2. DEEP MODEL COMPRESSION AND ACCELERATION:

Early approaches for deep model compression include (LeCun et al., 1989; Hassibi & Stork, 1993) that prune the connections in networks based on the second order information. Most recent works in network compression adopt weight pruning (Han et al., 2015; Collins & Kohli, 2014; Anwar et al., 2017; Lebedev & Lempitsky, 2016; Kim et al., 2015; Luo et al., 2017; Li et al., 2017) , filter decomposition (Sindhwani et al., 2015; Denton et al., 2014; Jaderberg et al., 2014) , hashed networks (Chen et al., 2015; 2016) and weight quantization (Han et al., 2016) . However, although those works reduce model size, they also suffer from large performance drop. Bucilu et al. (2006) and Ba & Caruana (2014) are based on student-teacher approches which may be difficult to apply in new tasks since they require training a teacher network in advance. Denil et al. (2013) predicts parameters based on a few number of weight values. Jin et al. (2016) proposes an iterative hard thresholding method, but only achieve relatively small compression ratios. Gong et al. (2014) uses a binning method which can only be applied over fully connected layers. Hinton et al. (2015) compresses deep models by transferring the knowledge from pre-trained larger networks to smaller networks. In contrast, WS-Net is able to learn compact representation for both convolution layers and fully connected layers from scratch. The deep models learned by WSNet can significantly reduce model size compared to the baselines with comparable or even better performance.

In terms of deep model acceleration, the factorization and quantization methods listed above can also reduce computation latency in inference. While irregular pruning (as done in most pruning methods (Han et al., 2016) ) incurs computational overhead, grouped pruning (Lebedev & Lempitsky, 2016) is able to accelerate networks. FFT (Mathieu et al., 2013) and LCNN (Bagherinezhad et al., 2016) are also used to speed up computation in pratice. Comparatively, WSNet is superior because it learns networks that have both smaller model size and faster computation versus baselines.

2.3. EFFICIENT MODEL DESIGN:

WSNet presents a class of novel models with the appealing properties of a small model size and small computation cost. Some recently proposed efficient model architectures include the class of Inception models (Szegedy et al., 2015; Ioffe & Szegedy, 2015; Chollet, 2016) which adopts depthwise separable convolutions, the class of Residual models (He et al., 2016; Xie et al., 2017; Chen et al., 2017) which uses residual path for efficient optimization, and the factorized networks which use fully factorized convolutions. MobileNet (Howard et al., 2017) and Flattened networks (Jin et al., 2014) are based on factorization convolutions. ShuffleNet (Zhang et al., 2017) uses group convolution and channel shuffle to reduce computational cost. Compared with above works, WS-Net presents a new model design strategy which is more flexible and generalizable: the parameters in deep networks can be obtained conveniently from a more compact representation, e.g. through the weight sampling method proposed in this paper or other more complex methods based on the learned statistic models.

3. METHOD:

In this section, we describe details of the proposed WSNet for 1D CNNs. First, the notations are introduced. Secondly, we elaborate on the core components in WSNet: weight sampling along the spatial dimension and channel dimension. Thirdly, we introduce the denser weight sampling to enhance the learning capability of WSNet. Finally, we propose an integral image method for accelerating WSNet in both training and inference.

3.1. NOTATIONS:

Before diving into the details, we first introduce the notations used in this paper. The traditional 1D convolution layer takes as input the feature map F ∈ R T ×M and produces an output feature map G ∈ R T ×N where (T, M, N ) denotes the spatial length of input, the channel of input and the number of filters respectively. We assume that the output has the same spatial size as input which holds true by using zero padded convolution. The 1D convolution kernel K used in the actual convolution of WSNet has the shape of (L, M, N ) where L is the kernel size. Let k n , n ∈ {1, • • • N } denotes a filter and f t , t ∈ {1, • • • T } denotes a input patch that spatially spans from t to t + L -1, then the convolution assuming stride one and zero padding is computed as:

EQUATION

where • stands for the vector inner product. Note we omit the element-wise activation function to simplify the notation.

In WSNet, instead of learning each weight independently, K is obtained by sampling from a learned condensed filter Φ which has the shape of (L * , M * ). The goal of training WSNet is thus cast to learn more compact DNNs which satisfy the condition of L * M * < LM N . To quantize the advantage of WSNet in achieving compact networks, we define the compactness of K in a learned layer in WSNet w.r.t. the conventional layer with independently learned weights as: compactness = LM N L * M * . In the following section, we demonstrate WSNet learn compact networks by sampling weights in two dimensions: the spatial dimension and the channel dimension.

3.2. WEIGHT SAMPLING:



3.2.1. ALONG SPATIAL DIMENSION:

In conventional CNNs, the filters in a layer are learned independently which presents two disadvantages. Firstly, the resulted DNNs have a large number of parameters, which impedes their deploy-

Channel Sampling:



Spatial Sampling:

Sampling Stride: S

Shapes of Filters:

Figure 1 : Illustration of WSNet that learns small condensed filters with weight sampling along two dimensions: spatial dimension (the bottom panel) and channel dimension (the top panel). The figure depicts procedure of generating two continuous filters (in pink and purple respectively) that convolve with input. In spatial sampling, filters are extracted from the condensed filter with a stride of S. In channel sampling, the channel of each filter is sampled repeatedly for C times to achieve equal with the input channel. Please refer to Section 3.2 for detailed explanations. ment in computation resource constrained platforms. Second, such over-parameterization makes the network prone to overfitting and getting stuck in (extra introduced) local minimums. To solve these two problems, a novel weight sampling method is proposed to efficiently reuse the weights among filters. Specifically, in each convolutional layer of WSNet, all convolutional filters K are sampled from the condensed filter Φ, as illustrated in Figure 1 . By scanning the weight sharing filter with a window size of L and stride of S, we could sample out N filters with filter size of L. Formally, the equation between the filter size of the condensed filter and the sampled filters is:

L * = L + (N -1)S.

(2)

The compactness along spatial dimension is

LM * N L * M * ≈ L S .

Note that since the minimal value of S is 1, the minimal value of L * (i.e. the minimum spatial length of the condensed filter) is L + N -1 and the maximal achievable compactness is therefore L.

3.2.2. ALONG CHANNEL DIMENSION:

Although it is experimentally verified that the weight sampling strategy could learn compact deep models with negligible loss of classification accuracy (see Section 4), the maximal compactness is limited by the filter size L, as mentioned in Section 3.2.1.

In order to seek more compact networks without such limitation, we propose a channel sharing strategy for WSNet to learn by weight sampling along the channel dimension. As illustrated in Figure 1 (top panel), the actual filter used in convolution is generated by repeating sampling for C times. The relation between the channels of filters before and after channel sampling is:

EQUATION

Therefore, the compactness of WSNet along the channel dimension achieves C. As introduced later in Experiments (Section 4), we observe that the repeated weight sampling along the channel dimension significantly reduces the model size of WSNet without significant performance drop. One notable advantage of channel sharing is that the maximum compactness can be as large as M (i.e. when the condensed filter has channel of 1), which paves the way for learning much more aggressively smaller models (e.g. more than 100× smaller models than baselines).

The above analysis for weight sampling along spatial/channel dimensions can be conveniently generalized from convolution layers to fully connected layers. For a fully connected layer, we treat its weights as a flattened vector with channel of 1, along which the spatial sampling (ref. Section 3.2.1) is performed to reduce the size of learnable parameters. For example, for the fully connected layer "fc1" in the baseline network in Table 1 , its filter size, channel number and filter number are 1536, 1 and 256 respectively. We can therefore perform spatial sampling for "fc1" to learn a more compact representation. Compared with convolutional layers which generally have small filter sizes and thus have limited compactnesses along the spatial dimenstion, the fully connected layers can achieve larger compactnesses along the spatial dimension without harming the performance, as demonstrated in experimental results (ref.

to Section 4.2).

3.2.3. THE TRAINING OF CONDENSED FILTERS:

WSNet is trained from the scratch in a similar way to conventional deep convolutional networks by using standard error back-propagation. Since every weight K l,m,n in the convolutional kernel K is sampled from the condensed filter Φ along the spatial and channel dimension, the only difference is the gradient of Φ i,j is the summation of all gradients of weights that are tied to it. Therefore, by simply recording the position mapping M : (i, j) → (l, m, n) from Φ i,j to all the tied weights in K, the gradient of Φ i,j is calculated as:

EQUATION

)

where L is the conventional cross-entropy loss function. In open-sourced machine learning libraries which represent computation as graphs, such as TensorFlow Abadi et al. (2016) , Equation ( 4) can be calculated automatically.

3.3. DENSER WEIGHT SAMPLING:

The performance of WSNet might be adversely affected when the size of condensed filter is decreased aggressively (i.e. when S and C are large). To enhance the learning capability of WSNet, we could sample more filters for layers with significantly reduced sizes. Specifically, we use a smaller sampling stride S ( S < S) when performing spatial sampling. In order to keep the shape of weights unchanged in the following layer, we append a 1×1 convolution layer with the shape of (1, n, n) to reduce the channels of densely sampled filters. It is experimentally verified that denser weight sampling can effectively improve the performance of WSNet in Section 4. However, since it also brings extra parameters and computational cost to WSNet, denser weight sampling is only used in lower layers of WSNet whose filter number (n) is small. Besides, one can also conduct channel sampling on the added 1×1 convolution layers to further reduce their sizes.

3.4. EFFICIENT COMPUTATION WITH INTEGRAL IMAGE:

According to Equation 1, the computation cost in terms of the number of multiplications and adds (i.e. Mult-Adds) in a conventional convolutional layer is:

T M LN (5)

However, as illustrated in Figure 2 , since all filters in a layer in WSNet are sampled from a condensed filter Φ with stride S, calculating the results of convolution in the conventional way as in Eq. ( 1) incurs severe computational redundance. Concretely, as can be seen from Eq. ( 1), one item in the ouput feature map is equal to the summation of L inner products between the row vector of f and the column vector of k. Therefore, when two overlapped filters that are sampled from the condensed filter (e.g. k 1 and k 2 in Fig. 2 ) convolves with the overlapped input windows (e.g. f 1 and f 2 in Fig. 2 )), some partially repeated calculations exist (e.g. the calculations highlight in green and indicated by arrow in Fig. 2 ). To eliminate such redundancy in convolution and speed-up WSNet, we propose a novel integral image method to enable efficient computation via sharing computations.

We first calculate an inner product map P ∈ R T ×L * which stores the inner products between each row vector in the input feature map (i.e. F) and each column vector in the condensed filter (i.e. Φ): ...

EQUATION

I(u 1 +L-1,v 1 +L-1) I(u 1 ,v 1 ) v 2 +L-1 v 1 +L-1 u 2 u 1 +L-1 u 2 +L-1 v 1 u 1

Figure 2 : Illustration of efficient computation with integral image in WSNet. The inner product map P ∈ R T ×L * calculates the inner product of each row in F and each column in Φ as in Eq. ( 6). The convolution result between a filter k 1 which is sampled from Φ and the input patch f 1 is then the summation of all values in the segment between (u, v) and (u + L -1, v + L -1) in P (recall that L is the convolutional filter size). Since there are repeated calculations when the filter and input patch are overlapped, e.g. the green segment indicated by arrow when performing convolution between k 2 and s 2 , we construct the integral image I using P according to Eq. ( 7). Based on I, the convolutional results between any sampled filter and input patch can be retrieved directly in time complexity of O(1) according to Eq. ( 8), e.g. the results of

k 1 • s 1 is I(u 1 + L -1, v 1 + L -1) -I(u 1 -1, v 1 -1).

For notation definitions, please refer to Sec. 3.1. The comparisons of computation costs between WSNet and the baselines using conventional architectures are introduced in Section 3.4.

The integral image for speeding-up convolution is denoted as I. It has the same size as P and can be conveniently obtained throught below formulation:

EQUATION

Based on I, all convolutional results can be obtained in time complexity of O(1) as follows

EQUATION

Recall that the n-th filter lies in the spatial range of (nS, nS + L -1) in the condensed filter Φ.

Since G ∈ R T ×N , it thus takes T N times of calculating Eq. ( 8) to get G. In Eq. ( 6) ∼ Eq. ( 8), we omit the case of padding for clear description. When zero padding is applied, we can freely get the convolutional results for the padded areas even without using Eq. ( 8) since

I(u, v) = I(T, v-1), u > T .

Based on Eq. ( 6) ∼ Eq. ( 8), the computation cost of the proposed integral image method is

T M L *

Eq. ( 6)

+ T L *

Eq. ( 7)

EQUATION

Note the computation cost of P (i.e. Eq. ( 6)) is the dominating term in Eq. ( 9). Based on Eq. ( 5), Eq. ( 9) and Eq. ( 2), the theoretical acceleration ratio is

T M LN T (M + 1)L * + T N ≈ L S

Recall that L is the filter size and S is the pre-defined stride when sampling filters from the condensed filter Φ (ref. to Eq. ( 2)).

In practice, we adopt a variant of above method to further boost the computation efficiency of WS-Net, as illustrated in Fig 3 . In Eq. ( 6), we repeat Φ by C times along the channel dimension to channel wrapping

v 2 +L-1 v 1 +L-1 u 1 +L-1 u 2 +L-1 v 1 v 2 u 1 u 2

Figure 3 : A variant of the integral image method used in practice which is more efficient than that illustrated in Figure 2 . Instead of repeatedly sampling along the channel dimension of Φ to convolve with the input F, we wrap the channels of F by summing up C matrixes that are evenly divided from F along the channels, i.e. F(i, j) =

C-1 c=0 F(i, j + cM * ). Since the channle of F is only 1/C of the channel of F, the overall computation cost is reduced as demonstrated in Eq. ( 10). make it equal with the channel of the input F. However, we could first wrap the channels of F by accumulating the values with interval of L along its channel dimension to a thinner feature map F ∈ R T ×M * which has the same channel number as Φ, i.e. F(i, j) =

C-1 c=0 F(i, j + cM * ). Both Eq. ( 7) and Eq. ( 8) remain the same. Then the computational cost is reduced to

T M * (C -1) channel warp + T M * L * Eq. (6)

+ T L * Eq. ( 7) + T N Eq. ( 8) (10) where the first item is the computational cost of warping the channels of F to obtain F. Since the dominating term (i.e. Eq. ( 6)) in Eq (10) is smaller than in Eq. ( 9), the overall computation cost is thus largely reduced. By combining Eq. ( 10) and Eq. ( 5), the theoretical acceleration compared to the baseline is

EQUATION

Finally, we note that the integral image method applied in WSNet naturally takes advantage of the property in weight sampling: redundant computations exist between overlapped filters and input patches. Different from other deep model speedup methods (Sindhwani et al., 2015; Denton et al., 2014) which require to solve time-consuming optimization problems and incur performance drop, the integral image method can be seamlessly embeded in WSNet without negatively affecting the final performance.

4. EXPERIMENTS:

In this section, we present the details and analysis of the results in our experiments. Extensive ablation studies are conducted to verify the effectiveness of the proposed WSNet on learning compact and efficient networks. On all tested datasets, WSNet is able to improve the classification performance over the baseline networks while using 100× smaller models. When using even smaller (e.g. 180×) model size, WSNet achieves comparable performance w.r.t the baselines. In addition, WSNet achieves 2× ∼ 4× acceleration compared to the baselines with a much smaller model (more than 100× smaller).

4.1. EXPERIMENTAL SETTINGS:

Datasets We collect a large-scale music detection dataset (MusicDet200K) from publicly available platforms (e.g. Facebook, Twitter, etc.) for conducting experiments. For fair comparison with previous literatures, we also test WSNet on three standard, publicly available datasets, i.e ESC-50, UrbanSound8K and DCASE. The details of used datasets are as follows.

MusicDet200K aims to assign a sample a binary label to indicate whether it is music or not. Mu-sicDet200K has overall 238,000 annotated sound clips. Each has a time duration of 4 seconds and is resampled to 16000 Hz and normalized (Piczak, 2015b). Among all samples, we use 200,000/20,000/18,000 as train/val/test set. The samples belonging to "non-music" count for 70% of all samples, which means if we trivially assign all samples to be "non-music", the classification accuracy is 70%.

ESC-50 (Piczak, 2015a) is a collection of 2000 short (5 seconds) environmental recordings comprising 50 equally balanced classes of sound events in 5 major groups (animals, natural soundscapes and water sounds, human non-speech sounds, interior/domestic sounds and exterior/urban noises) divided into 5 folds for cross-validation. Following Aytar et al. (2016) , we extract 10 sound clips from each recording with length of 1 second and time step of 0.5 second (i.e. two neighboring clips have 0.5 seconds overlapped). Therefore, in each cross-validation, the number of training samples is 16000. In testing, we average over ten clips of each recording for the final classification result.

UrbanSound8K (Salamon et al., 2014 ) is a collection of 8732 short (around 4 seconds) recordings of various urban sound sources (air conditioner, car horn, playing children, dog bark, drilling, engine idling, gun shot, jackhammer, siren and street music). As in ESC-50, we extract 8 clips with the time length of 1 second and time step of 0.5 second from each recording. For those that are less than 1 second, we pad them with zeros and repeat for 8 times (i.e. time step is 0.5 second). Evaluation criteria To demonstrate that WSNet is capable of learning more compact and efficient models than conventional CNNs, three evaluation criteria are used in our experiments: model size, the number of multiply and adds in calculation (mult-adds) and classification accuracy.

Baseline networks To test the scability of WSNet to different network architectures (e.g. whether having fully connected layers or not), two baseline networks are used in comparision. The baseline network used on MusicDet200K consists of 7 convolutional layers and 2 fully connected layers, using which we demonstrate the effectiveness of WSNet on both convolutional layers and fully connected layers. For fair comparison with previous literatures, we firstly modify the state-of-theart SoundNet (Aytar et al., 2016) by applying pooling layers to all but the last convolutional layer. As can be seen in Table 5 , this modification significantly boosts the performance of original SoundNet.

We then use the modified SoundNet as a baseline on all three public datasets. The architectures of the two baseline networks are shown in Table 1 and Table 2 respectively.

Weight Quantization Similar to other works (Han et al., 2016; Rastegari et al., 2016) , we apply weight quantization to further reduce the size of WSNet. Specifically, the weights in each layer are linearly quantized to q bins where q is a pre-defined number. By setting all weights in the same bin to the same value, we only need to store a small index of the shared weight for each weight. The size of each bin is calculated as (max(Φ) -min(Φ))/q. Given q bins, we only need log 2 (q) bits to encode the index. Assuming each weight in WSNet is represented using 4 bytes float number (32 bits) without weight quantization, the ratio of each layer's size before and after weight quantization is

32L * M * L * M * log 2 (q)+32q

. Recall that L * and M * are the spatial size and the channel number of condensed filter. Since the condition L * M * q generally holds in most layers of WSNet, weight quantization is able to reduce the model size by a factor of 32 log 2 (q) . Different from (Han et al., 2016; Rastegari et al., 2016) which learns the quantization during training, we apply weight quantization to WSNet Table 1 : Baseline-1: configurations of the baseline network used on MusicDet200K. Each convolutional layer is followed by a nonlinearity layer (i.e. ReLU), batch normalization layer and pooling layer, which are omitted in the table for brevity. The strides of all pooling layers are 2. The padding strategies adopted for both convolutional layers and fully connected layers are all "size preserving". Implementation details WSNet is implemented and trained from scratch in Tensorflow (Abadi et al., 2016) . Following Aytar et al. (2016) , the Adam (Kingma & Ba, 2014) optimizer, a fixed learning rate of 0.001, and momentum term of 0.9 and batch size of 64 are used throughout experiments.

We initialized all the weights to zero mean gaussian noise with a standard deviation of 0.01. In the network used on MusicDet200K, the dropout ratio for the dropout layers (Srivastava et al., 2014) after each fully connected layer is set to be 0.8. The overall training takes 100,000 iterations.

4.2. RESULTS AND ANALYSIS:



4.2.1. MUSICDET200K:

Ablation analysis Through controlled experiments, we investigate the effects of each component in WSNet on the model size, computational cost and classification accuracy. The comparative study results of different settings of WSNet are listed in Table 3 . For clear description, we name WSNets with different settings by the combination of symbols S/C/SC † /D/Q. Please refer to the caption of Table 3 for detailed meanings.

(1) Spatial sampling. We test the performance of WSNet by using different sampling stride S in spatial sampling. As listed in Table 3 , S 2 and S 4 slightly outperforms the classification accuracy of the baseline, possibly due to reducing the overfitting of models. When the sampling stride is 8, i.e. the compactness in spatial dimension is 8 (ref. to Section 3.2.1), the classification accuracy of S 8 only drops slightly by 0.6%. Note that the maximum compactness along the spatial dimension is equal to the filter size, thus for the layer "conv7" which has a filter size of 4, its compactness is limited by 4 (highlighted by underline in Table 3 ) in S 8 . Above results clearly demonstrate that the Table 3 : Ablative study of the effects of different settings of WSNet on the model size, computation cost (in terms of #mult-adds) and classification accuracy on MusicDet200K. For clear description, we name WSNets with different settings by the combination of symbols S/C/SC † /D/Q. "S" denotes the weight sampling along spatial dimension; "C" denotes the weight sampling along the channel dimension. "SC † " denotes the weight sampling of fully connected layers whose parameters can be seen as flattened vectors with channel of 1. "D" denotes denser filter sampling. "Q" denotes weight quantization. With a symbol occurred in the name, the corresponding component is used in WSNet. The numbers in subscripts of S/C/SC † /D/Q denotes the maximum compactness (ref. to Sec. 3 .1 for the definition of compactness) on spatial/channel dimension in all layers, the ratio of the number of filters in WSNet versus in the baseline, the compactness of fully connected layers and the ratio of WSNet's size before and after weight quantization, respectively. To avoid confusion, SC † only occured in the names when both spatial and channel sampling are applied for convolutional layers. The model size and the computational cost are provided for the baseline. For the model size and #mult-adds of WSNet, we provide the ratio of the baseline's model size versus WSNet's model size and the ratio of the baseline's #Mult-Adds versus WSNet's #Mult-Adds. (2) Channel sampling. Three different compactness along the channel dimension, i.e. 2, 4 and 8 are tested by comparing with baslines. It can be observed from Table 3 that C 2 and C 4 and C 8 have linearly reduced model size without incurring noticeable drop of accuracy. In fact, C 2 can even improve the accuracy upon the baseline, demonstrating the effectiveness of channel sampling in WSNet. When learning more compact models, C 8 demonstrates better performance compared to S 8 tha has the same compactness in the spatial dimension, which suggests we should focus on the channel sampling when the compactness along the spatial dimension is high.

WSNet's conv{1-3} conv4 conv5 conv6 conv7 fc1/2 Acc. Model Mult- settings S C A S C A S C A S C A S C A SC † size Adds Baseline 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 88.9 ± 0.1 3M (1×) 1.2e10 (1×) BaselineQ4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 88.8 ± 0.1 4× 1× S2 2 1 1 2 1 1 2 1 1 2 1 1 2 1 1 2 89.0 ± 0.0 2× 1× S4 4 1 1 4 1 1 4 1 1 4 1 1 4 1 1 4 89.0 ± 0.0 4× 1.8× S8 8 1 1 8 1 1 8 1 1 8 1 1 4 1 1 8 88.3 ± 0.1 5.7× 3.4× C2 1 2 1 1 2 1 1 2 1 1 2 1 1 2 1 2 89.1 ± 0.2 2× 1× C4 1 4 1 1 4 1 1 4 1 1 4 1 1 4 1 4 88.7 ± 0.1 4× 1.4× C8 1 8 1 1 8 1 1 8 1 1 8 1 1 8

We then simultaneously perform weight sampling on both the spatial and channel dimensions. As demonstrated by the results of S 4 C 4 SC † 4 and S 8 C 8 SC † 8 , WSNet can learn highly compact models (more than 20× smaller than baselines) without noticeable performance drop (less than 0.5%).

(3) Denser weight sampling. Denser weight sampling is used to enhance the learning capability of WSNet with aggressive compactness (i.e. when S and C are large) and make up the performance loss caused by sharing too much parameters among filters. As shown in Table 3 , by sampling 2× more filters in conv1, conv2 and conv3, S 8 C 8 SC † 8 D 2 significantly outperforms the S 8 C 8 SC † 8 . Above results demonstrate the effectiveness of denser weight sampling to boost the performance of WSNet.

(4) Integral image for efficient computation. As evidenced in the last column in Table 3 , the proposed integral image method consistently reduces the computation cost of WSNet. For S 8 C 8 SC † 8 which is 23× smaller than the baseline, the computation cost (in terms of #mult-adds) is significantly reduced by 16.4 times. Due to the extra computation cost brought by the 1×1 convolution in denser 3 for the meaning of symbols S/C/D. Since the input lengths for the baseline are different in each dataset, we only provide the #Mult-Adds for UrbanSound8K. Note that since we use the ratio of baseline's #Mult-Adds versus WSNet's #Mult-Adds for one WSNet, the numbers corresponding to WSNets in the column of #Mult-Adds are the same for all dataset. Baseline 1 1 1 1 1 1 1 1 1 1 1 1 1 1 (Aytar et al., 2016) scratch init.; provided data 51.1 13M SoundNet (Aytar et al., 2016) pre-training; extra data 72.9 13M filter sampling, S 8 C 8 SC † 8 D 2 achieves lower acceleration (3.8×). Group convolution (Xie et al., 2017) can be used to alleviate the computation cost of the added 1×1 convolution layers. We will explore this direction in our future work.

(5) Weight quantization. It can be observed from Table 3 that by using 256 bins to represent each weight by one byte (i.e. 8bits), S 8 C 8 SC † 15 A 2 Q 4 is reduced to 1/168 of the baseline's model size while incurring only 0.1% accuracy loss. The above result demonstrates that the weight quantization is complementary to WSNet and they can be used jointly to effectively reduce the model size of WSNet. Since we do not use weight quantization to accelerate models in this paper, the WSNets before and after weight quantization have the same computational cost.

4.2.2. ESC-50:

The comparison of WSNet with other state-of-the-arts on ESC-50 is listed in Table 5 . The settings of WSNet used on ESC-50, UrbanSound8K and DCASE are listed in Table 4 . Compared with the baseline, WSNet is able to significantly reduce the model size of the baseline by 25 times and 45 times, while at the same time improving the accuracy of the baseline by 0.5% and 0.1% respectively. The computation costs of WSNet are listed in Table 4 , from which one can observe that WSNet achieves higher computational efficiency by reducing the #Mult-Adds of the baseline by 2.3× and 2.4×, respectively. Such promising results again demonstrate the effectiveness of WS-Net on learning compact and efficient networks. After applying weight quantization to WSNet, its model size is reduced to only 1/180 of the baseline while the accuracy only slightly drops by 0.2%. Compared with the SoundNet trained from scratch with provided data, WSNets significantly outperform its classification accuracy by over 10% with more than 100× smaller models. Using a transfer learning approach, SoundNet (Aytar et al., 2016) that is pre-trained using a large number of unlabeled videos achieves better accuracy than WSNet. However, since the training method is 

86 ± 0 25× WSNet S 8 C 4 D 2 Q 4 86 ± 0 100× WSNet S 8 C 8 D 2 84 ± 0 45× WSNet S 8 C 8 D 2 Q 4 84 ± 0 180×

RNH (Roma et al., 2013) scratch init.; provided data 77 -Ensemble (Stowell et al., 2015b) scratch init.; provided data 78 -SoundNet (Aytar et al., 2016) pre-training; extra data 88 13M

orthogonal to WSNet, we believe that WSNet can achieve better performance by training in a similar way as SoundNet (Aytar et al., 2016) on a large amount of unlabeled video data.

4.2.3. URBANSOUND8K:

We report the comparison results of WSNet with state-of-the-arts on UrbanSound8k in Table 6 . It is again observed that WSNet significantly reduces the model size of baseline while obtaining comparative results. Both Piczak (2015b) and Salamon & Bello (2015) use pre-computed 2D features after log-mel transformation as input. In comparison, the proposed WSNet simply takes the raw wave of recordings as input, enabling the model to be trained in an end-to-end manner.

4.2.4. DCASE:

As evidenced in Since both WSNet and HashNet Chen et al. (2015; 2016) explore weights tying, we compare them on MNIST and CIFAR10. For fair comparison, we use the same baselines used in Chen et al. (2015; 2016) . The baseline used for MNIST is a 3-layer fully connected network with a single hidden layer containing 1,000 hidden units.

The configuration of the baseline network used for CIFAR10 is listed in Table 9 . All hyperparameters used training including learning rate, momentum, drop out and so on follow Chen et al. (2015; 2016) . For each dataset, we hold out 20% of the whole training samples to form a validation set. The comparison results between WSNet and HashNet on MNIST/CIFAR10 are listed in Table 10 /Table 11 , respectively.

As one can observe in Table 10 and Table 11 , when learning networks with the same sizes, WSNet achieves significant lower error rates than HashNet on both datasets. Above results clearly demonstrate the advantages of WSNet in learning compact models.

Furthermore, we also conduct experiment on CIFAR10 with the state-of-the-art ResNet18 He et al. (2016) network as baseline. Both the network architecture and training hyperparameters follow He et al. (2016) .

WSNet is able to achieve 20× smaller model size with slight performance drop (0.6%). Such promising results further demonstrate the effectiveness of WSNet.

:



Figures and Tables:

(Stowell et al., 2015a)  is used in the Detection and Classification of Acoustic Scenes and Events Challenge (DCASE). It contains 10 acoustic scene categories, 10 training examples per category and 100 testing examples. Each sample is a 30-second audio recording. During training, we evenly extract 12 sound clips with time length of 5 seconds and time step of 2.5 seconds from each recording.

Baseline-2: configuration of the baseline network used on ESC-50, UrbanSound8K and DCASE. This baseline is adapted from SoundNet(Aytar et al., 2016) as detailed in Section 4.1. For brevity, the nonlinearity layer (i.e. ReLU), batch normalization layer and pooling layer following each convolutional layer are omitted. The kernel sizes for pooling layers following conv1-conv4 and conv5-conv7 are 8 and 4 respectively. The stride of every pooling layers is 2.

Table content: <table><tr><td>Layer</td><td>conv1</td><td>conv2</td><td>conv3</td><td>conv4</td><td>conv5</td><td>conv6</td><td>conv7</td><td>fc1</td><td>fc2</td></tr><tr><td>Filter sizes</td><td>32</td><td>32</td><td>16</td><td>8</td><td>8</td><td>8</td><td>4</td><td>1536</td><td>256</td></tr><tr><td>#Filters</td><td>32</td><td>64</td><td>128</td><td>128</td><td>256</td><td>512</td><td>512</td><td>256</td><td>128</td></tr><tr><td>Stride</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td></tr><tr><td>#Params</td><td>1K</td><td>65K</td><td>130K</td><td>130K</td><td>260K</td><td>1M</td><td>1M</td><td>390K</td><td>33K</td></tr><tr><td>Params (%)</td><td>0.03</td><td>2.1</td><td>4.2</td><td>4.2</td><td>8.4</td><td>33.7</td><td>33.7</td><td>12.6</td><td>1.1</td></tr><tr><td>#Mult-Adds (10 8 )</td><td>4.1</td><td>65.5</td><td>32.7</td><td>8.2</td><td>4.1</td><td>4.2</td><td>1.0</td><td>0.1</td><td>0.007</td></tr><tr><td>Mult-Adds (%)</td><td>3.4</td><td>54.5</td><td>27.3</td><td>6.8</td><td>3.4</td><td>3.5</td><td>0.9</td><td>0.1</td><td>0.005</td></tr><tr><td>Layer</td><td colspan="9">conv1 conv2 conv3 conv4 conv5 conv6 conv7 conv8</td></tr><tr><td>Filter sizes</td><td/><td>64</td><td>32</td><td>16</td><td>8</td><td>4</td><td>4</td><td>4</td><td>8</td></tr><tr><td>#Filters</td><td/><td>16</td><td>32</td><td>64</td><td>128</td><td>256</td><td>512</td><td>1024</td><td>1401</td></tr><tr><td>Stride</td><td/><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>#Params</td><td/><td>1K</td><td>16K</td><td>32K</td><td>65K</td><td colspan="2">130K 520K</td><td>2M</td><td>11M</td></tr><tr><td>Params (%)</td><td colspan="2">0.01</td><td>0.11</td><td>0.22</td><td>0.45</td><td>0.90</td><td>3.63</td><td colspan="2">14.55 79.61</td></tr><tr><td>#Mult-Adds (10 8 )</td><td/><td>2.3</td><td>9.0</td><td>4.5</td><td>2.3</td><td>1.2</td><td>1.2</td><td>1.2</td><td>2.3</td></tr><tr><td>Mult-Adds (%)</td><td/><td>9.4</td><td>37.7</td><td>18.8</td><td>9.5</td><td>4.8</td><td>4.8</td><td>5.3</td><td>9.6</td></tr><tr><td colspan="10">after its training. In the experiments, we find that such an off-line way is sufficient to reduce model</td></tr><tr><td colspan="2">size without losing accuracy.</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

The configurations of the WSNet used on ESC-50, UrbanSound8K and DCASE. Please refer to Table

Table content: <table/>

Comparison with state-of-the-arts on ESC-50. All results of WSNet are obtained by 10folder validation. Please refer to Table3for the meaning of symbols S/C/D/Q. The baseline used here is a simple modification of SoundNet with 8 convolution layers (refer to Section 4.1 for details), thus they have the same model size.

Table content: <table><tr><td>Model</td><td>Settings</td><td>Acc. (%)</td><td>Model size</td></tr><tr><td>baseline</td><td colspan="2">scratch init.; provided data 66.0 ± 0.25</td><td>13M (1×)</td></tr><tr><td>baselineQ 4</td><td colspan="2">scratch init.; provided data 65.8 ± 0.25</td><td>4×</td></tr><tr><td>WSNet</td><td>S 8 C 4 D 2</td><td>66.5 ± 0.10</td><td>25×</td></tr><tr><td>WSNet</td><td>S 8 C 4 D 2 Q 4</td><td>66.25 ± 0.25</td><td>100×</td></tr><tr><td>WSNet</td><td>S 8 C 8 D 2</td><td>66.1 ± 0.15</td><td>45×</td></tr><tr><td>WSNet</td><td>S 8 C 8 D 2 Q 4</td><td>65.8 ± 0.25</td><td>180×</td></tr><tr><td>Piczak ConvNet (Piczak, 2015b)</td><td>scratch init.; provided data</td><td>64.5</td><td>28M</td></tr><tr><td>SoundNet</td><td/><td/><td/></tr></table>

Comparison with state-of-the-arts on UrbanSound8K. All results of WSNet are obtained by 5-folder validation. Please refer to Table3for the meaning of symbols S/C/D/Q.

Table content: <table><tr><td>Model</td><td>Settings</td><td>Acc. (%)</td><td>Model size</td></tr><tr><td>baseline</td><td colspan="3">scratch init.; provided data 70.39 ± 0.31 13M (1×)</td></tr><tr><td>baselineQ 4</td><td colspan="2">scratch init.; provided data 70.10 ± 0.31</td><td>4×</td></tr><tr><td>WSNet</td><td>S 8 C 4 D 2</td><td>70.76 ± 0.15</td><td>25×</td></tr><tr><td>WSNet</td><td>S 8 C 4 D 2 Q 4</td><td>70.61 ± 0.20</td><td>100×</td></tr><tr><td>WSNet</td><td>S 8 C 8 D 2</td><td>70.14 ± 0.23</td><td>45×</td></tr><tr><td>WSNet</td><td>S 8 C 8 D 2 Q 4</td><td>70.03 ± 0.11</td><td>180×</td></tr><tr><td>Piczak ConvNet (Piczak, 2015b)</td><td>pre-computed features</td><td>73.1</td><td>28M</td></tr></table>

Comparison with state-of-the-arts on DCASE. Note there are only 100 samples in testing set. Please refer to Table3for the meaning of symbols S/C/D/Q.

Table content: <table><tr><td>Model</td><td>Settings</td><td colspan="2">Acc. (%) Model size</td></tr><tr><td>baseline</td><td>scratch init.; provided data</td><td>85 ± 0</td><td>13M (1×)</td></tr><tr><td>baselineQ 4</td><td>scratch init.; provided data</td><td>84 ± 0</td><td>4×</td></tr><tr><td>WSNet</td><td>S 8 C 4 D 2</td><td/><td/></tr></table>

WSNet outperforms the classification accuracy of the baseline by 1% with a 100× smaller model. When using an even more compact model, i.e. 180× smaller in model size. The classification accuracy of WSNet is only one percentage lower than the baseline (i.e. has only one more incorrectly classified sample), verifying the effectiveness of WSNet. Compared with SoundNet(Aytar et al., 2016) that utilizes a large number of unlabeled data during training, WSNet (S 8 C 4 D 2 Q 4 ) that is 100× smaller achieves comparable results only by using the provided data.5 CONCLUSIONIn this paper, we present a class of Weight Sampling networks (WSNet) which are highly compact and efficient. A novel weight sampling method is proposed to sample filters from condensed filters which are much smaller than the independently trained filters in conventional networks. The weight sampling in conducted in two dimensions of the condensed filters, i.e. by spatial sampling and channel sampling. Taking advantage of the overlapping property of the filters in WSNet, we propose an integral image method for efficient computation. Extensive experiments on four audio classification datasets including MusicDet200K, ESC-50, UrbanSound8K and DCASE clearly demonstrate that WSNet can learn compact and efficient networks with competitive performance.

Table content: <table/>

Test error rates (in %) of WSNet and HashNet on CIFAR10. The model size is provided for the baseline. For the model size of WSNet/HashNet, we provide the ratio of the baseline's model size versus the model size of WSNet/HashNet. For each convolutional layer in WSNet with compactness of 16, we set its compactness along spatial/channel dimension to be 4/4, respectively. For each convolutional layer in WSNet with compactness of 64, we set its compactness along spatial/channel dimension to be 8/8, respectively. The compactness of fully connected layers in WSNet with compactness of 16 and 64 are set to be 16 and 64 respectively.

Table content: <table><tr><td>Model</td><td colspan="2">Model size Error rate</td></tr><tr><td>baseline</td><td>1.2M (×)</td><td>14.91</td></tr><tr><td>HashNet Chen et al. (2016)</td><td>16×</td><td>21.42</td></tr><tr><td>HashNet Chen et al. (2016)</td><td>64×</td><td>30.79</td></tr><tr><td>WSNet</td><td>16×</td><td>17.82</td></tr><tr><td>WSNet</td><td>64×</td><td>23.59</td></tr><tr><td>6.3.2 PRELIMINARY EXPERIMENTAL RESULTS</td><td/><td/></tr></table>

Peer review (iclr2018):
 In this work, the authors propose a technique to compress convolutional and fully-connected layers in a network by tying various weights in the convolutional filters: specifically within a single channel (weight sampling) and across channels (channel sampling). When combined with quantization, the proposed approach allows for large compression ratios with minimal loss in performance on various audio classification tasks. Although the results are interesting, I have a number of concerns about this work, which are listed below: 1. The idea of tying weights in the neural network in order to compress the model is not entirely new. This has been proposed previously in the context of feed-forward networks [1], and convolutional networks [2] where the choice of parameter tying is based on hash functions which ensure a random (but deterministic) mapping from a small set of “true” weights to a larger set of “virtual” weights. I think it would be more fair to compare against the HashedNet technique. References: [1] Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. 2015. Compressing neural networks with the hashing trick. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (ICML*15), Francis Bach and David Blei (Eds.), Vol. 37. JMLR.org 2285-2294. [2] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. 2016. Compressing Convolutional Neural Networks in the Frequency Domain. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD *16). ACM, New York, NY, USA, 1475-1484. DOI: https://doi.org/10.1145/2939672.2939839 2. Given that the experiments are conducted on tasks where there isn’t a large amount of training data, one concern is that the baseline model used by the authors might be overparameterized. It would be interesting to see how performance varies as a function of number of parameters for these tasks without any “compression”, i.e., just by reducing filter sizes, for example. 3. It seems somewhat surprising that repeating the filter weights across channels as is done in the channel sharing technique yields no loss in accuracy, especially for the deeper convolutional layers. Could this perhaps be a function of the tasks that the binary “music detection” task that these models are evaluated on? Do the authors have any comments on why this doesn*t hurt performance? 4. In citing relevant previous work, the authors should also include student-teacher approaches [1, 2] and distillation [3], and work by Denil et al. [4] on compression. References: [1] C. Bucilua, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541. ACM, 2006 [2] J. Ba and R. Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pages 2654–2662, 2014. [3] G. Hinton, O. Vinyals, J. Dean. Distilling the Knowledge in a Neural Network, NIPS 2014 Deep Learning Workshop. 2014. [4] M. Denil, B. Shakibi, L. Dinh, N. de Freitas, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pages 2148–2156, 2013. 5. Section 3, where the authors describe the proposed techniques is somewhat confusing to read, because of a lack of detailed mathematical explanations of the proposed techniques. This makes the paper harder to understand, in my view. Please re-write these sections in order to clearly express the parameter tying mechanism. In particular, I had the following questions: - Are weights tied across layers i.e., are the “weight sharing” matrices shared across layers? - There appears to be a typo in Equation 3: I believe it should be m = m* C. - Filter augmentation/Weight quantization are applicable to all methods, including the baseline. It would therefore be interesting to examine how they affect the baseline, not just the proposed system. - Section 3.5, on using the “Integral Image” to speed up computation was not clear to me. In particular, could the authors re-write to explain how the computation is computed efficiently with “two subtraction operations”. Could the authors also clarify the savings achieved by this technique? 6. Results are reported on the various test sets without any discussion of statistical significance. Could the authors describe whether the differences in performance on the various test sets are statistically significant? 7. On the ESC-50, UrbanSound8K, and DCASE tasks, it is a bit odd to compare against previous baselines which use different input features, use different model configurations, etc. It would be much better to use one of the previously published configurations as the baseline, and apply the proposed techniques to that configuration to examine performance. In particular, could the authors also use log-Mel filterbank energies as input features similar to (Piczak, 2015) and (Salomon and Bello, 2015), and apply the proposed techniques starting from those input features? Also, it would be useful when comparing against previously published baselines to indicate total number of independent parameters in the system in addition to accuracy numbers. 8. Minor Typographical Errors: There are a number of minor typographical/grammatical errors in the paper, some of which are listed below: - Abstract: “Combining weight quantization ...” → “Combining with weight quantization ...” - Sec 1: “... without sacrificing the loss of accuracy” → “... without sacrificing accuracy” - Sec 1: “Above experimental results strongly evident the capability of WSNet …” → “Above experimental results strongly evidence the capability of WSNet …” - Sec 2: “... deep learning based approaches has been recently proven ...” → “... deep learning based approaches have been recently proven ...” - The work by Aytar et al., 2016 is repeated twice in the references.