Title: SELECTIVE CONVOLUTIONAL UNITS: IMPROVING CNNS VIA CHANNEL SELECTIVITY

Abstract: Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently. In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., redistributing its computations to important channels. In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks. During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of (a) pruning unimportant channels, and (b) rewiring the pruned parameters to important channels. The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it. Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.

1. INTRODUCTION:

Nowadays, convolutional neural networks (CNNs) have become one of the most effective approaches in various fields of artificial intelligence. With a growing interest of CNNs, there has been a lot of works on designing more advanced CNN architectures (Szegedy et al., 2015; Simonyan & Zisserman, 2014; Ioffe & Szegedy, 2015) . In particular, the simple idea of adding identity connection in ResNet (He et al., 2016a) has enabled breakthroughs in this direction, as it allows to train substantially deeper/wider networks than before by alleviating existed optimization difficulties in previous CNNs. Recent CNNs can scale over a thousand of layers (He et al., 2016b) or channels (Huang et al., 2017b) without much overfitting, and most of these "giant" models consider identity connections in various ways (Xie et al., 2017; Huang et al., 2017b; Chen et al., 2017) . However, as CNN models grow rapidly, deploying them in the real-world becomes increasingly difficult due to computing resource constraints. This has motivated the recent literature such as network pruning (Han et al., 2015; He et al., 2017; Liu et al., 2017; Neklyudov et al., 2017) , weight quantization (Rastegari et al., 2016; Courbariaux & Bengio, 2016; Chen et al., 2018) , adaptive networks (Teerapittayanon et al., 2016; Figurnov et al., 2017; Bolukbasi et al., 2017; Huang et al., 2018) , and resource-efficient architectures (Huang et al., 2017a; Sandler et al., 2018; Ma et al., 2018) .

For designing a resource-efficient CNN architecture, it is important to process succinct representations of large-scale channels. To this end, the identity connections are useful since they allow to reduce the representation dimension to a large extent while "preserving" information from the previous layer. Such bottleneck architectures are now widely used in modern CNNs such as ResNet (He et al., 2016a) and DenseNet (Huang et al., 2017b) for parameter efficiency, and many state-of-the-art mobile-targeted architectures such as SqueezeNet (Iandola et al., 2016) , ShuffleNet (Zhang et al., 2017b; Ma et al., 2018) , MoblileNet (Howard et al., 2017; Sandler et al., 2018) , and CondenseNet (Huang et al., 2017a) commonly address the importance of designing efficient bottlenecks.

Contribution. In this paper, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit for efficient utilization of parameters in particular as a bottleneck upon identity connection. At a high-level, SCU performs a convolutional operation to transform a given input. The main goal of SCU, however, is rather to re-distribute their computations only to selected channels of importance, instead of processing the entire input naively. To this end, SCU has two special operations: (a) de-allocate unnecessary input channels (dealloc), and (b) re-allocate the obstructed channels to other channels of importance (realloc) (see Figure 1a ). They are performed without damaging the network output (i.e., function-preserving operations), and therefore one can call them safely at any time during training. Consequently, training SCU is a process that increases the efficiency of CNN by iteratively pruning or rewiring its parameters on-the-fly along with learning them.

In some sense, it is similar to how hippocampus in human brain learn, where new neurons are generated daily, and rewired into the existing network while maintaining them via neuronal apoptosis or pruning (Sahay et al., 2011a; b) .

We combine several new ideas to tackle technical challenges for such on-demand, efficient trainable SCU. First, we propose expected channel damage score (ECDS), a novel metric of channel importance that is used as the criterion to select channels for dealloc or realloc. Compared to other popular magnitude-based metrics (Li et al., 2016; Liu et al., 2017; Neklyudov et al., 2017) , ECDS allows capturing not only low-magnitude channels but also channels of low-contribution under the input distribution. Second, we impose channel-wise spatial shifting bias when a channel is reallocated, providing much diversity in the input distribution. It also has an effect of enlarging the convolutional kernel of SCU. Finally, we place a channel-wise scaling layer inside SCU with sparsity-inducing regularization, which also promotes dealloc (and consequently realloc as well), without further overhead in inference and training.

We evaluate the effectiveness of SCU by applying it to several modern CNN models including ResNet (He et al., 2016a) , DenseNet (Huang et al., 2017b) , and ResNeXt (Xie et al., 2017) , on various classification datasets. Our experimental results consistently show that SCU improves the efficiency of bottlenecks both in model size and classification accuracy. For example, SCU reduces the error rates of DenseNet-40 model (without any post-processing) by using even less parameters: 6.57% → 5.95% and 29.97% → 28.64% on CIFAR-10/100 datasets, respectively. We also apply SCU to a mobile-targeted CondenseNet (Huang et al., 2017a) model, and further improve its efficiency: it even outperforms NASNet-C (Zoph et al., 2018) , an architecture searched with 500 GPUs for 4 days, while our model is constructed with minimal efforts automatically via SCU.

There have been significant interests in the literature on discovering which parameters to be pruned during training of neural networks, e.g., see the literature of network sparsity learning (Wen et al., 2016; Lebedev & Lempitsky, 2016; Scardapane et al., 2017; Molchanov et al., 2017; Neklyudov et al., 2017; Louizos et al., 2017; 2018; Dai et al., 2018) . On the other hand, the progress is, arguably, slower for how to rewire the pruned parameters of a given model to maximize its utility. Han et al. (2016) proposed Dense-Sparse-Dense (DSD), a multi-step training flow applicable for a wide range of DNNs showing that re-training with re-initializing the pruned parameters can improve the performance of the original network. Dynamic network surgery (Guo et al., 2016) , on the other hand, proposed a methodology of splicing the pruned connections so that mis-pruned ones can be recovered, yielding a better compression performance. In this paper, we propose a new way of rewiring for parameter efficiency, i.e., rewiring for channel-selectivity, and a new architectural framework that enables both pruning and rewiring in a single pass of training without any postprocessing or re-training (as like human brain learning). Under our framework, one can easily set a targeted trade-off between model compression and accuracy improvement depending on her purpose, simply by adjusting the calling policy of dealloc and realloc. We believe that our work sheds a new direction on the important problem of training neural networks efficiently.

2. SELECTIVE CONVOLUTIONAL UNITS:

In this section, we describe Selective Convolutional Unit (SCU), a generic architectural unit for bottleneck CNN architectures. The overall structure of SCU is described in Section 2.1 and 2.2. In Section 2.3, we introduce a metric deciding channel-selectivity in SCU. We present in Section 2.4 how to handle a network including SCUs in training and inference.

2.1. OVERVIEW:

Bottleneck structures in modern CNNs. We first consider a residual function defined in ResNet (He et al., 2016a) which has an identity mapping: for a given input random variable X ∈ R I×H×W (H and W are the height and width of each channel, respectively, and I is the number of channels or feature maps) and a non-linear function F, the output of a residual function is written by Y = X + F(X). This function has been commonly used as a building block for designing recent deep CNN models, in a form that F is modeled by a shallow CNN. However, depending on how F is designed, computing F(X) can be expensive when I is large. For tackling the issue, bottleneck structure is a prominent approach, that is, to model F by F • R by placing a bottleneck R that firstly maps X into a lower dimension of I < I features. This approach, in essence, requires the identity connection, for avoiding information loss from X to Y. Namely, the identity connection enables a layer to save redundant computation (or parameters) for just "keeping" information from the input. Bottleneck structures can be used other than ResNet as well, as long as the identity connection exists. Recent architectures including DenseNet (Huang et al., 2017b) , PyramidNet (Han et al., 2017) and DPN (Chen et al., 2017) develop this idea with using a different aggregation function W instead of addition in ResNet, e.g., W(X, X ) = [X, X ] (channel-wise concatenation) for DenseNet. Designing R-F -W is now a common way of handling large features.

Channel-selectivity for efficient bottlenecks. Although placing a bottleneck R reduces much computation of the main function F , we point out the majority of modern CNNs currently use inefficient design of R itself, so that even the computation of R often dominates the remaining. In ResNet and DenseNet models, for example, bottlenecks are designed using a pointwise convolution with a batch normalization layer (BN) (Ioffe & Szegedy, 2015) and ReLU (Nair & Hinton, 2010) :

EQUATION

where Conv 1×1 I→I denotes a pointwise convolution that maps I features into I features, i.e., its parameters can be represented by a I × I matrix. This means that the parameters of R grows linearly on I, and it can be much larger than F if I I . For example, in case of DenseNet-BC-190 (Huang et al., 2017b) , 70% of the total parameters are devoted for modeling R, which is inefficient as the expressivity of a pointwise convolution is somewhat limited. In this paper, we attempt to improve the efficiency of R in two ways: (a) reducing the parameters in Conv 1×1 I→I by channel pruning, and (b) improving its expressivity by using the pruned parameters again. This motivates our goal to learn both channel-selectivity and parameters jointly.

Overall architecture of SCU. SCU is designed to learn the channel-selectivity via dynamic pruning and rewiring of channels during training. In this paper, we focus on putting SCU as a bottleneck R, and show that the channel-selectivity of SCU improves its parameter efficiency. Our intuition is that (a) the information-preserving nature of identity connection brings optimization benefits if neurons in its structure are dynamically pruned during training, and (b) such pruning can be particularly effective on bottlenecks as their outputs are in a much lower dimension compared to the input. Nevertheless, we believe that our ideas on SCU are not limited to the bottleneck structures, as the concept of channel-selectivity can be generalized to other structures. At a high level, SCU follows the bottleneck structure from (1), but for two additional layers: Channel Distributor (CD) and Noise Controller (NC) whose details are presented in Section 2.2. We model a non-linear function SCU : R I×H×W → R I ×H×W as follows (see Figure 1b ):

EQUATION

SCU has two special operations which control its input channels to process: (a) channel deallocation (dealloc), which obstructs unnecessary channels from being used in future computations, and (b) channel re-allocation (realloc), which allocates more parameters to important, non-obstructed channels by copying them into the obstructed areas. We design those operations to be function preserving, i.e. they do not change the original function of the unit, so that can be called at anytime during training without damage. Repeating dealloc and realloc alternatively during training translates the original input to what has only a few important channels, potentially duplicated multiple times. Namely, the parameters originally allocated to handle the entire input now operate on its important subset. On the way of designing the operations of function preserving, we propose Expected Channel Damage Score (ECDS) that leads to an efficient, safe way to capture unimportant channels by measuring how much the output of SCU changes on average (w.r.t. data distribution) after removing each channel. The details of ECDS are in Section 2.3.

2.2. DESIGN OF SCU: CD AND NC:

Channel Distributor (CD) is the principal mechanism of SCU and is placed at the beginning of the unit. The role of CD is to "rebuild" the input, so that unnecessary channels can be discarded, and important channels are copied to be emphasized. In essence, we implement this function by re-indexing and blocking the input channel-wise: CD(X) i := g i • X πi with an index pointer

π i ∈ {1, 2, • • • , I}, a gate variable g i ∈ {0, 1} for i = 1, 2, • • • , I.

Here, we notice that CD(X) may contain a channel copied multiple times, i.e., multiple π i 's can have the same value. Since SCU has different parameters for each channel, setting multiple π i 's has an effect of allocating more parameters to better process the channel pointed by π i .

Figure 2 : The kernel enlarging effect of spatial shifting.

We found that, however, it is hard to take advantage of the newly allocated parameters by simply copying a channel due to symmetry, i.e., the parameters for each channel usually degenerates.

Due to this, we consider spatial shifting biases b i = (b h i , b w i ) ∈ R 2 for each channel, as illustrated in Figure 2 . This trick can provide the copied channels much diversity in input distributions (and hence relaxing degeneracy), in a way that it is effective for the convolutional layer in SCU: it enlarges the convolutional kernel from 1 × 1 for the re-allocated channels only.

To summerize, CD(X) i takes (a) the channel which π i is pointing, in the spatially shifted form with bias b i , or (b) 0 if the gate g i is closed. Formally, CD can be represented by (π, g, b) = (π i , g i , b i ) I i=1 . CD(X) has the same size to X, and defined as follows:

EQUATION

Here, shift(X, b h , b w ) denotes the "shifting" operation along spatial dimensions of X. For each pixel location (i, j) in X, we define shift(X, b h , b w ) i,j as:

shift X, b h , b w i,j := H n=1 W m=1 X n,m • max 0, 1 -|i -n + b h | • max (0, 1 -|j -m + b w |) (4)

using a bilinear interpolation kernel. This formulation allows b h and b w to be continuous real values, thereby to be learned via gradient-based methods with other parameters jointly.

Noise Controller (NC) is a component for more effective training of SCU. As SCU continuously performs channel pruning via dealloc during training, the efficiency of SCU depends on which regularization is used. The key role of NC is to induce the training of SCU to get more channel-wise sparsity, so that more channels can be de-allocated safely. Formally, NC is a channel-wise re-scaling layer: NC(X) := X θ, 1 where θ = (θ i ) I i=1 are parameters to be learned. For the channel-wise sparsity, we impose sparsity-inducing regularization specifically on θ.

Although any sparsity-inducing regularization can be used for θ (Liu et al., 2017; Wen et al., 2016) , in this paper we adopt the Bayesian pruning approach proposed by Neklyudov et al. (2017) 2 for two reasons: (a) it is easy to incorporate into training process, and (b) we found that noise incurred from Bayesian parameters helps to recover damage from channel pruning. In general, a Bayesian scheme regards each parameter θ as a random variable with prior p(θ). Updating the posterior p(θ|D) from data D often leads the model to have much sparsity, if p(θ) is set to induce sparsity, e.g., by 1 denotes the element-wise product. 2 For completeness, we present for the readers an overview of Neklyudov et al. (2017) in Appendix B. log-uniform prior (Kingma et al., 2015) . Meanwhile, p(θ|D) is usually approximated with a simpler model q φ (θ), where φ are parameters to be learned. In case of NC, we regard each scaling parameter as a random variable, so that they become channel-wise multiplicative noises on input. We follow Neklyudov et al. (2017) for the design choices on q φ (θ) and p(θ), by fully-factorized log-normal distribution q φ (θ) = i LogN(θ i |µ i , σ 2 i ) and log-uniform prior p(θ) = i LogU(θ i ) ∝ 1 θi .

2.3. METRIC FOR CHANNEL-SELECTIVITY: ECDS:

Consider an input random variable X = (X i ∈ R H×W ) I i=1 of I features. From now on, we denote a SCU by S = (W NC , W CD , W BN , W Conv ), where each denotes the parameters in NC, CD, BN, and Conv, respectively. Here, W BN = (γ i , β i ) I i=1 , and

W Conv = (W Conv i ∈ R I ×1×1 ) I i=1 . Expected channel damage score (ECDS) aims for measuring E[SCU(X; S) -SCU(X; S -i )] ∈ R I ×H×W

, where S -i denotes a SCU identical to S but g i = 0. In other words, it is the expected amount of changes in outputs when S i is "damaged" or "pruned". The primary goal of this criteria is to make dealloc to be function preserving. We define ECDS(S) i by the (Euclidean) norm of the averaged values of E[SCU(X; S) -SCU(X; S -i )] over the spatial dimensions:

EQUATION

Notice that the above definition requires a marginalization over random variable X. One can estimate it via Monte Carlo sampling using training data, but this is computationally too expensive compared to other popular magnitude-based metrics (Li et al., 2016; Liu et al., 2017; Neklyudov et al., 2017) . Instead, we utilize the BN layer inside SCU, to infer the current input distribution of each channel at any time of training. This trick enables to approximate ECDS(S) i by a closed formula of S i , avoiding expensive computations of SCU(X; •), as in what follows.

Consider a hidden neuron x following BN and ReLU, i.e., y = ReLU(BN(x)), and suppose one wants to estimate E[y] without sampling. To this end, we exploit the fact that BN already "accumulates" its input statistics continuously during training. Under assuming that BN(x) ∼ N (β, γ 2 ) where γ and β are the scaling and shifting parameter in BN, respectively, it is elementary to check:

EQUATION

where φ N and Φ N denote the p.d.f. and the c.d.f. of the standard normal distribution, respectively.

The assumption is quite reasonable during training BN as each mini-batch is exactly normalized before applying the scaling and shifting inside BN. The idea is directly extended to obtain a closed form formula of ECDS(S) i under some assumptions, as stated in the following proposition.

Proposition 1. Assume BN(CD(X; W CD ); W BN ) i,h,w ∼ N (β i , γ 2 i ) for all i, h, w. 3 Then, it holds ECDS(S) i = g i • |γ i |φ N β i |γ i | + β i Φ N β i |γ i | (a) • E[θ i ] (b) • W Conv i (c)

, for all i.

The proof of the above proposition is given in Appendix D. In essence, there are three main terms in the formula: (a) a term that measures how much the input channel is active, (b) how much the NC amplifies the input, and (c) the total magnitude of weights in the convolutional layer. Therefore, it allows a way to capture not only low-magnitude channels but also channels of low-contribution under the input distribution (see Section 3.2 for comparisons with other metrics).

2.4. TRAINING AND INFERENCE PROCEDURES:

Consider a CNN model p(Y|X, Θ) employing SCU, where Θ denotes the collection of model parameters. For easier explanation, we rewrite Θ by (V, W): V consists of (π, g) in CDs, and W is the remaining ones. Given dataset D = {(x n , y n )} N n=1 , (V, W) is trained via alternating two phases: (a) training W via stochastic gradient descent (SGD), and (b) updating V via dealloc or realloc. The overall training process is mainly driven by (a), and the usage of (b) is optional. In (a), we use stochastic variational inference (Kingma & Welling, 2013) in order to incorporate stochasticity incurred from NC, so that SCU can learn its Bayesian parameters in NC jointly with the others via SGD. On the other hand, in (b), dealloc and realloc are called on demand during training depending on the purpose. For example, one may decide to call dealloc only throughout the training to obtain a highly compressed model, or one could use realloc as well to utilize more model parameters. Once (b) is called, (a) is temporally paused and V are updated.

Training via stochastic variational inference. We can safely ignore the effect of V during training of W, since they remain fixed. Recall that, each noise θ from a NC is assumed to follow q φ (θ) = LogN(θ|µ, σ 2 ). Then, θ can be "re-parametrized" with a noise ε from the standard normal distribution as follows:

θ = f (φ = (µ, σ), ε) = exp (µ + σ • ε)

, where ε ∼ N (0, 1 2 ). Stochastic variational inference (Kingma & Welling, 2013) allows a minibatch-based stochastic gradient method for θ, in such case that θ can be re-parametrized with an non-parametric noise. The final loss we minimize for a minibatch {(x i k , y i k )} M k=1 becomes (see Appendix F for more details):

EQUATION

where

ε i k = (ε i k ,u )

|φ| u=1 is a sampled vector from the fully-factorized standard normal distribution. Channel de-allocation and re-allocation. Consider a SCU S = (W NC , W CD , W BN , W Conv ). The main role of dealloc and realloc is to update W CD in S that are not trained directly via SGD. They are performed as follows: select slices to operate by thresholding ECDS(S), and update S from the selected channels. More formally, when dealloc is called, S i 's where ECDS(S) i < T l for a fixed threshold T l are selected, and g i 's in W CD are set by 0. If one chooses small T l , this operation does not hurt the original function. On the other hand, realloc selects channels by collecting S i where ECDS(S) i > T h , for another threshold T h . Each of the selected channels can be re-allocated only if there is a closed channel in S. If there does not exist a enough space, channels with higher ECDS have priority to be selected. A single re-allocation of a channel S i to a closed channel S j consists of several steps: (i) open S j by g j ← 1, (ii) copy

W NC j , W BN j ← W NC i , W BN i (iii) set W Conv j

← 0, (iv) re-initialize the shifting bias b j , and (v) set π j ← π i . This procedure is function-preserving, due to (iii).

After training a SCU S, one can safely remove S i 's that are closed, to yield a compact unit. Then, CDs are now operated by "selecting" channels rather than by obstructing, thereby the subsequent layers play with smaller dimensions. Hence, at the end, SCU is trained to select only a subset of the input for performing the bottleneck operation. For NC, on the other hand, one can still use it for inference, but efficient inference can be performed by replacing each noise θ i by constant E[θ i ], following the well-known approximation used in many dropout-like techniques (Hinton et al., 2012) .

3. EXPERIMENTAL RESULTS:

In our experiments, we apply SCU to several well-known CNN architectures that uses bottlenecks, and perform experiments on CIFAR-10/100 (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015) classification datasets. The more details on our experimental setups, e.g., datasets, training details, and configurations of SCU, are given in Appendix G.

3.1. IMPROVED CNN MODELS WITH SCU:

Improving existing CNNs with SCU. We consider models using ResNet (He et al., 2016a) , DenseNet (Huang et al., 2017b) and ResNeXt (Xie et al., 2017) architectures. In general, every model we used in this paper forms a stack of multiple bottlenecks, where the definition of each bottleneck differs depending on its architecture except that it can be commonly expressed by R-F -W (the details are given in Table 5 in the appendix). We compare the existing models with the corresponding new ones in which the bottlenecks are replaced by SCU. For each SCU-based model, we consider three cases: (a) neither dealloc nor realloc is used during training, (b) only dealloc is used, and (c) both dealloc and realloc are used. We measure the total number of parameters in bottlenecks, and error rates.

Table 1 : Comparison of performances on CIFAR-10/100 datasets in terms of their total parameter usage in R-parts of the models, and their classification error rates. Here, "S" denotes whether SCU is used, and "D", "R" denote the use of dealloc and realloc, respectively. We indicate k by the growth rate of DenseNet. All the values in the table are taken from averaging over 5 trials. Table 1 compares the existing CNN models with the corresponding ones using SCU, on CIFAR-10/100. The results consistently demonstrate that SCU improves the original models, showing their effectiveness in different ways. When only dealloc is used, the model tends to be trained with minimizing their parameter to use. Using realloc, SCU now can utilize the de-allocated parameters to improve their accuracy aggressively. Note that SCU can improve the accuracy of the original model even neither dealloc nor realloc is used. This gain is from the regularization effect of stochastic NC, acting a dropout-like layer. We also emphasize that one can set a targeted trade-off between compression of SCU and accuracy improvement depending on her purpose, simply by adjusting the calling policy of dealloc and realloc. For example, in case of DenseNet-100 model on CIFAR-10, one can easily trade-off between reductions in (compression, error) = (-51.4%, -1.78%) and (-18.2%, -8.24%). In overall, SCU-based models achieve both model compression and accuracy improvement under all tested architectures. Table 2 shows the results on ImageNet, which are consistent to those on CIFAR-10/100. Notice that reducing parameters and error simultaneously is much more non-trivial in the case of ImageNet, e.g., reducing error 23.6% → 23.0% requires to add 51 more layers to ResNet-101 (i.e., ResNet-152), as reported in the official repository of ResNet (He et al., 2016c) .

Designing efficient CNNs with SCU. We also demonstrate that SCU can be used to design a totally new efficient architecture. Recall that, in this paper, SCU focus on the bottlenecks inside the overall structure. The other parts, F or W, are other orthogonal design choices. To improve the efficiency of the parts, we adopt some components from CondenseNet (Huang et al., 2017a) , which is one of the state-of-the-art architectures in terms of computational efficiency, designed for mobile devices. Although we do not adopt their main component, i.e., learned group convolution (LGC) as it also targets for the bottleneck as like SCU, we can still utilize other components of CondenseNet: increasing growth rate (IGR) (doubles the growth rate of DenseNet for every N blocks starting from 8) and the use of group convolution for F . Namely, we construct a new model, coined CondenseNet-SCU by adopting IGR and GC upon a DenseNet-182 model with SCU. We replace each 3×3 convolution for F by a group convolution of 4 groups. We train this model using dealloc only to maximize the computational efficiency. In Table 3 , we compare our model with state-of-the-art level CNNs, including ResNet-1001 (He et al., 2016b) , WRN-28-10 (Zagoruyko & Komodakis, 2016), NASNet-C (Zoph et al., 2018) , and the original CondenseNet-182. As one can observe, our model shows better efficiency compared to the corresponding CondenseNet, suggesting the effectiveness of SCU over

LGC. Somewhat interestingly, ours even outperforms NASNet-C that is an architecture searched over thousands of candidates, in both model compression and accuracy improvement. We finally remark that CondenseNet-SCU-182 model presented in Table 3 originally has 6.29M parameters in total before training, devoting 5.89M for bottlenecks, i.e., it is about 93.7% of the total number of parameters. This is indeed an example in that reducing overhead from bottlenecks is important for better efficiency, which is addressed by SCU.

3.2. ABLATION STUDY:

We also perform numerous ablation studies on the proposed SCU, investigating the effect of the key components: CD, NC, and ECDS. For evaluation, we use the DenseNet-SCU-40 model (DenseNet-40 using SCU) trained for CIFAR-10. We also follow the training details described in Appendix G.

Spatial shifting and re-allocation. We propose spatial shifting as a trick in realloc procedure to provide diversity in input distributions. To evaluate its effect, we compare three DenseNet-SCU-40 models with different configurations of SCU: (D) only dealloc during training, (+R) realloc together but without spatial shifting, and (+R+S) further with the shifting. Figure 3a shows that +R does not improve the model performance much compared to D, despite +R+S outperforms both of them. This suggests that copying a channel naively is not enough to fully utilize the rewired parameters, and spatial shifting is an effective way to overcome the issue. 

ECDS M2 ECDS M3 SNR < 1 M4 SNR < 2.3 M5 2 < 0.25

Sparsity-inducing effect of NC. We place NC in SCU to encourage more sparse channels. To verify such an effect, we consider DenseNet-SCU-40 model (say M1) and its variant removing NC from SCU (say M2). We first train M1 and M2 calling neither dealloc nor realloc, and compare them how the ECDS of each channel is distributed. Figure 3b shows that M1 tends to have ECDS closer to zero, i.e., more channels will be de-allocated than M2. Next, we train these models using dealloc, to confirm that NC indeed leads to more deallocation. The left panel of Figure 3c shows that the number of de-allocated channels of M1 is relatively larger than that of M2, which is the desired effect of NC. Note that M1 also outperforms M2 on error rates, which is an additional advantage of NC from its stochastic regularization effect.

Effectiveness of ECDS. Nevertheless, remark that M2 in Figure 3c already de-allocates many channels, which suggests that SBP (used in NC) is not crucial for efficient de-allocation. Rather, the efficiency mainly comes from ECDS. To prove this claim, we evaluate three variants of M1 which use different de-allocation policies than ECDS < T l : (a) SNR < 1 (thresholding the signal-to-noise ratio of NC in each channel by 1, proposed by the original SBP; M3), (b) SNR < 2.3 (M4) and (c) 2 < 0.25 (thresholding W Conv i 2 ; M5). We train them using only dealloc, and compare the performances with the proposed model (M1). The right panel of Figure 3c shows the results of the three variants. First, we found that the M3 could not de-allocate any channel in our setting (this is because we prune a network on-the-fly during training, while the original SBP only did it after training). When we de-allocate competitive numbers of channels against M1 by tuning thresholds of others (M4 and M5), the error rates are much worse than that of M1. These observations confirm that ECDS is a more effective de-allocation policy than other magnitude-based metrics.

A STOCHASTIC VARIATIONAL INFERENCE FOR BAYESIAN MODELS:

Consider a probabilistic model p(Y|X, θ) between two random variables X and Y, and suppose one wants to infer θ from a dataset D = {(x n , y n )} N n=1 consisting N i.i.d. samples from the distribution of (X, Y). In Bayesian inference, θ is regarded as a random variable, under assuming some prior knowledge in terms of a prior distribution p(θ). The dataset D is then used to update the posterior belief on θ, namely p(θ|D) = p(D|θ)p(θ)/p(D) from the Bayes rule. In many cases, however, computing p(θ|D) through Bayes rule is intractable since it requires to compute intractable integrals. To address the issue, variational inference approximates p(θ|D) by another parametric distribution q φ (θ), and tries to minimize the KL-divergence D KL (q φ (θ) p(θ|D)) between q φ (θ) and p(θ|D). Instead of directly minimizing it, one typically maximizes the variational lower bound L(φ), due to the following:

EQUATION

where

EQUATION

In case of complex models, however, expectations in ( 9) are still intractable. Kingma & Welling (2013) proposed an unbiased minibatch-based Monte Carlo estimator for them, which can be used when q φ (θ) is representable by

EQUATION

EQUATION

Now we can solve optimize L(φ) by stochastic gradient ascent methods, if f is differentiable. For a model having non-Bayesian parameters, say W, we can still apply the above approach by maximizing

EQUATION

where φ and W can be jointly optimized under L SGVB (φ, W) L(φ, W).

B STRUCTURED BAYESIAN PRUNING:

Structured Bayesian pruning (SBP) (Neklyudov et al., 2017) is a good example to show how stochastic variational inference can be incorporated into deep neural networks. The SBP framework assumes X to be an object of I features, that is, X = (X i ) I i=1 . For example, X ∈ R I×H×W can be a convolutional input consisting I channels, of the form X = (X i ∈ R W ×H ) I i=1 where W and H denote the width and the height of each channel, respectively. It considers a dropout-like layer with a noise vector θ = (θ i ) I i=1 ∼ p noise (θ), which outputs X θ of the same size as X. 4 Here, θ is treated as a random vector, and the posterior p(θ|D) is approximated by a fully-factorized truncated log-normal distribution q φ (θ):

EQUATION

where 1 Meanwhile, the prior p(θ) is often chosen by a fully-factorized log-uniform distribution, e.g., Sparse Variational Dropout (Molchanov et al., 2017) , and SBP use the truncated version:

EQUATION

The reason why they use truncations for q φ (θ) and p(θ) is to prevent D KL (q φ (θ) p(θ)) to be improper. Previous works (Kingma et al., 2015; Molchanov et al., 2017) ignore this issue by implicitly regarding them as truncated distributions on a broad interval, but SBP treats this issue explicitly.

Note that, each θ i ∼ q φ (θ i ) = LogN(θ i |µ i , σ 2 i ) in the noise vector θ can be re-parametrized with a non-parametric uniform noise ε i ∼ U(ε|0, 1) by:

EQUATION

where α i = a-µi σi , β i = b-µi σi , and Φ denotes the cumulative distribution function of the standard normal distribution. Now one can optimize φ = (µ, σ) jointly with the weights W of a given neural network via stochastic variational inference described in Section A. Unlike Molchanov et al. (2017) , SBP regards W as a non-Bayesian parameter, and the final loss L SBP to optimize becomes

EQUATION

Here, the KL-divergence term is scaled by α to compensate the trade-off between sparsity and accuracy. In practice, SBP starts from a pre-trained model, and re-trains it using the above loss. Due to the sparsity-inducing behavior of log-uniform prior, θ is forced to become more noisy troughout the re-training. Neurons with θ of signal-to-noise ratio (SNR) below 1 are selected, and removed after the re-training:

EQUATION

)

C BAYESIAN PRUNING AND IDENTITY CONNECTIONS SCU requires "training-time removal" of input channels for the channel de-allocation and reallocation to work. But usually, this process should be done carefully since it can make the optimization much difficult and put the network into a bad local minima. In particular, it occurs if we select channels to remove too aggressively. It is known that this issue becomes more pronounced in Bayesian neural networks (Sønderby et al., 2016; Molchanov et al., 2017; Neklyudov et al., 2017; Louizos et al., 2017) , such as SBP we use in this paper. Recall the variational lower bound objective in (12), for Bayesian parameters φ and non-Bayesian W. If the gradient of the first term N n=1 E q φ [log p(y n |x n , θ, W)] on the right-hand side does not contribute much on ∇ φ L(φ, W), then φ will be optimized mostly by -∇ φ D KL (q φ (θ) p(θ)), that is, to follow the prior p(θ). Unfortunately, in practice, we usually observe this phenomena at the early stage of training, when W are randomly initialized. In that case then, q φ (θ) will become p(θ) too fast because of the "uncertain" W, thereby many channels will be pruned forever, in SBP for example. This problem is usually dealt with in one of two ways: (a) using a pre-trained network as a starting point of W (Molchanov et al., 2017; Neklyudov et al., 2017) , and (b) a "warm-up" strategy, where the KL-divergence term is rescaled by β that increases linearly from 0 to 1 during training (Sønderby et al., 2016; Louizos et al., 2017) . In this paper, however, neither methods are used, but instead we have found that the problem can be much eased with identity connections, as it can eliminate a possible cause of the optimization difficulty from removing channels: optimization difficulty from losing information as an input passes through a deep network. The presence of identity connection implies that the information of an input will be fully preserved even in the case when all the parameters in a layer are pruned. This may not be true in models without identity, for example, in VGGNet (Simonyan & Zisserman, 2014) , one can see that the information of an input will be completely lost if any of the layers removes its entire channels. This suggests us that identity connections can be advantageous not only for scaling up the network architectures, but also for reducing the size of them. ). Then, we have:

D PROOF:

SCU(X; S) -SCU(X; S -i ) =: S i (X) = Conv 1×1,(i) I→I ((NC • ReLU • BN • CD)(X) i ) = Conv 1×1,(i) I→I (NC(ReLU(Y)) i ) = Conv 1×1,(i) I→I (max(Y, 0) i θ) = Conv 1×1,(i) I→I (max(Y i,:,: , 0) • θ i ) =: Conv 1×1,(i) I→I (Z i ).

(19) Now, check that ECDS(S i ) becomes:

EQUATION

By the assumption that Y i,h,w ∼ N (β i , γ 2 i ) for all h, w, we get:

EQUATION

EQUATION

where φ N and Φ N denote the probability distribution function and the cumulative distribution function of the standard normal distribution.

Therefore, the desired formula for ECDS(S i ) can be obtained by using the linearity of expectation:

ECDS(S i ) = 1 HW h,w E[Conv 1×1,(i) I→I (Z i )] :,h,w = 1 HW h,w   E   1/2 x=-1/2 1/2 y=-1/2 W Conv i,j,x,y • Z i,h+x,w+y     I j=1 = 1 HW h,w (W Conv i,j • E[Z i,h,w ]) I j=1 = f (S i ) • 1 HW h,w (W Conv i,j ) I j=1 = f (S i ) • W Conv i = g i • f (S i ) • W Conv i .

(23) This completes the proof of Proposition 1.

E EMPIRICAL SUPPORTS ON THE ASSUMPTION OF PROPOSITION 1:

To validate whether the assumption BN(CD(X; W CD ); W BN ) i,h,w ∼ N (β i , γ 2 i ) holds in modern CNNs, we first observe that, once we ignore the effects from spatial shifting,5 a necessary condition of the assumption is that (X i,:,: ) are identically distributed normal for a given channel X i . This is because BN and CD do not change the "shape" of pixel-wise distributions of X i . From this observation, we conduct a set of experiments focusing on a randomly chosen hidden layer in a DenseNet-40 model. We analyze the empirical distribution of the hidden activation incoming to the layer calculated from CIFAR-10 test dataset. Since the data consists of 10,000 samples, we get an hidden activation X test ∈ R 10000×C×32×32 ,6 where C denotes the number of channels of the input. Now, suppose that we focus on a specific channel in X test , say X test c ∈ R 10000×32×32 . Notice that if our assumption is perfectly satisfied, then all the slices X test c,h,w ∈ R 10000 will represent a fixed normal distribution for any h, w. Interestingly, by analyzing empirical distributions of X test c,h,w for varying h and w, we found that: (a) for a fixed c, most of the empirical distributions from X test c,h,w have unimodal shapes, except for the pixels in boundaries of the channel (Figure 4a , 4b), and (b) for a large portion of c ∈ {1, • • • , C}, the empirical means and variances of X test c,h,w 's are concentrated in a cluster (Figure 4c, 4d ).

These observations suggest us that the assumption of Proposition 1 can be reasonable except for the boundaries. We also emphasize that these trends we found are also appeared even when the model is not trained at all (Figure 4c ), that is, all the weights are randomly initialized, which implies that these properties are not "learned", but come from a structural property of CNN, e.g. equivariance on translation, or the central limit theorem. This observation provides us another support why the ECDS formula stated in Proposition 1 is valid at any time during training.

F TRAINING SCU VIA STOCHASTIC VARIATIONAL INFERENCE:

From Θ = (V, W): V consists of (π, g) in CDs, we further rewrite W by (W NC , W C ): W NC the parameters in NCs, and W C is the remaining ones. One can safely ignore the effect of V during training of (W NC , W C ), since they remain fixed. Recall that each noise θ from a NC is assumed to follow LogN(θ|µ, σ 2 ). They can be re-written with a noise ε from the standard normal distribution, i.e., θ = f ((µ, σ), ε) = exp (µ + σ • ε), where ε ∼ N (0, 1 2 ). In such case that each noise θ from NC can be "re-parametrized" with an non-parametric noise and the corresponding parameters φ = (µ, σ), we can then use stochastic variational inference (Kingma & Welling, 2013) for the optimization of (W NC , W C ) with a minibatch-based stochastic gradient method (see Appendix A for more details). Then, the final loss we minimize for a minibatch {(x i k , y i k )} M k=1 becomes:

L SCU (W NC , W C ) = - 1 M M k=1 log p(y i k |x i k , f (W NC , ε i k ), W C ) + 1 N (θ,φ) D KL (q φ (θ) p(θ)) (24)

where

ε i k = (ε i k ,u )

|φ| u=1 is a sampled vector from the fully-factorized standard normal distribution, and D KL (• •) denotes the KL-divergence. Although not shown in (24), an extra regularization term R(W C ) can be added to the loss for the non-Bayesian parameters W C , e.g., weight decays.

In fact, in our case, i.e. q φ (θ) = LogN(θ|µ, σ 2 ) and p(θ) = LogU(θ), D KL (q φ (θ) p(θ)) becomes improper:

D KL (LogN(θ|µ, σ 2 ) LogU(θ)) = C -log σ,

where C → ∞.

(25) As we explain in Appendix B, SBP bypasses this issue by using truncated distributions on a compact interval [a, b] for q φ (θ) and p(θ). We found that, however, this treatment also imposes extra computational overheads on several parts of training process, such as on sampling noises and computing D KL (q φ (θ) p(θ)). These overheads are non-negligible on large models like ResNet or DenseNet, which we are mainly focusing on. Therefore, unlike SBP, here we do not take truncations on q φ (θ) and p(θ) due to practical consideration, assuming an approximated form between the truncated distributions of q φ (θ) and p(θ) on a large interval. Then we can replace each D KL (q φ (θ) p(θ)) in ( 24) by -log σ for optimization. In other words, each noise θ in NC is regularized to a larger variance, i.e., the more "noisy". We observed that this approximation does not harm much on the performance of SCU. Nevertheless, one should be careful that q φ (θ) and p(θ) should not be assumed as the un-truncated forms itself, but instead as approximated forms of truncated distributions on a large interval, not to make the problem ill-posed. As used in SBP, if they are truncated, the KL-divergence becomes: G EXPERIMENT SETUPS Datasets. We perform our experiments extensively on CIFAR-10 and CIFAR-100 (Krizhevsky, 2009) classification datasets. CIFAR-10/100 contains 60,000 RGB images of size 32 × 32 pixels, 50,000 for training and 10,000 for test. Each image in the two datasets is corresponded to one of 10 and 100 classes, respectively, and the number of data is set evenly for each class. We use a common scheme for data-augmentation (Srivastava et al., 2015; Lin et al., 2013; He et al., 2016a; Huang et al., 2017b) . ImageNet classification dataset, on the other hand, consists of 1.2 million training images and 50,000 validation images, which are labeled with 1,000 classes. We follow (Huang et al., 2017a; He et al., 2016a) for preprocessing of data in training and inference time.

D KL (LogN

Training details. All models in our experiments is trained by stochastic gradient descent (SGD) method, with Nesterov momentum of weight 0.9 without dampening. We use a cosine shape learning rate schedule (Loshchilov & Hutter, 2016) , i.e., decreasing the learning rate gradually from 0.1 to 0 throughout the training. We set the weight decay 10 -4 by for non-Bayesian parameters of each model. We train each CIFAR model for 300 epochs with mini-batch size 64 following Huang et al. (2017b) , except for the "DenseNet-BC-190+mixup" models as they are trained for 200 epochs following the original setting (Zhang et al., 2018) . For ImageNet models, on the other hand, we train for 120 epochs with mini-batch size 256.

Configurations of SCU. When a SCU S = (W NC , W CD , W BN , W Conv ) is employed in a model, we initialize W NC = (µ, σ) by (0, e -3 ), and W CD = (π i , g i , b i ) I i=1 by (i, 1, 0) I i=1 . Initializations of W BN and W Conv may differ depending on models, and we follow the initialization scheme of the given model. In our experiments, we follow a pre-defined calling policy when dealloc and realloc will be called throughout training. If dealloc is used, it is called at the end of each epoch of training. On the other hand, if realloc is used, it start to be called after 10% of the training is done, called for every 3 epochs, and stopped in 50% of training is done. The thresholds for dealloc and realloc, i.e. T l and T h , is set by 0.0025 and 0.05, respectively, except for CondenseNet-SCU-182 (Table 3 ), in which T l is adjusted by 0.001 for an effective comparison with the baseline. For all the CIFAR-10/100 models, we re-initialize b i by a random sample from [-1.5, 1.5] × [-1.5, 1.5] pixels uniformly whenever a channel slice S i is re-open via realloc process. We set the weight decay on each b i to 10 -5 separately from the other parameters. For the ImageNet results (Table 2 ), however, we did not jointly train b for faster training. Instead, each b i is set fixed unless it is re-initialized via realloc. In this case, we sampled a point from [-2.5, 2.5 ] × [-2.5, 2.5] pixels uniformly for the re-initialization. We found that this simple reallocation scheme can also improve the efficiency of SCU.

Models.:

In general, every model we used here forms a stack of multiple bottlenecks, where the definition of each bottleneck differs depending on its architecture (see Table 5 ). Each stack is separated into three (CIFAR-10/100) or four (ImageNet) stages by average pooling layers of kernel 2 × 2 to perform down-sampling. Each of the stages consists N bottleneck blocks, and we report which N is used for all the tested models in Table 6 . The whole stack of each model follows a global average pooling layer (Lin et al., 2013) and a fully connected layer, and followed by single convolutional

:

In Appendix E, we also provide empirical supports on why the assumption holds in modern CNNs.

CONCLUSIONWe demonstrate that CNNs of large-scale features can be trained effectively via channel-selectivity, primarily focusing on bottleneck architectures. The proposed ideas on channel-selectivity, however, would be applicable other than the bottlenecks, which we believe is an interesting future research direction. We also expect that channel-selectivity has a potential to be used for other tasks as well, e.g., interpretability (Selvaraju et 2017), robustness(Goodfellow et al., 2014), and memorization(Zhang et al., 2017a).

However, we emphasize that one can safely ignore this effect, since it can be successfully bypassed in practice by padding the input with boundary pixels.

Here, we get a tensor of 32 × 32 channels, since we choose a layer from the first block of the model.

Figures and Tables:

Figure 1: (a) An illustration of channel de-allocation and re-allocation procedures. The higher the saturation of the channel color, the higher the ECDS value. (b) The overall structure of SCU.

Figure 3: Ablation study on SCU. The model configurations for (b, c) are presented in Table4. (a) Comparison of error rates between SCU (green) and the ablation on shifting (red). (b) Histograms on ECDS for all the channels of SCUs, using neither dealloc nor realloc. (c) Comparisons of the number of de-allocated channels and error rates. Here, all the models are trained with dealloc.

[a,b] denotes the indicator function for the inveral [a, b].4 denotes the element-wise product.

OF PROPOSITION 1One can assume that S i is open, i.e. g i = 1, otherwise ECDS(S) i = 0 by definition since S = S -i . Say Y := BN(CD(X; W CD ); W BN ), and let Conv 1×1,(i) I→I : R 1×H×W → R I ×H×W be the i-th slice of Conv 1×1 I→I , which can be defined by the convolution with W Conv i so that Conv 1×1 I→I (X; W Conv ) =

Figure 4: Pixel-wise input distributions of a layer in DenseNet-40 model. (a, b) Empirical distributions of three randomly chosen pixels in a fixed channel of input, which are inferred from CIFAR-10 test dataset. (c, d) Scatter plots between empirical mean and standard deviation of each pixel distributions, plotted for 3 representative channels in the input. Each plot consists 1,024 points, as a channel have 32 × 32 pixels. Pixels in boundaries are specially marked as ×.

[a,b]  (θ|µ, σ 2 ) LogU [a,b] (θ)) = log b -a √ 2πeσ 2 -log Φ(β) -Φ(α) -αφ(α) -βφ(β) 2(Φ(β) -Φ(α)) ,(26) where α = a-µ σ , β = b-µ σ , φ(•) and Φ(•) are the p.d.f. and c.d.f. of the standard normal distribution, respectively.

Comparison of model performances onImageNet classification dataset. Here, we measure the single-crop validation error rates. "S" denotes whether the model uses SCU or not, and "D", "R" denote the use of dealloc and realloc, respectively.

Table content: <table><tr><td>Model</td><td colspan="2">S D R R-Params Error (%)</td></tr><tr><td>ResNet-50</td><td>3.50M</td><td>23.19</td></tr><tr><td>(bottleneck)</td><td>3.52M</td><td>23.22</td></tr><tr><td/><td>2.22M</td><td>23.19</td></tr><tr><td/><td>3.04M</td><td>22.82</td></tr><tr><td>DenseNet-121</td><td>1.00M</td><td>23.63</td></tr><tr><td>(k = 32)</td><td>1.01M</td><td>23.62</td></tr><tr><td/><td>0.79M</td><td>23.62</td></tr><tr><td/><td>0.97M</td><td>23.24</td></tr></table>



Table content: <table><tr><td>Model</td><td>Params</td><td>FLOPs Error</td></tr><tr><td>ResNet-1001</td><td>16.1M</td><td>2,357M 4.62%</td></tr><tr><td>WRN-28-10</td><td>36.5M</td><td>5,248M 4.17%</td></tr><tr><td>ResNeXt-29</td><td colspan="2">68.1M 10,704M 3.58%</td></tr><tr><td>DenseNet-190</td><td>25.6M</td><td>9,388M 3.46%</td></tr><tr><td>NASNet-C</td><td>3.1M</td><td>-3.73%</td></tr><tr><td>VGGNet-Pruned</td><td>2.30M</td><td>391M 6.20%</td></tr><tr><td>ResNet-164-Pruned</td><td>1.10M</td><td>275M 5.27%</td></tr><tr><td>DenseNet-40-Pruned</td><td>0.35M</td><td>381M 5.19%</td></tr><tr><td>CondenseNet-182</td><td>4.20M</td><td>513M 3.76%</td></tr><tr><td colspan="2">CondenseNet-SCU-182 2.59M</td><td>286M 3.63%</td></tr></table>

Configurations.

Table content: <table><tr><td>Model Metric</td><td>NC</td></tr><tr><td>M1</td><td/></tr></table>

Peer review (iclr2019):
 This is an architecture design paper. It proposes a general structure called Selective Convolutional Unit that the authors claim to be useful for various CNN models. The SCU structure contains two major parts: CD and NC. CD for compressing/pruning channels and NC for multiplicative noise. The paper gives a measure, called expected channel damage score, on the change of the output for SCU. It also shows the effectiveness of SCU on CIFAR-10, CIFAR-100 and imagenet. Some questions and concerns: 1. The paper spends too much space introducing the bottleneck structures and a whole lot of the details on the optimization of NC and CD are put in the appendix. I would suggest to reduce the section of introductory part and put a shorter version of appendix A and B to the main text so that the readers know more about the architecture and how it is optimized. In particular, the description on NC is confusing since without looking at the appendix it is not clear how the prior p(.theta) is used. 2. The experiment shows improvement on densenet and resnetXT, but the result is not the state-of-the-art. Wide-Resnet seems to get better accuracy on both CIFAR-10 and CIFAR-100 compared to the best accuracy reported in this paper. Also the number reported by the original densenet paper on imagenet seems to be better (densenet-264 has an error rate of 22.15/20.80) 3. In your CD design, channel assignment .pi is a discrete variable. How is it optimized in the training process? 4. The proof of proposition 1 does not look correct to me. The optimization procedure makes use of the data X to determine your NC variable .theta so .theta depends on X. In this way you cannot factorize the expectation in the equation below (20) in your appendix.