Year: 2019

Abstract:
Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectified linear function-like activations (e.g. ReLU) as building blocks. Previous work pointed out that learning feature channels with equal magnitudes is important for a CNN to achieve good generalization ability. However, the above “Norm+ReLU-like” basic block often learns inhibited channels that has small magnitude (i.e. contributes little to the feature representation), impeding both learning and generalization ability of CNNs. This problem is seldom explored in the literature. To mitigate the inhibited channels and encourage channels to contribute equally to the feature representation, we propose a new building block, Channel Equilibrium (CE), which is able to prevent inhibited channels in both experiments and theory. CE has several appealing properties. First, CE can be stacked after many different normalization methods such as BN and Group Normalization (GN), as well as integrated into many advanced CNN architectures such as ResNet and MobileNet V2 to form a series of CE networks (CENets), outperforming existing network architectures. Second, CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. Third, extensive experiments show that CE achieves state-of-the-art results on various challenging benchmarks such as ImageNet and COCO. The models and codes will be released.

heading: 1 INTRODUCTION
text: Normalization is an important technique for a wide range of tasks such as image classification (Ioffe & Szegedy, 2015), object detection (He et al., 2017a; Wu & He, 2018), and image generation (Miyato et al., 2018). In recent years, a lot of work improved normalization methods, such as batch normalization (BN) (Ioffe & Szegedy, 2015), layer normalization (LN) (Ba et al., 2016) and switchable normalization (SN) (Luo et al., 2018). These methods are often used together with the ReLU-like activation functions such as ReLU (Glorot et al., 2011; Nair & Hinton, 2010), ELU (Clevert et al., 2015) and Leaky ReLU (LReLU) (Maas et al., 2013), making the “Norm+ReLU-like” module become one of the most widely-used building blocks of modern CNNs. This work investigates and alleviates the inhibited channels emerged in the “Norm+ReLU-like” building block, which consists of a normalization layer and a ReLU-like activation function given by
yncij = g(x̃ncij), x̃ncij = γcx̄ncij + βc, (1)
where subscript n, c, i, and j denote indices of a sample, a channel, height and width of a feature channel respectively. For instance, yncij indicates output value of the location (i, j) in the c-th channel of the n-th sample. And x̃ncij and x̄ncij represent normalized channel features and standardized channel features respectively. γ and β are two vectors of parameters, where each element re-scales and re-shifts the standardized features for each channel c. Moreover, g(·) denotes a ReLU-like activation function.
As pointed out in (Morcos et al., 2018), a CNN that generalizes well would have its channels contributed equally in its feed-forward computations. We term this desired property “channel equalization”. However, a recent study disclosed a critical problem of the “Norm+ReLU-like” builiding block, known as “channel collapse”, where certain channels always produce small output values given any input. For example, the lottery hypothesis (Frankle & Carbin, 2018) claimed that one over-parameterized CNN always contains unimportant channels that contribute little to the network’s prediction. This paper shows that these unimportant channels are inhibited channels, which usually
associated with small values of γ in Eqn.(1) or small values of the channel features (In this paper, inhibited channel is defined as channel whose all feature values are less than 1e− 2). For example, as shown in Fig.1(a,b), the inhibited channels exist in many “Norm+ReLU-like” basic block such as ‘LN+ReLU’, ‘BN+ELU’ and ‘BN+ReLU’. This phenomenon has motivated many investigations to directly remove these inhibited channels, such as network slimming (Liu et al., 2017; Yu et al., 2018) and channel pruning (He et al., 2017b). However, disequilibrium among channels caused by the inhibited channels would do harm to generalization ability of the network and simply removing the inhibited channels that are inactive during training does not help improve learning capacity.
Instead of removing the inhibited channels, we present an alternative perspective by proposing a novel building block, named Channel Equilibrium (CE), to recover and equalize the inhibited channels by encouraging channels to contribute equally in the feature representation learning process, thus enhancing the representation and generalization of CNNs. To this end, a key observation from Eqn.(1) is that the dependency (covariance) matrix of all the feature channels after normalization is scaled by γγT. Let this covariance matrix be Σ. We will see that by applying a decorrelation operator, i.e. Σ− 1 2 , we can not only effectively eliminate the correlations between computations of channel features and the magnitude of γ, but also equalize the channels’ magnitudes (Barlow et al., 1961; Bengio & Bergstra, 2009). This operator enables all the channels to play an equal role in the computations of a CNN, improving its generalization ability. For example, as shown in Fig.1, the VGGNet (Simonyan & Zisserman, 2014) equipped with CE is able to effectively prevent inhibited channels and achieves channel equalization in various of “Norm+RelU-like” blocks, consistently improving their recognition performance.
The main contributions of this work are three-fold. First, we introduce an efficient and effective building block, Channel Equilibrium (CE), which encourages channel equalization and enhances representation learning of CNNs. Second, CE blocks can be stacked after common normalizers and plugged into various advanced architectures, consistently improving their performance by a large margin. For example, CE can be integrated into ResNet and MobileNet V2, forming a series of CE-Networks by replacing the ordinary ‘BN-ReLU’ block by ‘BN-CE-ReLU’ block, which merely introduces subtle extra computational complexity. As a result, the CE-ResNet50 and CE-MobileNet V2 outperform their counterparts by 1.7% and 2.1% top-1 accuracy with nearly the same FLOPs. We also show that combining CE with synchronization across GPUs increases the AP metric on the MS-COCO dataset to 42.0, surpassing its counterpart by 3.4. Third, the learned equalized feature representation of CENet can be better transferred to many other tasks like object detection and segmentation.

heading: 2 RELATED WORK
text: Channel equalization. Channel equalization indicates that channels in a layer of CNNs contribute equally to network’s computation. The success of the two most commonly used regularization techniques, i.e. BN (Ioffe & Szegedy, 2015) and Dropout (Srivastava et al., 2014), is attributed to channel or neuron equalization. For example, Mianjy et al. (2018) showed that Dropout makes the norm of incoming/outgoing weight vectors of all the hidden nodes equal, indicating a kind of equalization between neurons. Moreover, Morcos et al. (2018) pointed out that BN implicitly discourages single direction reliance, indicating that equalizing different channels is able to enhance the generalization of learned feature representation. Note that the squeeze-and-excitation (SE) network (Hu et al., 2018) is a pioneer work that explicitly modeling interdependencies among channels by investigating network design. However, SE selectively emphasizes informative channels and suppresses less useful ones. In contrast, the proposed CE block encourages all the channels to play an equal role in network’s computation, which, as will be shown, can be linked with Nash Equilibrium. More related work on sparsity in ReLU and normalization methods are provided in Sec.A of Appendix.

heading: 3 METHOD
text: In this section, we first review the normalization method and then introduce the proposed Channel Equilibrium (CE) block. CE contains two complementary branches, i.e. batch decorrelation (BD) and adaptive instance inverse (AII). We show how BD and AII benefit from each other through parameter γ and how CE is linked with Nash Equilibrium.
Notations. For CNNs, we use x ∈ RN×C×H×W to represent the feature in a layer, specifically, xncij denotes a pixel (i, j) in the c-th channel of the n-th sample. Sometimes, we ignore the subscript ‘n’ and denote it as xcij for clarity of notation. xnij ∈ RC is obtained by stacking elements in all channels of xncij into a column vector. Diag(·) returns a matrix with the given diagonal and zero offdiagonal entries, and diag(·) extracts the diagonal of the given matrix. γ, β ∈ RC are normalization parameters.

heading: 3.1 OVERVIEW OF NORMALIZATION
text: Normalization is usually employed after convolution layers to stabilize the training of CNNs. Given a hidden feature x ∈ RN×C×H×W , a normalizer first standardizes it to x̄, and then maps it to x̃ by an affine transformation, as written by
x̃ncij = γcx̄ncij + βc, x̄ncij = (xncij − µs)/σs (2) where s ∈ Ω = {IN,BN,LN, · · · } indicates a normalizer and µs, σs are the mean and standard deviation of the given normalizer. For simplicity, in original formulation is omitted (Ioffe & Szegedy, 2015). From Eqn.(2), we claim that normalization would lead to an unequal feature representation in a channel basis, based on the fact that common-used normalizers like IN and BN are performed channel-wisely. It is known that importance of channels are quantified by the learned parameters γ with its magnitude since channel features are scaled by γ in a channel basis. Previous work (Frankle & Carbin, 2018; Mehta et al., 2019) revealed that some inhibited channels emerge when the associated γ or feature map gets small (Mehta et al., 2019; Lu et al., 2019). Obviously, the inhibited channels would cause disequilibrium among channels, resulting in limited generalization ability. To alleviate such a disequilibrium, Channel Equilibrium (CE) block is proposed in the following section.

heading: 3.2 CHANNEL EQUILIBRIUM (CE) BLOCK
text: A Channel Equilibrium (CE) block is a computational unit which aims to equalize feature representation capacity among channels. To this end, decorrelation method is adopted. Different from previous methods (Huang et al., 2018; 2019) decorrelating features by a single batch estimated covariance matrix Σ, the proposed method brings in an adaptive instance variance, Sn, on the diagonal of the covariance matrix Σ, considering that channel dependency is specific to each input (Hu et al., 2018), as formulated in the following,
Dn = λΣ + (1− λ)Diag(Sn), Sn = F (σ2(x̃n)), (3)
where the subscript n is the sample index, λ ∈ (0, 1) is a trainable ratio used to switch between batch and instance statistics, F : RC → RC is a transformation conditioned on the current input x̃ and σ2(x̃n) computes instance variance of x̃n within each channel. On the issue of channel disequilibrium, CE block works by decorrelating feature maps after normalization using D− 1 2
n . Further, the Jensen inequality for matrix functions (Pečarić, 1996) can be employed to obtain a relaxed decorrelation operator D− 1 2 n :
D − 12 n = [λΣ + (1− λ)Diag(Sn)]− 1 2 λΣ− 12 + (1− λ) [Diag(Sn)]− 1 2 , (4)
where A B indicates B − A is semi-definite positive. We introduce this relaxation for the following two reasons. (1) Computation reduction. It allows less computational cost for each training step since the relaxed form only needs to calculate the inverse of square root Σ− 1 2 once, and the other branch Diag(Sn)− 1 2 is easy to compute. (2) Inference acceleration. Σ− 1 2 is a moving-average statistic in inference which can be absorbed into previous layer, therefore, enabling fast inference. Note that Eqn.(4) transforms the combination of covariance and adaptive instance variance into the combination of their inverse square roots.
In the following, we refer Σ− 1 2 in Eqn.(4) as batch decorrelation (BD) and refer [Diag(Sn)] − 12 as adaptive instance inverse (AII). The former decorrelates channels by a batch covariance, while the latter adjusts the extend of inverse for each channel and instance in an adaptive manner. Integrating both of them yields the forward representation of CE block:
pnij = D − 12 n (Diag(γ)x̄nij + β) (5)
where pnij ∈ RC denotes the output of CE, as illustrated in Fig.2(b). Note that CE is performed after the normalization layer, BN is taken as an example to introduce these two branches in the following sections.

heading: 3.2.1 BATCH DECORRELATION (BD)
text: Although a lot of previous work (Huang et al., 2018; 2019; Pan et al., 2019) has investigated whitening method using covariance matrix, all of them are applied after the convolution layer. Thus, inhibited channels still exist since whitened channel features are also scaled by γ. Instead, decorrelation in CE is applied after the normalization layer to equalize magnitude of all channels. Consider a tensor x̃ after a BN layer, it can be reshaped as x̃ ∈ RC×M and M = N · H · W . Then the covariance matrix Σ of x̃ can be written as (details are presented in Sec. B of Appendix)
Σ = γγT 1 M x̄x̄T (6)
where x̄ is a standardized feature with zero mean and unit variance and indicates elementwise multiplication. Eqn.(6) implies that the covariance matrix Σ of x̃ can be decomposed into two parts. The first part depends on normalization parameter γ and the second part becomes correlation matrix of x̃. It is observed that Σij , which represents dependency between i-th channel and j-th channel, is scaled by γiγj after BN is applied.
The Batch Decorrelation (BD) branch requires computing Σ− 1 2 , which is usually related to eigendecomposition or SVD and involves heavy computation (Huang et al., 2018). Instead, here we adopt an efficient approach, i.e., Newton’s Iteration to obtain Σ− 1 2 (Bini et al., 2005; Higham, 1986). Given covariance matrix Σ, Newton’s Iteration calculates Σ− 1 2 by the following iterations:{
Σ0 = I Σk = 1 2 (3Σk−1 − Σ 3 k−1Σ), k = 1, 2, · · · , T.
(7)
where T is the iteration number (T = 3 in our experiments). Note that the convergence of Eqn.(7) is guaranteed if ‖Σ‖2 < 1 (Bini et al., 2005). To this end, Σ is normalized as Σ/tr(Σ) where tr(·) is the trace operator (Huang et al., 2019). In this way, the normalized covariance matrix is written as Σ = γγ T
‖γ‖22 1M x̄x̄ T. To sum up, the batch decorrelation branch firstly calculates a normalized covariance matrix and then applies Newton’s Iteration to obtain its inverse square root, reducing lots of computational cost compared with SVD decomposition in the training stage. Furthermore, BD branch can be merged into convolutional layers in the inference stage, which adds extra computation marginally.

heading: 3.2.2 ADAPTIVE INSTANCE INVERSE (AII)
text: Channel dependencies are specific to each sample. Consequently, a conditional decorrelation is desired for each sample. The adaptive instance inverse (AII) branch only uses diagonal entries to model channel dependencies, as shown in Eqn.(3). Since a diagonal matrix can be inverted easily, this approach can avoid the computation of Eqn.(4).
To construct the AII branch, we analyze its input (the output of a BN layer), which is formulated as x̃ncij = γcx̄ncij +βc. The input of AII is the instance variance of each channel (details are provided in Appendix Sec. B),
σ2nc = γ2c (σ 2 IN) nc
(σ2BN) c
(8)
where σ2IN and σ 2 BN represent the variances in IN and BN respectively. The ratio of them measures the relative fluctuation of how much the instance statistic are deviated from the batch-estimated statistic. Similar to Eqn.(6), the input of AII is also scaled by γ2c .
The AII branch takes σ2nc as input and computes an adaptive instance inverse, i.e. [Diag(Sn)] − 12 . It needs to satisfy two requirements. First, as is desired, dependencies among channels should be embedded in [Diag(Sn)]
− 12 for each sample. Second, the output of AII should have the same philosophy as inverse square root of variance or covariance in BD branch. To achieve this, a reparameterization trick is employed to generate adaptive instance inverse. Let s be an estimate of variance, the AII branch can be reparameterized as below,
[Diag(Sn)] − 12 = Diag(F̃ (σ2n)) · s− 1 2 , (9)
F̃ (σ2n) = δ2(W2δ1(LN(W1σ 2 n))), s = σ 2(x̃), (10)
where δ1 and δ2 are ReLU and sigmoid activation function respectively, W1 ∈ R C r ×C and W2 ∈ RC×Cr and r is reduction ratio, s ∈ R denotes variance of all elements in x̃, which is a batch statistic in training and is obtained using moving average for inference. F̃ (σ2n) ∈ (0, 1)C is treated as a gating mechanism in order to control the strength of instance inverse for each channel. Following the best practice (Hu et al. (2018); Cao et al. (2019)) in characterizing channel relationships, F̃ is expressed by a bottleneck architecture that is able to model channel dependencies and limit model complexity. Layer normalization (LN) is used inside the bottleneck transform (before ReLU) to ease optimization. It is seen from Eqn.(9) that s− 1 2 represents the quantity of inverse square root of variance and F̃ (σ2n) regulates the extend of variance inverse. F̃ maps the instance variance to a set of channel weights. In this sense, the AII branch intrinsically introduces channel dependencies conditioned on each input.

heading: 3.3 DISCUSSIONS
text: Instantiations. Our CE block can be integrated into various advanced architectures, such as ResNet, VGGNet, ShuffleNet V2 and MobileNet V2, by inserting it in ’norm+ReLU-like’ building block. The CE block is described in Fig.2(b). As discussed earlier, CE processes incoming features after the normalization layer by combining two branches, i.e. batch decorrelation (BD) and adaptive instance inverse (AII). Compared with SE block in Fig.2(a), the proposed CE block combines both instance and batch statistics, and it can consequently model dependencies among channels better.
A series of CENets can be constructed by integrating CE block into various advanced CNN architectures. For example, we consider the residual networks (ResNet). The core unit of the ResNet is the residual block that consists of ‘1× 1’, ‘3× 3’ and ‘1× 1’ convolution layers, sequentially. Since the CE block is expected to help channel equalization, it would benefit from larger number of channels. Therefore, we employ CE block in the last ‘1 × 1’ convolution layer by plugging the CE module before ReLU non-linearity, as shown in Fig.2(c). As for CE-MobileNet V2, since the last ‘1 × 1’ convolution layer in the bottleneck is not followed by a ReLU activation, we insert CE in the ‘3× 3’ convolution layer that also has a largest number of channels in the bottleneck. Following similar strategies, CE is further integrated into ShuffleNet V2 to construct CE-ShuffleNet V2. We provide extensive experiments evaluating all these CENets in Sec.4 and computational details in training and inference in Sec.F of Appendix.
Equivalent γ. Here we show how BD and AII benefit from each other through parameter γ. First, we disclose the mechanism in preventing inhibited channels behind the BD branch. Previous work (He et al., 2017b; Yu et al., 2018; Frankle & Carbin, 2018) revealed that γ in BN can be used to prune less unimportant channels, implying that the representational power of feature map largely depends on the magnitude of γ. Combining Eqn.(4) and Eqn.(5), the output of BD can be expressed as pBDnij = Diag(Σ − 12 γ)x̄nij + Σ − 12 β. Comparing with Eqn.(2), an equivalent γ can be defined as γ̂ = Σ− 1 2 γ for BD branch. The proposition 1 shows that BD explicitly increases the magnitude of γ̂ in a feed-forward way, encouraging all channels to contribute to the feature learning process. The representational power is thus boosted in a channel basis. We provide the proof of proposition 1 in Sec.C of Appendix.
Furthermore, the original γ in BN is also implicitly enlarged. It can be seen from Eqn.(6), a sufficient small γc can cause degradation of the covariance matrix and then the convergence of Newton’s iteration (Bini et al., 2005) cannot be guaranteed. As a result, once the network converges, γc is not supposed to degrade. This will in turn bring many benefits to AII branch. Eqn.(8) shows that the input of AII is proportional to γ, meaning that the features fed into AII branch are enlarged as γ increases. In this way, a bottleneck architecture in AII can learn more compact global information and model channel dependencies better.
Proposition 1. Let Σ be covariance matrix of feature maps after batch normalization. Assume that Σk = Σ − 12 , ∀k = 2, 3, · · · , T , then ‖γ̂‖1 > ‖γ‖1. Especially, we have |γ̂i| > |γi|.
Connection with Nash Equilibrium. We show an interesting connection between the proposed CE block and the well-known Nash Equilibrium in game theory (Leshem & Zehavi, 2009). To be specific, we bring novel insights on normalization from an optimization perspective. Suppose each channel obtains its output by maximizing capacity available to itself under some constraints. Especially, we restrict that each channel has a maximum budget and all the outputs are non-negative. Further, if we consider dependencies among channels, the channels are thought to play a noncooperative game, named Gaussian interference game which admits a unique Nash Equilibrium solution (Laufer et al., 2006). In Sec.D of Appendix, we present the detailed construction of Gaussian interference game in the context of CNNs. It is worth noting that when all the outputs are activated (larger than 0), this Nash Equilibrium solution has an explicit expression. Under some mild approximations, it can be shown that the explicit Nash Equilibrium solution can surprisingly match the representation of CE in Eqn.(5). It shows that decorrelating features after normalization layer can be connected with Nash Equilibrium, implying that the proposed CE block indeed encourages every channel to contribute to the network’s computation. We present detailed explanations about the connection between CE and Nash Equilibrium in Sec.D of Appendix.

heading: 4 EXPERIMENTS
text: We evaluate our methods on two basic vision tasks, image classification on ImageNet and object detection/segmentation on COCO, where we demonstrate the effectiveness of the CE block.

heading: 4.1 IMAGE CLASSIFICATION ON IMAGENET
text: We first evaluate CE on the ImageNet benchmark. The training details are illustrated in Sec.F of Appendix.
Performance comparison on ResNet. We evaluate on representative residual network structures including ResNet18, ResNet50 and ResNet101. The CE-ResNet is then compared with baseline (plain ResNet) and SE-ResNet. For fair comparisons, we use publicly available code and re-implement baseline models and SE modules with their respective best settings in a unified Pytorch framework. To save computation, the CE blocks are selectively inserted into the last normalization layer of each residual block. Specifically, for ResNet18, we plug the CE block into each residual block. For ResNet50, CE is inserted into all residual blocks except for those layers with 2048 channels. For ResNet101, the CE blocks are employed in the first seven residual blocks.
As shown in Table 1, our proposed CE outperforms the BN baseline and SE block by a large margin with little increase of GFLOPs. Concretely, CE-ResNet18, CE-ResNet50 and CE-ResNet101 obtain top-1 accuracy increase of 1.5%, 1.7% and 1.0% compared with the corresponding plain ResNet architectures. The CE-ResNet50 even outperforms the plain ResNet101 (78.0). We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and CE-ResNet50 in Sec.E of Appendix.
We also analyze the complexity of BN, SE, and CE in terms of GFLOPs, GPU and CPU running time. We evaluate the inference time1 with a mini-batch of 32. In term of GFLOPs, the CEResNet18, CE-ResNet50, CE-ResNet101 has only 0.242% and 0.241% relative increase in GFLOPs compared with plain ResNet. Additionally, the CPU and GPU inference time of CENet is nearly the same with SENet.
Performance comparison on light-weight networks. We further verify the effectiveness of our proposed CE in two representative light-weight networks, MobileNet V2 and ShuffleNet V2. The results of comparison are listed in Table 2. It is seen that CE blocks bring conspicuous improvements
1The CPU type is Intel Xeon CPU E5-2682 v4, and the GPU is NVIDIA GTX1080TI. The implementation is based on Pytorch
in performance at a minimal increase in computational burden on mobile settings. For MobileNet V2, we see that CE blocks even improves top-1 accuracy of baseline by 2.1%.
Other Normalizers. In addition to BN, CE is also effective for other normalization technologies, since inhibited channels emerges in many well-known normalizers as shown in Fig.1. To prove this, we conduct experiments using ResNet-50 under different normalizers including batch normalization (BN), group normalization (GN), instance normalization (IN), and layer normalization (LN). For these experiments, we stack CE block after the above normalizers to see whether CE helps other normalization methods. As shown in Table 3, our CE generalize well over different normalization technology, improving the performance by 0.6-1.8 top-1 accuracy.

heading: 4.2 ANALYSIS OF CE
text: In this section, we first demonstrate that CE is able to equalize the importance of all channels and then analyze the effects of BD and AII branches separately on CIFAR10 and ImageNet datasets. More discussions about CE are provided in Sec.E of Appendix.
CE is effective in channel equalization. In Fig.1, we have demonstrated that CE is able to alleviate inhibited channels, which is a necessary condition of channel equalization. Here, we further show that the ability that CE can prevent inhibited channels is robust to a wide range of strength of weight decay. As shown in Fig.3(c,d), CE prevents inhibited channels and retains higher performance under different strengths of weight decay.
Next, we verify whether CE can help channel equalization by an ablation approach used in Morcos et al. (2018). Typically, the importance of a single channel to the network’s computation can be measured by the relative performance drop once that channel is removed (clamping activity a feature map to zero). In this regard, the more reliant a network is on a small set of channels, the more quickly the accuracy will drop if those channels are ablated. On the contrary, if the importance of channels to the network’s computation are more equal, the accuracy will drop more gently. With this powerful technique, we see how ResNet50 and MobileNet V2 with CE blocks respond to cumulative random ablation of channels. We plot the ablation ratio versus the top-1 accuracy in Fig.3(a,b). As we can see, our CE block is able to resist the cumulative random ablation of channels on both ResNet50 and MobileNet V2, showing that CE can effectively equalize the importance of channels. For example, the top-1 accuracy of our CE-ResNet50 is 1.7 higher
than the original ResNet50 if no channels are ablated, but when 70% channels are ablated, CEResNet50 still obtain 23.0 top-1 accuracy, while the original ResNet50 gets only 4.6 top-1 accuracy.
BD is able to mitigate inhibited channels. As proved in Proposition 1, equivalent γ in the BD branch is explicitly enlarged, leading to the expansion of representational power of all channels. Here we investigate this property experimentally with a single BD branch. The inhibited channels ratio are measured by the percentage of feature maps whose values are less than 1e-2. Fig.3(d) shows inhibited channels ratio under a wide range of weight decay for BN, BD and CE. It is observed that the top-1 accuracy of VGGNet with BN drops significantly as the weight decay increases, but BD can reduce accuracy drop. For example, when the weight decay is 1e-3, the top-1 accuracy of BD is only 0.1 higher than BN, but when the weight decay reaches to 1e-2, BD is 4.95 higher. Moreover, the inhibited channels ratio of CE is even lower than BD while the top-1 accuracy is higher, which can demonstrate that CE can strengthen the effect of alleviation of inhibited channels compared with single BD.
AII helps CE learn preciser feature representation. First, as discussed in Sec.3.3, AII benefits from BD such that the features fed into AII branch are more informative. To see this, we train ResNet50 with a single AII or CE branch, termed AII-ResNet50 or CE-ResNet50. We do principal component analysis (PCA) on the inputs of AII branch in AII-ResNet50 and the counterpart in CEResNet50 and plot the box chart of principal components. As is shown in Fig.4, the input of AII branch in CE-ResNet50 gets much lower means and variances, meaning that the input feature has more valid basis and thus more informative.From the Grad-CAM visualization provided in Sec.E of Appendix, we find that AII helps CE learn preciser feature representation.
BD and AII are complementary. Here, we verify that BD and AII are complementary to each other. We train plain ResNet50, BD-ResNet50, AII-ResNet50, and CE-ResNet50 for comparison. The top-1 accuracy is reported in Table 4. It is observed that the BD-ResNet50 and AII-ResNet50 are 0.4 and 0.7 higher than the plain ResNet-50 respectively. However, when they are combined, the top-1 accuracy improves by 1.7, higher than combined accuracy increase (1.1), which demonstrates that they benefit from each other.

heading: 4.3 OBJECT DETECTION AND INSTANCE SEGMENTATION ON COCO
text: We assess the generalization of our CE block on detection/segmentation track using the COCO2017 dataset ( Lin et al. (2014)). We train our model on the union of 80k training images and 35k validation images and report the performance on the mini-val 5k images. Mask-RCNN is used as the base detection/segmentation framework. The standard COCO metrics of Average Precision (AP) for bounding box detection (APbb) and instance segmentation (APm) is used to evaluate our methods. In addition, we adopt two common training settings for our models, (1) freezing the vanilla batch normalization and channel equilibrium layer and (2) updating parameters with the synchronized version. For vanilla BN and CE layers, all the gamma, beta parameters, and the tracked running
statistics are frozen. In contrast, for the synchronized version, the running mean and variance for batch normalization and the covariance for CE layers are computed across multiple GPUs. The gamma and beta parameters are updated during training while F̃ and λ are frozen to prevent overfitting. We use MMDetection training framework with ResNet50/ResNet101 as basic backbones and all the hyper-parameters are the same as Chen et al. (2019). Table 5 shows the detection and segmentation results. The results show that compared with vanilla BN, our CE block can consistently improve the performance. For example, our fine-tuned CE-ResNet50 is 2.2 AP higher in detection and 2.7 AP higher in segmentation. For the sync BD version, CE-ResNet50 gets 42.0 AP in detection and 37.5 AP in segmentation, which is the best performance for ResNet50 to the best of our knowledge. To sum up, these experiments demonstrate the generalization ability of CE blocks in other tasks.

heading: 5 CONCLUSION
text: In this paper, we presented a novel network block, termed as Channel Equilibrium (CE). The CE block conditionally decorrelates feature maps after normalization layer by switching between batch decorrelation branch and adaptive instance inverse branch. We show that CE is able to explicitly alleviate the inhibited channels and help channel equalization, enhancing the representational power of a neural network in a feed-forward way. Specifically, CE can be stacked between the normalization layer and the ReLU function, making it flexible to be integrated into many advanced CNN architectures. The superiority of CE blocks has been demonstrated on the task of image classification and instance segmentation. We hope that the analysis of channel equalization in CE could bring a new perspective for future work in architecture design.

heading: A MORE RELATED WORK
text: Sparsity in ReLU. An attractive property of ReLU (Sun et al., 2015; Nair & Hinton, 2010) is sparsity, which brings potential advantages such as information disentangling and linear separability. However, Lu et al. (2019) and Mehta et al. (2019) pointed out that some ReLU neurons may become inactive and output 0 values for any input. Previous work tackled this issue by designing new activation functions, such as ELU (Clevert et al., 2015) and Leaky ReLU (Maas et al., 2013). Recently, Lu et al. (2019) also tried to solve this problem by modifying initialization scheme. Different from these work, we focus on explicitly preventing inhibited channel in a feed-forward way by the proposed CE blocks.
Normalization and decorrelation. There are many practices on normalizer development, such as Batch Normalization (BN) (Ioffe & Szegedy, 2015), Group normalization (GN) (Wu & He, 2018) and Switchable Normalization (Luo et al., 2018). A normalization scheme is typically applied after a convolution layer and contains two stages: standardization and rescaling. Another type of normalization methods not only standardizes but also decorrelates features, like DBN (Huang et al., 2018), IterNorm (Huang et al., 2019) and switchable whitening (Pan et al., 2019). Despite their success in performance improvement, little is explored about relation between those methods and inhibited channels. Fig.1 shows that inhibited channels emerges in VGGNet where ‘BN+ReLU’ or ‘LN+ReLU’ are used. Unlike previous decorrelated normalizations where decorrelation operation is applied after a convolution layer, our CE explicitly decorrelates features after normalization.

heading: B COMPUTATION DETAILS IN ’BN-CE-RELU’ BLOCK
text: As discussed before, CE processes incoming features after normalization layer by combining two branches, i.e. batch decorrelation and adaptive instance inverse. The former computes a covariance matrix and the latter calculates instance variance. We now take ’BN-CE-ReLU’ block as an example to show the computation details of statistics in it. Given a tensor x ∈ RN×C×H×W , the mean and variance in IN (Ulyanov et al., 2016) are calculated as:
µncIN = 1
HW H,W∑ i,j xncij , (σ 2 IN) nc = 1 HW H,W∑ i,j (xncij − µncIN)2 (11)
Hence, we have µIN, σ2IN ∈ RN×C . Then, the statistics in BN can be reformulated as follows:
µcBN = 1
NHW N,H,W∑ n,i,j xncij = 1 N N∑ i 1 HW H,W∑ i,j xncij
(σ2BN) c =
1
NHW N,H,W∑ n,i,j (xncij − µcBN)2
= 1
N N∑ n 1 HW H,W∑ i,j (xncij − µncIN + µncIN − µcBN)2
= 1
N N∑ n ( 1 HW H,W∑ i,j (xncij − µncIN)2 + (µncIN − µcBN)2)
= 1
N N∑ n (σ2IN) nc + 1 N N∑ n (µncIN − µcBN)2
(12)
Then, we have µBN = E[µIN] and σ2BN = E[σ2IN] + D[µIN], where E[·] and D[·] denote expectation and variance operators over N samples. Further, the input of AII is instance variance of features
after BN, which can be calculated as follows:
σ2nc = 1
HW H,W∑ i,j [ (γc xncij − µcBN σcBN + βc)− (γc µncIN − µcBN σcBN + βc) ]2
= γ2c
(σ2BN) c
1
HW H,W∑ i,j (xncij − µncIN)2
= γ2c (σ 2 IN) nc
(σ2BN) c
(13)
At last, the output of BN is x̃ncij = γcx̄ncij + βc, then the entry in c-th row and d-th column of covariance matrix Σ of x̃ is calculated as follows:
Σcd = 1
NHW N,H,W∑ n,i,j (γcx̄ncij)(γdx̄ndij) = γcγdρcd (14)
where ρcd is the element in c-th row and j-th column of correlation matrix of x̄. Thus, we can write Σ into vector form: Σ = γγT 1M x̄x̄ T if we reshape x̃ to x̃ ∈ RC×M and M = N ·H ·W .

heading: C PROOF OF PROPOSITION 1
text: Proposition 1. Let Σ be covariance matrix of feature maps after batch normalization. Assume that Σk = Σ − 12 , ∀k = 2, 3, · · · , T , then ‖γ̂‖1 > ‖γ‖1. Especially, we have |γ̂i| > |γi|
Proof. Since Σk = Σ− 1 2 , ∀k = 2, 3, · · · , T , we have Σkγ = 12Σk−1(3I − Σ 2 k−1Σ)γ = Σk−1γ. Therefore, we only need to show ‖γ̂‖1 = ‖ΣT γ‖1 = · · · = ‖Σ2γ‖1 > ‖γ‖1. Now, we show that for k = 2 we have ∥∥ 1 2 (3I − Σ)γ ∥∥ 1 > ‖γ‖1. From Eqn.(6), we know that Σ = γγT ‖γ‖22 ρ where ρ is the correlation matrix of x̃ and −1 ≤ ρij ≤ 1, ∀i, j ∈ [C]. Then, we have
1 2 (3I − Σ)γ = 1 2 (3I − γγ
T
‖γ‖22 ρ)γ
= 1 2 (3γ − ( γγ
T
‖γ‖22 ρ)γ)
= 1 2 (3γ − 1
‖γ‖22  C∑ j γ1γjρ1jγj , C∑ j γ2γjρ2jγj , · · · , C∑ j γCγjρCjγj T) = 1
2 (3γ − 1
‖γ‖22  C∑ j γ1γjρ1jγj , C∑ j γ2γjρ2jγj , · · · , C∑ j γCγjρCjγj T) = 1
2 (3− C∑ j γ2j ρ1j ‖γ‖22 )γ1, (3− C∑ j γ2j ρ2j ‖γ‖22 )γ2, · · · , (3− C∑ j γ2j ρCj ‖γ‖22 )γC T
(15)
Note that |3− ∑C j γ2j ρij
‖γ‖22 | ≥ 3− |
∑C j γ2j ρij
‖γ‖22 | ≥ 3−
∑C j γ2j ‖γ‖22
= 2, where the last equality holds iff ρij = 1, ∀i, j ∈ [C]. However, this is not the case in practice. Hence we have∣∣∣∣[12(3I − Σ)γ ] i ∣∣∣∣ = ∣∣∣∣∣∣12(3− C∑ j γ2j ρij ‖γ‖22 )γi
∣∣∣∣∣∣ > |γi| (16) Therefore, we have ‖γ̂‖1 > ‖γ‖1. Here completes the proof.

heading: D CONNECTION BETWEEN CE BLOCK AND NASH EQUILIBRIUM
text: We first introduce the definition of Gaussian interference game in context of CNN and then build the connection between a CE block and Nash Equilibrium. For clarity of notation, we omit the subscript n for a concrete sample.
Let the channels 1, 2, · · · , C operate over H ×W pixels. Assume that the C channels have dependencies G = {gcd(i, j)}C,Cc,d=1. Each pixel is characterized by a power gain hcij ≥ 0 and channel noise strength σc > 0. In context of normalization, we suppose hcij = x̄cij + δ where x̄cij is standardized pixel in Eqn.(2) and δ is sufficiently large to guarantee a non-negative power gain. Assume that c-th channel is allowed to transmit a total power of Pc and we have ∑H,W i,j=1 pcij = Pc. Besides, each channel can transmit a power vector pc = (pc11, · · · , pcHW ). Since normalization layer is often followed by a ReLU activation, we restrict pcij ≥ 0. What we want to maximize the capacity transmitted over the c-th channel, ∀c ∈ [C], then the maximization problem is given by:
max Cc(p1, p2, · · · , pC) = h,W∑ i,j=1 ln
( 1 +
gccpcij∑ d 6=c gcdpdij + σc/hcij
)
s.t.
{ ∑H,W i,j=1 pcij = Pc,
pcij ≥ 0, ∀i ∈ [H], j ∈ [W ]
(17)
where Cc is the capacity available to the c-th channel given power distributions p1, p2, · · · , pC . In game theory, C channels and solution space of {pcij}C,H,Wc,i,j=1 together with pay-off vector C = (C1, C2, · · · , CC) form a Gaussian interference game G. Different from basic settings in G, here we do not restrict dependencies gcd to (0, 1). It is known that G has a unique Nash Equilibrium point whose definition is given as below,
Definition 1. An C-tuple of strategies (p1, p2, · · · , pC) for channels 1, 2, · · · , C respectively is called a Nash equilibrium iff for all c and for all p (p a strategy for channel c)
Cc(p1, · · · , pc−1, p, pc+1, · · · , pC) ≤ Cc(p1, p2, · · · , pC) (18)
i.e., given that all other channels d 6= c use strategies pd, channel c best response is pc. Since C1, C2, · · · , CC are concave in p1, p2, · · · , pC respectively, KKT conditions imply the following theorem.
Theorem 1. Given pay-off in Eqn.(17), (p∗1, · · · , p∗C) is a Nash equilibrium point if and only if there exist v0 = (v10 , · · · , vC0 ) (Lagrange multiplier) such that for all i ∈ [H] and j ∈ [W ],
gcc∑ d gcdp ∗ dij + σc/hcij
{ = vc0 for p ∗ cij > 0
≤ vc0 for p∗cij = 0 (19)
Proof. The Lagrangian corresponding to minimization of −Cc subject to the equality constraint and non-negative constraints on pcij is given by
Lc = − h,W∑ i,j=1 ln
( 1 +
gccpcij∑ d6=c gcdpdij + σc/hcij
) + vc0( H,W∑ i,j=1 pcij − Pc) + H,W∑ i,j=1 vcij1 (−pcij). (20)
Differentiating the Lagrangian with respect to pcij and equating the derivative to zero, we obtain
gcc∑ d gcdpcij + σc/hcij + vcij1 = v c 0 (21)
Now, using the complementary slackness condition vcij1 pcij = 0 and v cij 1 ≥ 0, we obtain condition (19). This completes the proof.
By Theorem 1, the unique Nash Equilibrium point can be explicitly written as follows when p∗cij > 0,
p∗ij = G −1 (Diag(v0)−1diag(G)−Diag(hij)−1σ) (22)
where p∗ij , hij , σ ∈ RC and v0 ∈ RC are Lagrangian multipliers corresponding to equality constraints. Note that a approximation can be made using Taylor expansion as follow: − σchcij = σc(2 + hcij +O((1 + hcij)2)). Thus, a linear proxy to Eqn.(22) can be written as
p∗ij = G −1 (Diag(σ)x̄ij + Diag(v0)−1diag(G) + (2 + δ)σ) (23)
Let G = [Dn] 1 2 , γ = σ and β = Diag(v0)−1diag(G) + (2 + δ)σ, Eqn.(23) can surprisingly match CE unit in Eqn.(5), implying that the proposed CE block indeed performs a mechanism on channel equalization. In Gaussian interference game, σ is known and v0 can be determined when budget Pc’s are given. However, γ and β are learned by SGD in deep neural networks.

heading: E EXPERIMENTS
text: E.1 TRAINING AND VALIDATION CURVES
We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and CE-ResNet50 in Fig.5. We can observe that CE-ResNet50 consistently have lower training and validation errors over the whole training period, indicating that CE improves both learning capacity and generalization ability.
E.2 MORE DISCUSSION ABOUT CE
As discussed in related work in Sec.A, many methods have been proposed to improve normalizers and ReLU activation. The ablation approach in Morcos et al. (2018) is used to see whether and how these methods help channel equalization. We demonstrate the effectiveness of CE by answering the following questions.
Do other ReLU-like activation functions help channel equalization? Two representative improvements on ReLU function, i.e. ELU (Clevert et al., 2015) and LReLU (Maas et al., 2013), are employed to see whether other ReLU-like activation functions can help channel equalization. We plot the cumulative ablation curve that depicts ablation ratio versus the top-1 accuracy on CIFAR10 dataset in Fig.6(a). The baseline curve is ’BN+ReLU’. As we can see, the top-1 accuracy curve of ’BN+LReLU’ drops more gently, implying that LReLU helps channel equalization. But ’ELU+ReLU’ has worse cumulative ablation curve than ’BN+ReLU’. By contrast, the proposed CE block improves the recognition performance of ’BN+ReLU’ (higher top-1 accuracy) and promotes channel equalization most (the most gentle cumulative ablation curve).
Do the adaptive normalizers help channel equalization? We experiment on a representative adaptive normalization method (i.e. SN), to see whether it helps channel equalization. SN learns to
select an appropriate normalizer from IN, BN and LN for each channel. The cumulative ablation curves are plotted on ImageNet dataset with ResNet-50 under blocks of ’BN+ReLU’, ‘SN+ReLU’ and ’BN+CE+ReLU’. As shown in Fig.6(b), SN even does damage to channel equalization when it is used to replace BN. However, ’BN+CE+ReLU’ shows the most gentle cumulative ablation curve, indicating the effectiveness of CE block in channel equalization. Compared with SN, ResNet-50 with CE block also achieves better top-1 accuracy (78.3 vs 76.9), showing that channel equalization is important for block design in a CNN.
Integration strategy of CE block. We put CE in different position of a bottleneck in ResNet50, which consists of three ”Conv-BN-ReLU” basic blocks. The channel of the third block is 4 times than that of the second one. We compare the performance of CE-ResNet50 by putting CE in the second block (CE2-ResNet50) or the third block (CE3-ResNet50). As shown in Table 6, the top-1 accuracy of CE3-ResNet-50 outperforms CE2-ResNet50 by 0.4, which indicates that our CE block benefits from larger number of channels.
E.3 GRAD-CAM VISUALIZATION
We claim that AII learns adaptive inverse of variance for each channel in a self-attention manner. Fed into more informative input, AII is expected to make the network respond to different inputs in a highly class-specific manner. In this way, it helps CE learn preciser feature representation. To verify this, we employ an off-the-shelf tool to visualize the class activation map (CAM) Selvaraju et al. (2017). We use ResNet50, BD-ResNet50, and CE-ResNet50 trained on ImageNet for comparison. As shown in Fig.7, the heat maps extracted from CAM for CE-ResNet50 have more coverage on the
object region and less coverage on the background region. It shows that the AII branch helps CE learn preciser information from the images.

heading: F TRAINING AND INFERENCE
text: Moving average in inference. Unlike previous methods in manual architecture design that do not depend on batch estimated statistics, the proposed CE block requires computing the inverse square root of a batch covariance matrix Σ and a global variance scale s in each training step. To make the output depend only on the input, deterministically in inference, we use the moving average to calculate the population estimate of Σ̂− 1 2 and ŝ− 1 2 by following the below updating rules:
Σ̂− 1 2 = (1−m)Σ̂− 12 +mΣ− 12 , ŝ− 12 = (1−m)ŝ− 12 +m · s− 12 (24)
where s and Σ are the variance scale and covariance calculated within each mini-batch during training, and m denotes the momentum of moving average. It is worth noting that since Σ̂− 1 2 is fixed
during inference, the BD branch does not introduce extra costs in memory or computation except for a simple linear transformation ( Σ̂− 1 2 x̃).
Model and computational complexity. The main computation of our CE includes calculating the covariance and inverse square root of it in the BD branch and computing two FC layers in the AII branch. We see that there is a lot of space to reduce computational cost of CE. For BD branch, given an internal feature x ∈ RN×C×H×W , the cost of calculating a covariance matrix is 2NHWC2, which is comparable to the cost of convolution operation. A pooling operation can be employed to downsample featuremap for too large H and W . In this way, the complexity can be reduced to 2NHWC2/k2 + CHW where k is kernel size of the window for pooling. Further, we can use group-wise whitening to improve efficiency, reducing the cost of computing Σ− 1 2 from TC3 to TCg2 (g is group size). For AII branch, we focus on the additional parameters introduced by two FC layers. In fact, the reduction ratio r can be appropriately chosen to balance model complexity and representational power. Besides, the majority of these parameters come from the final block of the network. For example, a single AII in the final block of ResNet-50 has 2 ∗ 20482/r parameters. In practice, the CE blocks in the final stages of networks are removed to reduce additional parameters. We provide the measurement of computational burden and Flops in Table 1.
ResNet Training Setting. All networks are trained using 8 GPUs with a mini-batch of 32 per GPU. We train all the architectures from scratch for 100 epochs using stochastic gradient descent (SGD) with momentum 0.9 and weight decay 1e-4. The base learning rate is set to 0.1 and is multiplied by 0.1 after 30, 60 and 90 epochs. Besides, the covariance matrix in BD branch is calculated within each GPU. Since the computation of covariance matrix involves heavy computation when the size of feature map is large, a 2× 2 maximum pooling is adopted to down-sample the feature map after the first batch normalization layer. Like (Huang et al., 2019), we also use group-wise decorrelation with group size 16 across the network to improve the efficiency in the BD branch. By default, the reduction ratio r in AII branch is set to 4.
MobileNet V2 training Setting. All networks are trained using 8 GPUs with a mini-batch of 32 per GPU for 150 epochs with cosine learning rate. The base learning rate is set to 0.05 and the weight decay is 4e-5.
ShuffleNet V2 training Setting. All networks are trained using 8 GPUs with a mini-batch of 128 per GPU for 240 epochs with poly learning rate. The base learning rate is set to 0.5 and weight decay is 4e-5. We also adopt warmup and label smoothing tricks.
VGG networks on CIFAR10 training setting. For CIFAR10, we train VGG networks with a batch size of 256 on a single GPU for 160 epochs. The initial learning rate is 0.1 and is decreased by 10 times every 60 epochs. The inhibited channel ratios in Fig. 1 and Fig.3(c) is measured by the average ratio for the first 6-th layers since the bottom layers can extract rich feature representation and the sparsity is not desired. For inference drop experiments in Fig.1(c), we randomly drop channels in the third layer with different dropout ratio. For each ratio, we run the experiment 5 times and average the top 1 accuracy.
Mask-RCNN training setting in COCO. We fine-tune the ImageNet pretrained model in COCO for 24 epoch with base learning rate 0.02 and multiply it by 0.1 after 16 and 22 epochs. All the models are trained using 8 GPUs with a mini-batch of 2 images. The basic backbone structure is adopted from the ResNet50/ResNet101 trained on ImageNet.

Review summaries:
The authors point out that CNNs can develop collapsed channels that limit their capacity. They propose to remedy this with batch decorrelation (BD), which focuses on ensuring that channels play an equal role in the feature map and are less likely to collapse. The claim is supported with experiments on CIFAR10, ImageNet and COCO.

This paper studies the channel-collapsed problem in CNNs using 'BN+ReLU' . The Channel Equilibrium block which consists of batch decorrelation branch and adaptive instance inverse branch are proposed to reduce the channel-level sparsity. Experiments on ImageNet and COCO demonstrate that the proposed CE block can achieve higher performance than the conventional CNNs by introducing little computational complexity. The author also discuss the relationship between the proposed method and Nash Equilibrium.