Title: CHANNEL EQUILIBRIUM NETWORKS

Abstract: Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectified linear function-like activations (e.g. ReLU) as building blocks. Previous work pointed out that learning feature channels with equal magnitudes is important for a CNN to achieve good generalization ability. However, the above "Norm+ReLU-like" basic block often learns inhibited channels that has small magnitude (i.e. contributes little to the feature representation), impeding both learning and generalization ability of CNNs. This problem is seldom explored in the literature. To mitigate the inhibited channels and encourage channels to contribute equally to the feature representation, we propose a new building block, Channel Equilibrium (CE), which is able to prevent inhibited channels in both experiments and theory. CE has several appealing properties. First, CE can be stacked after many different normalization methods such as BN and Group Normalization (GN), as well as integrated into many advanced CNN architectures such as ResNet and MobileNet V2 to form a series of CE networks (CENets), outperforming existing network architectures. Second, CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. Third, extensive experiments show that CE achieves state-of-the-art results on various challenging benchmarks such as ImageNet and COCO. The models and codes will be released.

1. INTRODUCTION:

Normalization is an important technique for a wide range of tasks such as image classification (Ioffe & Szegedy, 2015) , object detection (He et al., 2017a; Wu & He, 2018) , and image generation (Miyato et al., 2018) . In recent years, a lot of work improved normalization methods, such as batch normalization (BN) (Ioffe & Szegedy, 2015) , layer normalization (LN) (Ba et al., 2016) and switchable normalization (SN) (Luo et al., 2018) . These methods are often used together with the ReLU-like activation functions such as ReLU (Glorot et al., 2011; Nair & Hinton, 2010) , ELU (Clevert et al., 2015) and Leaky ReLU (LReLU) (Maas et al., 2013) , making the "Norm+ReLU-like" module become one of the most widely-used building blocks of modern CNNs. This work investigates and alleviates the inhibited channels emerged in the "Norm+ReLU-like" building block, which consists of a normalization layer and a ReLU-like activation function given by

EQUATION

where subscript n, c, i, and j denote indices of a sample, a channel, height and width of a feature channel respectively. For instance, y ncij indicates output value of the location (i, j) in the c-th channel of the n-th sample. And xncij and xncij represent normalized channel features and standardized channel features respectively. γ and β are two vectors of parameters, where each element re-scales and re-shifts the standardized features for each channel c. Moreover, g(•) denotes a ReLU-like activation function.

As pointed out in (Morcos et al., 2018) , a CNN that generalizes well would have its channels contributed equally in its feed-forward computations. We term this desired property "channel equalization". However, a recent study disclosed a critical problem of the "Norm+ReLU-like" builiding block, known as "channel collapse", where certain channels always produce small output values given any input. For example, the lottery hypothesis (Frankle & Carbin, 2018) claimed that one over-parameterized CNN always contains unimportant channels that contribute little to the network's prediction. This paper shows that these unimportant channels are inhibited channels, which usually show that inhibited channels emerge in many 'Norm+ReLU-like' blocks such as 'BN+ReLU', 'LN+ReLU' and 'BN+ELU'. (c) plots cumulative ablation curves that is a technique demonstrating channel equalization (Morcos et al., 2018) . Equipped with CE, both 'BN+ReLU' and 'BN+ELU' presents a more gentle drop curve of top-1 accuracy versus cumulative ablation of channels, implying that CE helps channel equalization.

associated with small values of γ in Eqn.( 1) or small values of the channel features (In this paper, inhibited channel is defined as channel whose all feature values are less than 1e -2). For example, as shown in Fig. 1 (a,b), the inhibited channels exist in many "Norm+ReLU-like" basic block such as 'LN+ReLU', 'BN+ELU' and 'BN+ReLU'. This phenomenon has motivated many investigations to directly remove these inhibited channels, such as network slimming (Liu et al., 2017; Yu et al., 2018) and channel pruning (He et al., 2017b) . However, disequilibrium among channels caused by the inhibited channels would do harm to generalization ability of the network and simply removing the inhibited channels that are inactive during training does not help improve learning capacity.

Instead of removing the inhibited channels, we present an alternative perspective by proposing a novel building block, named Channel Equilibrium (CE), to recover and equalize the inhibited channels by encouraging channels to contribute equally in the feature representation learning process, thus enhancing the representation and generalization of CNNs. To this end, a key observation from Eqn.( 1) is that the dependency (covariance) matrix of all the feature channels after normalization is scaled by γγ T . Let this covariance matrix be Σ. We will see that by applying a decorrelation operator, i.e. Σ -1 2 , we can not only effectively eliminate the correlations between computations of channel features and the magnitude of γ, but also equalize the channels' magnitudes (Barlow et al., 1961; Bengio & Bergstra, 2009) . This operator enables all the channels to play an equal role in the computations of a CNN, improving its generalization ability. For example, as shown in Fig. 1 , the VGGNet (Simonyan & Zisserman, 2014) equipped with CE is able to effectively prevent inhibited channels and achieves channel equalization in various of "Norm+RelU-like" blocks, consistently improving their recognition performance.

The main contributions of this work are three-fold. First, we introduce an efficient and effective building block, Channel Equilibrium (CE), which encourages channel equalization and enhances representation learning of CNNs. Second, CE blocks can be stacked after common normalizers and plugged into various advanced architectures, consistently improving their performance by a large margin. For example, CE can be integrated into ResNet and MobileNet V2, forming a series of CE-Networks by replacing the ordinary 'BN-ReLU' block by 'BN-CE-ReLU' block, which merely introduces subtle extra computational complexity. As a result, the CE-ResNet50 and CE-MobileNet V2 outperform their counterparts by 1.7% and 2.1% top-1 accuracy with nearly the same FLOPs. We also show that combining CE with synchronization across GPUs increases the AP metric on the MS-COCO dataset to 42.0, surpassing its counterpart by 3.4. Third, the learned equalized feature representation of CENet can be better transferred to many other tasks like object detection and segmentation.

2. RELATED WORK:

Channel equalization. Channel equalization indicates that channels in a layer of CNNs contribute equally to network's computation. The success of the two most commonly used regularization techniques, i.e. BN (Ioffe & Szegedy, 2015) and Dropout (Srivastava et al., 2014) , is attributed to channel or neuron equalization. For example, Mianjy et al. (2018) showed that Dropout makes the norm of incoming/outgoing weight vectors of all the hidden nodes equal, indicating a kind of equalization between neurons. Moreover, Morcos et al. (2018) pointed out that BN implicitly discourages single direction reliance, indicating that equalizing different channels is able to enhance the generalization of learned feature representation. Note that the squeeze-and-excitation (SE) network (Hu et al., 2018) is a pioneer work that explicitly modeling interdependencies among channels by investigating network design. However, SE selectively emphasizes informative channels and suppresses less useful ones. In contrast, the proposed CE block encourages all the channels to play an equal role in network's computation, which, as will be shown, can be linked with Nash Equilibrium. More related work on sparsity in ReLU and normalization methods are provided in Sec.A of Appendix.

3. METHOD:

In this section, we first review the normalization method and then introduce the proposed Channel Equilibrium (CE) block. CE contains two complementary branches, i.e. batch decorrelation (BD) and adaptive instance inverse (AII). We show how BD and AII benefit from each other through parameter γ and how CE is linked with Nash Equilibrium.

Notations. For CNNs, we use x ∈ R N ×C×H×W to represent the feature in a layer, specifically, x ncij denotes a pixel (i, j) in the c-th channel of the n-th sample. Sometimes, we ignore the subscript 'n' and denote it as x cij for clarity of notation. x nij ∈ R C is obtained by stacking elements in all channels of x ncij into a column vector. Diag(•) returns a matrix with the given diagonal and zero offdiagonal entries, and diag(•) extracts the diagonal of the given matrix. γ, β ∈ R C are normalization parameters.

3.1. OVERVIEW OF NORMALIZATION:

Normalization is usually employed after convolution layers to stabilize the training of CNNs. Given a hidden feature x ∈ R N ×C×H×W , a normalizer first standardizes it to x, and then maps it to x by an affine transformation, as written by

EQUATION

where s ∈ Ω = {IN, BN, LN, • • • } indicates a normalizer and µ s , σ s are the mean and standard deviation of the given normalizer. For simplicity, in original formulation is omitted (Ioffe & Szegedy, 2015) . From Eqn.

(2), we claim that normalization would lead to an unequal feature representation in a channel basis, based on the fact that common-used normalizers like IN and BN are performed channel-wisely. It is known that importance of channels are quantified by the learned parameters γ with its magnitude since channel features are scaled by γ in a channel basis. Previous work (Frankle & Carbin, 2018; Mehta et al., 2019) revealed that some inhibited channels emerge when the associated γ or feature map gets small (Mehta et al., 2019; Lu et al., 2019) . Obviously, the inhibited channels would cause disequilibrium among channels, resulting in limited generalization ability.

To alleviate such a disequilibrium, Channel Equilibrium (CE) block is proposed in the following section.

3.2. CHANNEL EQUILIBRIUM (CE) BLOCK:

A Channel Equilibrium (CE) block is a computational unit which aims to equalize feature representation capacity among channels. To this end, decorrelation method is adopted. Different from previous methods (Huang et al., 2018; 2019) decorrelating features by a single batch estimated covariance matrix Σ, the proposed method brings in an adaptive instance variance, S n , on the diagonal of the covariance matrix Σ, considering that channel dependency is specific to each input (Hu et al., 2018) , as formulated in the following,

EQUATION

where the subscript n is the sample index, λ ∈ (0, 1) is a trainable ratio used to switch between batch and instance statistics, F : R C → R C is a transformation conditioned on the current input x and σ 2 (x n ) computes instance variance of xn within each channel. On the issue of channel disequilibrium, CE block works by decorrelating feature maps after normalization using D -1 2 n . Further, the Jensen inequality for matrix functions (Pečarić, 1996) can be employed to obtain a relaxed decorrelation operator D

EQUATION

where A B indicates B -A is semi-definite positive. We introduce this relaxation for the following two reasons. (1) Computation reduction. It allows less computational cost for each training step since the relaxed form only needs to calculate the inverse of square root Σ -1 2 once, and the other branch Diag(S n ) -1 2 is easy to compute. (2) Inference acceleration. Σ -1 2 is a moving-average statistic in inference which can be absorbed into previous layer, therefore, enabling fast inference. Note that Eqn.(4) transforms the combination of covariance and adaptive instance variance into the combination of their inverse square roots.

In the following, we refer Σ -1 2 in Eqn.(4) as batch decorrelation (BD) and refer [Diag(S n )]

-1 2 as adaptive instance inverse (AII). The former decorrelates channels by a batch covariance, while the latter adjusts the extend of inverse for each channel and instance in an adaptive manner. Integrating both of them yields the forward representation of CE block:

EQUATION

where p nij ∈ R C denotes the output of CE, as illustrated in Fig. 2 (b). Note that CE is performed after the normalization layer, BN is taken as an example to introduce these two branches in the following sections.

3.2.1. BATCH DECORRELATION (BD):

Although a lot of previous work (Huang et al., 2018; 2019; Pan et al., 2019) has investigated whitening method using covariance matrix, all of them are applied after the convolution layer. Thus, inhibited channels still exist since whitened channel features are also scaled by γ. Instead, decorrelation in CE is applied after the normalization layer to equalize magnitude of all channels. Consider a tensor x after a BN layer, it can be reshaped as x ∈ R C×M and M = N • H • W . Then the covariance matrix Σ of x can be written as (details are presented in Sec. B of Appendix)

Σ = γγ T 1 M xx T (6)

where x is a standardized feature with zero mean and unit variance and indicates elementwise multiplication. Eqn.( 6) implies that the covariance matrix Σ of x can be decomposed into two parts.

The first part depends on normalization parameter γ and the second part becomes correlation matrix of x. It is observed that Σ ij , which represents dependency between i-th channel and j-th channel, is scaled by γ i γ j after BN is applied.

The Batch Decorrelation (BD) branch requires computing Σ -1 2 , which is usually related to eigendecomposition or SVD and involves heavy computation (Huang et al., 2018) . Instead, here we adopt an efficient approach, i.e., Newton's Iteration to obtain Σ -1 2 (Bini et al., 2005; Higham, 1986) . Given covariance matrix Σ, Newton's Iteration calculates Σ -1 2 by the following iterations:

EQUATION

)

where T is the iteration number (T = 3 in our experiments). Note that the convergence of Eqn.( 7) is guaranteed if Σ 2 < 1 (Bini et al., 2005) . To this end, Σ is normalized as Σ/tr(Σ) where tr(•) is the trace operator (Huang et al., 2019) . In this way, the normalized covariance matrix is written as

Σ = γγ T γ 2 2 1 M xx T .

To sum up, the batch decorrelation branch firstly calculates a normalized covariance matrix and then applies Newton's Iteration to obtain its inverse square root, reducing lots of computational cost compared with SVD decomposition in the training stage. Furthermore, BD branch can be merged into convolutional layers in the inference stage, which adds extra computation marginally.

3.2.2. ADAPTIVE INSTANCE INVERSE (AII):

Channel dependencies are specific to each sample. Consequently, a conditional decorrelation is desired for each sample. The adaptive instance inverse (AII) branch only uses diagonal entries to model channel dependencies, as shown in Eqn.(3). Since a diagonal matrix can be inverted easily, this approach can avoid the computation of Eqn.(4).

To construct the AII branch, we analyze its input (the output of a BN layer), which is formulated as xncij = γ c xncij + β c . The input of AII is the instance variance of each channel (details are provided in Appendix Sec. B),

EQUATION

where σ 2 IN and σ 2 BN represent the variances in IN and BN respectively. The ratio of them measures the relative fluctuation of how much the instance statistic are deviated from the batch-estimated statistic. Similar to Eqn.( 6), the input of AII is also scaled by γ 2 c .

The AII branch takes σ 2 nc as input and computes an adaptive instance inverse, i.e. [Diag(S n )]

-1

2 . It needs to satisfy two requirements. First, as is desired, dependencies among channels should be embedded in [Diag(S n )] -1 2 for each sample. Second, the output of AII should have the same philosophy as inverse square root of variance or covariance in BD branch. To achieve this, a reparameterization trick is employed to generate adaptive instance inverse. Let s be an estimate of variance, the AII branch can be reparameterized as below,

EQUATION

where δ 1 and δ 2 are ReLU and sigmoid activation function respectively,

W 1 ∈ R C r ×C and W 2 ∈ R C× C

r and r is reduction ratio, s ∈ R denotes variance of all elements in x, which is a batch statistic in training and is obtained using moving average for inference. F (σ 2 n ) ∈ (0, 1) C is treated as a gating mechanism in order to control the strength of instance inverse for each channel. Following the best practice (Hu et al. (2018) ; Cao et al. (2019) ) in characterizing channel relationships, F is expressed by a bottleneck architecture that is able to model channel dependencies and limit model complexity. Layer normalization (LN) is used inside the bottleneck transform (before ReLU) to ease optimization. It is seen from Eqn.(9) that s -1 2 represents the quantity of inverse square root of variance and F (σ 2 n ) regulates the extend of variance inverse. F maps the instance variance to a set of channel weights. In this sense, the AII branch intrinsically introduces channel dependencies conditioned on each input.

3.3. DISCUSSIONS:

Instantiations. Our CE block can be integrated into various advanced architectures, such as ResNet, VGGNet, ShuffleNet V2 and MobileNet V2, by inserting it in 'norm+ReLU-like' building block. The CE block is described in Fig. 2(b ). As discussed earlier, CE processes incoming features after the normalization layer by combining two branches, i.e. batch decorrelation (BD) and adaptive instance inverse (AII). Compared with SE block in Fig. 2 (a), the proposed CE block combines both instance and batch statistics, and it can consequently model dependencies among channels better.

A series of CENets can be constructed by integrating CE block into various advanced CNN architectures. For example, we consider the residual networks (ResNet). The core unit of the ResNet is the residual block that consists of '1 × 1', '3 × 3' and '1 × 1' convolution layers, sequentially. Since the CE block is expected to help channel equalization, it would benefit from larger number of channels. Therefore, we employ CE block in the last '1 × 1' convolution layer by plugging the CE module before ReLU non-linearity, as shown in Fig. 2(c ). As for CE-MobileNet V2, since the last '1 × 1' convolution layer in the bottleneck is not followed by a ReLU activation, we insert CE in the '3 × 3' convolution layer that also has a largest number of channels in the bottleneck. Following similar strategies, CE is further integrated into ShuffleNet V2 to construct CE-ShuffleNet V2. We provide extensive experiments evaluating all these CENets in Sec.4 and computational details in training and inference in Sec.F of Appendix. Equivalent γ. Here we show how BD and AII benefit from each other through parameter γ. First, we disclose the mechanism in preventing inhibited channels behind the BD branch. Previous work (He et al., 2017b; Yu et al., 2018; Frankle & Carbin, 2018) revealed that γ in BN can be used to prune less unimportant channels, implying that the representational power of feature map largely depends on the magnitude of γ. Combining Eqn.(4) and Eqn.( 5), the output of BD can be expressed as

p BD nij = Diag(Σ -1 2 γ)x nij + Σ -1 2 β.

Comparing with Eqn.( 2), an equivalent γ can be defined as γ = Σ -1 2 γ for BD branch. The proposition 1 shows that BD explicitly increases the magnitude of γ in a feed-forward way, encouraging all channels to contribute to the feature learning process. The representational power is thus boosted in a channel basis. We provide the proof of proposition 1 in Sec.C of Appendix.

Furthermore, the original γ in BN is also implicitly enlarged. It can be seen from Eqn.(6), a sufficient small γ c can cause degradation of the covariance matrix and then the convergence of Newton's iteration (Bini et al., 2005 ) cannot be guaranteed. As a result, once the network converges, γ c is not supposed to degrade. This will in turn bring many benefits to AII branch. Eqn.(8) shows that the input of AII is proportional to γ, meaning that the features fed into AII branch are enlarged as γ increases. In this way, a bottleneck architecture in AII can learn more compact global information and model channel dependencies better.

Proposition 1. Let Σ be covariance matrix of feature maps after batch normalization. Assume that

Σ k = Σ -1 2 , ∀k = 2, 3, • • • , T , then γ 1 > γ 1 . Especially, we have | γi | > |γ i |.

Connection with Nash Equilibrium. We show an interesting connection between the proposed CE block and the well-known Nash Equilibrium in game theory (Leshem & Zehavi, 2009) . To be specific, we bring novel insights on normalization from an optimization perspective. Suppose each channel obtains its output by maximizing capacity available to itself under some constraints. Especially, we restrict that each channel has a maximum budget and all the outputs are non-negative.

Further, if we consider dependencies among channels, the channels are thought to play a noncooperative game, named Gaussian interference game which admits a unique Nash Equilibrium solution (Laufer et al., 2006) . In Sec.D of Appendix, we present the detailed construction of Gaussian interference game in the context of CNNs. It is worth noting that when all the outputs are activated (larger than 0), this Nash Equilibrium solution has an explicit expression. Under some mild approximations, it can be shown that the explicit Nash Equilibrium solution can surprisingly match the representation of CE in Eqn.( 5). It shows that decorrelating features after normalization layer can be connected with Nash Equilibrium, implying that the proposed CE block indeed encourages every channel to contribute to the network's computation. We present detailed explanations about the connection between CE and Nash Equilibrium in Sec.D of Appendix. on ImageNet dataset respectively. We randomly ablate channels with an increasing fraction in the first normalization layers. Note that CE-ResNet50 and CE-MobileNet V2 are more robust to cumulative ablation of channels than those networks trained with only BN, suggesting that CE also helps channels equalization on ImageNet. For (c) and (d), we train VGGNet in CIFAR-10 under different weight decays. It is observed that the networks trained with the proposed BD and CE were consistently and substantially more robust to the increasing strength of weight decay than those trained with single BN.

in performance at a minimal increase in computational burden on mobile settings. For MobileNet V2, we see that CE blocks even improves top-1 accuracy of baseline by 2.1%.

Other Normalizers. In addition to BN, CE is also effective for other normalization technologies, since inhibited channels emerges in many well-known normalizers as shown in Fig. 1 CE is effective in channel equalization. In Fig. 1 , we have demonstrated that CE is able to alleviate inhibited channels, which is a necessary condition of channel equalization. Here, we further show that the ability that CE can prevent inhibited channels is robust to a wide range of strength of weight decay. As shown in Fig. 3(c, d ), CE prevents inhibited channels and retains higher performance under different strengths of weight decay.

Next, we verify whether CE can help channel equalization by an ablation approach used in Morcos et al. (2018) . Typically, the importance of a single channel to the network's computation can be measured by the relative performance drop once that channel is removed (clamping activity a feature map to zero). In this regard, the more reliant a network is on a small set of channels, the more quickly the accuracy will drop if those channels are ablated. On the contrary, if the importance of channels to the network's computation are more equal, the accuracy will drop more gently. With this powerful technique, we see how ResNet50 and MobileNet V2 with CE blocks respond to cumulative random ablation of channels. We plot the ablation ratio versus the top-1 accuracy in Fig. 3(a, b ). As we can see, our CE block is able to resist the cumulative random ablation of channels on both ResNet50 and MobileNet V2, showing that CE can effectively equalize the importance of channels. For example, the top-1 accuracy of our CE-ResNet50 is 1.7 higher BD is able to mitigate inhibited channels. As proved in Proposition 1, equivalent γ in the BD branch is explicitly enlarged, leading to the expansion of representational power of all channels.

Here we investigate this property experimentally with a single BD branch. The inhibited channels ratio are measured by the percentage of feature maps whose values are less than 1e-2. Fig. 3(d) shows inhibited channels ratio under a wide range of weight decay for BN, BD and CE. It is observed that the top-1 accuracy of VGGNet with BN drops significantly as the weight decay increases, but BD can reduce accuracy drop. For example, when the weight decay is 1e-3, the top-1 accuracy of BD is only 0.1 higher than BN, but when the weight decay reaches to 1e-2, BD is 4.95 higher. Moreover, the inhibited channels ratio of CE is even lower than BD while the top-1 accuracy is higher, which can demonstrate that CE can strengthen the effect of alleviation of inhibited channels compared with single BD.

AII helps CE learn preciser feature representation. First, as discussed in Sec.3.3, AII benefits from BD such that the features fed into AII branch are more informative. To see this, we train ResNet50 with a single AII or CE branch, termed AII-ResNet50 or CE-ResNet50. We do principal component analysis (PCA) on the inputs of AII branch in AII-ResNet50 and the counterpart in CE-ResNet50 and plot the box chart of principal components. As is shown in Fig. 4 , the input of AII branch in CE-ResNet50 gets much lower means and variances, meaning that the input feature has more valid basis and thus more informative.From the Grad-CAM visualization provided in Sec.E of Appendix, we find that AII helps CE learn preciser feature representation.

BD and AII are complementary. Here, we verify that BD and AII are complementary to each other. We train plain ResNet50, BD-ResNet50, AII-ResNet50, and CE-ResNet50 for comparison. The top-1 accuracy is reported in Table 4 . It is observed that the BD-ResNet50 and AII-ResNet50 are 0.4 and 0.7 higher than the plain ResNet-50 respectively. However, when they are combined, the top-1 accuracy improves by 1.7, higher than combined accuracy increase (1.1), which demonstrates that they benefit from each other.

4.3. OBJECT DETECTION AND INSTANCE SEGMENTATION ON COCO:

We assess the generalization of our CE block on detection/segmentation track using the COCO2017 dataset ( Lin et al. (2014) ). We train our model on the union of 80k training images and 35k validation images and report the performance on the mini-val 5k images. Mask-RCNN is used as the base detection/segmentation framework. The standard COCO metrics of Average Precision (AP) for bounding box detection (APbb) and instance segmentation (APm) is used to evaluate our methods.

In addition, we adopt two common training settings for our models, (1) freezing the vanilla batch normalization and channel equilibrium layer and (2) updating parameters with the synchronized version. For vanilla BN and CE layers, all the gamma, beta parameters, and the tracked running statistics are frozen. In contrast, for the synchronized version, the running mean and variance for batch normalization and the covariance for CE layers are computed across multiple GPUs. The gamma and beta parameters are updated during training while F and λ are frozen to prevent overfitting. We use MMDetection training framework with ResNet50/ResNet101 as basic backbones and all the hyper-parameters are the same as Chen et al. (2019) . Table 5 shows the detection and segmentation results. The results show that compared with vanilla BN, our CE block can consistently improve the performance. For example, our fine-tuned CE-ResNet50 is 2.2 AP higher in detection and 2.7 AP higher in segmentation. For the sync BD version, CE-ResNet50 gets 42.0 AP in detection and 37.5 AP in segmentation, which is the best performance for ResNet50 to the best of our knowledge. To sum up, these experiments demonstrate the generalization ability of CE blocks in other tasks.

5. CONCLUSION:

In this paper, we presented a novel network block, termed as Channel Equilibrium (CE). The CE block conditionally decorrelates feature maps after normalization layer by switching between batch decorrelation branch and adaptive instance inverse branch. We show that CE is able to explicitly alleviate the inhibited channels and help channel equalization, enhancing the representational power of a neural network in a feed-forward way. Specifically, CE can be stacked between the normalization layer and the ReLU function, making it flexible to be integrated into many advanced CNN architectures. The superiority of CE blocks has been demonstrated on the task of image classification and instance segmentation. We hope that the analysis of channel equalization in CE could bring a new perspective for future work in architecture design.

after BN, which can be calculated as follows:

EQUATION

At last, the output of BN is xncij = γ c xncij + β c , then the entry in c-th row and d-th column of covariance matrix Σ of x is calculated as follows:

Σ cd = 1 N HW N,H,W n,i,j (γ c xncij )(γ d xndij ) = γ c γ d ρ cd (14)

where ρ cd is the element in c-th row and j-th column of correlation matrix of x. Thus, we can write Σ into vector form:

Σ = γγ T 1 M xx T if we reshape x to x ∈ R C×M and M = N • H • W .

C PROOF OF PROPOSITION 1

Proposition 1. Let Σ be covariance matrix of feature maps after batch normalization. Assume that

Σ k = Σ -1 2 , ∀k = 2, 3, • • • , T , then γ 1 > γ 1 . Especially, we have | γi | > |γ i | Proof. Since Σ k = Σ -1 2 , ∀k = 2, 3, • • • , T , we have Σ k γ = 1 2 Σ k-1 (3I -Σ 2 k-1 Σ)γ = Σ k-1 γ. Therefore, we only need to show γ 1 = Σ T γ 1 = • • • = Σ 2 γ 1 > γ 1

. Now, we show that for k = 2 we have 1 2 (3I -Σ)γ 1 > γ 1 . From Eqn.(6), we know that Σ = γγ T γ 2 2 ρ where ρ is the correlation matrix of x and -1 ≤ ρ ij ≤ 1, ∀i, j ∈ [C]. Then, we have

1 2 (3I -Σ)γ = 1 2 (3I - γγ T γ 2 2 ρ)γ = 1 2 (3γ -( γγ T γ 2 2 ρ)γ) = 1 2 (3γ - 1 γ 2 2   C j γ 1 γ j ρ 1j γ j , C j γ 2 γ j ρ 2j γ j , • • • , C j γ C γ j ρ Cj γ j   T ) = 1 2 (3γ - 1 γ 2 2   C j γ 1 γ j ρ 1j γ j , C j γ 2 γ j ρ 2j γ j , • • • , C j γ C γ j ρ Cj γ j   T ) = 1 2   (3 - C j γ 2 j ρ 1j γ 2 2 )γ 1 , (3 - C j γ 2 j ρ 2j γ 2 2 )γ 2 , • • • , (3 - C j γ 2 j ρ Cj γ 2 2 )γ C   T (15) Note that |3 - C j γ 2 j ρij γ 2 2 | ≥ 3 -| C j γ 2 j ρij γ 2 2 | ≥ 3 - C j γ 2 j γ 2 2 = 2

, where the last equality holds iff ρ ij = 1, ∀i, j ∈ [C]. However, this is not the case in practice. Hence we have

EQUATION

Therefore, we have γ 1 > γ 1 . Here completes the proof.

D CONNECTION BETWEEN CE BLOCK AND NASH EQUILIBRIUM:

We first introduce the definition of Gaussian interference game in context of CNN and then build the connection between a CE block and Nash Equilibrium. For clarity of notation, we omit the subscript n for a concrete sample.

Let the channels 1, 2, • • • , C operate over H × W pixels. Assume that the C channels have dependencies G = {g cd (i, j)} C,C c,d=1 . Each pixel is characterized by a power gain h cij ≥ 0 and channel noise strength σ c > 0. In context of normalization, we suppose h cij = xcij + δ where xcij is standardized pixel in Eqn.(2) and δ is sufficiently large to guarantee a non-negative power gain. Assume that c-th channel is allowed to transmit a total power of P c and we have H,W i,j=1 p cij = P c . Besides, each channel can transmit a power vector p c = (p c11 , • • • , p cHW ). Since normalization layer is often followed by a ReLU activation, we restrict p cij ≥ 0. What we want to maximize the capacity transmitted over the c-th channel, ∀c ∈ [C], then the maximization problem is given by:

EQUATION

where C c is the capacity available to the c-th channel given power distributions p 1 , p 2 , • • • , p C . In game theory, C channels and solution space of {p cij } C,H,W c,i,j=1 together with pay-off vector C = (C 1 , C 2 , • • • , C C ) form a Gaussian interference game G. Different from basic settings in G, here we do not restrict dependencies g cd to (0, 1). It is known that G has a unique Nash Equilibrium point whose definition is given as below,

Definition 1. An C-tuple of strategies (p 1 , p 2 , • • • , p C ) for channels 1, 2, • • • , C

respectively is called a Nash equilibrium iff for all c and for all p (p a strategy for channel c) 

EQUATION

EQUATION

Proof. The Lagrangian corresponding to minimization of -C c subject to the equality constraint and non-negative constraints on p cij is given by

EQUATION

)

Differentiating the Lagrangian with respect to p cij and equating the derivative to zero, we obtain

g c c d g cd p cij + σ c /h cij + v cij 1 = v c 0 (21)

Now, using the complementary slackness condition v cij 1 p cij = 0 and v cij 1 ≥ 0, we obtain condition (19). This completes the proof.

By Theorem 1, the unique Nash Equilibrium point can be explicitly written as follows when p where p * ij , h ij , σ ∈ R C and v 0 ∈ R C are Lagrangian multipliers corresponding to equality constraints. Note that a approximation can be made using Taylor expansion as follow:

EQUATION

-σc hcij = σ c (2 + h cij + O((1 + h cij ) 2 ))

. Thus, a linear proxy to Eqn.( 22) can be written as 23) can surprisingly match CE unit in Eqn.( 5), implying that the proposed CE block indeed performs a mechanism on channel equalization. In Gaussian interference game, σ is known and v 0 can be determined when budget P c 's are given. However, γ and β are learned by SGD in deep neural networks.

p * ij = G -1 Diag(σ)x ij + Diag(v 0 ) -1 diag(G) + (2 + δ)σ (23) Let G = [D n ] 1 2 , γ = σ and β = Diag(v 0 ) -1 diag(G) + (2 + δ)σ, Eqn.(

E EXPERIMENTS E.1 TRAINING AND VALIDATION CURVES:

We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and CE-ResNet50 in Fig. 5 . We can observe that CE-ResNet50 consistently have lower training and validation errors over the whole training period, indicating that CE improves both learning capacity and generalization ability.

E.2 MORE DISCUSSION ABOUT CE:

As discussed in related work in Sec.A, many methods have been proposed to improve normalizers and ReLU activation. The ablation approach in Morcos et al. ( 2018) is used to see whether and how these methods help channel equalization. We demonstrate the effectiveness of CE by answering the following questions. Do other ReLU-like activation functions help channel equalization? Two representative improvements on ReLU function, i.e. ELU (Clevert et al., 2015) and LReLU (Maas et al., 2013) , are employed to see whether other ReLU-like activation functions can help channel equalization. We plot the cumulative ablation curve that depicts ablation ratio versus the top-1 accuracy on CI-FAR10 dataset in Fig. 6 (a). The baseline curve is 'BN+ReLU'. As we can see, the top-1 accuracy curve of 'BN+LReLU' drops more gently, implying that LReLU helps channel equalization. But 'ELU+ReLU' has worse cumulative ablation curve than 'BN+ReLU'. By contrast, the proposed CE block improves the recognition performance of 'BN+ReLU' (higher top-1 accuracy) and promotes channel equalization most (the most gentle cumulative ablation curve). Integration strategy of CE block. We put CE in different position of a bottleneck in ResNet50, which consists of three "Conv-BN-ReLU" basic blocks. The channel of the third block is 4 times than that of the second one. We compare the performance of CE-ResNet50 by putting CE in the second block (CE2-ResNet50) or the third block (CE3-ResNet50). As shown in Table 6 , the top-1 accuracy of CE3-ResNet-50 outperforms CE2-ResNet50 by 0.4, which indicates that our CE block benefits from larger number of channels.

E.3 GRAD-CAM VISUALIZATION:

We claim that AII learns adaptive inverse of variance for each channel in a self-attention manner. Fed into more informative input, AII is expected to make the network respond to different inputs in a highly class-specific manner. In this way, it helps CE learn preciser feature representation. To verify this, we employ an off-the-shelf tool to visualize the class activation map (CAM) Selvaraju et al. (2017) . We use ResNet50, BD-ResNet50, and CE-ResNet50 trained on ImageNet for comparison.

As shown in Fig. 7 , the heat maps extracted from CAM for CE-ResNet50 have more coverage on the

:

The CPU type is Intel Xeon CPU E5-2682 v4, and the GPU is NVIDIA GTX1080TI. The implementation is based on Pytorch

Figures and Tables:

Figure 1: We train VGGNet on CIFAR-10 with various 'Norm+ReLU' blocks (details about training setings are provided in Sec.F of Appendix). The inhibited channel ratio is defined as the average percentage of values less than 1e-2 in the first six feature maps of VGGNet. (a) & (b) show that inhibited channels emerge in many 'Norm+ReLU-like' blocks such as 'BN+ReLU', 'LN+ReLU' and 'BN+ELU'. (c) plots cumulative ablation curves that is a technique demonstrating channel equalization (Morcos et al., 2018). Equipped with CE, both 'BN+ReLU' and 'BN+ELU' presents a more gentle drop curve of top-1 accuracy versus cumulative ablation of channels, implying that CE helps channel equalization.

Figure 2: Illustrations of SE block (Hu et al., 2018) (a), CE block (b) and CE residual block in ResNet (c). denotes broadcast element-wise multiplication, + denotes broadcast elementwise addition and ⊗ denotes matrix multiplication. (b) shows CE has two lightweight branches, BN and AII. (c) shows CE can be easily stacked into many advanced networks such as ResNet with merely small extra computation.

Figure 3: (a) & (b) show channel drop ratios versus top-1 accuracy for MobileNet V2 and ResNet50on ImageNet dataset respectively. We randomly ablate channels with an increasing fraction in the first normalization layers. Note that CE-ResNet50 and CE-MobileNet V2 are more robust to cumulative ablation of channels than those networks trained with only BN, suggesting that CE also helps channels equalization on ImageNet. For (c) and (d), we train VGGNet in CIFAR-10 under different weight decays. It is observed that the networks trained with the proposed BD and CE were consistently and substantially more robust to the increasing strength of weight decay than those trained with single BN.

Figure 4: We do principal component analysis (PCA) on the input of AII sub-network, the variance of each channel. This figure show the box chart of principal components. CENet has lower means and variances than AIINet, indicating the input of AII sub-network in CENet is more equal and informative.

Figure 5: Training and validation error curves on ImageNet with ResNet50 as backbone for BN, SE and CE.

Figure 6: (a) compares the cumulative ablation curves of 'BN+ReLU', 'BN+ELU', 'BN+LReLU' and 'BN+CE+ReLU' with VGGNet on CIFAR-10 dataset. We see that the Both LReLU and CE can improve the channel equalization in 'BN+ReLU' block. (b) compares the cumulative ablation curves of 'BN+ReLU', 'SN+ReLU' and 'BN+CE+ReLU' with ResNet-50 on ImageNet dataset.The proposed CE consistently improves the channel equalization of 'BN+RelU' block. Note that 'BN+CE+ReLU' achieves the highest top-1 accuracy on both two datasets compared to its counterparts (when drop ration is 0).Top-1 acc CE2-ResNet5077.9 CE3-ResNet50 78.3Table6: We add CE after the second (CE2-ResNet50) and third (CE3-ResNet50) batch normalization layer in each residual block. The channel of the third batch normalization is 4 times than that of the second one but the top-1 accuracy of CE3-ResNet50 outperforms CE2-ResNet50 by 0.4, which indicates CE benefits from larger number of channels.

Comparisons with baseline and SENet on ResNet-18, -50, and -101 in terms of accuracy, GFLOPs, CPU and GPU inference time on ImageNet. The top-1,-5 accuracy of our CE-ResNet is higher than SE-ResNet while the computational cost in terms of GFLOPs, GPU and CPU inference time remain nearly the same.

Table content: <table><tr><td/><td/><td colspan="2">ResNet18</td><td/><td colspan="2">ResNet50</td><td/><td colspan="2">ResNet101</td></tr><tr><td/><td colspan="2">Baseline</td><td>SE</td><td>CE</td><td>Baseline</td><td>SE</td><td>CE</td><td>Baseline</td><td>SE</td><td>CE</td></tr><tr><td>Top-1</td><td>70.4</td><td colspan="2">71.4</td><td>71.9</td><td>76.6</td><td>77.6</td><td>78.3</td><td>78.0</td><td>78.5</td><td>79.0</td></tr><tr><td>Top-5</td><td>89.4</td><td colspan="2">90.4</td><td>90.8</td><td>93.0</td><td>93.7</td><td>94.1</td><td>94.1</td><td>94.1</td><td>94.6</td></tr><tr><td>GFLOPs</td><td>1.82</td><td colspan="2">1.82</td><td>1.83</td><td>4.14</td><td>4.15</td><td>4.16</td><td>7.87</td><td>7.88</td><td>7.89</td></tr><tr><td>CPU (s)</td><td>3.69</td><td colspan="2">3.69</td><td>4.13</td><td>8.61</td><td colspan="2">11.08 11.06</td><td>15.58</td><td colspan="2">19.34 17.05</td></tr><tr><td>GPU (s)</td><td>0.003</td><td colspan="3">0.005 0.006</td><td>0.005</td><td colspan="2">0.010 0.009</td><td>0.011</td><td colspan="2">0.040 0.015</td></tr><tr><td/><td colspan="3">MobileNet V2 1×</td><td/><td colspan="3">ShuffleNet V2 0.5×</td><td colspan="3">ShuffleNet V2 1×</td></tr><tr><td/><td colspan="10">Top-1 Top-5 GFLOPs Top-1 Top-5 GFLOPs Top-1 Top-5 GFLOPs</td></tr><tr><td>Baseline</td><td>72.5</td><td>90.8</td><td colspan="2">0.33</td><td>59.2</td><td>82.0</td><td>0.05</td><td>69.0</td><td>88.6</td><td>0.15</td></tr><tr><td>SE</td><td>73.5</td><td>91.7</td><td colspan="2">0.33</td><td>60.2</td><td>82.4</td><td>0.05</td><td>70.7</td><td>89.6</td><td>0.15</td></tr><tr><td>CE</td><td>74.6</td><td>91.7</td><td colspan="2">0.33</td><td>60.5</td><td>82.7</td><td>0.05</td><td>71.2</td><td>89.8</td><td>0.16</td></tr></table>

Comparisons with baseline and SE on lightweight networks, MobileNet V2 and ShuffleNet V2, in terms of accuracy and GFLOPs on ImageNet. Our CENet improves the top-1 accuracy by a large margin compared with SENet with nearly the same GFLOPs.

Table content: <table><tr><td>As shown in Table 1, our proposed CE outperforms the BN baseline and SE block by a large margin</td></tr><tr><td>with little increase of GFLOPs. Concretely, CE-ResNet18, CE-ResNet50 and CE-ResNet101 obtain</td></tr><tr><td>top-1 accuracy increase of 1.5%, 1.7% and 1.0% compared with the corresponding plain ResNet</td></tr><tr><td>architectures. The CE-ResNet50 even outperforms the plain ResNet101 (78.0). We plot training</td></tr><tr><td>and validation loss during the training process for ResNet50, SE-ResNet50 and CE-ResNet50 in</td></tr><tr><td>Sec.E of Appendix.</td></tr><tr><td>We also analyze the complexity of BN, SE, and CE in terms of GFLOPs, GPU and CPU run-</td></tr><tr><td>ning time. We evaluate the inference time 1 with a mini-batch of 32. In term of GFLOPs, the CE-</td></tr><tr><td>ResNet18, CE-ResNet50, CE-ResNet101 has only 0.242% and 0.241% relative increase in GFLOPs</td></tr><tr><td>compared with plain ResNet. Additionally, the CPU and GPU inference time of CENet is nearly the</td></tr><tr><td>same with SENet.</td></tr></table>

In this section, we first demonstrate that CE is able to equalize the importance of all channels and then analyze the effects of BD and AII branches separately on CIFAR10 and ImageNet datasets. More discussions about CE are provided in Sec.E of Appendix.

Table content: <table><tr><td>. To prove this,</td></tr><tr><td>we conduct experiments using ResNet-50 under different normalizers including batch normalization</td></tr><tr><td>(BN), group normalization (GN), instance normalization (IN), and layer normalization (LN). For</td></tr><tr><td>these experiments, we stack CE block after the above normalizers to see whether CE helps other</td></tr><tr><td>normalization methods. As shown in Table 3, our CE generalize well over different normalization</td></tr><tr><td>technology, improving the performance by 0.6-1.8 top-1 accuracy.</td></tr><tr><td>4.2 ANALYSIS OF CE</td></tr></table>

Results of batch covariance decorrelation, adaptive variance inverse and channel equilibrium. We use ResNet-50 as the basic structure. The top-1 accuracy increase (1.7) of CE-ResNet is higher than combined top-1 accuracy increase (1.1) of BD-ResNet and AII-ResNet, indicating the effects of BD and AII branch is complementary. than the original ResNet50 if no channels are ablated, but when 70% channels are ablated, CE-ResNet50 still obtain 23.0 top-1 accuracy, while the original ResNet50 gets only 4.6 top-1 accuracy.

Table content: <table/>

Detection and segmentation results in COCO using Mask-RCNN We use the pretrained CE-ResNet50 model (78.3) and CE-ResNet101 (79.0) in ImageNet to train our model. CENet can consistently improve both box AP and segmentation AP by a large margin.

Table content: <table><tr><td>Backbone</td><td colspan="2">AP b AP b .5</td><td>AP b .75</td><td colspan="2">AP m AP m .5</td><td>AP m .75</td></tr><tr><td>ResNet50</td><td>38.6</td><td>59.5</td><td>41.9</td><td>34.2</td><td>56.2</td><td>36.1</td></tr><tr><td>CE-ResNet50</td><td>40.8</td><td>62.7</td><td>44.3</td><td>36.9</td><td>59.2</td><td>39.4</td></tr><tr><td colspan="2">SyncCE-ResNet50 42.0</td><td>62.6</td><td>46.1</td><td>37.5</td><td>59.5</td><td>40.3</td></tr><tr><td>ResNet101</td><td>40.3</td><td>61.5</td><td>44.1</td><td>36.5</td><td>58.1</td><td>39.1</td></tr><tr><td>CE-ResNet101</td><td>41.6</td><td>62.8</td><td>45.8</td><td>37.4</td><td>59.4</td><td>40.0</td></tr></table>

Peer review (iclr2020):
This paper proposes a new block that gives diminishing Pros) (+) This paper is well-written and the idea looks interesting. (+) The provided experiments look quite extensive. I like the experiments with both heavy and light network architectures to show the effectiveness of the proposed method. (+) The authors provided plausible and sufficient backups for the claim. (+) Connection with Nash equilibrium seems to be slightly overclaimed, but the attempt is novel and interesting. Cons) (-) The overall paper flow could be organized much better. I think some of the materials in the appendix needs to be placed in the main paper. (-) Some of the notations in Section 3 need to be rewritten for better clarity. (-) The terminology such as channel-level sparsity, channel equalization, and equivalent lambda looks vague. I recommend the authors define them for clarity. Comments) - I think the relationship between diminishing the sparsity on channels and improving the evaluation performance should be more addressed. If we change ReLU after Conv with other nonlinear functions such as ELU or PReLU, then the output hardly becomes zero, but as you know the performance usually gets worse. Then, how do you explain the necessity of reducing sparsities? - How did you measure the sparsity in a plain network in Figure 1 and Figure 3? Did you pick a single layer or gather all the layers* sparsity? Please specify the way of measuring it. - How did you plot the graphs in Figure 3.(a) and 3.(b). Did you pick a single channel randomly? - To support the authors* claim, it would be better to show the sparsity ratio for a model (ResNet18, MobileNetV2, and so on) which shows better performance than each original one. - How did you determine the transformation of F in the AII branch? Please provide any intuitions why the authors choose the transformation like in eq.(9). - If CE-block actually conducts well right after a BN layer, then why CE block is attached only after the final BN layer in a bottleneck module? It would be better to provide any results by doing some studies when dealing with the intermediate BNs in a bottleneck module. - I am wondering whether adaptive normalization techniques such as Instance-Batchnormalziation or Switchable Normalization methods could also give less sparsified features. Please clarify this by comparing with the proposed method. - It seems that CE would have some extra computational costs compared with SE*s. Please clarify why CE-module does have small computational costs. - Please specify what MobileNetV2 has used. Looks like MobileNetV2x1.0 would be used. - Why do you think the Top-5 accuracy of MobileNetV2 has not been improved? - Can your method combine with ResNet with SE such as ResNet50SE? - Please clarify the way of measuring the sparsity ratio in Figure 3d. Why did you consider the sparseness of a feature by measuring the gamma in Figure 4? There may exist a channel that contains large magnitude values then gamma cannot make the output close to zero. About overall rating) The theoretical backups for the authors* claim look sound, and all the experiments including the performance improvement with several models seem to support the effectiveness of the idea very well. Specifically, I like all the analyses and the connection with Nash equilibrium, which are very intuitive and may provide further insights to researchers in this field. I think this paper is clearly above the standard of ICLR. If the authors could address all the concerns above and refine the paper with better readability, then I could increase the score.