Channel Equilibrium Networks
Authors: Wenqi Shao
Shitao Tang
Xingang Pan
Ping Tan
Xiaogang Wang
Ping Luo
Venue:  (2020)
Keywords: Deep learning
convolutional neural networks
building block design

Abstract: Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectified linear function (ReLU) as building blocks. Previous work showed that this basic block would lead to channel-level sparsity (i.e. channel of zero values), reducing computational complexity of CNNs. However, over-sparse CNNs have many collapsed channels (i.e. many channels with undesired zero values), impeding their learning ability. This problem is seldom explored in the literature. To recover the collapsed channels and enhance learning capacity, we propose a  building block, Channel Equilibrium (CE), which takes the output of a normalization layer as input and switches between two branches, batch decorrelation (BD) branch and adaptive instance inverse (AII) branch. CE is able to prevent implicit channel-level sparsity in both experiments and theory. It has several appealing properties. First, CE can be stacked after many normalization methods such as BN and Group Normalization (GN), and integrated into many advanced CNN architectures such as ResNet and MobileNet V2 to form a series of CE networks (CENets), consistently improving their performance. Second, extensive experiments show that CE achieves state-of-the-art results on various challenging benchmarks such as ImageNet and COCO. Third, we show an interesting connection between CE and Nash Equilibrium, a well-known solution of a non-cooperative game. The models and code will be released soon.

Under review as a conference paper at ICLR 2020
CHANNEL EQUILIBRIUM NETWORKS
Anonymous authors
Paper under double-biee -hind review
ABSTRACT
Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectified linear function- activations (e.g. ReLU) as building blocks. Previous work pointed out that learning feature chan- nels with equal magnitudes is important for a CNN to achieve good generalization ability. However, the above "Nerm-ReLUUlikee" basic block often learns inhibited channels that has small magnitude (i.e. contributes little to the feature representa- tion), impeding both learning and generalization ability of CNNs. This problem is seldom explored in the literature. To mitigate the inhibited channels and en- courage channels to contribute equally to the feature representation, we propose a new building block, Channel Equilibrium (CE), which is able to prevent inhib- ited channels both experiments and theory. CE has several appealing proper- ties. First, CE can be stacked after many different normalization methods such as BN and Group Normalization (GN), as well as integrated into many advanced CNN architectures such ResNet and MobileNet V2 to form series of CE networks (CENets), outperforming existing network architectures. Second, CE has interesting connection with the Nash Fquilibrium, well-known solution of a non-cooperative game. Third, extensive experiments show that CE achieves tfate-of-the-at results on various challenging benchmarks such ImageNet and COCO. The models and codes will be released.
INTRODUCTION
Normalization is an important technique for a wide range of tasks such as image classification (Toffe & Szegedy| (2015). object detection (He et al.) 7a] Wu & He) (018)), and image generation (My-- ato et al. 2018). In recent years, a lot of work improved normalization methods, such as batch nor- malization (BN) (Toffe& & Szegedy 2015), layer normalization (LN) (Ba et al. (016] and switchable normalization (SN) (Luo et al., 2018). These methods are often used together with the ReLU-like activation functions such as ReLU (Glorot et al. [201 Nair & Hinton 2010). ELU (Clevert et al. 2013) and Leaky ReLU (LReLU) (Maas et al. 2013). making the "Norm+ReLU-like"' module be- come of the most widely-used building blocks of modern CNNs. This work investigates and alleviates the inhibited channels emerged in the "Norm+ReLU-like building block, which consists of normalization layer and ReLU-like activation function given by
Yneij g(āneiz).
Encij = YoTnnij + Be,
where subscript n, c, i, and denote indices of a sample, a channel, height and width of a feature channel respectively. For instance, Mneiy indicates output value of the location (i,j) in the c-th chan- nel of the n-th sample. And Encij and Encis represent normalized channel features and standardized channel features respectively. ? and B are two vectors of parameters, where each element re-scales and re-shifts the standardized features for each channel c. Moreover, g()) denotes ReLU-likee activation function.
As pointed out in (Morcos et al. 2018), CNN that generalizes well would have its channels con- tributed equally in its feed-forward computations. We term this desired property "channel equal- ization". However, recent study disclosed critical problem of the "Norm+ReLU-like builiding block, known as "channel collapse". where certain channels always produce small output values given any input. For example, the lottery hypothesis (Frankle & Carbin 2018) claimed that one over-paranee parametterre CNN always contains unimportant channels that contribute little to the network's 's prediction. This paper shows that these unimportant channels are inhibited channels, which usually

---

Under review as a conference paper at ICLR 2020
kadi
Heah RACE
90-
Top Litrr
80
70
Top
I4A
BN+CE+
BN+ELU
BN+CE+ELU
600
CL-410
CEAERALI
0.2
0.4
0.6
0.8
1.0
Drop Ratio
(a) "Norm+ReLU" blocks
(b) 'BN+ReLU-like" blocks
(c) Cumalative ablation curves
Figure I: We train VGGNet CIFAR-10 10 with various 'Norm+ReLU" blocks (details about training setings are provided in See F of Appendix). The inhibited channel ratio is defined as the average percentage of values less than 1e-2 in the first six feature maps of VGGNet. (a) & (b) show that in- hibited channels emerge in many "Norm+ReLU-like blocks such as 'BN+ReLU', "LN+ReLLU' and "BN+ELU'. (c) plots cumulative ablation curves that is technique demonstrating channel equal- ization (Morcos et al. 2018). Equipped with CE, both 'BN+ReLU" and 'BN+ELU" presents a more gentle drop curve of top-I accuracy versus cumulative ablation of channels, implying that CE helps channel equalization.
associated with small values of in Eqn. or small values of the channel features (In this paper, inhibited channel is defined as channel whose all feature values are less than le - 2). For example, as shown in Fig I (a.b). the inhibited channels exist in many Norm+RRLU--iks basic block such as 'LN+ReLU', "BN+ELU" and 'BN+ReLU'. This phenomenon has motivated many investigations to directly remove these inhibited channels, such as network slimming (Liu et al.] 2017] Yu et al., (018) and channel pruning (He et al.) 2017b). However, disequilibrium among channels caused by the inhibited channels would do harm to generalization ability of the network and simply removing the inhibited channels that are inactive during training does help improve learning capacity.
Instead of removing the inhibited channels, we present an alternative perspective by proposing a novel building block, named Channel Equilibrium (CE), to recover and equalize the inhibited chan- nels by encouraging channels to contribute equally in the feature representation learning process, thus enhancing the representation and generalization of CNNs. To this end, key observation from Eqn.(1 is that the dependency (covariance) matrix of all the feature channels after normalization scaled by 77 Let this covariance matrix be Σ. We will see that by applying decorrelation operator, i.e. Σー 2, we can not only effectively eliminate the correlations between computations of channel features and the magnitude of y, but also equalize the channels' magnitudes (Barlow al. 1961 Bengio & Bergstral (2009). This operator enables all the channels to play an equal role in the computations of CNN, improving its generalization ability. For example, as shown in Fig T the VGGNet (Simonyan & Zisserrann 2014) equipped with CE is able to effectively prevent inhibited channels and achieves channel equalization in various of "Norm-RelU-like" blocks, consistently improving their recognition performance.
The main contributions of this work three-fold. First, we introduce efficient and effective building block, Channel Equilibrium (CE), which encourages channel equalization and enhances representation learning of CNNs. Second, CE blocks can be stacked after common normalizers and plugged into various advanced architectures, consistently improving their performance by a large margin. For example, CE can be integrated into ResNet and MobileNet V2, forming series of CE-Networks by replacing the ordinary 'BN-ReLU' block by "BN-CE-ReLLU" block, which merely introduces subtle extra computational complexity. As result, the CE-ResNet50 and CE-MobileNet V2 outperform their counterparts by 1.7% and 2.1% top-1 accuracy with nearly the same FLOPs. We also show that combining CE with synchronization across GPUs increases the AP metric on the MS-COCO dataset to 42.0, surpassing its counterpart by 3.4. Third, the learned equalized feature representation of CENet can be better transferred to many other tasks like object detection and segmentation.
2

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Channel equalization. Channel equalization indicates that channels in a layer of CNNs contribute equally to network's computation. The success of the two most commonly used regularization tech- niques, i.e BN (loffe & Szegedy (2115) and Dropout (Srivastava et al.) 2014). is attributed to channel or neuron equalization. For example, |Mianjy et al. (2018) showed that Dropout makes the norm of inconing/outgoing weight vectors of all the hidden nodes equal, indicating a kind of equalization between neurons. Moreover, Morcos et al. (2018) pointed out that BN implicitly discourages single direction reliance, indicating that equalizing different channels is able to enhance the generaliza- tion of learned feature representation. Note that the squeeze-and exciitaitttionn (SE) network (Hu al.) 2018) is a pioneer work that explicitly modeling interdependencies among channels by investigat- ing network design. However, SE selectively emphasizes informative channels and suppresses less useful ones. In contrast, the proposed CE block encourages all the channels to play an equal role in network's computation, which, as will be shown, can be linked with Nash Equilibrium. More Appendix.
related work on sparsity in ReLU and normalization methods are provided in
3 METHOD
In this section, we first review the normalization method and then introduce the proposed Channel Equilibrium (CE) block. CE contains two complementary branches, i.e. batch decorrelation (BD) and adaptive instance inverse (All). We show how BD and All benefit from each other through parameter 7 and how CE linked with Nash Equilibrium.
Notations. For CNNs, we use E NxCx XxW to represent the feature in a layer, specifically, Eneiy denotes a pixel (i,j) in the e-th channel of the n-th sample. Sometimes, we ignore the subscript 'n' and denote as Tetf for clarity of notation. Znij E RC is obtained by stacking elements in all channels of Encij into column vector. Diag(-) returns matrix with the given diagonal and zero off- diagonal entries, and diag(-) extracts the diagonal of the given matrix. 7 B E IR. are normalization parameters.
3.1 OVERVIEW OF NORMALIZATION
Normalization is usually employed after convolution layers to stabilize the training of CNNs. Given a hidden feature x E V xXXXXX a normalizer first standardizes it to ह, and then maps it to by an affine transformation, as written by
Encij = VeTneij + Bo, Enccj (Incis ua)/a,
where s E Ω2 = (IN, BN, L..... } indicates normalizer and Ms. d, are the mean and standard de- viation of the given normalizer. For simplicity, in original formulation is omitted (loffe & Szegedy) 2013). From Eqn. we claim that normalization would lead to an unequal feature representation in channel basis, based on the fact that common-used normalizers like IN and BN are performed channel-wisely. It known that importance of channels are quantified by the learned parameters y with its magnitude since channel features are scaled by 7 in a channel basis. Previous work (Frankle) & Carbin 2018 Mchta et al., 2019) revealed that some inhibited channels emerge when the asso- ciated 7 feature map gets small (Mehta et al., 2019 Lu al. 2019). Obviously, the inhibited channels would cause disequilibrium among channels, resulting in limited generalization ability. To alleviate such disequilibrium, Channel Equilibrium (CE) block is proposed in the following section.
3.2 CHANNEL EQUILIBRIUM (CE) BLOCK
Channel Equilibrium (CE) block is a computational unit which aims equalize feature repre- sentation capacity among channels. To this end, decorrelation method is adopted. Different from previous methods (Huang et al.) [2088] 2019) decorrelating features by a single batch estimated co- variance matrix Σ, the proposed method brings in an adaptive instance variance, S., on the diagonal of the covariance matrix Σ, considering that channel dependency is specific to each input (Hu al. 2018), as formulated in the following.
D, AΣ (1 A)Diag(Sn),
S. (ā.)))
3

---

Under review as a conference paper at ICLR 2020
where the subscript n is the sample index, À E (0, 1) is trainable ratio used to switch between batch and instance statistics, F RC + R is a transformation conditioned on the current in- put Itr and o² (%,,) computes instance variance of ž, within each channel. On the issue of channel disequilibrium, CE block works by decorrelating feature maps after normalization using Dr. à Fur- ther, the Jensen inequality for matrix functions (Pečarid 1996 can be employed to obtain a relaxed decorrelation operator D
(4) where 3 B indicates B A is semi-definite positive. We introduce this relaxation for the fol- lowing two reasons. (1) Computation reduction. It allows less computational cost for each training step since the relaxed form only needs to calculate the inverse of square root Σ once, and the other branch (S.. is easy to compute. (2) Inference acceleration. Σ is moving- statistic inference which can absorbed into previous layer, therefore, enabling fast inference. Note that Eqn. transforms the combination of covariance and adaptive instance variance into the combination of their inverse square roots.
D - F + (1 X)Diag(S.)) AΣ + (1-A) DDiigg(S.))
In the following, we refer Σー in Eqn. batch decorrelation (BD) and refer [S,a as adaptive instance inverse (AII). The former decorrelates channels by a batch covariance, while the latter adjusts the extend of inverse for each channel and instance in an adaptive manner. Integrating both of them yields the forward representation of CE block:
Pnij Dn 3 (Diag())*na) + :)
(5)
where Pnij E RCC denotes the output of CE, as illustrated in Fig b). Note that CE is performed after the normalization layer, BN is taken as an example to introduce these two branches in the following sections.
3.2. BATCU DECORRELATION (BD)
Although a lot of previous work (Huang al. 2018 2019 Pan et al. 2019) has investigated whiten- ing method using covariance matrix, all them are applied after the convolution layer. Thus, inhibited channels still exist since whitened channel features are also scaled by y. Instead, decorre- lation in CE is applied after the normalization layer to equalize magnitude of all channels. Consider tensor 0 after BN layer, it be reshaped as ă RC M and M N И W. Then the covariance matrix of ā can be written as (details are presented in Sec.
B of Appendix)
Σ ==
M DT
(6)
where is standardized feature with zero mean and unit variance and indicates elementwise multiplication. Eqn.(6) implies that the covariance matrix Σ of â can be decomposed into two parts. The first part depends on normalization parameter yy and the second part becomes correlation matrix of म. It is observed that Eij. which represents dependency between i-th channel and j-th channel, is scaled by "yy after BN is applied.
The Batch Decorrelation (BD) branch requires computing Σ-1, which is usually related to eigen- decomposition SVD and involves heavy computation (Huang et al. 2018). Instead, here adopt
an efficient approach, i.e., Newton's Iteration to obtain Σ- (Bini al. 2005 Higham 1986). Given
covariance matrix E, Newton's 's Iteration calculates Σ 1/2 by the following iterations:
{
Σg Σx 133 > E), = 1,2,
where T' the iteration number (T' in our experiments). Note that the convergence Eqn. is guaranteed if || < 1 (Bini et al. 2005). To this end, Σ is normalized as E/tr(E) where tr(.) the trace operator (Huang et al. 2019p. In this way, the normalized covariance matrix is written as Σ 77
@ 37 _. To sum up, the batch decorrelation branch firstly calculates normalized
covariance matrix and then applies Newton's 's Iteration to obtain its inverse square root, reducing lots of computational cost compared with SVD decomposition in the training stage. Furthermore, BD branch can be merged into convolutional layers in the inference stage, which adds extra computation marginally.
4

---

Under review as a conference paper at ICLR 2020
3.2.2 ADAPTIVE INSTANCE INVERSE (AII)
Channel dependencies are specific to each sample. Consequently, conditional decorrelation is desired for each sample. The adaptive instance inverse (AII) branch only uses diagonal entries to model channel dependencies, as shown Eqn. Since a diagonal matrix can inverted easily, this approach can avoid the computation of Eqn.
To construct the All branch, we analyze its input (the output of a BN layer), which is formulated as Enoij neineij + Be. The input of All is the instance variance of each channel (details are provided in Appendix Sec. B
(2( (N
(8)
where iN and TE represent the variances in IN and BN respectively. The ratio of them measures the relative fluctuation of how much the instance statistic are deviated from the batch-estimated statistic. Similar to Eqn.(6) the input of All is also scaled by 22-
The All branch takes of as input and computes an adaptive instance inverse, i.c. [Diag(S,.]] It needs to satisfy two requirements. First, as is desired, dependencies among channels should be embedded in [Diag(S.)] ) for each sample. Second, the output of All should have the same phi- losophy as inverse square root of variance covariance in BD branch. To achieve this, reparame- terization trick employed to generate adaptive instance inverse. Let 8 be an estimate of variance, the All branch can be reparameterized as below,
[Ding(S.)] Diag(F(o 5-
(9)
F ((a 52 (W25 (IN((oo2)) s s= o
(10)
where 8, and 82 are ReLU and sigmoid activation function respectively, Wi E R and W2 RCx and is reduction ratio, E I denotes variance of all elements in ā, which is batch statistic in training and is obtained using moving average for inference. F(a2) E (0, 1) is treated as a gating mechanism in order control the strength of instance inverse for each channel. Following the best practice (Hu et al. (2018); Cao et al. (2119) in characterizing channel relationships, F' is expressed by a bottleneck architecture that is able to model channel dependencies and limit model complexity. Layer normalization (LN) is used inside the bottleneck transform (before ReLU) to ease optimization. It is seen from Eqn. 9 that s represents the quantity of inverse square root of variance and F(aR) regulates the extend of variance inverse. F maps the instance variance to set of channel weights. In this sense, the All branch intrinsically introduces channel dependencies conditioned on each input.
3.3 DISCUSSIONS
Instantiations. Our CE block can be integrated into various advanced architectures, such as ResNet, VGGNet, ShuffleNet V2 and MobileNet V2, by inserting it in horm+Rel.U-iiks building block. The CE block is deseribed in bb). As discussed earlier, CE processes incoming features after the normalization layer by combining two branches, i.c. batch decorrelation (BD) and adaptive instance inverse (AII). Compared with SE block in Fig a)) the proposed CE block combines both instance and batch statistics, and it can consequently model dependencies among channels better.
A series of CENets can be constructed by integrating CE block into various advanced CNN architec- tures. For example, we consider the residual networks (ResNet). The core unit of the ResNet the residual block that consists of "1 x 1', '3 x 3' and '1 x 1' convolution layers, sequentially. Since the CE block is expected to help channel equalization, it would benefit from larger number of channels. Therefore, we employ CE block in the last '1 1 convolution layer by plugging the CE module before ReLU non-linearity, as shown in Fig 2 c). As for CE-MobileNet V2, since the last '1 x 1' convolution layer in the bottleneck is not followed by ReLU activation, we insert CE in the '3 x 3 convolution layer that also has a largest number of channels in the bottleneck. Following similar strategies, CE is further integrated into ShuffleNet V2 to construct TE-ShuffleNet V2. We provide extensive experiments evaluating all these CENets in See 4 and computational details in training and inference in Sec of Appendix.
5

---

Under review as a conference paper at ICLR 2020
Norm
Norm
Input
Global avg pooling
Instance Variance
Batch Covariance
Conv (IxI).
Residual
BN, ReLU
Conv (3x3),
FC, ReLU
Reparam & A ttention
Newton 's Iteration +
BN. ReLU
Conv (1xl),
FC, Sigmoid
All
BD
BN
CE
ReLU
SE block
(b) CE block
CE reskdual block in ResNet
Figure 2: Illustrations of SE block (Hu et al. 2018) (a), CE block (b) and CE residual block in
ResNet (c). denotes broadcast element-wise multiplication, + denotes broadcast elementwise addition and @ denotes matrix multiplication. (b) shows CE has two lightweight branches, BN and All. (c) shows CE can easily stacked into many advanced networks such as ResNet with merely small extra computation.
Equivalent Here we show how BD and All benefit from each other through parameter y. First, we disclose the mechanism in preventing inhibited channels behind the BD branch. Previous work (He al. 2017b; Yu al. 2018; Frankle & Carbin] 2018) revealed that y in BN can be used to prune less unimportant channels, implying that the representational power feature map largely depends the magnitude of Combining Eqn. and Eqn. (5), the output of BD can be expressed Pnij ESag(EE=F (-=+t)=+-ayyeee Enjj 23-23. Comparing with Eqn. ( equivalent y be defined as
f Σ y for BD branch. The proposition 1 shows that BD explicitly increases the magnitude of f a feed-forward way, encouraging all channels to contribute to the feature learning process. The representational power is thus boosted in a channel basis. We provide the proof of proposition i Ce Appendix.
Furthermore, the original y in BN is also implicitly enlarged. It can be seen from Eqn. 6). a sufficient small To can cause degradation of the covariance matrix and then the convergence Newton's iteration (Bini al. 2005) cannot be guaranteed. As result, once the network converges, Yo is not supposed to degrade. This will in turn bring many benefits to All branch. Eqn.(8) shows that the input of All is proportional to 7 meaning that the features fed into All branch are enlarged as y increases. In this way, a bottleneck architecture All can learn more compact global information and model channel dependencies better.
Proposition 1. Let E be covariance matrix of feature maps after batch normalization. Assume that Σ Σ 1, vk- 2,3, 2.3..... .T. then [| 11//l1- Especially, we have Hil Iyal-
Connection with Nash Equilibrium. We show an interesting connection between the proposed CE block and the well-known Nash Equilibrium in game theory (Leshem & Zehavi 2009). To be specific, we bring novel insights on normalization from an optimization perspective. Suppose each channel obtains its output by maximizing capacity available to itself under some constraints. Espe- cially, we restrict that each channel has maximum budget and all the outputs are non-negative. Further, if we consider dependencies among channels, the channels are thought to play non- cooperative game. named Gaussian interference game which admits a unique Nash Equilibrium solution (Laufer et al. (206)). In De Appendix, we present the detailed construction of Gaus- sian interference game in the context of CNNs. It is worth noting that when all the outputs are activated (larger than 0), this Nash Equilibrium solution has an explicit expression. Under some mild approximations, it can be shown that the explicit Nash Equilibrium solution can surprisingly match the representation of CE in (5.. It shows that decorrelating features after normalization layer can be connected with Nash Equilibrium, implying that the proposed CE block indeed encour- ages every channel to contribute to the network's computation. We present detailed explanations about the connection between CE and Nash Equilibrium in Sec of Appendix.

---

Under review as a conference paper at ICLR 2020
8
ResNet50
ResNet101
Baseline
SE
CE
Baseline
SE
CE
Bascline
SE
CE
Top-1
70.4
71.4
71.9
76.6
77.6
78.3
78.0
78.5
79.0
Top-5
89.4
90.4
90.8
93.0
93.7
94.1
94.1
94.1
94.6
GFLOPs
1.82
1.82
1.83
4.14
4.15
4.16
7.87
7.88
7.89
CPU (s)
3.69
3.69
4.13
8.61
11.08
11.06
15.58
19.34
17.05
GPU (8)
0.003
0.005
0.006
0.005
0.010
0.009
0.011
0.040
0.015
Table 1: Comparisons with baseline and SENet on ResNet-18, -50, and -101 in terms of accuracy, GFLOPs, CPU and GPU inference time ImageNet. The top-1,-5 accuracy of our CE-ResNet is higher than SE-ResNet while the computational cost in terms GFLOPs, GPU and CPU inference time remain nearly the same.
MobileNet V2
1x
ShuffleNet V2 0.5%
ShuffleNet V2
Top-1
Top-5
GFLOPs
Top-1
Top-5
GFLOPs
Top-1
Top-S
GFLOPs
Baseline
72.5
90.8
0.33
59.2
82.0
0.05
69.0
88.6
0.15
SE
73.5
91.7
0.33
60.2
82.4
0.05
70.7
89.6
0.15
CE
74.6
91.7
0.33
60.5
82.7
0.05
71.2
89.8
0.16
Table 2: Comparisons with baseline and SE on lightweight networks, MobileNet V2 and ShuffleNet V2, in terms of accuracy and GFLOPs on ImageNet. Our CENet improves the top-1 accuracy by large margin compared with SENet with nearly the same GFL.OPs.
EXPERIMENTS
We evaluate our methods on two basic vision tasks, image classification on ImageNet and object detection/seemmetttiion COCO, where we demonstrate the effectiveness of the CE block.
4.1
IMAGE CLASSIFCATION ON IMAGENET
We first evaluate CE on the ImageNet benchmark. The training details are illustrated in Sec F of
Appendix.
Performance comparison on ResNet. We evaluate on representative residual network structures in- cluding ResNet18. ResNet50 and ResNet101. The CEi-ReNNet is then compared with baseline (plain ResNet) and SE-ResNet. For fair comparisons, we use publicly available code and re-implement baseline models and SE modules with their respective best settings To save computation, the CE blocks are selectively inserted into the last normalization layer of each residual block. Specifically, for ResNet18, we plug the CE block into each residual block. For ResNe150, CE is inserted into all residual blocks except for those layers with 2048 channels. For ResNet101, the CE blocks are employed in the first seven residual blocks.
unified Pytorch framework.
As shown in Table our proposed CE outperforms the BN baseline and SE block by a large margin with little increase of GFLOPs. Concretely, Ce--ReNNet18, CE-ResNet50 and CE-ResNet101 obtain top-1 accuracy increase of 1.5%. 1.7% and 1.0% compared with the corresponding plain ResNet architectures. The CE-Res ResNett0 even outperforms the plain ResNet101 (78.0). We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and CE- Ce-ResNet50 in Sec Appendix.
We also analyze the complexity of BN, SE, and CE in terms of GFLOPs, GPU and CPU run- ning time. We evaluate the inference time") with a mini-batch of 32. In term of GFLOPs, the CE- ResNet18, CE-ResNet50, CE-ResNet101 has only 0.242% and 0.241 % relative increase in GFLOPs compared with plain ResNet. Additionally, the CPU and GPU inference time of CENet is nearly the same with SENet.
Performance comparison on light- networks. We further verify the effectiveness of our proposed CE two representative light- networks, MobileNet V2 and ShuffleNet V2. The results of comparison are listed in Table 2 It is seen that CE blocks bring conspicuous improvements
'The CPU type is Intel Xeon CPU ES-2682 v4, and the GPU is NVIDIA GTX 1080TT. The implementation based on Pytorch
7

---

Under review as a conference paper at ICLR 2020
BN
ON
IN
LN
top-1
top-5
top-1
top-5
top-1
top-5
top-1
top-5
Baseline
76.6
93.0
75.6
92.8
74.2
91.9
71.6
89.9
Baseline+CI
78.3
94.1
76.2
92.9
76.0
92.7
73.3
91.3
Increase
+1.7
+1.1
+0.6
+0.1
+1.8
+0.8
+1.7
+1,4
Table CE improves top-1 and top-5 accuracy of various normalization methods on ImageNet with ResNet50 backbone.
Heiibited Charnel Radio
80
BR
Top 3,61e
10
Top
Drop Ratis
Drop
Weight Dacay (2a-3)
(a)
(b)
(c)
(d)
Figure 3: (a) & (b) show channel drop ratios versus top-1 accuracy for MobileNet V2 and ResNer50 on ImageNet dataset respectively. We randomly ablate channels with an increasing fraction in the first normalization layers. Note that CE-ResNet50 and CE-MobileNet V2 are more robust to cumu- lative ablation of channels than those networks trained with only BN, suggesting that CE also helps channels equalization ImageNet. For (c) and (d), we train VGGNet in CIFAR-10 under different weight decays. It is observed that the networks trained with the proposed BD and CE were consis- tently and substantially more robust to the increasing strength of weight decay than those trained with single BN.
in performance minimal increase in computational burden mobile settings. For MobileNet V2, we see that CE blocks even improves top-1 accuracy of baseline by 2.1%.
Other Normalizers. In addition to BN, CE is also effective for other normalization technologies, since inhibited channels emerges in many well-known normalizers as shown in Fig[T] To prove this, conduct experiments using ResNet-50 under different normalizers including batch normalization (BN), group normalization (GN), instance normalization (IN), and layer normalization (LNN. For these experiments, we stack CE block after the above normalizers to see whether CE helps other normalization methods. As shown in Table 3 our CE generalize well over different normalization technology, improving the performance by 0.6- top-1 accuracy.
4.2 ANALYSIS OF CE
In this section, we first demonstrate that CE is able to equalize the importance of all channels and then analyze the effects of BD and All branches separately CIFAR10 and ImageNet datasets. More discussions about CE are provided in See of Appendix.
CE is effective in channel equalization. In Fig inhibited channels, which is a necessary condition of channel equalization. Here, we further show that the ability that CE can prevent inhibited channels is robust wide range of strength of weight decay. As shown in Fig (c,d), CE prevents inhibited channels and retains higher performance under different strengths of weight decay.
have demonstrated that CE is able alleviate
Next, we verify whether CE can help channel equalization by an ablation approach used in Mor- cos et al. (2018). Typically, the importance of a single channel to the network's computation can be measured by the relative performance drop once that channel is removed (clamping activity feature map to zero). In this regard, the more reliant a network is on small set of channels, the more quickly the accuracy will drop if those channels are ablated. On the contrary, if the im- portance of channels to the network's computation are more equal, the accuracy will drop more gently. With this powerful technique, we see how ResNet50 and MobileNet V2 with CE blocks respond to cumulative random ablation of channels. We plot the ablation ratio versus the top-1 accuracy in Fig a.b.. As lation of channels on both ResNet50 and MobileNet V2, showing that CE can effectively equalize the importance of channels. For example, the top-1 accuracy of Res is 1.7 higher
see, CE block is able to resist the cumulative random ab-
8

---

Under review as a conference paper at ICLR 2020
Method
Plain ResNet50
BD-ResNett0
All-ResNet50
Ci-ResNet50
top-1
76.6
77.0 (+0.4)
77.3 (+3.7)
78.3 (+1.7)
Table 4: Results of batch covariance decorrelation, adaptive variance inverse and channel equilib- rium. We use ResNet-50 as the basic structure. The top-1 accuracy increase (1.7) of CE-ResNet is higher than combined top-1 accuracy increase (1.1) of BD-ResNet and All-ResNet, indicating the effects of BD and All branch is complementary.
than the original ResNet50 no channels are ablated, but when 70% channels are ablated, CE- ResNet50 still obtain 23.0 top-1 accuracy, while the original ResNer50 gets only 4.6 top-1 accuracy.
BD is able to mitigate inhibited channels. As proved in Proposition []] equivalent y in the BD branch is explicitly enlarged, leading to the ex- pansion of representational power of all channels.
Here we investigate this property experimentally with a single BD branch. The inhibited channels ratio are measured by the percentage of feature maps whose values are less than le-2. 3)) shows inhibited channels ratio under wide range of weight decay for BN, BD and CE. Is observed that the t accuracy of VGGNet with BN drops significantly as the weight decay increases, but BD can reduce accuracy drop. For example, when the weight decay is le-3, the top-1 accuracy of BD is only 0.1 higher than BN, but when the weight de- cay reaches to le-2, BD is 4.95 higher. Moreover, the inhibited channels ratio of CE is even lower than BD while the top-1 accuracy is higher, which can demonstrate that CE can strengthen the effect of alleviation of inhibited channels compared with single BD.
Layer Number
Figure 4: We do principal component analy- sis (PCA) on the input of All sub-network, the variance of cach channel. This figure show the box chart of principal components. CENet has lower means and variances than AllNet, indi- cating the input of All sub-network in CENet is more equal and informative.
AII helps CE learn preciser feature representation. First, discussed in Sec 3.3 All benefits from BD such that the features fed into All branch are more informative. To see this, we train ResNet50 with single All or CE branch, termed All-ResNet50 or CE-ResNet50. We do principal component analysis (PCA) on the inputs of All branch in All-ResNer50 and the counterpart in CE- ResNe150 and plot the box chart of principal components. As is shown in Fig 4 the input of All branch in CE-Resh CE-ResNet50 gets much lower means and variances, meaning that the input feature has more valid basis and thus more informative. From the Grad-CAM visualization provided in Sec/E of Appendix, we find that All helps CE learn preciser feature representation.
BD and All are complementary. Here, we verify that BD and All are complementary to each other. We train plain ResNet50, BD-Re ResNet50, All-ResNet50, and CE-ResNet5h for comparison. The top-1 accuracy reported in Table are 0.4 and 0.7 higher than the plain ResNet-50 respectively. However, when they are combined, the top-1 accuracy improves by 1.7, higher than combined accuracy increase (1.1), which demonstrates that they benefit from each other.
is observed that the BD-ResNet50 and All-ResNe150
4.3 OBJECT DETECTION AND INSTANCE SEGMENTATION ON COCO
We assess the generalization of our CE block on detection/seemmenationn track using the COC02017 dataset (Lin et al. (2014)). We train our model on the union of 80k training images and 35k vali- dation images and report the performance on the mini-val 5k images. Mask-RCNN is used as the base detection/seemmenttiion framework. The standard COCO metrics of Average Precision (AP) for bounding box detection (APbb) and instance segmentation (APm) is used to evaluate our methods. In addition, adopt two common training settings for models, (1) freezing the vamilla batch normalization and channel equilibrium layer and (2) updating parameters with the synchronized version. For vanilla BN and CE layers, all the gamma, beta parameters, and the tracked running

---

Under review as a conference paper at ICLR 2020
Backbone
AP
AP
AP7s
APm
APS
APT
ResNet50
38.6
59.5
41.9
34.2
56.2
36.
CE-ResNet50
40.8
62.7
44.3
36,9
59.2
39.4
SyneCE-ResNetthy
42.0
62.6
46.1
37.5
59,5
40.3
ResNer101
40.3
61.5
44.1
36.5
58.1
39.1
CF-ResNet101
41.6
62.8
45.8
37.4
59.4
40.0
Table 5: Detection and segmentation results in COCO using Mask-RCNN We use the pretrained CE-ResNet50 model (78.3) and CE-ResNet101 (79.0) in ImageNet to train our model. CENet can consistently improve both box AP and segmentation AP by large margin.
statistics are frozen. contrast, for the synchronized version, the running mean and variance for batch normalization and the covariance for CE layers computed across multiple GPUs. The gamma and beta parameters are updated during training while F and À are frozen to prevent overfit- ting. We use MMDetection training framework with ResNet50/ResNettll all the hyper-parameters are the same as Chen et al. (2019). Table 5 shows the detection and seg- mentation results. The results show that compared with vanilla BN, our CE block can consistently improve the performance. For example, our fine-tuned CE-ResNet50 is 2.2 AP higher in detection and 2.7 AP higher in segmentation. For the syne BD version, CE-ResNec50 gets 42.0 AP in detec- tion and 37.5 AP in segmentation, which the best performance for ResNet50 to the best of our knowledge. To sum up, these experiments demonstrate the generalization ability of CE blocks in other tasks.
as basic backbones and
CONCLUSION
In this paper, we presented novel network block, termed Channel Equilibrium (CE). The CE block conditionally decorrelates feature maps after normalization layer by switching between batch decorrelation branch and adaptive instance inverse branch. We show that CE is able to explicitly alleviate the inhibited channels and help channel equalization, enhancing the representational power of a neural network in fred-forward way. Specifically. CE can be stacked between the normal- ization layer and the ReLU function, making it flexible to be integrated into many advanced CNN architectures. The superiority of CE blocks has been demonstrated on the task of image classifica- tion and instance segmentation. We hope that the analysis of channel equalization in CE could bring a new perspective for future work in architecture design.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint crXiv:1607004495 2016.
Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1:217-234, 1961.
Yoshua Bengio and James S Bergstra. Slow, decorrelated features for pretraining complex cell-like networks. In Advances in neural information processing systems, pp. 99-1979 2009.
Dario A Bini, Nicholas J Higham, and Beatrice Meini. Algorithms for the matrix pth root. Numerical Algorithms, 3944):349-378, 2005,
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeere-excitation networks and beyond. arXiv preprint arXiv:1904. 1492, 2019.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arxiv: XXX:1196.00755, 2019.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.0728) 2015.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: / 803,00335, 2018.
10

---

Under review as a conference paper at ICLR 2020
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteetth international conference on artificial intelligence and statistics, pp- 315-323, 2011.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017a.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017b.
Nicholas Higham. Newton's method for the matrix square root. Mathematics of Computation, 46 (174):537-549, 1986.
Jie Hu, Shen, and Gang Sun. Squeere-and-excitatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 791-800, 2018.
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardiza- tion towards efficient whitening. In Proceedings the the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4874-4883 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Amir Laufer, Amir Leshem, and Hagit Messer. Game theoretic aspects of distributed spectral coor- dination with application to dsl networks. arXiv preprint cs/0602014, 2006.
Amir Leshem and Ephraim Zehavi. Game theory and the frequency selective interference channel. IEEE Signal Processing Magazine, 26(5):28 40, 2009.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference Computer Vision, pp. 7736-2744,, 2017.
Lu Lu, Yeonjong Shin, Yanhui Su, and George Em Kamiadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint XXXiv:90330077. 2019.
Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning-to- normalize via switchable normalization. arXiv preprint arXiv:1806. 10779, 2018.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlincarities improve neural net- work acoustic models. In Proc. ieml, volume 30, pp. 2013.
Dushyant Mehta, Kwang In Kim, and Christian Theobalt. On implicit filter level sparsity in convo- lutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 520-528, 2019.
Poorya Mianjy, Raman Arora, and Rene Vidal. On the implicit bias of dropout. arXiv preprint 18Xv01180660977,, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 1rXi::8822.0557. 2018.
Ari Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv: arXiv: 1803.06999 2018.

---

Under review as a conference paper at ICLR 2020
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Xingang Pan, Xiaohang Zhan, Jianping Shi, Xiaoou Tang, and Ping Luo. Switchable whitening for deep representation learning. Proceedings of the IEEE International Conference on Computer Vision, 2019.
Josip Pečarić. Power matrix means and related inequalities. Mathematical Communications, 1(2): 91-110, 1996.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference Computer Vision, pp. 618-626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Dexxvv:4499 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ily Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting, The journal of machine learning research, 15(1):1999-1958, 2014.
Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deeply learned face representations are sparse, selective, and robust. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2892-2900, 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint 12Xiv:1607008022 2016.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arX1v:1812.08898, 2018.
12

---

Under review as a conference paper at ICLR 2020
Appendix
A MORE RELATED WORK
Sparsity in ReLU. An attractive property of ReLU (Sun et al. 2015 Nair & Hinton, 2010 is spar- sity, which brings potential advantages such information disentangling and linear separability. However, Lu et al. (2019) and Mehta et al. (2019) pointed out that some ReLU neurons may be- come inactive and output values for any input. Previous work tackled this issue by designing new activation functions, such as ELU CClevert et al. 2015) and Leaky ReLU (Maas et al. (013). Re- cently, [uu et al. (2019) also tried to solve this problem by modifying initialization scheme. Different from these work, we focus explicitly preventing inhibited channel in feed-forward way by the proposed CE blocks.
Normalization and decorrelation. There are many practices on normalizer development, such as Batch Normalization (BN) (Ioffe Szegedy, 2015). Group normalization (GN) (Wu & Hc 2018)) and Switchable Normalization (Luo et al. 2018). A normalization scheme is typically applied after a convolution layer and contains two stages: standardization and rescaling. Another type of nor- malization methods not only standardizes but also decorrelates features, like DBN (Huang et al.) 2018). IterNorm (Huang et al.) 2019) and switchable whitening (Pan et al., 2019). Despite their success in performance improvement, little is explored about relation between those methods and inhibited channels. Fig shows that inhibited channels emerges in VGGNet where 'BN+ReLU' or "LN+ReLU" are used. Unlike previous decorrelated normalizations where decorrelation operation is applied after a convolution layer, our CE explicitly decorrelates features after normalization.
B COMPUTATION DETAILS IN 'BN-CE-RELU" BLOCK
As discussed before, CE processes incoming features after normalization layer by combining two branches, i.c. batch decorrelation and adaptive instance inverse. The former computes a covariance matrix and the latter calculates instance variance. We now take 'BN-CE-ReLU" block as an example to show the computation details of statistics in it. Given tensor NxCxXxX the mean and variance in IN (Ulyanov et al. 2016) are calculated as:
H.W Σ
H.W
1
1
MIN
Encijs
(oix) )TC
Σ
(oncij - MPN)"
(11)
HW
HW
1
1
Hence, we have MIN, oN E NXC Then, the statistics BN can be reformulated as follows:
N.H.W
H.W
1
Σ
Encij
Σ
Σ
Encij
NHW
N
HW
n.J.J
N.H.W
1
(O
Σ
(Incij MBNN
NHW
N
H.W
Σ
(Incij MAN + MIN MiiN) )²
(12)
N
HW
N
H.W
Σ
(Enei)
1 2 + (NIN - MBN)2
N
HW
N
+
(PIN - PEN)2
N
Then, we have Man = E[pin] and of = E [ + D[pin]. where E|:] and D[:] denote expectation and variance operators over N samples. Further, the input of All is instance variance of features
13

---

Under review as a conference paper at ICLR 2020
after BN, which can be caleulated as follows:
H.W
Encij - MiNN
MIN -MEN + +B2)
Σ
(1e
+ +c) - (%
ne
HW
BN
OBN
12
H.W
Σ
(Enci) (iN)
(13)
BN )
HW
1e (N
BoN)
At last, the output BN is Enesy = YoIncis + Se, then the entry in e-th row and d-th column of covariance matrix E of 0 is calculated follows:
N.H.W
Ed
Σ
(NeFnes))((RRnnt)) = YaPed n.C.j
(14)
NHW
where Ped is the element in e-th and j-th column of correlation matrix of σ. Thus, write
into vector form: = my @ M
-r we reshape to E RCxM and M N. H W.
C PROOF OF PROPOSITION
Proposition 1. Let 2 be covariance matrix of feature maps after batch normalization. Assume that Σg = Σ- Ak- = 2.3, ... T. then || > 117/11- Especially, we have Hi 1%/
Proof. Since Σg Σー 3 Yk 2,3, T, we have E&a 1 (37 ΣZ Eyy Ex-17- Therefore, we only need show || ||Er>|l1
|| 1/ Now, we show that
for k = 2 have [3 Σ>||, > ||/l|, From Eqn.(6), know that Σ
2n Tr
where p is
the correlation matrix of and 1 S Pij < 1, Vi,j E [C]. Then, we have
1
(31 E}> = (31 2
2 p)r
2
111l22
77
(37
p))
1
1
(3y
1
Σ
E
?27)p22
Σ
ToTSOONT
)
17>11
(15)
= (37
Σ
Mor61593 Σ
Σ
1m
T
1
(3 Σ
mm, (3
Σ
TÿP25
... (3
C
17/12
1|
111l
5
Tspy Note that Σ3 1
23 3- - و 73py
2
2, where the last equality holds iff
Psj 1, Vi E [C]. However, this is not the case in practice. Hence we have
C
[ 31 - Σ>>
3
Σ
2
> hal
(16)
2
17
12
3
Therefore, have ||9//, 1| Here completes the proof.
14

---

Under review as a conference paper at ICLR 2020
D CONNECTION BETWEEN CE BLOCK AND NASH EQUILIBRIUM
We first introduce the definition of Gaussian interference game in context of CNN and then build the connection between CE block and Nash Equilibrium. For clarity of notation, we omit the subscript for concrete sample.
Let the channels 1,2, C operate over x W pixels. Assume that the C channels have depen- C.C
dencies G = (t, (,) Each pixel characterized by a power gain hasj 2 0 and channel noise strength de > 0. In context normalization, we suppose heir = Σaj + where Eeis is stan- dardized pixel in Eqn.(2) and ó is sufficiently large to guarantee non-negative power gain. Assume that e-th channel is allowed transmit total power of P. and have -i.j=1 Peij Pe. Besides, each channel can transmit power vector Pe = (Pel1, .Porw). Since normalization layer often followed by a ReLU activation, we restrict Peij 2 O. What we want to maximize the capacity transmitted over the c-th channel, Vc [C], then the maximization problem given by:
max Co(p1,P2.... ... .pc) ,..)))).).))))............))))")
Σ iJ
1+
JecPeij
Edee JodPass + oo/hoss
(17)
Σ H.W Eij= Peij = Pe:
s.t.
Potj 20,
Vi E [H] E [W]
where C. the capacity available to the e-th channel given power distributions P1- P2." PPc- In game theory, C channels and solution space of {Peis (peis)e.j.-1 together with pay-off vector (C1, C2. ... Cc) form a Gaussian interference game G. Different from basic settings in G, here do not restrict dependencies Ied to (0, 1). It is known that G has unique Nash Equilibrium point whose definition is given as below,
C.H.W
Definition 1. An C-tuple of strategies (pi, (p1,P2...- ... PCC) for channels 1,2,..., , respectively is called a Nash equilibrium iff for all and all p (p strategy for channel c)
Celpr, PP- Pctt1 .Pc) S Cc(P1.P2, ,Pc)
(18)
ie., given that all other channels * c use strategies Pa. channel c best response is Pe- Since Cr,Cr C2,.. Cc are concave in P1, P2, PCC respectively, KKT conditions imply the following theorem.
Theorem 1. Given pay-off in Equ. 17 (pi,... .PC) is a Nash equilibrium point if and only if there ,00)
exist th = (og. (00)) (Lagrange multiplier) such that for all é E [H] and j E [W],
Ice
2 for Pos > 0 S o for Posj = 0
(19)
Ea Bej oe/hesj
Proof. The Lagrangian corresponding to minimization of - Ce subject to the equality constraint and non-negative constraints on Peig given by
N.W Σ In 1+ ( iJ=1
H.W P.)
H.W
L.
E
EdteeeePaus PedPais + ae/heis
9cePeij
+ vg
Σ +,==1
Paj
Σ
(-Pea))) (20)
Differentiating the Lagrangian with respect to Peij and equating the derivative to zero, we obtain
JeC
+ Di
oij
(21)
Ea PedPeis + Be/heis
Now, using the complementary slackness condition uf Pesj = and of 2 0, we obtain condition This completes the proof.
19
By Theorem the unique Nash Equilibrium point can be explicitly written as follows when Pai
0.
P/ G-1 (Diag(un) -1 diag(() (Diag(hy))10
(22)
15

---

Under review as a conference paper at ICLR 2020
60
60
BN
BN
55
55
CE
CE
50
50-
45
45-
train (%)
40
40
validattion
35
35
30
30
25
25
20
epochs
epochs
Figure 5: Training and validation error curves on ImageNet with ResNet50 as backbone for BN, SE and CE.
where Piy.hig. RC and to E RC are Lagrangian multipliers corresponding to equality straints. Note that approximation can be made using Taylor expansion as follow: hedj
0.(2 + hai + +O((1 + h)) Thus, a linear proxy to (2) can written as Pij = G-1 (Diag(a) (Gs + Diag(ro) diag G)) (2+))
(23)
and 8 = Diag( ( - (G) (2 + 8)o, Eqn. Eqn.(23) can surprisingly match CE unit Eqn. implying that the proposed CE block indeed performs mechanism on channel equalization. In Gaussian interference game, is known and to can be determined when budget Pa's are given. However, 7 and are learned by SGD in deep neural networks.
Let G = [Dx]+
E EXPERIMENTS
E. TRAINING AND VALIDATION CURVES
We plot training and validation loss during the training process for ResNet50, SE-ResNet50 and CE-ResNer50 in 5 We observe that CE-ResNet50 consistently have lower training and validation errors over the whole training period, indicating that CE improves both learning capacity and generalization ability.
E.2 MORE DISCUSSION ABOUT CE
As discussed in related work in Sec many methods have been proposed to improve normalizers and ReLU activation. The ablation approach Morcos al. (2018) used see whether and how these methods help channel equalization. We demonstrate the effectiveness of CE by answering the following questions.
Do other ReLU-like activation functions help channel equalization? Two representative im- provements on ReLU function, i.e. ELU (Clevert et al.] 2013) and LReLU (Maas al. 2013), are employed to see whether other ReL.U-like activation functions can help channel equalization. We plot the cumulative ablation curve that depicts ablation ratio versus the top-1 accuracy on CI- FARI0 dataset in bla). The baseline curve is BeN+ReLU'. As we see, the top-1 accuracy curve of "N++ReeUU" drops more gently, implying that LReLU helps channel cqualization. But 'ELU+ReLU" has worse cumulative ablation eurve than "BN+ReLU', By contrast, the proposed CE block improves the recognition performance of "eN+ReLU' (higher top-1 accuracy) and promotes channel equalization most (the most gentle cumulative ablation curve).
Do the adaptive normalizers help channel equalization? We experiment on representative adaptive normalization method (i.e. SN), whether it helps channel equalization. SN learns to
16

---

Under review as a conference paper at ICLR 2020
90
80
BN
80
CE
60
SN
Top 1 Acc
70
Top 1 Acc
40
50
BN+ReLU
40
UN+CE+ReLU
20
BN+ELU
30
BN+LeakyReLU
0
0.2
0.4
0.6
1.0
0.2
0.4
0.8
1.0
Drop Ratio
Drop Ratio
(a)
(b)
Figure 6: (a) compares the cumulative ablation curves of 'BN+ReLU', 'BN+ELU', 'BN+LReLU" and "BN+CE+ReLU" with VGGNet on CIFAR-10 dataset. We see that the Both LReLU and CE improve the channel equalization in "BN+ReLU' block. (b) compares the cumulative ablation curves of 'BN+ReLU', 'SN+ReLU' and "BN+CE+ReLU" with ResNet-50 on ImageNet dataset. The proposed CE consistently improves the channel equalization of "BN+ReIU" block. Note that "BN+CE+ReLU" achieves the highest top-1 accuracy on both two datasets compared to its counter- parts (when drop ration 0).
Top- acc
CE2-ResNet50
77.9
CE3-ResNet50
78.3
Table 6: We add CE after the second (CE2-ResNet50) and third (CE3-ResNeth) batch normaliza- tion layer each residual block. The channel of the third batch normalization is times than that of the second one but the top-1 accuracy of Ce3-ResNet50 outperforms CH2-ResNer50 by 0.4, which indicates CE benefits from larger number of channels.
select an appropriate normalizer from IN, BN and LN for each channel. The cumulative ablation curves are plotted on ImageNet dataset with ResNet-50 under blocks of 'BN+ReLU', 'SN+ReLU' and "BN+CE+ReLU". As shown in Fig 6 b), SN even does damage to channel equalization when it used to replace BN. However, "BN+CE+ shows the most gentle cumulative ablation curve, indicating the effectiveness of CE block in channel equalization. Compared with SN, ResNet-50 with CE block also achieves better top-1 accuracy (78.3 vs 76.9), showing that channel cqualization important for block design in CNN.
Integration strategy CE block. We put CE in different position of a bottleneck ResNet50, which consists of three "Conv-BN-Rell" basic blocks. The channel of the third block is 4 times than that of the second one. We compare the performance of CE-ResNet50 by putting CE in the the third block (CE3-ResNet5u) As shown in Table 6 the top-1
second block accuracy of CE3-ResNer-50 outperforms CF2-ResNet50 by 0.4, which indicates that our CE block benefits from larger number of channels.
E.3 GRAD-CAM VISUALIZATION
We claim that All learns adaptive inverse of variance for each channel a self-attention manner. Fed into more informative input, All is expected to make the network respond different inputs in highly class-specific manner. In this way, it helps CE learn preciser feature representation. To verify this, employ an off-the-shelf tool to visualize the class activation map (CAM) (eelaraju et al. (2017). We use ResNet50, DD-ResNet50, and rE-ReeNet50 trained on ImageNet for comparison. As shown in Fig 7 the heat maps extracted from CAM for CE-ResNet50 have more coverage on the
17

---

Under review as a conference paper at ICLR 2020
(a) Origin
(b) ResNet50
(c) BD-ResNet50
(d) CE-ResNet550
Figure 7: Grad-cam visualization results from the final convolutional layer for plain ResNet50, SE SE-ResNet50, and CE-ResNet50.
object region and less coverage on the background region. shows that the All branch helps CE learn preciser information from the images.
F TRAINING AND INFERENCE
Moving average in inference. Unlike previous methods in manual architecture design that do depend on batch estimated statistics, the proposed CE block requires computing the inverse square root of batch covariance matrix Σ and a global variance scale in each training step. To make the output depend only on the input, deterministically in inference, we use the moving average to calculate the population estimate of Đー and $ by following the below updating rules:
Σー = (1 m m -
(1 m)s -
(24)
where and E are the variance scale and covariance caleulated within each mini mati-batch during train- ing, and denotes the momentum of moving average. It is worth noting that since シ- is fixed
18

---

Under review as a conference paper at ICLR 2020
during inference, the BD branch does not introduce extra costs in memory computation except for simple linear transformation (8)
Model and computational complexity. The main computation of our CE includes calculating the covariance and inverse square root of it in the BD branch and computing two FC layers in the All branch. We see that there is a lot of space to reduce computational cost of CE. For BD branch, given the cost of calculating covariance matrix is 2N HWC*
an internal feature E R- which is comparable to the cost of convolution operation. A pooling operation can be employed to downsample featuremap for too large H and W. In this way, the complexity can be reduced to HN + CHW where l is kernel size of the window for pooling. Further, we can use group-wise whitening to improve efficiency, reducing the cost of computing Σー from TC to TCg² (g is group size). For All branch, we focus on the additional parameters introduced by two FC layers. In fact, the reduction ratio r' can be appropriately chosen to balance model complexity and representational power. Besides, the majority of these parameters come from the final block of the network. For example, single All in the final block of ResNet-50 has 2 2048 /r parameters. In practice, the CE blocks in the final stages of networks are removed to reduce additional parameters. We provide the measurement of computational burden and Flops in Table 1.
ResNet Training Setting. All networks are trained using 8 GPUs with a mini-batch of 32 per GPU. We train all the architectures from scratch for 100 epochs using stochastic gradient descent (SGD) with momentum 0.9 and weight decay le-4. The base learning rate is set to 0.1 and is multiplied by 0.1 after 30, 60 and 90 epochs. Besides, the covariance matrix BD branch calculated within each GPU. Since the computation of covariance matrix involves heavy computation when the size of feature map is large, a x 2 maximum pooling is adopted to down-sample the feature map after the first batch normalization layer. Like (Huang et al.) 2201)), we also use group-wise decorrelation with group size 16 across the network to improve the efficiency in the BD branch. By default, the reduction ratio in All branch is set to 4.
MobileNet V2 training Setting, All networks are trained using 8 GPUs with a mini-batch of 32 per GPU for 150 epochs with cosine learning rate. The base learning rate is set to 0.05 and the weight decay is 4e-5.
NeuffleNet V2 training Setting. All networks are trained using 8 GPUs with mini-batch of 128 per GPU for 240 epochs with poly learning rate. The base learning rate is set to 0.5 and weight decay is 4e-5. We also adopt warmup and label smoothing tricks.
VGG networks on CIFARIO training setting. For CIFARIO 10, we train VGG networks with a batch size of 256 on single GPU for 160 epochs. The initial learning rate is 0.1 and is decreased by 10 times every 60 epochs. The inhibited channel ratios in Fig. [ and Fig (() is measured by the average ratio for the first 6-th layers since the bottom layers can extract rich feature representation and the sparsity is not desired. For inference drop experiments in Fig [(c), we randomly drop channels in the third layer with different dropout ratio. For each ratio, we run the experiment times and average the top I accuracy.
-CCN training setting in COCO. We fine-tune the ImageNet pretrained model in COCO for 24 epoch with base learning rate 0.02 and multiply it by 0.1 after 16 and 22 epochs. All the models are trained using GPUs with mini-batch of images. The basic backbone structure adopted from the ResNec50/ResNettl trained on ImageNet.
19


Reviews:
[{"review_id": "SklZzltOqS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the channel-collapsed problem in CNNs using 'BN+ReLU' . The Channel Equilibrium block which consists of batch decorrelation branch and adaptive instance inverse branch are proposed to reduce the channel-level sparsity. Experiments on ImageNet and COCO demonstrate that the proposed CE block can achieve higher performance than the conventional CNNs by introducing little computational complexity. The author also discuss the relationship between the proposed method and Nash Equilibrium.\n\nPros:\n\n+ The experimental results are impressive, the proposed block can improve the accuracy of CNNs while requires little additional computation cost.\n+ This paper is well-written and easy to follow. The authors give a explicit explanation as well as prove of the proposed scheme. \n\nCons:\n\n- The motivation of this paper seems to be weak. The author argues that popular CNNs with 'BN+ReLU' have certain channels which would always output 0 for any input. Why not directly remove this channel to achieve speed-up? \n- Moreover, the author argues that 'BN+ReLU' block would lead to channel-level sparsity according to [1]. However, [1] says that this sparsity relies on weight decay. Figure 3 (d) also proves that the sparsity ratios of BN and CE are all 0 when weight decay is set as 0 (also notes that they achieve best accuracy when weight decay is 0). The results demonstrate that the higher accuracy of CE does not rely on its lower sparsity ratio.\n\nIn conclusion, the proposed CE is effective for achieving higher accuracy. However, the motivation and argument of the proposed method seems to be invalid, which prevents this work to be accepted.\n\n[1] On implicit filter level sparsity in convolutional neural networks. CVPR, 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HygbkYmJ9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new block that gives diminishing \n\nPros)\n(+) This paper is well-written and the idea looks interesting.\n(+) The provided experiments look quite extensive. I like the experiments with both heavy and light network architectures to show the effectiveness of the proposed method.\n(+) The authors provided plausible and sufficient backups for the claim.\n(+) Connection with Nash equilibrium seems to be slightly overclaimed, but the attempt is novel and interesting.\n\n\nCons)\n(-) The overall paper flow could be organized much better. I think some of the materials in the appendix needs to be placed in the main paper.\n(-) Some of the notations in Section 3 need to be rewritten for better clarity. \n(-) The terminology such as channel-level sparsity, channel equalization, and equivalent lambda looks vague. I recommend the authors define them for clarity.\n\nComments) \n- I think the relationship between diminishing the sparsity on channels and improving the evaluation performance should be more addressed. If we change ReLU after Conv with other nonlinear functions such as ELU or PReLU, then the output hardly becomes zero, but as you know the performance usually gets worse. Then, how do you explain the necessity of reducing sparsities? \n- How did you measure the sparsity in a plain network in Figure 1 and Figure 3? Did you pick a single layer or gather all the layers' sparsity? Please specify the way of measuring it.  \n- How did you plot the graphs in Figure 3.(a) and 3.(b). Did you pick a single channel randomly?\n- To support the authors' claim, it would be better to show the sparsity ratio for a model (ResNet18, MobileNetV2, and so on) which shows better performance than each original one. \n- How did you determine the transformation of F in the AII branch? Please provide any intuitions why the authors choose the transformation like in eq.(9). \n- If CE-block actually conducts well right after a BN layer, then why CE block is attached only after the final BN layer in a bottleneck module? It would be better to provide any results by doing some studies when dealing with the intermediate BNs in a bottleneck module.\n- I am wondering whether adaptive normalization techniques such as Instance-Batchnormalziation or Switchable Normalization methods could also give less sparsified features. Please clarify this by comparing with the proposed method.\n- It seems that CE would have some extra computational costs compared with SE's. Please clarify why CE-module does have small computational costs.\n- Please specify what MobileNetV2 has used. Looks like MobileNetV2x1.0 would be used.\n- Why do you think the Top-5 accuracy of MobileNetV2 has not been improved?\n- Can your method combine with ResNet with SE such as ResNet50SE? \n- Please clarify the way of measuring the sparsity ratio in Figure 3d.   Why did you consider the sparseness of a feature by measuring the gamma in Figure 4? There may exist a channel that contains large magnitude values then gamma cannot make the output close to zero.\n\nAbout overall rating)\nThe theoretical backups for the authors' claim look sound, and all the experiments including the performance improvement with several models seem to support the effectiveness of the idea very well. Specifically, I like all the analyses and the connection with Nash equilibrium, which are very intuitive and may provide further insights to researchers in this field.  I think this paper is clearly above the standard of ICLR. If the authors could address all the concerns above and refine the paper with better readability, then I could increase the score. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HygV4Vq0tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors point out that CNNs can develop collapsed channels that limit their capacity. They propose to remedy this with batch decorrelation (BD), which focuses on ensuring that channels play an equal role in the feature map and are less likely to collapse. The claim is supported with experiments on CIFAR10, ImageNet and COCO.\n\nFigures 1 and 3 show convincingly that BD reduces channel collapse. The overall results are modest improvements, but the point the authors make is that reducing collapse is correlated with improved performance.\n\nI selected to reject the paper because in its current form a general readership will struggle to understand it. An expert familiar with the details of the field (not just the general area) can probably disentangle the text, but otherwise it's too convoluted.\n\nAn example note on math:\nIt would be great to reduce confusion and improve the consistency of math in the paper. In some parts of the manuscript, like section 3 \"Notations\", the symbols are well defined and clear to understand. In other locations, like section 1, an undefined \\gamma is mentioned in the text, without definition. Is that \\gamma the same as in section 3? What is its definition in section 1? \n\nAssuming that \\gamma is as (loosely) defined in section 3, the reader can go back to section 1. This is an example of the convolution I mention above. For someone who has read the paper (and already read section 3), the \\gamma in section 1 makes sense. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed Channel Equilibrium (CE) to overcome the over-sparsity problem in CNNs using 'BN+ReLU'. Experiments on ImageNet and COCO show its effectiveness by introducing little computational complexity. However the reviewers pointed a number of problems in the writing and the clarity of the paper. Although the authors addressed all the se concerns in details and agreed to make revisions in the paper, it's better for the authors to submit the revised version to another opportunity.
Decision: Reject