Channel Equilibrium Networks

1 INTRODUCTION . Normalization is an important technique for a wide range of tasks such as image classification ( Ioffe & Szegedy , 2015 ) , object detection ( He et al. , 2017a ; Wu & He , 2018 ) , and image generation ( Miyato et al. , 2018 ) . In recent years , a lot of work improved normalization methods , such as batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) , layer normalization ( LN ) ( Ba et al. , 2016 ) and switchable normalization ( SN ) ( Luo et al. , 2018 ) . These methods are often used together with the ReLU-like activation functions such as ReLU ( Glorot et al. , 2011 ; Nair & Hinton , 2010 ) , ELU ( Clevert et al. , 2015 ) and Leaky ReLU ( LReLU ) ( Maas et al. , 2013 ) , making the “ Norm+ReLU-like ” module become one of the most widely-used building blocks of modern CNNs . This work investigates and alleviates the inhibited channels emerged in the “ Norm+ReLU-like ” building block , which consists of a normalization layer and a ReLU-like activation function given by yncij = g ( x̃ncij ) , x̃ncij = γcx̄ncij + βc , ( 1 ) where subscript n , c , i , and j denote indices of a sample , a channel , height and width of a feature channel respectively . For instance , yncij indicates output value of the location ( i , j ) in the c-th channel of the n-th sample . And x̃ncij and x̄ncij represent normalized channel features and standardized channel features respectively . γ and β are two vectors of parameters , where each element re-scales and re-shifts the standardized features for each channel c. Moreover , g ( · ) denotes a ReLU-like activation function . As pointed out in ( Morcos et al. , 2018 ) , a CNN that generalizes well would have its channels contributed equally in its feed-forward computations . We term this desired property “ channel equalization ” . However , a recent study disclosed a critical problem of the “ Norm+ReLU-like ” builiding block , known as “ channel collapse ” , where certain channels always produce small output values given any input . For example , the lottery hypothesis ( Frankle & Carbin , 2018 ) claimed that one over-parameterized CNN always contains unimportant channels that contribute little to the network ’ s prediction . This paper shows that these unimportant channels are inhibited channels , which usually associated with small values of γ in Eqn . ( 1 ) or small values of the channel features ( In this paper , inhibited channel is defined as channel whose all feature values are less than 1e− 2 ) . For example , as shown in Fig.1 ( a , b ) , the inhibited channels exist in many “ Norm+ReLU-like ” basic block such as ‘ LN+ReLU ’ , ‘ BN+ELU ’ and ‘ BN+ReLU ’ . This phenomenon has motivated many investigations to directly remove these inhibited channels , such as network slimming ( Liu et al. , 2017 ; Yu et al. , 2018 ) and channel pruning ( He et al. , 2017b ) . However , disequilibrium among channels caused by the inhibited channels would do harm to generalization ability of the network and simply removing the inhibited channels that are inactive during training does not help improve learning capacity . Instead of removing the inhibited channels , we present an alternative perspective by proposing a novel building block , named Channel Equilibrium ( CE ) , to recover and equalize the inhibited channels by encouraging channels to contribute equally in the feature representation learning process , thus enhancing the representation and generalization of CNNs . To this end , a key observation from Eqn . ( 1 ) is that the dependency ( covariance ) matrix of all the feature channels after normalization is scaled by γγT . Let this covariance matrix be Σ . We will see that by applying a decorrelation operator , i.e . Σ− 1 2 , we can not only effectively eliminate the correlations between computations of channel features and the magnitude of γ , but also equalize the channels ’ magnitudes ( Barlow et al. , 1961 ; Bengio & Bergstra , 2009 ) . This operator enables all the channels to play an equal role in the computations of a CNN , improving its generalization ability . For example , as shown in Fig.1 , the VGGNet ( Simonyan & Zisserman , 2014 ) equipped with CE is able to effectively prevent inhibited channels and achieves channel equalization in various of “ Norm+RelU-like ” blocks , consistently improving their recognition performance . The main contributions of this work are three-fold . First , we introduce an efficient and effective building block , Channel Equilibrium ( CE ) , which encourages channel equalization and enhances representation learning of CNNs . Second , CE blocks can be stacked after common normalizers and plugged into various advanced architectures , consistently improving their performance by a large margin . For example , CE can be integrated into ResNet and MobileNet V2 , forming a series of CE-Networks by replacing the ordinary ‘ BN-ReLU ’ block by ‘ BN-CE-ReLU ’ block , which merely introduces subtle extra computational complexity . As a result , the CE-ResNet50 and CE-MobileNet V2 outperform their counterparts by 1.7 % and 2.1 % top-1 accuracy with nearly the same FLOPs . We also show that combining CE with synchronization across GPUs increases the AP metric on the MS-COCO dataset to 42.0 , surpassing its counterpart by 3.4 . Third , the learned equalized feature representation of CENet can be better transferred to many other tasks like object detection and segmentation . 2 RELATED WORK . Channel equalization . Channel equalization indicates that channels in a layer of CNNs contribute equally to network ’ s computation . The success of the two most commonly used regularization techniques , i.e . BN ( Ioffe & Szegedy , 2015 ) and Dropout ( Srivastava et al. , 2014 ) , is attributed to channel or neuron equalization . For example , Mianjy et al . ( 2018 ) showed that Dropout makes the norm of incoming/outgoing weight vectors of all the hidden nodes equal , indicating a kind of equalization between neurons . Moreover , Morcos et al . ( 2018 ) pointed out that BN implicitly discourages single direction reliance , indicating that equalizing different channels is able to enhance the generalization of learned feature representation . Note that the squeeze-and-excitation ( SE ) network ( Hu et al. , 2018 ) is a pioneer work that explicitly modeling interdependencies among channels by investigating network design . However , SE selectively emphasizes informative channels and suppresses less useful ones . In contrast , the proposed CE block encourages all the channels to play an equal role in network ’ s computation , which , as will be shown , can be linked with Nash Equilibrium . More related work on sparsity in ReLU and normalization methods are provided in Sec.A of Appendix . 3 METHOD . In this section , we first review the normalization method and then introduce the proposed Channel Equilibrium ( CE ) block . CE contains two complementary branches , i.e . batch decorrelation ( BD ) and adaptive instance inverse ( AII ) . We show how BD and AII benefit from each other through parameter γ and how CE is linked with Nash Equilibrium . Notations . For CNNs , we use x ∈ RN×C×H×W to represent the feature in a layer , specifically , xncij denotes a pixel ( i , j ) in the c-th channel of the n-th sample . Sometimes , we ignore the subscript ‘ n ’ and denote it as xcij for clarity of notation . xnij ∈ RC is obtained by stacking elements in all channels of xncij into a column vector . Diag ( · ) returns a matrix with the given diagonal and zero offdiagonal entries , and diag ( · ) extracts the diagonal of the given matrix . γ , β ∈ RC are normalization parameters . 3.1 OVERVIEW OF NORMALIZATION . Normalization is usually employed after convolution layers to stabilize the training of CNNs . Given a hidden feature x ∈ RN×C×H×W , a normalizer first standardizes it to x̄ , and then maps it to x̃ by an affine transformation , as written by x̃ncij = γcx̄ncij + βc , x̄ncij = ( xncij − µs ) /σs ( 2 ) where s ∈ Ω = { IN , BN , LN , · · · } indicates a normalizer and µs , σs are the mean and standard deviation of the given normalizer . For simplicity , in original formulation is omitted ( Ioffe & Szegedy , 2015 ) . From Eqn . ( 2 ) , we claim that normalization would lead to an unequal feature representation in a channel basis , based on the fact that common-used normalizers like IN and BN are performed channel-wisely . It is known that importance of channels are quantified by the learned parameters γ with its magnitude since channel features are scaled by γ in a channel basis . Previous work ( Frankle & Carbin , 2018 ; Mehta et al. , 2019 ) revealed that some inhibited channels emerge when the associated γ or feature map gets small ( Mehta et al. , 2019 ; Lu et al. , 2019 ) . Obviously , the inhibited channels would cause disequilibrium among channels , resulting in limited generalization ability . To alleviate such a disequilibrium , Channel Equilibrium ( CE ) block is proposed in the following section . 3.2 CHANNEL EQUILIBRIUM ( CE ) BLOCK . A Channel Equilibrium ( CE ) block is a computational unit which aims to equalize feature representation capacity among channels . To this end , decorrelation method is adopted . Different from previous methods ( Huang et al. , 2018 ; 2019 ) decorrelating features by a single batch estimated covariance matrix Σ , the proposed method brings in an adaptive instance variance , Sn , on the diagonal of the covariance matrix Σ , considering that channel dependency is specific to each input ( Hu et al. , 2018 ) , as formulated in the following , Dn = λΣ + ( 1− λ ) Diag ( Sn ) , Sn = F ( σ2 ( x̃n ) ) , ( 3 ) where the subscript n is the sample index , λ ∈ ( 0 , 1 ) is a trainable ratio used to switch between batch and instance statistics , F : RC → RC is a transformation conditioned on the current input x̃ and σ2 ( x̃n ) computes instance variance of x̃n within each channel . On the issue of channel disequilibrium , CE block works by decorrelating feature maps after normalization using D− 1 2 n . Further , the Jensen inequality for matrix functions ( Pečarić , 1996 ) can be employed to obtain a relaxed decorrelation operator D− 1 2 n : D − 12 n = [ λΣ + ( 1− λ ) Diag ( Sn ) ] − 1 2 λΣ− 12 + ( 1− λ ) [ Diag ( Sn ) ] − 1 2 , ( 4 ) where A B indicates B − A is semi-definite positive . We introduce this relaxation for the following two reasons . ( 1 ) Computation reduction . It allows less computational cost for each training step since the relaxed form only needs to calculate the inverse of square root Σ− 1 2 once , and the other branch Diag ( Sn ) − 1 2 is easy to compute . ( 2 ) Inference acceleration . Σ− 1 2 is a moving-average statistic in inference which can be absorbed into previous layer , therefore , enabling fast inference . Note that Eqn . ( 4 ) transforms the combination of covariance and adaptive instance variance into the combination of their inverse square roots . In the following , we refer Σ− 1 2 in Eqn . ( 4 ) as batch decorrelation ( BD ) and refer [ Diag ( Sn ) ] − 12 as adaptive instance inverse ( AII ) . The former decorrelates channels by a batch covariance , while the latter adjusts the extend of inverse for each channel and instance in an adaptive manner . Integrating both of them yields the forward representation of CE block : pnij = D − 12 n ( Diag ( γ ) x̄nij + β ) ( 5 ) where pnij ∈ RC denotes the output of CE , as illustrated in Fig.2 ( b ) . Note that CE is performed after the normalization layer , BN is taken as an example to introduce these two branches in the following sections .

Summary: The authors point out that CNNs can develop collapsed channels that limit their capacity. They propose to remedy this with batch decorrelation (BD), which focuses on ensuring that channels play an equal role in the feature map and are less likely to collapse. The claim is supported with experiments on CIFAR10, ImageNet and COCO.