Improving Neural Network Generalization via Promoting Within-Layer Diversity

1 INTRODUCTION . Neural networks are a powerful class of non-linear function approximators that have been successfully used to tackle a wide range of problems . They have enabled breakthroughs in many tasks , such as image classification ( Krizhevsky et al. , 2012 ) , speech recognition ( Hinton et al. , 2012a ) , and anomaly detection ( Golan & El-Yaniv , 2018 ) . Formally , the output of a neural network consisting of P layers can be defined as follows : f ( x ; W ) = φP ( W P ( φP−1 ( · · ·φ2 ( W 2φ1 ( W 1x ) ) ) ) , ( 1 ) where φi ( . ) is the element-wise activation function , e.g. , ReLU or Sigmoid , of the ith layer and W = { W 1 , . . . , W P } are the corresponding weights of the network . The parameters of f ( x ; W ) are optimized by minimizing the empirical loss : L̂ ( f ) = 1 N N∑ i=1 l ( f ( xi ; W ) , yi ) , ( 2 ) where l ( · ) is the loss function , and { xi , yi } Ni=1 are the training samples and their associated groundtruth labels . The loss is minimized using the gradient decent-based optimization coupled with backpropagation . However , neural networks are often over-parameterized , i.e. , have more parameters than data . As a result , they tend to overfit to the training samples and not generalize well on unseen examples ( Goodfellow et al. , 2016 ) . While research on double descent ( Belkin et al. , 2019 ; Advani et al. , 2020 ; Nakkiran et al. , 2020 ) shows that over-parameterization does not necessarily lead to overfitting , avoiding overfitting has been extensively studied ( Neyshabur et al. , 2018 ; Nagarajan & Kolter , 2019 ; Poggio et al. , 2017 ; Dziugaite & Roy , 2017 ; Foret et al. , 2020 ) and various approaches and strategies have been proposed , such as data augmentation ( Goodfellow et al. , 2016 ; Zhang et al. , 2018 ) , regularization ( Kukačka et al. , 2017 ; Bietti et al. , 2019 ; Arora et al. , 2019 ) , and Dropout ( Hinton et al. , 2012b ; Wang et al. , 2019 ; Lee et al. , 2019 ; Li et al. , 2016 ) , to close the gap between the empirical loss and the expected loss . Diversity of learners is widely known to be important in ensemble learning ( Li et al. , 2012 ; Yu et al. , 2011 ) and , particularly in deep learning context , diversity of information extracted by the network neurons has been recognized as a viable way to improve generalization ( Xie et al. , 2017a ; 2015b ) . In most cases , these efforts have focused on making the set of weights more diverse ( Yang et al . ; Malkin & Bilmes , 2009 ) . However , diversity of the activations has not received much attention . Here , we argue that due to the presence of non-linear activations , diverse weights do not guarantee diverse feature representation . Thus , we propose applying the diversity on top of feature mapping instead of the weights . To the best of our knowledge , only Cogswell et al . ( 2016 ) have considered diversity of the activations directly in the neural network context . They proposed an additional loss term using cross-covariance of hidden activations , which encourages the neurons to learn diverse or non-redundant representations . Their proposed approach , known as DeCov , was empirically proven to alleviate overfitting and to improve the generalization ability of neural network , yet a theoretical analysis to prove this has so far been lacking . Moreover , modeling diversity as the sum of the pairwise cross-covariance , it can capture only the pairwise diversity between components and is unable to capture the ” higher-order diversity ” . In this work , we start by theoretically showing that the within-layer activation diversity boosts the generalization ability of neural networks and reduces overfitting . Moreover , we propose a novel approach to encourage activation diversity within a layer . We propose complementing the ’ betweenlayer ’ feedback with additional ’ within-layer ’ feedback to penalize similarities between neurons on the same layer . Thus , we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer . We propose three variants for our approach that are based on different global diversity definitions . Our contributions in this paper are as follows : • Theoretically , we derive novel generalization bounds for neural networks depending on the within-layer activation diversity . As shown in Section 2 , we express the upper-bound of the estimation error as a function of the diversity factor . Thus , we provide theoretical evidence that the within-layer activation diversity helps reduce the generalization error . • Methodologically , we propose a new approach to encourage the ’ diversification ’ of the layers ’ output feature maps in neural networks . The proposed approach has three variants . The main intuition is that by promoting the within-layer activation diversity , neurons within a layer learn distinct patterns and , thus , increase the overall capacity of the model . • Empirically , we show that the proposed within-layer activation diversification boosts the performance of neural networks . Experimental results on several tasks show that the proposed approach outperforms competing methods . 2 GENERALIZATION ERROR ANALYSIS . In this section , we derive generalization bounds of neural networks depending on the within-layer activation diversity . Generalization theory ( Zhang et al. , 2017 ; Kawaguchi et al. , 2017 ) focuses on the relation between the empirical loss defined in equation 2 and the expected risk , for any f in the hypothesis class F , defined as follows : L ( f ) = E ( x , y ) ∼Q [ l ( f ( x ) , y ) ] , ( 3 ) where Q is the underlying distribution of the dataset . Let f∗ = argminf∈F L ( f ) be the expected risk minimizer and f̂ = argminf∈F L̂ ( f ) be the empirical risk minimizer . We are interested in the estimation error , i.e. , L ( f∗ ) −L ( f̂ ) , defined as the gap in the loss between both minimizers ( Barron , 1994 ) . The estimation error represents how well an algorithm can learn . It usually depends on the complexity of the hypothesis class and the number of training samples ( Barron , 1993 ; Zhai & Wang , 2018 ) . In this work , we are interested in the effect of the within-layer activation diversity on the estimation error . In order to study this effect , given a layer with M units , we assume that with a high proba- bility τ , the average pairwise distance between the outputs of the units , 12M ( M−1 ) ∑M i 6=j ( φn ( x ) − φm ( x ) ) 2 , is lower bounded by d2min for any input x . Intuitively , if two neurons n and m have similar outputs for many samples , the corresponding distance ( φn ( x ) − φm ( x ) ) 2 will be small . If the average distance is small , the lower bound dmin is also small and the units within this layer are considered redundant and “ not diverse ” . Otherwise , if the average distance between the different pairs is large , their corresponding dmin is large and they are considered “ diverse ” . By studying how the lower bound dmin affects the generalization of the model , we can theoretically understand how diversity affects the performance of neural networks . To this end , we derive generalization bounds for neural networks using dmin . Several techniques have been used to quantify the estimation error , such as PAC learning ( ShalevShwartz & Ben-David , 2014 ; Valiant , 1984 ) , VC dimension ( Sontag , 1998 ) , and the Rademacher complexity ( Shalev-Shwartz & Ben-David , 2014 ) . The Rademacher complexity has been widely used as it usually leads to a tighter generalization error bound ( Sokolic et al. , 2016 ; Neyshabur et al. , 2018 ; Golowich et al. , 2018 ) . The formal definition of the empirical Rademacher complexity is given as follows : Definition 1 . ( Shalev-Shwartz & Ben-David , 2014 ; Bartlett & Mendelson , 2002 ) For a given dataset with N samples D = { xi , yi } Ni=1 generated by a distribution Q and for a model space F : X → R with a single dimensional output , the empirical Rademacher complexity RN ( F ) of the set F is defined as follows : RN ( F ) = Eσ [ sup f∈F 1 N N∑ i=1 σif ( xi ) ] , ( 4 ) where the Rademacher variables σ = { σ1 , · · · , σN } are independent uniform random variables in { −1 , 1 } . In this work , we rely on the Rademacher complexity to study diversity . Our analysis starts with the following lemma : Lemma 1 . ( Bartlett & Mendelson , 2002 ) With a probability of at least 1− δ L ( f̂ ) − L ( f∗ ) ≤ 4RN ( A ) +B √ 2 log ( 2/δ ) N ( 5 ) where B ≥ supx , y , f |l ( f ( x ) , y ) | andRN ( A ) is the Rademacher complexity of the loss set A . It upper-bounds the estimation error using the Rademacher complexity defined over the loss set and supx , y , f |l ( f ( x ) , y ) | . Our analysis continues by seeking a tighter upper bound of this error and showing how the within-layer diversity , expressed with dmin , affects the bound . In this paper , we derive such an upper-bound for a simple network with one hidden layer trained for a regression task . We show how to extend it for classification , general multi-layer network and for different losses in the Appendix . The proofs are provided as supplementary material . 2.1 SINGLE HIDDEN-LAYER NETWORK . Here , we consider a simple neural network with one hidden-layer with M neurons and onedimensional output trained for a regression task . The full characterization of the setup can be summarized in the following assumptions : Assumptions 1 . • The input vector x ∈ RD satisfies ||x||2 ≤ C1 and the output scalar y ∈ R satisfies |y| ≤ C2 . The activation function of the hidden layer , φ ( · ) , is a positive Lφ-Lipschitz continuous function . • The weight matrix W = [ w1 , w2 , · · · , wM ] ∈ RD×M connecting the input to the hidden layer satisfies ||wm||2 ≤ C3 . The weight vector v ∈ RM connecting the hidden-layer to the output neuron satisfies ||v||∞ ≤ C4 . • The hypothesis class is F = { f |f ( x ) = ∑M m=1 vmφm ( x ) = ∑M m=1 vmφ ( w T mx ) } . • Loss function set is A = { l|l ( f ( x ) , y ) = 12 |f ( x ) − y| 2 } . • With a probability τ , 12M ( M−1 ) ∑M n 6=m ( φn ( x ) − φm ( x ) ) 2 ≥ d2min . The main idea of our proof is to find a diversity-dependant bound for both terms in Lemma 1 . To this end , we derive a novel bound for loss hypothesis class A depending on dmin and use it for both terms . The loss depends on the hypothesis class F . Thus , we start by deriving an upper bound for this class : Lemma 2 . Under Assumptions 1 , with a probability at least τ , we have sup x , f∈F |f ( x ) | ≤ √ J , ( 6 ) where J = C24 ( MC25 +M ( M − 1 ) ( C25 − d2min ) ) and C5 = LφC1C3 + φ ( 0 ) , Note that in Lemma 2 , we have expressed the upper-bound of supx , f |f ( x ) | in terms of dmin . Using this bound , we can now find an upper-bound for supx , f , y |l ( f ( x ) , y ) | in the following lemma : Lemma 3 . Under Assumptions 1 , with a probability at least τ , we have sup x , y , f |l ( f ( x ) , y ) | ≤ 1 2 ( √ J + C2 ) 2 . ( 7 ) Our main goal is to analyze the estimation error bound of the neural network and to see how its upper-bound is linked to the diversity , expressed by dmin , of the different neurons . The main result of the paper is presented in Theorem 1 . Theorem 1 . Under Assumptions 1 , there exist a constant A , such that with probability at least τ ( 1− δ ) , we have L ( f̂ ) − L ( f∗ ) ≤ ( √ J + C2 ) A√ N + 1 2 ( √ J + C2 ) 2 √ 2 log ( 2/δ ) N ( 8 ) where J = C24 ( MC25 +M ( M − 1 ) ( C25 − d2min ) ) , and C5 = LφC1C3 + φ ( 0 ) . Theorem 1 provides an upper-bound for the generalization gap . We note that it is a decreasing function of dmin . Thus , this suggests that higher dmin , i.e. , more diverse activations , yields a lower estimation error bound . In other words , by promoting the within-layer diversity , we can reduce the generalization error of neural networks . It should be noted that in the last item of our assumptions , we considered a relaxed variant of the following assumption ’ H * : There exists a lower bound to the distance , valid for any input x ’ . H * is impractical especially if the intermediate layer has ReLu activations . Therefore , we considered a relaxed variant of this assumption by introducing the probability τ . This makes the theoretical findings useful in practice .

Summary: In this paper, the authors propose a regularization technique encourage the "activation diversity". Specifically, they design a within-layer loss that add penalty to the similar neurons with the same activation pattern. They also showed that encouraging the within-layer diversity can be used to control the Rademacher complexity of model.