Improving Neural Network Generalization via Promoting Within-Layer Diversity

1 INTRODUCTION . Neural networks are a powerful class of non-linear function approximators that have been successfully used to tackle a wide range of problems . They have enabled breakthroughs in many tasks , such as image classification ( Krizhevsky et al. , 2012 ) , speech recognition ( Hinton et al. , 2012a ) , and anomaly detection ( Golan & El-Yaniv , 2018 ) . Formally , the output of a neural network consisting of P layers can be defined as follows : f ( x ; W ) = φP ( W P ( φP−1 ( · · ·φ2 ( W 2φ1 ( W 1x ) ) ) ) , ( 1 ) where φi ( . ) is the element-wise activation function , e.g. , ReLU or Sigmoid , of the ith layer and W = { W 1 , . . . , W P } are the corresponding weights of the network . The parameters of f ( x ; W ) are optimized by minimizing the empirical loss : L̂ ( f ) = 1 N N∑ i=1 l ( f ( xi ; W ) , yi ) , ( 2 ) where l ( · ) is the loss function , and { xi , yi } Ni=1 are the training samples and their associated groundtruth labels . The loss is minimized using the gradient decent-based optimization coupled with backpropagation . However , neural networks are often over-parameterized , i.e. , have more parameters than data . As a result , they tend to overfit to the training samples and not generalize well on unseen examples ( Goodfellow et al. , 2016 ) . While research on double descent ( Belkin et al. , 2019 ; Advani et al. , 2020 ; Nakkiran et al. , 2020 ) shows that over-parameterization does not necessarily lead to overfitting , avoiding overfitting has been extensively studied ( Neyshabur et al. , 2018 ; Nagarajan & Kolter , 2019 ; Poggio et al. , 2017 ; Dziugaite & Roy , 2017 ; Foret et al. , 2020 ) and various approaches and strategies have been proposed , such as data augmentation ( Goodfellow et al. , 2016 ; Zhang et al. , 2018 ) , regularization ( Kukačka et al. , 2017 ; Bietti et al. , 2019 ; Arora et al. , 2019 ) , and Dropout ( Hinton et al. , 2012b ; Wang et al. , 2019 ; Lee et al. , 2019 ; Li et al. , 2016 ) , to close the gap between the empirical loss and the expected loss . Diversity of learners is widely known to be important in ensemble learning ( Li et al. , 2012 ; Yu et al. , 2011 ) and , particularly in deep learning context , diversity of information extracted by the network neurons has been recognized as a viable way to improve generalization ( Xie et al. , 2017a ; 2015b ) . In most cases , these efforts have focused on making the set of weights more diverse ( Yang et al . ; Malkin & Bilmes , 2009 ) . However , diversity of the activations has not received much attention . Here , we argue that due to the presence of non-linear activations , diverse weights do not guarantee diverse feature representation . Thus , we propose applying the diversity on top of feature mapping instead of the weights . To the best of our knowledge , only Cogswell et al . ( 2016 ) have considered diversity of the activations directly in the neural network context . They proposed an additional loss term using cross-covariance of hidden activations , which encourages the neurons to learn diverse or non-redundant representations . Their proposed approach , known as DeCov , was empirically proven to alleviate overfitting and to improve the generalization ability of neural network , yet a theoretical analysis to prove this has so far been lacking . Moreover , modeling diversity as the sum of the pairwise cross-covariance , it can capture only the pairwise diversity between components and is unable to capture the ” higher-order diversity ” . In this work , we start by theoretically showing that the within-layer activation diversity boosts the generalization ability of neural networks and reduces overfitting . Moreover , we propose a novel approach to encourage activation diversity within a layer . We propose complementing the ’ betweenlayer ’ feedback with additional ’ within-layer ’ feedback to penalize similarities between neurons on the same layer . Thus , we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer . We propose three variants for our approach that are based on different global diversity definitions . Our contributions in this paper are as follows : • Theoretically , we derive novel generalization bounds for neural networks depending on the within-layer activation diversity . As shown in Section 2 , we express the upper-bound of the estimation error as a function of the diversity factor . Thus , we provide theoretical evidence that the within-layer activation diversity helps reduce the generalization error . • Methodologically , we propose a new approach to encourage the ’ diversification ’ of the layers ’ output feature maps in neural networks . The proposed approach has three variants . The main intuition is that by promoting the within-layer activation diversity , neurons within a layer learn distinct patterns and , thus , increase the overall capacity of the model . • Empirically , we show that the proposed within-layer activation diversification boosts the performance of neural networks . Experimental results on several tasks show that the proposed approach outperforms competing methods . 2 GENERALIZATION ERROR ANALYSIS . In this section , we derive generalization bounds of neural networks depending on the within-layer activation diversity . Generalization theory ( Zhang et al. , 2017 ; Kawaguchi et al. , 2017 ) focuses on the relation between the empirical loss defined in equation 2 and the expected risk , for any f in the hypothesis class F , defined as follows : L ( f ) = E ( x , y ) ∼Q [ l ( f ( x ) , y ) ] , ( 3 ) where Q is the underlying distribution of the dataset . Let f∗ = argminf∈F L ( f ) be the expected risk minimizer and f̂ = argminf∈F L̂ ( f ) be the empirical risk minimizer . We are interested in the estimation error , i.e. , L ( f∗ ) −L ( f̂ ) , defined as the gap in the loss between both minimizers ( Barron , 1994 ) . The estimation error represents how well an algorithm can learn . It usually depends on the complexity of the hypothesis class and the number of training samples ( Barron , 1993 ; Zhai & Wang , 2018 ) . In this work , we are interested in the effect of the within-layer activation diversity on the estimation error . In order to study this effect , given a layer with M units , we assume that with a high proba- bility τ , the average pairwise distance between the outputs of the units , 12M ( M−1 ) ∑M i 6=j ( φn ( x ) − φm ( x ) ) 2 , is lower bounded by d2min for any input x . Intuitively , if two neurons n and m have similar outputs for many samples , the corresponding distance ( φn ( x ) − φm ( x ) ) 2 will be small . If the average distance is small , the lower bound dmin is also small and the units within this layer are considered redundant and “ not diverse ” . Otherwise , if the average distance between the different pairs is large , their corresponding dmin is large and they are considered “ diverse ” . By studying how the lower bound dmin affects the generalization of the model , we can theoretically understand how diversity affects the performance of neural networks . To this end , we derive generalization bounds for neural networks using dmin . Several techniques have been used to quantify the estimation error , such as PAC learning ( ShalevShwartz & Ben-David , 2014 ; Valiant , 1984 ) , VC dimension ( Sontag , 1998 ) , and the Rademacher complexity ( Shalev-Shwartz & Ben-David , 2014 ) . The Rademacher complexity has been widely used as it usually leads to a tighter generalization error bound ( Sokolic et al. , 2016 ; Neyshabur et al. , 2018 ; Golowich et al. , 2018 ) . The formal definition of the empirical Rademacher complexity is given as follows : Definition 1 . ( Shalev-Shwartz & Ben-David , 2014 ; Bartlett & Mendelson , 2002 ) For a given dataset with N samples D = { xi , yi } Ni=1 generated by a distribution Q and for a model space F : X → R with a single dimensional output , the empirical Rademacher complexity RN ( F ) of the set F is defined as follows : RN ( F ) = Eσ [ sup f∈F 1 N N∑ i=1 σif ( xi ) ] , ( 4 ) where the Rademacher variables σ = { σ1 , · · · , σN } are independent uniform random variables in { −1 , 1 } . In this work , we rely on the Rademacher complexity to study diversity . Our analysis starts with the following lemma : Lemma 1 . ( Bartlett & Mendelson , 2002 ) With a probability of at least 1− δ L ( f̂ ) − L ( f∗ ) ≤ 4RN ( A ) +B √ 2 log ( 2/δ ) N ( 5 ) where B ≥ supx , y , f |l ( f ( x ) , y ) | andRN ( A ) is the Rademacher complexity of the loss set A . It upper-bounds the estimation error using the Rademacher complexity defined over the loss set and supx , y , f |l ( f ( x ) , y ) | . Our analysis continues by seeking a tighter upper bound of this error and showing how the within-layer diversity , expressed with dmin , affects the bound . In this paper , we derive such an upper-bound for a simple network with one hidden layer trained for a regression task . We show how to extend it for classification , general multi-layer network and for different losses in the Appendix . The proofs are provided as supplementary material . 2.1 SINGLE HIDDEN-LAYER NETWORK . Here , we consider a simple neural network with one hidden-layer with M neurons and onedimensional output trained for a regression task . The full characterization of the setup can be summarized in the following assumptions : Assumptions 1 . • The input vector x ∈ RD satisfies ||x||2 ≤ C1 and the output scalar y ∈ R satisfies |y| ≤ C2 . The activation function of the hidden layer , φ ( · ) , is a positive Lφ-Lipschitz continuous function . • The weight matrix W = [ w1 , w2 , · · · , wM ] ∈ RD×M connecting the input to the hidden layer satisfies ||wm||2 ≤ C3 . The weight vector v ∈ RM connecting the hidden-layer to the output neuron satisfies ||v||∞ ≤ C4 . • The hypothesis class is F = { f |f ( x ) = ∑M m=1 vmφm ( x ) = ∑M m=1 vmφ ( w T mx ) } . • Loss function set is A = { l|l ( f ( x ) , y ) = 12 |f ( x ) − y| 2 } . • With a probability τ , 12M ( M−1 ) ∑M n 6=m ( φn ( x ) − φm ( x ) ) 2 ≥ d2min . The main idea of our proof is to find a diversity-dependant bound for both terms in Lemma 1 . To this end , we derive a novel bound for loss hypothesis class A depending on dmin and use it for both terms . The loss depends on the hypothesis class F . Thus , we start by deriving an upper bound for this class : Lemma 2 . Under Assumptions 1 , with a probability at least τ , we have sup x , f∈F |f ( x ) | ≤ √ J , ( 6 ) where J = C24 ( MC25 +M ( M − 1 ) ( C25 − d2min ) ) and C5 = LφC1C3 + φ ( 0 ) , Note that in Lemma 2 , we have expressed the upper-bound of supx , f |f ( x ) | in terms of dmin . Using this bound , we can now find an upper-bound for supx , f , y |l ( f ( x ) , y ) | in the following lemma : Lemma 3 . Under Assumptions 1 , with a probability at least τ , we have sup x , y , f |l ( f ( x ) , y ) | ≤ 1 2 ( √ J + C2 ) 2 . ( 7 ) Our main goal is to analyze the estimation error bound of the neural network and to see how its upper-bound is linked to the diversity , expressed by dmin , of the different neurons . The main result of the paper is presented in Theorem 1 . Theorem 1 . Under Assumptions 1 , there exist a constant A , such that with probability at least τ ( 1− δ ) , we have L ( f̂ ) − L ( f∗ ) ≤ ( √ J + C2 ) A√ N + 1 2 ( √ J + C2 ) 2 √ 2 log ( 2/δ ) N ( 8 ) where J = C24 ( MC25 +M ( M − 1 ) ( C25 − d2min ) ) , and C5 = LφC1C3 + φ ( 0 ) . Theorem 1 provides an upper-bound for the generalization gap . We note that it is a decreasing function of dmin . Thus , this suggests that higher dmin , i.e. , more diverse activations , yields a lower estimation error bound . In other words , by promoting the within-layer diversity , we can reduce the generalization error of neural networks . It should be noted that in the last item of our assumptions , we considered a relaxed variant of the following assumption ’ H * : There exists a lower bound to the distance , valid for any input x ’ . H * is impractical especially if the intermediate layer has ReLu activations . Therefore , we considered a relaxed variant of this assumption by introducing the probability τ . This makes the theoretical findings useful in practice .

Summary: The authors prove an upper bound on the estimation error that depends on pairwise distances between hidden unit values per layer. The bound itself is quite likely very loose in standard deep learning settings due to the dependence on various norms of the weights. Loosely inspired by the bound, the authors then propose an optimization algorithm with a particular regularizer that encourages within-layer activation value diversity. The empirical results suggest that the proposed algorithm boosts the performance in most regimes studied (CIFAR 10, CIFAR100 trained on ResNets, Dense nets). The boost is minimal for standard datasets, but it increases when the labels are randomized. Compared to standard empirical risk minimization, a small boost in test accuracy on transfer learning tasks is also reported.