ON NEURAL NETWORK GENERALIZATION VIA PROMOTING WITHIN-LAYER ACTIVATION DIVERSITY

1 INTRODUCTION . Neural networks are a powerful class of non-linear function approximators that have been successfully used to tackle a wide range of problems . They have enabled breakthroughs in many tasks , such as image classification ( Krizhevsky et al. , 2012 ) , speech recognition ( Hinton et al. , 2012a ) , and anomaly detection ( Golan & El-Yaniv , 2018 ) . Formally , the output of a neural network consisting of P layers can be defined as follows : f ( x ; W ) = φP ( W P ( φP−1 ( · · ·φ2 ( W 2φ1 ( W 1x ) ) ) ) , ( 1 ) where φi ( . ) is the element-wise activation function , e.g. , ReLU and Sigmoid , of the ith layer and W = { W 1 , . . . , W P } are the corresponding weights of the network . The parameters of f ( x ; W ) are optimized by minimizing the empirical loss : L̂ ( f ) = 1 N N∑ i=1 l ( f ( xi ; W ) , yi ) , ( 2 ) where l ( · ) is the loss function , and { xi , yi } Ni=1 are the training samples and their associated groundtruth labels . The loss is minimized using the gradient decent-based optimization coupled with backpropagation . However , neural networks are often over-parameterized , i.e. , have more parameters than data . As a result , they tend to overfit to the training samples and not generalize well on unseen examples ( Goodfellow et al. , 2016 ) . While research on Double descent ( Belkin et al. , 2019 ; Advani et al. , 2020 ; Nakkiran et al. , 2020 ) shows that over-parameterization does not necessarily lead to overfitting , avoiding overfitting has been extensively studied ( Neyshabur et al. , 2018 ; Nagarajan & Kolter , 2019 ; Poggio et al. , 2017 ) and various approaches and strategies have been proposed , such as data augmentation ( Goodfellow et al. , 2016 ) , regularization ( Kukačka et al. , 2017 ; Bietti et al. , 2019 ; Arora et al. , 2019 ) , and dropout ( Hinton et al. , 2012b ; Wang et al. , 2019 ; Lee et al. , 2019 ; Li et al. , 2016 ) , to close the gap between the empirical loss and the expected loss . Diversity of learners is widely known to be important in ensemble learning ( Li et al. , 2012 ; Yu et al. , 2011 ) and , particularly in deep learning context , diversity of information extracted by the network neurons has been recognized as a viable way to improve generalization ( Xie et al. , 2017a ; 2015b ) . In most cases , these efforts have focused on making the set of weights more diverse ( Yang et al . ; Malkin & Bilmes , 2009 ) . However , diversity of the activation has not received much attention . Inspired by the motivation of dropout to co-adapt neuron activation , Cogswell et al . ( 2016 ) proposed to regularize the activations of the network . An additional loss using cross-covariance of hidden activations was proposed , which encourages the neurons to learn diverse or non-redundant representations . The proposed approach , known as Decov , has empirically been proven to alleviate overfitting and to improve the generalization ability of neural network , yet a theoretical analysis to prove this has so far been lacking . In this work , we propose a novel approach to encourage activation diversity within the same layer . We propose complementing ’ between-layer ’ feedback with additional ’ within-layer ’ feedback to penalize similarities between neurons on the same layer . Thus , we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer . Moreover , inspired by Xie et al . ( 2015b ) , we provide a theoretical analysis showing that the within-layer activation diversity boosts the generalization performance of neural networks and reduces overfitting . Our contributions in this paper are as follows : • Methodologically , we propose a new approach to encourage the ’ diversification ’ of the layer-wise feature maps ’ outputs in neural networks . The proposed approach has three variants based on how the global diversity is defined . The main intuition is that by promoting the within-layer activation diversity , neurons within the same layer learn distinct patterns and , thus , increase the overall capacity of the model . • Theoretically , we analyse the effect the within-layer activation diversity on the generalization error bound of neural network . The analysis is presented in Section 3 . As shown in Theorems 3.7 , 3.8 , 3.9 , 3.10 , 3.11 , and 3.12 , we express the upper-bound of the estimation error as a function of the diversity factor . Thus , we provide theoretical evidence that the within-layer activation diversity can help reduce the generalization error . • Empirically , we show that the within-layer activation diversity boosts the performance of neural networks . Experimental results show that the proposed approach outperforms the competing methods . 2 WITHIN-LAYER ACTIVATION DIVERSITY . We propose a diversification strategy , where we encourage neurons within a layer to activate in a mutually different manner , i.e. , to capture different patterns . To this end , we propose an additional within-layer loss which penalizes the neurons that activate similarly . The loss function L̂ ( f ) defined in equation 2 is augmented as follows : L̂aug ( f ) = L̂ ( f ) + λ P∑ i=1 J i , ( 3 ) where J i expresses the overall pair-wise similarity of the neurons within the ith layer and λ is the penalty coefficient for the diversity loss . As in ( Cogswell et al. , 2016 ) , our proposed diversity loss can be applied to a single layer or multiple layers in a network . For simplicity , let us focus on a single layer . Let φin ( xj ) and φ i m ( xj ) be the outputs of the n th and mth neurons in the ith layer for the same input sample xj . The similarity snm between the the nth and mth neurons can be obtained as the average similarity measure of their outputs for N input samples . We use the radial basis function to express the similarity : snm = 1 N N∑ j=1 exp ( − γ||φin ( xj ) − φim ( xj ) ||2 ) , ( 4 ) where γ is a hyper-parameter . The similarity snm can be computed over the whole dataset or batchwise . Intuitively , if two neurons n andm have similar outputs for many samples , their corresponding similarity snm will be high . Otherwise , their similarity smn is small and they are considered “ diverse ” . Based on these pair-wise similarities , we propose three variants for the global diversity loss J i of the ith layer : • Direct : J i = ∑ n 6=m snm . In this variant , we model the global layer similarity directly as the sum of the pairwise similarities between the neurons . By minimizing their sum , we encourage the neurons to learn different representations . • Det : J i = −det ( S ) , where S is a similarity matrix defined as Snm = snm . This variant is inspired by the Determinantal Point Process ( DPP ) ( Kulesza & Taskar , 2010 ; 2012 ) , as the determinant of S measures the global diversity of the set . Geometrically , det ( S ) is the volume of the parallelepiped formed by vectors in the feature space associated with s. Vectors that result in a larger volume are considered to be more “ diverse ” . Thus , maximizing det ( · ) ( minimizing −det ( · ) ) encourages the diversity of the learned features . • Logdet : J i = −logdet ( S ) 1 . This variant has the same motivation as the second one . We use logdet instead of det as logdet is a convex function over the positive definite matrix space . It should be noted here that the first proposed variant , i.e. , direct , similar to Decov ( Cogswell et al. , 2016 ) , captures only the pairwise diversity between components and is unable to capture the higherorder “ diversity ” , whereas the other two variants consider the global similarity and are able to measure diversity in a more global manner . Our newly proposed loss function defined in equation 3 has two terms . The first term is the classic loss function . It computes the loss with respect to the ground-truth . In the back-propagation , this feedback is back-propagated from the last layer to the first layer of the network . Thus , it can be considered as a between-layer feedback , whereas the second term is computed within a layer . From equation 3 , we can see that our proposed approach can be interpreted as a regularization scheme . However , regularization in deep learning is usually applied directly on the parameters , i.e. , weights ( Goodfellow et al. , 2016 ; Kukačka et al. , 2017 ) , while in our approach , similar to ( Cogswell et al. , 2016 ) , an additional term is defined over the output maps of the layers . For a layer with C neurons and a batch size of N , the additional computational cost is O ( C2 ( N + 1 ) ) for direct variant and O ( C3 + C2N ) ) for both the determinant and log of the determinant variants . 3 GENERALIZATION ERROR ANALYSIS . In this section , we analyze how the proposed within-layer diversity regularizer affects the generalization error of a neural network . Generalization theory ( Zhang et al. , 2017 ; Kawaguchi et al. , 2017 ) focuses on the relation between the empirical loss , as defined in equation 2 , and the expected risk defined as follows : L ( f ) = E ( x , y ) ∼Q [ l ( f ( x ) , y ) ] , ( 5 ) where Q is the underlying distribution of the dataset . Let f∗ = argminf L ( f ) be the expected risk minimizer and f̂ = argminf L̂ ( f ) be the empirical risk minimizer . We are interested in the estimation error , i.e. , L ( f∗ ) −L ( f̂ ) , defined as the gap in the loss between both minimizers ( Barron , 1994 ) . The estimation error represents how well an algorithm can learn . It usually depends on the complexity of the hypothesis class and the number of training samples ( Barron , 1993 ; Zhai & Wang , 2018 ) . 1This is defined only if S is positive definite . It can be shown that in our case S is positive semi-definite . Thus , in practice we use a regularized version ( S + I ) to ensure the positive definiteness . Several techniques have been used to quantify the estimation error , such as PAC learning ( Hanneke , 2016 ; Arora et al. , 2018 ) , VC dimension ( Sontag , 1998 ; Harvey et al. , 2017 ; Bartlett et al. , 2019 ) , and the Rademacher complexity ( Xie et al. , 2015b ; Zhai & Wang , 2018 ; Tang et al. , 2020 ) . The Rademacher complexity has been widely used as it usually leads to a tighter generalization error bound ( Sokolic et al. , 2016 ; Neyshabur et al. , 2018 ; Golowich et al. , 2018 ) . The formal definition of the empirical Rademacher complexity is given as follows : Definition 3.1 . ( Bartlett & Mendelson , 2002 ) For a given dataset with N samples D = { xi , yi } Ni=1 generated by a distribution Q and for a model space F : X → R with a single dimensional output , the empirical Rademacher complexityRN ( F ) of the set F is defined as follows : RN ( F ) = Eσ [ sup f∈F 1 N N∑ i=1 σif ( xi ) ] , ( 6 ) where the Rademacher variables σ = { σ1 , · · · , σN } are independent uniform random variables in { −1 , 1 } . In this work , we analyse the estimation error bound of a neural network using the Rademacher complexity and we are interested in the effect of the within-layer diversity on the estimation error . In order to study this effect , inspired by ( Xie et al. , 2015b ) , we assume that with a high probability τ , the distance between the output of each pair of neurons , ( φn ( x ) −φm ( x ) ) 2 , is lower bounded by dmin for any input x . Note that this condition can be expressed in terms of the similarity s defined in equation 4 : snm ≤ e ( −γdmin ) = smin for any two distinct neurons with the probability τ . Our analysis starts with the following lemma : Lemma 3.2 . ( Xie et al. , 2015b ; Bartlett & Mendelson , 2002 ) With a probability of at least 1− δ L ( f̂ ) − L ( f∗ ) ≤ 4RN ( A ) +B √ 2 log ( 2/δ ) N ( 7 ) for B ≥ supx , y , f |l ( f ( x ) , y ) | , whereRN ( A ) is the Rademacher complexity of the loss set A . It upper-bounds the estimation error using the Rademacher complexity defined over the loss set and supx , y , f |l ( f ( x ) , y ) | . Our analysis continues by seeking a tighter upper bound of this error and showing how the within-layer diversity , expressed with dmin , affects this upper bound . We start by deriving such an upper-bound for a simple network with one hidden layer trained for a regression task and then we extend it for a general multi-layer network and for different losses .

Summary: This paper proposes adding regularization terms to encourage diversity of the layer outputs in order to improve the generalization performance. The proposed idea is an extension of Cogswell's work with different regularization terms. In addition, the authors performed detailed generalization analysis based on the Rademacher complexity. The appearance of the term related to the layer output diversity in the generalization bound provides theoretical support for the proposed idea.