Year: 2021

Abstract:
Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are backpropagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional ’between-layer’ feedback with additional ’within-layer’ feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer’s overall diversity. By penalizing similarities and promoting diversity, we encourage each unit within the layer to learn a distinctive representation and, thus, to enrich the data representation learned and to increase the total capacity of the model. We derive novel generalization bounds for neural networks depending on the within-layer activation diversity and prove that increasing the diversity of hidden activations reduces the estimation error. In addition to the theoretical guarantees, we present an extensive empirical study confirming that the proposed approach enhances the performance of state-of-the-art neural network models and decreases the generalization gap in multiple tasks.

heading: 1 INTRODUCTION
text: Neural networks are a powerful class of non-linear function approximators that have been successfully used to tackle a wide range of problems. They have enabled breakthroughs in many tasks, such as image classification (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012a), and anomaly detection (Golan & El-Yaniv, 2018). Formally, the output of a neural network consisting of P layers can be defined as follows:
f(x;W) = φP (W P (φP−1(· · ·φ2(W 2φ1(W 1x)))), (1)
where φi(.) is the element-wise activation function, e.g., ReLU or Sigmoid, of the ith layer and W = {W 1, . . . ,W P } are the corresponding weights of the network. The parameters of f(x;W) are optimized by minimizing the empirical loss:
L̂(f) = 1
N N∑ i=1 l ( f(xi;W), yi ) , (2)
where l(·) is the loss function, and {xi, yi}Ni=1 are the training samples and their associated groundtruth labels. The loss is minimized using the gradient decent-based optimization coupled with backpropagation.
However, neural networks are often over-parameterized, i.e., have more parameters than data. As a result, they tend to overfit to the training samples and not generalize well on unseen examples (Goodfellow et al., 2016). While research on double descent (Belkin et al., 2019; Advani et al., 2020; Nakkiran et al., 2020) shows that over-parameterization does not necessarily lead to overfitting, avoiding overfitting has been extensively studied (Neyshabur et al., 2018; Nagarajan & Kolter, 2019; Poggio et al., 2017; Dziugaite & Roy, 2017; Foret et al., 2020) and various approaches and strategies have been proposed, such as data augmentation (Goodfellow et al., 2016; Zhang et al., 2018), regularization (Kukačka et al., 2017; Bietti et al., 2019; Arora et al., 2019), and Dropout
(Hinton et al., 2012b; Wang et al., 2019; Lee et al., 2019; Li et al., 2016), to close the gap between the empirical loss and the expected loss.
Diversity of learners is widely known to be important in ensemble learning (Li et al., 2012; Yu et al., 2011) and, particularly in deep learning context, diversity of information extracted by the network neurons has been recognized as a viable way to improve generalization (Xie et al., 2017a; 2015b). In most cases, these efforts have focused on making the set of weights more diverse (Yang et al.; Malkin & Bilmes, 2009). However, diversity of the activations has not received much attention. Here, we argue that due to the presence of non-linear activations, diverse weights do not guarantee diverse feature representation. Thus, we propose applying the diversity on top of feature mapping instead of the weights.
To the best of our knowledge, only Cogswell et al. (2016) have considered diversity of the activations directly in the neural network context. They proposed an additional loss term using cross-covariance of hidden activations, which encourages the neurons to learn diverse or non-redundant representations. Their proposed approach, known as DeCov, was empirically proven to alleviate overfitting and to improve the generalization ability of neural network, yet a theoretical analysis to prove this has so far been lacking. Moreover, modeling diversity as the sum of the pairwise cross-covariance, it can capture only the pairwise diversity between components and is unable to capture the ”higher-order diversity”.
In this work, we start by theoretically showing that the within-layer activation diversity boosts the generalization ability of neural networks and reduces overfitting. Moreover, we propose a novel approach to encourage activation diversity within a layer. We propose complementing the ’betweenlayer’ feedback with additional ’within-layer’ feedback to penalize similarities between neurons on the same layer. Thus, we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer. We propose three variants for our approach that are based on different global diversity definitions.
Our contributions in this paper are as follows:
• Theoretically, we derive novel generalization bounds for neural networks depending on the within-layer activation diversity. As shown in Section 2, we express the upper-bound of the estimation error as a function of the diversity factor. Thus, we provide theoretical evidence that the within-layer activation diversity helps reduce the generalization error.
• Methodologically, we propose a new approach to encourage the ’diversification’ of the layers’ output feature maps in neural networks. The proposed approach has three variants. The main intuition is that by promoting the within-layer activation diversity, neurons within a layer learn distinct patterns and, thus, increase the overall capacity of the model.
• Empirically, we show that the proposed within-layer activation diversification boosts the performance of neural networks. Experimental results on several tasks show that the proposed approach outperforms competing methods.

heading: 2 GENERALIZATION ERROR ANALYSIS
text: In this section, we derive generalization bounds of neural networks depending on the within-layer activation diversity. Generalization theory (Zhang et al., 2017; Kawaguchi et al., 2017) focuses on the relation between the empirical loss defined in equation 2 and the expected risk, for any f in the hypothesis class F , defined as follows:
L(f) = E(x,y)∼Q[l(f(x), y)], (3) where Q is the underlying distribution of the dataset. Let f∗ = argminf∈F L(f) be the expected risk minimizer and f̂ = argminf∈F L̂(f) be the empirical risk minimizer. We are interested in the estimation error, i.e., L(f∗)−L(f̂), defined as the gap in the loss between both minimizers (Barron, 1994). The estimation error represents how well an algorithm can learn. It usually depends on the complexity of the hypothesis class and the number of training samples (Barron, 1993; Zhai & Wang, 2018).
In this work, we are interested in the effect of the within-layer activation diversity on the estimation error. In order to study this effect, given a layer with M units, we assume that with a high proba-
bility τ , the average pairwise distance between the outputs of the units, 12M(M−1) ∑M i 6=j(φn(x) − φm(x)) 2, is lower bounded by d2min for any input x. Intuitively, if two neurons n and m have similar outputs for many samples, the corresponding distance (φn(x) − φm(x))2 will be small. If the average distance is small, the lower bound dmin is also small and the units within this layer are considered redundant and “not diverse”. Otherwise, if the average distance between the different pairs is large, their corresponding dmin is large and they are considered “diverse”. By studying how the lower bound dmin affects the generalization of the model, we can theoretically understand how diversity affects the performance of neural networks. To this end, we derive generalization bounds for neural networks using dmin.
Several techniques have been used to quantify the estimation error, such as PAC learning (ShalevShwartz & Ben-David, 2014; Valiant, 1984), VC dimension (Sontag, 1998), and the Rademacher complexity (Shalev-Shwartz & Ben-David, 2014). The Rademacher complexity has been widely used as it usually leads to a tighter generalization error bound (Sokolic et al., 2016; Neyshabur et al., 2018; Golowich et al., 2018). The formal definition of the empirical Rademacher complexity is given as follows:
Definition 1. (Shalev-Shwartz & Ben-David, 2014; Bartlett & Mendelson, 2002) For a given dataset with N samples D = {xi, yi}Ni=1 generated by a distribution Q and for a model space F : X → R with a single dimensional output, the empirical Rademacher complexity RN (F) of the set F is defined as follows:
RN (F) = Eσ [ sup f∈F 1 N N∑ i=1 σif(xi) ] , (4)
where the Rademacher variables σ = {σ1, · · · , σN} are independent uniform random variables in {−1, 1}.
In this work, we rely on the Rademacher complexity to study diversity. Our analysis starts with the following lemma:
Lemma 1. (Bartlett & Mendelson, 2002) With a probability of at least 1− δ L(f̂)− L(f∗) ≤ 4RN (A) +B √ 2 log(2/δ)
N (5)
where B ≥ supx,y,f |l(f(x), y)| andRN (A) is the Rademacher complexity of the loss set A.
It upper-bounds the estimation error using the Rademacher complexity defined over the loss set and supx,y,f |l(f(x), y)|. Our analysis continues by seeking a tighter upper bound of this error and showing how the within-layer diversity, expressed with dmin, affects the bound.
In this paper, we derive such an upper-bound for a simple network with one hidden layer trained for a regression task. We show how to extend it for classification, general multi-layer network and for different losses in the Appendix. The proofs are provided as supplementary material.

heading: 2.1 SINGLE HIDDEN-LAYER NETWORK
text: Here, we consider a simple neural network with one hidden-layer with M neurons and onedimensional output trained for a regression task. The full characterization of the setup can be summarized in the following assumptions:
Assumptions 1.
• The input vector x ∈ RD satisfies ||x||2 ≤ C1 and the output scalar y ∈ R satisfies |y| ≤ C2. The activation function of the hidden layer, φ(·), is a positive Lφ-Lipschitz continuous function.
• The weight matrix W = [w1,w2, · · · ,wM ] ∈ RD×M connecting the input to the hidden layer satisfies ||wm||2 ≤ C3. The weight vector v ∈ RM connecting the hidden-layer to the output neuron satisfies ||v||∞ ≤ C4.
• The hypothesis class is F = {f |f(x) = ∑M m=1 vmφm(x) = ∑M m=1 vmφ(w T mx)}.
• Loss function set is A = {l|l(f(x), y) = 12 |f(x)− y| 2}.
• With a probability τ , 12M(M−1) ∑M n 6=m(φn(x)− φm(x))2 ≥ d2min.
The main idea of our proof is to find a diversity-dependant bound for both terms in Lemma 1. To this end, we derive a novel bound for loss hypothesis class A depending on dmin and use it for both terms. The loss depends on the hypothesis class F . Thus, we start by deriving an upper bound for this class: Lemma 2. Under Assumptions 1, with a probability at least τ , we have
sup x,f∈F
|f(x)| ≤ √ J , (6)
where J = C24 ( MC25 +M(M − 1)(C25 − d2min) ) and C5 = LφC1C3 + φ(0),
Note that in Lemma 2, we have expressed the upper-bound of supx,f |f(x)| in terms of dmin. Using this bound, we can now find an upper-bound for supx,f,y |l(f(x), y)| in the following lemma: Lemma 3. Under Assumptions 1, with a probability at least τ , we have
sup x,y,f |l(f(x), y)| ≤ 1 2 ( √ J + C2)2. (7)
Our main goal is to analyze the estimation error bound of the neural network and to see how its upper-bound is linked to the diversity, expressed by dmin, of the different neurons. The main result of the paper is presented in Theorem 1. Theorem 1. Under Assumptions 1, there exist a constant A, such that with probability at least τ(1− δ), we have
L(f̂)− L(f∗) ≤ (√ J + C2 ) A√ N + 1 2 ( √ J + C2)2 √ 2 log(2/δ) N (8)
where J = C24 ( MC25 +M(M − 1)(C25 − d2min) ) , and C5 = LφC1C3 + φ(0).
Theorem 1 provides an upper-bound for the generalization gap. We note that it is a decreasing function of dmin. Thus, this suggests that higher dmin, i.e., more diverse activations, yields a lower estimation error bound. In other words, by promoting the within-layer diversity, we can reduce the generalization error of neural networks. It should be noted that in the last item of our assumptions, we considered a relaxed variant of the following assumption ’H*: There exists a lower bound to the distance, valid for any input x’. H* is impractical especially if the intermediate layer has ReLu activations. Therefore, we considered a relaxed variant of this assumption by introducing the probability τ . This makes the theoretical findings useful in practice.

heading: 3 WITHIN-LAYER ACTIVATION DIVERSITY
text: As shown in the previous section, promoting diversity of activations within a layer can lead to tighter generalization bound and can theoretically decrease the gap between the empirical and the true risks. In this section, we propose a novel diversification strategy, where we encourage neurons within a layer to activate in a mutually different manner, i.e., to capture different patterns. To this end, we propose an additional within-layer loss which penalizes the neurons that activate similarly. The loss function L̂(f) defined in equation 2 is augmented as follows:
L̂aug(f) = L̂(f) + λJ i, (9)
where J i expresses the overall similarity of the neurons within the ith layer and λ is the penalty coefficient for the diversity loss. As in (Cogswell et al., 2016), our proposed diversity loss can be applied to a single layer or multiple layers in a network. For simplicity, let us focus on a single layer.
Let φin(xj) and φ i m(xj) be the outputs of the n th and mth neurons in the ith layer for the same input sample xj . The similarity snm between the the nth and mth neurons can be obtained as the
average similarity measure of their outputs for N input samples. We use the radial basis function to express the similarity:
snm = 1
N N∑ j=1 exp ( − γ||φin(xj)− φim(xj)||2 ) , (10)
where γ is a hyper-parameter. The similarity snm can be computed over the whole dataset or batchwise. Intuitively, if two neurons n andm have similar outputs for many samples, their corresponding similarity snm will be high. Otherwise, their similarity smn is small and they are considered “diverse”. Based on these pairwise similarities, we propose three variants for the overall similarity J i in equation 9:
• Direct: J i = ∑ n 6=m snm. In this variant, we model the global layer similarity directly
as the sum of the pairwise similarities between the neurons. By minimizing their sum, we encourage the neurons to learn different representations.
• Det: J i = −det(S), where S is a similarity matrix defined as Snm = snm. This variant is inspired by the Determinantal Point Process (DPP) (Kulesza & Taskar, 2010; 2012), as the determinant of S measures the global diversity of the set. Geometrically, det(S) is the volume of the parallelepiped formed by vectors in the feature space associated with s. Vectors that result in a larger volume are considered to be more “diverse”. Thus, maximizing det(·) (minimizing −det(·)) encourages the diversity of the learned features.
• Logdet: J i = −logdet(S)1. This variant has the same motivation as the second one. We use logdet instead of det as logdet is a convex function over the positive definite matrix space.
It should be noted here that the first proposed variant, i.e., direct, similar to Decov (Cogswell et al., 2016), captures only the pairwise diversity between components and is unable to capture the higherorder “diversity”, whereas the other two variants consider the global similarity and are able to measure diversity in a more global manner.
Our newly proposed loss function defined in equation 9 has two terms. The first term is the classic loss function. It computes the loss with respect to the ground-truth. In the back-propagation, this feedback is back-propagated from the last layer to the first layer of the network. Thus, it can be considered as a between-layer feedback, whereas the second term is computed within a layer. From equation 9, we can see that our proposed approach can be interpreted as a regularization scheme. However, regularization in deep learning is usually applied directly on the parameters, i.e., weights (Goodfellow et al., 2016; Kukačka et al., 2017), while in our approach, similar to (Cogswell et al., 2016), an additional term is defined over the output maps of the layers. For a layer with C neurons and a batch size of N , the additional computational cost is O(C2(N + 1)) for direct variant and O(C3 + C2N)) for both the determinant and log of the determinant variants.
In connection to the bounds derived in Section 2 and to how diversity is theoretically defined in the last item in the Assumptions 1, we note that our regularizer relies on the pairwise RBF distance instead of the standard L2 distance. We note that minimizing the average RBF distance, i.e., our direct variant, is equivalent to maximizing the averageL2 distance (exp(-x) is a decreasing function). Moreover, we note that empirically the RBF-based distance is less sensitive to noise (Savas & Dovis, 2019; Haykin, 2010).

heading: 4 RELATED WORK
text: Diversity promoting strategies have been widely used in ensemble learning (Li et al., 2012; Yu et al., 2011), sampling (Derezinski et al., 2019; Bıyık et al., 2019; Gartrell et al., 2019), energybased models (Zhao et al., 2017; Laakom et al., 2021), ranking (Yang et al.; Gan et al., 2020), pruning by reducing redundancy (Kondo & Yamauchi, 2014; He et al., 2019; Singh et al., 2020; Lee et al., 2020), and semi-supervised learning (Zbontar et al., 2021). In the deep learning context, various approaches have used diversity as a direct regularizer on top of the weight parameters. Here,
1This is defined only if S is positive definite. It can be shown that in our case S is positive semi-definite. Thus, in practice we use a regularized version (S + I) to ensure the positive definiteness.
we present a brief overview of these regularizers. Based on the way diversity is defined, we can group these approaches into two categories. The first group considers the regularizers that are based on the pairwise dissimilarity of components, i.e., the overall set of weights are diverse if every pair of weights are dissimilar. Given the weight vectors {wm}Mm=1, Yu et al. (2011) define the regularizer as ∑ mn(1 − θmn), where θmn represents the cosine similarity between wm and wn.
Bao et al. (2013) proposed an incoherence score defined as− log (
1 M(M−1) ∑ mn β|θmn| 1 β ) , where
β is a positive hyperparameter. Xie et al. (2015a; 2016) used mean(θmn) − var(θmn) to regularize Boltzmann machines. They theoretically analyzed its effect on the generalization error bounds in (Xie et al., 2015b) and extended it to kernel space in (Xie et al., 2017a). The second group of regularizers considers a more globalist view of diversity. For example, in (Malkin & Bilmes, 2009; 2008; Xie et al., 2017b), a weight regularization based on the determinant of the weights’ covariance is proposed based on determinantal point process in (Kulesza & Taskar, 2012; Kwok & Adams, 2012).
Unlike the aforementioned methods which promote diversity on the weight level and similar to our method, (Cogswell et al., 2016) proposed to enforce dissimilarity on the feature map outputs, i.e., on the activations. To this end, they proposed an additional loss based on the pairwise covariance of the activation outputs. Their additional loss, LDecov is defined as the squared sum of the non-diagonal elements of the global covariance matrixC of the activations: LDecov = 12 (||C|| 2 F − ||diag(C)||22), where ||.||F is the Frobenius norm. Their approach, Decov, yielded superior empirical performance; however, it lacks theoretical proof. In this paper, we closed this gap and we showed theoretically how employing a diversity strategy on the network activations can indeed decrease the estimation error bound and improve the generalization of the model. Besides, we proposed variants of our approach which consider a global view of diversity.

heading: 5 EXPERIMENTAL RESULTS

heading: 5.1 IMAGE CLASSIFICATION
text: We start by evaluating our proposed diversity approach on two image datasets: CIFAR10 and CIFAR100 (Krizhevsky et al., 2009). We use our approach on three state-of-the-art CNNs: ResNext29-08-16: we consider the standard ResNext Model (Xie et al., 2017c) with a 29-layer architecture, a cardinality of 8, and a width of 16. DenseNet-12: we use DenseNet (Huang et al., 2017) with the 40- layer architecture and a growth rate of 12. ResNet50: we consider the standard ResNet model (He et al., 2016) with 50 layers. We compare against the standard networks2 as well as networks trained with DeCov diversity strategy (Cogswell et al., 2016). For each approach the hyperparameters are selected based on the validation set. The full experimental setup is presented in the Appendix. We report the average performance over three random seeds.
Table 1 reports the average top-1 errors of the different approaches with the three basis networks. We note that, compared to the standard approach, employing a diversity strategy generally boosts the results for all the three models and that our approach consistency outperforms both competing
2For the standard approach, the only difference is not using an additional diversity loss. The remaining regularizers, data augmentation, weight decay etc, are all applied as specified per-experiment.
methods (standard and DeCov) in all the experiments. For example with ResNet50, the three variants of our proposed approach significantly reduce the test errors over both datasets: 0.36% − 0.54% improvement on CIFAR10 and 1.86%− 1.94% on CIFAR100. ImageNet: To further demonstrate the effectiveness of our approach and its ability to reduce the generalization gap in neural networks, we conduct additional image classification experiments on the ImageNet-2012 classification dataset (Russakovsky et al., 2015) using the ResNet50 (He et al., 2016) and Wide-ResNet50 (Zagoruyko & Komodakis, 2016) models. The diversity term is applied on the last intermediate layer, i.e., the global average pooling layer for both DeCov and our method. For the hyperparameters, we use λ = 0.001 and γ = 10. The full experimental setup is presented in the Appendix. Table 2 reports the test errors of the different diversity strategies. To study the effect of diversity on the generalization gap, we also report the final training errors and the generalization gap, i.e., training accuracy - test accuracy. As it can be seen, diversity (our approach and DeCov) reduces the test error of the model and yields a better performance. We note that, in accordance with our theoretical findings in Section 2, using diversity indeed reduces overfitting and decreases the empirical generalisation gap of neural networks. In fact, our logdet variant reduces the empirical generalization gap of the model by 1.8% compared to the standard approach. We note that our approach has a small additional time cost. For example for ResNet50, our direct, det and logdet variants take only 0.29%, 0.39%, and 0.49% extra training time, respectively.
Sensitivity analysis: To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: γ, which is the RBF parameter used to measure the pairwise similarity between two units, and λ, which controls the contribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure 1 and in supplementary material. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparameters values. Moreover, we note that global modeling of diversity, i.e., det and logdet variants, yield tighter generalization gaps compared to the non-global direct approach. It is worth noting that Figure 1 shows that there is a trade-off between the generalization gap and the final error. Emphasizing
diversity and using a high weight for the diversity term significantly decreases the generalization gap, but this damages the performance of the model compared to the standard approach. For lower values of λ, the model is able to significantly outperform the standard approach on both the test error and the generalization gap.
Connection to Theory: In Section 2, we provided theoretical bounds for the generalization errors of neural networks, which are inversely proportional to the diversity term, dmin, the lower bound of 12M(M−1) ∑M i 6=j(φn(x) − φm(x))2. To show that the proposed regularizer indeed improves diversity, we track the empirical average of the aforementioned variable, i.e., 1
2M(M−1) ∑M n6=m ∑ i(φn(xi) − φm(xi))2 during the training. The results for both the standard approach and the logdet variant of our approach are reported in Figure 2. As it can be seen, our regularizer yields in higher diversity which reduces overfitting and leads to better generalization.
Compatibility with Dropout: Dropout (Srivastava et al., 2014) is a popular regularization technique that, similarly to our approach, is applied on top of the layer output and has been known to improve generalization and reduce overfitting. Here, we evaluate the compatibility of our approach with Dropout. We add a Dropout regularizer with 20% rate on top the last intermediate representation layer of both ResNet50 and Wide-ResNet50. The results are reported in Table 3. We note that adding a diversity regularizer alongside the Dropout consistently yields lower error rates compared to only Dropout except for the det variant on Wide-ResNet50. For Wide-ResNet50, the combination of our logdet variant and Dropout leads to 0.53%, 0.25%, and 0.38% improvement compared to standard approach, Dropout only, and our logdet variant only, respectively.
MLP-based models: Beyond CNN models, we also evaluate the performance of our diversity strategy on modern attention-free, multi-layer perceptron (MLP) based models for image classification (Tolstikhin et al., 2021; Liu et al., 2021; Lee-Thorp et al., 2021). Such models are known to exhibit high overfitting and require regularization. We evaluate how diversity affects the accuracy of such models on CIFAR10. In particular, we conduct a simple experiment using two models: MLP-Mixer (Tolstikhin et al., 2021), gMLP (Liu et al., 2021) with four blocks each. The full description of the experimental setup is presented in the Appendix. The results in Table 4 show that employing a diversity strategy can indeed improve the performance of these models thanks to its ability to learn rich and robust representation of the input.
Transfer learning: Beyond standard classification, the proposed approach can be useful, e.g., in transfer learning, where the main goal is to ’transfer’ previously learned representation to solve new
tasks. Thus, learning a rich and diverse representation is beneficial and can lead to better transferability. To demonstrate this, we conduct another experiment, where we use ImageNet-pretrained ResNet50 models with the different diversity approaches and we finetune them to CIFAR10 and CIFAR100. The results are reported in Table 5. As it can be seen, employing a diversity strategy helps in the transfer learning context and leads consistently to lower error rates. For example, the log variant of our approach leads to 0.94% and 1.27% gains on CIFAR10 and CIFAR100, respectively.

heading: 5.2 LEARNING IN THE PRESENCE OF LABEL NOISE
text: To further demonstrate the usefulness of promoting diversity, we test the robustness of our approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and not generalize well to the test set. Enforcing diversity can lead to better and richer representation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFAR10 and CIFAR100 using ResNet50. The results are reported in Table 6. As it can be seen, in the presence of noise, the gap between the standard approach and diversity (Decov and ours) increases. For example, our logdet variant boosts the results by 1.71% and 3.59% on CIFAR10 and CIFAR100 with 40% noise, respectively.

heading: 6 CONCLUSIONS
text: In this paper, we proposed a new approach to encourage ‘diversification’ of the layer-wise feature map outputs in neural networks. The main motivation is that by promoting within-layer activation diversity, units within the same layer learn to capture mutually distinct patterns. We proposed an additional loss term that can be added on top of any fully-connected layer. This term complements the traditional ‘between-layer’ feedback with an additional ‘within-layer’ feedback encouraging diversity of the activations. We theoretically proved that the proposed approach decreases the estimation error bound and, thus, improves the generalization ability of neural networks. This analysis was further supported by extensive experimental results showing that such a strategy can indeed improve the performance of different state-of-the-art networks across different datasets and different tasks, i.e., image classification, transfer learning, and label noise. We are confident that these results will spark further research in diversity-based approaches to improve the generalization ability of neural networks.

heading: A APPENDIX
text: The full characterization of the setup can be summarized in the following assumptions: Assumptions 2.
• The activation function of the hidden layer, φ(.), is a positive Lφ-Lipschitz continuous function.
• The input vector x ∈ RD satisfies ||x||2 ≤ C1.
• The output scalar y ∈ R satisfies |y| ≤ C2.
• The weight matrix W = [w1,w2, · · · ,wM ] ∈ RD×M connecting the input to the hidden layer satisfies ||wm||2 ≤ C3.
• The weight vector v ∈ RM connecting the hidden-layer to the output neuron satisfies ||v||∞ ≤ C4.
• The hypothesis class is F = {f |f(x) = ∑M m=1 vmφm(x) = ∑M m=1 vmφ(w T mx)}.
• Loss function set is A = {l|l(f(x), y) = 12 |f(x)− y| 2}.
• With a probability τ , 12M(M−1) ∑M n 6=m(φn(x)− φm(x))2 ≥ d2min.
We recall the following two lemmas related to the estimation error and the Rademacher complexity: Lemma 4. (Bartlett & Mendelson, 2002) For F ∈ RX , assume that g : R −→ R is a Lg-Lipschitz continuous function and A = {g ◦ f : f ∈ F}. Then we have
RN (A) ≤ LgRN (F). (11) Lemma 5. (Xie et al., 2015b) Under Assumptions 1, the Rademacher complexity RN (F) of the hypothesis class F = {f |f(x) = ∑M m=1 vmφm(x) = ∑M m=1 vmφ(w T mx)} can be upper-bounded as follows:
RN (F) ≤ 2LφC134M√ N + C4|φ(0)|M√ N , (12)
where C134 = C1C3C4 and φ(0) is the output of the activation function at the origin.

heading: A.1 PROOF OF LEMMA 2
text: Lemma 2 Under Assumptions 1, with a probability at least τ , we have
sup x,f |f(x)| ≤
√ J , (13)
where J = C24 ( MC25 +M(M − 1)(C25 − d2min) ) and C5 = LφC1C3 + φ(0).
Proof.
f2(x) = ( M∑ m=1 vmφm(x) )2 ≤ ( M∑ m=1 ||v||∞φm(x) )2 = ||v||2∞ ( M∑ m=1 φm(x) )2 ≤ C24 ( M∑ m=1 φm(x) )2
= C24 (∑ m,n φm(x)φn(x) ) = C24 ∑ m φm(x) 2 + ∑ m 6=n φn(x)φm(x)  . (14) We have supw,x φm(x) ≤ sup(Lφ|wTx| + φ(0)) because φ is Lφ-Lipschitz. Thus, ||φ||∞ ≤ LφC1C3 + φ(0) = C5. For the first term in equation 14, we have ∑ m φm(x)
2 < M(LφC1C3 + φ(0))
2 = MC25 . The second term, using the identity φm(x)φn(x) = 1 2 ( φm(x) 2 + φn(x) 2 − (φm(x)− φn(x))2 ) , can be rewritten as
∑ m 6=n φm(x)φn(x) = 1 2 ∑ m 6=n φm(x) 2 + φn(x) 2 − ( φm(x)− φn(x) )2 . (15)
In addition, we have with a probability τ , 1 2 ∑ m 6=n(φm(x)− φn(x))
2 ≥M(M − 1)d2min for m 6= n. Thus, we have with a probability at least τ :∑
m 6=n
φm(x)φn(x) ≤ 1
2 ∑ m 6=n (2C25 )−M(M − 1)d2min =M(M − 1)(C25 − d2min). (16)
By putting everything back to equation 14, we have with a probability τ , f2(x) ≤ C24 ( MC25 +M(M − 1)(C25 − d2min) ) = J . (17)
Thus, with a probability τ ,
sup x,f |f(x)| ≤ √ sup x,f f(x)2 ≤ √ J . (18)

heading: A.2 PROOF OF LEMMA 3
text: Lemma 3 Under Assumptions 1, with a probability at least τ , we have
sup x,y,f |l(f(x), y)| ≤ 1 2 ( √ J + C2)2 (19)
Proof. We have supx,y,f |f(x) − y| ≤ supx,y,f (|f(x)| + |y|) = √ J + C2. Thus supx,y,f |l(f(x), y)| ≤ 12 ( √ J + C2)2.

heading: A.3 PROOF OF THEOREM 1
text: Theorem 1 Under Assumptions 1, there exist a constant A, such that with probability at least τ(1− δ), we have
L(f̂)− L(f∗) ≤ (√ J + C2 ) A√ N + 1 2 ( √ J + C2)2 √ 2 log(2/δ) N (20)
where J = C24 ( MC25 +M(M − 1)(C25 − d2min) ) , and C5 = LφC1C3 + φ(0).
Proof. Given that l(·) is K-Lipschitz with a constant K = supx,y,f |f(x) − y| ≤ √ J + C2, and using Lemma 4, we can show that RN (A) ≤ KRN (F) ≤ ( √ J + C2)RN (F). For RN (F), we use the bound found in Lemma 5. Using Lemmas 1 and 3, we have
L(f̂) − L(f∗) ≤ 4 (√ J + C2 )( 2LφC134 + C4|φ(0)| ) M√ N + 1 2 ( √ J + C2)2 √ 2 log(2/δ)
N (21)
where C134 = C1C3C4, J = C24 ( MC25 +M(M − 1)(C25 − d2min) ) , and C5 = LφC1C3 + φ(0).
Thus, setting A = 4 ( 2LφC134 + C4|φ(0)| ) M completes the proof.

heading: B BINARY CLASSIFICATION
text: We now extend our analysis of the effect of the within-layer diversity on the generalization error in the case of a binary classification task, i.e., y ∈ {−1, 1}. The extensions of Theorem 1 in the case of a hinge loss and a logistic loss are presented in Theorems 2 and 3, respectively. Theorem 2. Using the hinge loss, there exist a constant A, such that with probability at least τ(1− δ), we have
L(f̂)− L(f∗) ≤ A/ √ N + (1 + √ J ) √ 2 log(2/δ)
N (22) where J = C24 (MC25 +M(M − 1)(C25 − d2min) ) and C5 = LφC1C3 + φ(0).
Theorem 3. Using the logistic loss l(f(x), y) = log(1 + e−yf(x)), there exist a constant A such that, with probability at least τ(1− δ), we have
L(f̂)− L(f∗) ≤ A (1 + e √ −J ) √ N + log(1 + e √ J )
√ 2 log(2/δ)
N (23)
where J = C24 (MC25 +M(M − 1)(C25 − d2min) ) and C5 = LφC1C3 + φ(0).
As we can see, also for the binary classification task, the error bounds of the estimation error for the hinge and logistic losses are decreasing with respect to dmin. Thus, employing a diversity strategy can improve the generalization also for the binary classification task.

heading: B.1 PROOFS OF THEOREMS 2 AND 3
text: Similar to the proofs of Lemmas 7 and 8 in Xie et al. (2015b), we can show the following two lemmas: Lemma 6. Using the hinge loss, we have with probability at least τ(1− δ)
L(f̂) − L(f∗) ≤ 4 ( 2LφC134 + C4|φ(0)| ) M√ N + (1 + √ J ) √ 2 log(2/δ) N (24)
where C134 = C1C3C4, J = C24 (MC25 +M(M − 1)(C25 − d2min) ) , and C5 = LφC1C3 + φ(0).
Lemma 7. Using the logistic loss l(f(x), y) = log(1 + e−yf(x)), we have with probability at least τ(1− δ)
L(f̂) − L(f∗) ≤ 4 1 + e √ −J
( 2LφC134 + C4|φ(0)| ) M√ N + log(1 + e √ J ) √ 2 log(2/δ) N (25)
where C134 = C1C3C4, J = C24 (MC25 +M(M − 1)(C25 − d2min) ) , and C5 = LφC1C3 + φ(0).
Taking A = 4 ( 2LφC134 + C4|φ(0)| ) M in Lemma 6 and Lemma 7 completes the proofs.

heading: C MULTI-LAYER NETWORKS
text: Here, we extend our result for networks with P (> 1) hidden layers. We assume that the pairwise distances between the activations within layer p are lower-bounded by d(p)min with a probability τ
(p). In this case, the hypothesis class can be defined recursively. In addition, we replace the fourth assumption in Assumptions 1 with: ||W (p)||∞ ≤ C(p)3 for every W (p), i.e., the weight matrix of the p-th layer. In this case, the main theorem is extended as follows:
Theorem 4. There exist a constant A such that, with probability of at least ∏P−1 p=0 (τ
(p))(1− δ), we have
L(f̂)− L(f∗) ≤ ( √ J P + C2)
A√ N + 1 2
(√ J P + C2 )2√2 log(2/δ) N
(26)
where J P is defined recursively using the following identities: J 0 = C03C1 and J (p) = M (p)Cp2 ( Mp2(LφJ p−1 + φ(0))2 −M(M − 1)d(p)min 2 ) ) , for p = 1, . . . , P .
In Theorem 4, we see that J P is decreasing with respect to d(p)min. Thus, we see that maximizing the within-layer diversity, we can reduce the estimation error of a multi-layer neural network.

heading: C.1 PROOF OF THEOREM 4
text: Theorem 4 There exist a constant A such that, with probability of at least ∏P−1 p=0 (τ
(p))(1− δ), we have
L(f̂)− L(f∗) ≤ ( √ J P + C2)
A√ N + 1 2
(√ J P + C2 )2√2 log(2/δ) N , (27)
where J P is defined recursively using the following identities: J 0 = C03C1 and J (p) = M (p)Cp2 ( Mp2(LφJ p−1 + φ(0))2 −M(M − 1)d(p)min 2 ) ) , for p = 1, . . . , P .
Proof. Lemma 5 in Xie et al. (2015b) provides an upper-bound for the hypothesis class. We denote by v(p) the outputs of the pth hidden layer before applying the activation function:
v0 = [w0 T 1 x, ....,w 0T M0x] (28)
v(p) = [ Mp−1∑ j=1 w (p) j,1φ(v p−1 j ), ...., Mp−1∑ j=1 w (p) j,M(p) φ(vp−1j )] (29)
v(p) = [w (p) 1
T φ(p), ...,w (p)
M(p)
T φ(p)], (30)
where φ(p) = [φ(vp−11 ), · · · , φ(v p−1 Mp−1)]. We have
||v(p)||22 = M(p)∑ m=1 (w(p)m T φ(p))2 (31)
and w(p)m T φ(p) ≤ C(p)3 ∑ n φ (p) n . Thus,
||v(p)||22 ≤ M(p)∑ m=1 (C (p) 3 ∑ n φ(p)n ) 2 =M (p)Cp3 2 ( ∑ n φ(p)n ) 2 =M (p)Cp3 2 ∑ mn φ(p)m φ (p) n . (32)
We use the same decomposition trick of φ(p)m φ (p) n as in the proof of Lemma 2. We need to bound supx φ (p):
sup x φ(p) < sup(Lφ|vp−1|+ φ(0)) < Lφ||vp−1||22 + φ(0). (33)
Thus, we have ||v(p)||22 ≤M (p)Cp 2(M2(Lφ||vp−1||22 + φ(0))2 −M(M − 1)d2min)) = J P . (34)
We found a recursive bound for ||v(p)||22, we note that for p = 0, we have ||v0||22 ≤ ||W 0||∞C1 ≤ C03C1 = J 0. Thus,
sup x,fP∈FP |f(x)| = sup x,fP∈FP
|vP | ≤ √ J P . (35)
By replacing the variables in Lemma 1, we have
L(f̂)−L(f∗) ≤ 4( √ J P+C2)
( (2Lφ)
PC1C 0 3√
N
P−1∏ p=0 √ M (p)C (p) 3 + |φ(0)|√ N P−1∑ p=0 (2Lφ) P−1−p P−1∏ j=p √ M jCj3
)
+ 1
2
(√ J P + C2 )2√2 log(2/δ) N
Taking A = 4 ∗ ((2Lφ)PC1C03 ∏P−1 p=0 √ M (p)C (p) 3 + |φ(0)| ∑P−1 p=0 (2Lφ) P−1−p∏P−1 j=p √ M jCj3) completes the proof.

heading: D MULTIPLE OUTPUTS
text: Finally, we consider the case of a neural network with a multi-dimensional output, i.e., y ∈ RD. In this case, we can extend Theorem 1 with the following two theorems: Theorem 5. For a multivariate regression trained with the squared error, there exist a constant A such that, with probability at least τ(1− δ), we have
L(f̂)− L(f∗) ≤ ( √ J + C2)
A√ N + D 2 ( √ J + C2)2
√ 2 log(2/δ)
N (36)
where J = C24 (MC25 +M(M − 1)(C25 − d2min) ) and C5 = LφC1C3 + φ(0).
Theorem 6. For a multi-class classification task using the cross-entropy loss, there exist a constant A such that, with probability at least τ(1− δ), we have
L(f̂)− L(f∗) ≤ A (D − 1 + e−2 √ J ) √ N
+ log ( 1 + (D − 1)e2 √ J )√2 log(2/δ)
N (37)
where J = C24 (MC25 +M(M − 1)(C25 − d2min) ) and C5 = LφC1C3 + φ(0).
Theorems 5 and 6 extend our result for the multi-dimensional regression and classification tasks, respectively. Both bounds are inversely proportional to the diversity factor dmin. We note that for the classification task, the upper-bound is exponentially decreasing with respect to dmin. This shows that increasing diversity within the layer yields a tighter generalization gap and, thus, theoretically guarantees a stronger generalization performance.

heading: D.1 PROOF OF THEOREM 5
text: Proof. The squared loss 12 ||f(x) − y|| 2 2 can be decomposed into D terms 1 2 (f(x)k − yk) 2. Using Theorem 1, we can derive the bound for each term and thus, we have:
L(f̂)−L(f∗) ≤ 4D( √ J +C2) ( 2LφC134+C4|φ(0)| ) M√ N + D 2 ( √ J +C2)2 √ 2 log(2/δ) N , (38)
where C134 = C1C3C4, J = C24 (MC25 +M(M − 1)(C25 − d2min) ) , and C5 = LφC1C3 + φ(0).
Taking A = 4D ( 2LφC134 + C4|φ(0)| ) M completes the proof.

heading: D.2 PROOF OF THEOREM 6
text: Proof. Using Lemma 9 in Xie et al. (2015b), we have supf,x,y l = log ( 1+ (D− 1)e2 √ J ) and l is
D−1 D−1+e−2 √ J -Lipschitz. Thus, using the decomposition property of the Rademacher complexity, we have Rn(A) ≤
4D(D − 1) D − 1 + e−2 √ J (2LφC134 + C4|φ(0)|) M√ N . (39)
Taking A = 4D(D − 1) (2LφC134 + C4|φ(0)|)M completes the proof.

heading: E EXPERIMENTAL RESULTS
text: Here, we report the experimental setup of the different experiments done in the paper along with a more detailed analysis of the results.
E.1 CIFAR10 & CIFAR100
We start by evaluating our proposed diversity approach on two image datasets: CIFAR10 and CIFAR100 (Krizhevsky et al., 2009). They contain 60,000 (50,000 train/10,000 test) 32 × 32 images grouped into 10 and 100 distinct categories, respectively. We split the original training set (50,000) into two sets: we use the first 40,000 images as the main training set and the last 10,000 as a validation set for hyperparameters optimization. We use our approach on three state-of-the-art CNNs: ResNext-29-08-16: we consider the standard ResNext Model (Xie et al., 2017c) with a 29-layer architecture, a cardinality of 8, and a width of 16. DenseNet-12: we use DenseNet (Huang et al., 2017) with the 40-layer architecture and a growth rate of 12. ResNet50: we consider the standard ResNet model (He et al., 2016) with 50 layers. We compare against the standard networks as well networks trained with DeCov diversity strategy (Cogswell et al., 2016).
All the models are trained using stochastic gradient descent (SGD) with a momentum of 0.9, weight decay of 0.0001, and a batch size of 128 for 200 epochs. The initial learning rate is set to 0.1 and is then decreased by a factor of 5 after 60, 120, and 160 epochs, respectively. We also adopt a standard data augmentation scheme that is widely used for these two datasets
(He et al., 2016; Huang et al., 2017). For all models, the additional diversity term is applied on top the last intermediate layer. For the hyperparameters: The loss weight is chosen from {0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01} for both our approach and Decov and γ in the radial basis function is chosen from {0.01, 0.1.1, 10, 50, 100}. For each approach, the model with the best validation performance is used in the test phase. We report the average performance over three random seeds.
E.2 IMAGENET AND SENSITIVITY ANALYSIS
To further demonstrate the effectiveness of our approach and its ability to reduce the generalization gap in neural networks, we conduct additional image classification experiments on the ImageNet2012 classification dataset (Russakovsky et al., 2015) using the ResNet50 model (He et al., 2016). The diversity term is applied on the last intermediate layer, i.e., the global average pooling layer for both DeCov and our method. For the hyperparameters, we use γ = 10 and λ = 0.0001 for all the different approaches. We use the standard augmentation practice for this dataset as in (Zhang et al., 2018; Huang et al., 2017; Cogswell et al., 2016). All the models are trained with a batch size of 256 for 100 epoch using SGD with Nesterov Momentum of 0.9. The learning rate is initially set to 0.1 and decreases at epochs 30, 60, 90 by a factor of 10.
To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: γ, which is the RBF parameter used to measure the pairwise similarity between two units, and λ, which controls the contribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure 1.
In Table 7, we report the same results of the main figure of the paper in a tabular form along with extra results. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparameters values. Moreover, we note that global modeling of diversity, i.e., det and logdet variants, yield tighter generalization gaps between the train and test errors compared to the non-global direct approach. In fact, while direct variant decreases the generalization gap compared to the standard approach, it decreases it only by 0.5% for most hyperparameter values, whereas, for the more global approaches, i.e., det and logdet, the generalization gap is less than 1.1% in multiple cases compared to the gaps 2.87% and 2.50% achieved by the standard approach and the direct variant, respectively.
For the direct variant (the curves in blue), we note that the performance of the method is not sensitive to the hyperparameters, and the method achieves its best performance for low values of λ and γ. For the det variant (the curves in orange), we note that it significantly improves the generalization ability of the model. However, there is a trade-off between the generalization gap and the final error. Emphasizing diversity and using a high weight for the diversity term significantly decreases the generalization gap, but this damages the performance of the model compared to the standard approach. For example, with λ = 0.01 and γ = 10, the generalization gap of the model is 0.9% compared to 2.87% of the standard. However, the test error for this model gets up to 24.42% compared to 23.87% for the standard. For lower values of λ, the model is able to significantly outperform the standard approach on both the test error and the generalization gap. For the logdet variant (green curves), we note that, in terms of generalization gap, the approach consistently outperforms the standard approach. Using a small value for λ, the model yields lower error rates than the standard approach. For high values of λ, the error rates become similar to the standard approach but with a lower generalization gap. This variant is not sensitive to the hyperparameter γ.

heading: E.3 ALL-MLP MODELS
text: Here, we evaluated the performance of our diversity strategy on modern attention-free, multi-layer perceptron (MLP) based models for image classification on CIFAR10. We conduct a simple experiment using two models: MLP-Mixer (Tolstikhin et al., 2021), gMLP (Liu et al., 2021) with four blocks each. For diversity strategies, i.e., ours and Decov, similar to our other experiments, the additional loss has been added on top of the last intermediate layer. The input image are resized to 72 × 72. We used a patch size of 8 × 8 and an embedding dimension of 256. all models has been trained for 100 epochs using Adam with learning rate of 0.002, weight decay with rate 0.0001, batch size 256. Standard data augmentation, i.e., Random horizontal Flip and random zoom with factor
20% has been used. We use 10% of the training data for validation. We also reduce the learning rate by a factor of 2 if the validation loss does not improve for 5 epochs and use early stopping when the validation loss does not improve for 10 epochs. All experiments has been repeated over 10 random seeds and the average results are reported.

heading: E.4 TRANSFER LEARNING
text: For the transfer learning experiment, we use ResNet50 models pre-trained on ImageNet and we finetune them on CIFAR10 and CIFAR100. Diversity strategy has been applied in both phases. The models are trained for 20 epoch using Adam optimizer with learning rate equal 0.0001 and standard data augmentation is applied. The original images of CIFAR are preprocessed and resized to (96,96,3) in order to be adequate for ResNet50 trained on ImageNet. As it can be seen, employing a diversity strategy helps in the transfer learning context and leads consistently to lower error rates on both datasets. For example, the log variant of our approach leads to 0.94% and 1.27% gains on CIFAR10 and CIFAR100, respectively.

heading: E.5 LEARNING IN THE PRESENCE OF LABEL NOISE
text: To further demonstrate the usefulness of promoting diversity, we test the robustness of our approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and not generalize well to the test set. Enforcing diversity can lead to better and richer representation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFAR10 and CIFAR100 using ResNet50. We use the same training protocol used for the original CIFAR10 and CIFAR100: all models are trained using stochastic gradient descent (SGD) with a momentum of 0.9, weight decay of 0.0001, and a batch size of 128 for 200 epochs. The initial learning rate is set to 0.1 and is then decreased by a factor of 5 after 60, 120, and 160 epochs, respectively. We also adopt a standard data augmentation scheme that is widely used for these two datasets (He et al., 2016; Huang et al., 2017). For all models, the additional diversity term is applied on top the last intermediate layer. For the hyperparameters: The loss weight is chosen from {0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01} for both our approach and De-
cov and γ in the radial basis function is chosen from {0.01, 0.1.1, 10, 50, 100}. For each approach, the model with the best validation performance is used in the test phase. The average errors over three random seed are reported.

heading: E.6 WEIGHT VS FEATURE DIVERSITY
text: Weight diversity is an active field of research which applies a diversity regularizer on top of the weights instead of the representation. Here, we compare the performance of two weight diversity approaches, i.e., (Yu et al., 2011; Xie et al., 2015b), against activation diversity approaches. Results are reported in 8. As it can be seen, activation-based diversity leads to superior results compared the weight-based.

Review summaries:
The paper proposes an interesting angle on the generalization of neural networks. Through proving a new generalization bound that has an internal distance term, the authors argue the neural networks’ generalization performance is related to the internal diversity of the neurons within each layer. As such, some regularization terms are introduced to encourage the internal ``diversity” within each layer. The effectiveness is verified empirically through experiments on a series of data sets including CIFAR and ImageNet.  

The authors prove an upper bound on the estimation error that depends on pairwise distances between hidden unit values per layer. The bound itself is quite likely very loose in standard deep learning settings due to the dependence on various norms of the weights. Loosely inspired by the bound, the authors then propose an optimization algorithm with a particular regularizer that encourages within-layer activation value diversity. The empirical results suggest that the proposed algorithm boosts the performance in most regimes studied (CIFAR 10, CIFAR100 trained on ResNets, Dense nets). The boost is minimal for standard datasets, but it increases when the labels are randomized. Compared to standard empirical risk minimization, a small boost in test accuracy on transfer learning tasks is also reported.

The authors propose a theoretically-grounded technique to improve neural network generalization. The loss is augmented with a term that encourages units to have a more diversity in its activations. More specifically, the authors use a radial basis function to measure pairwise similarity between activations and proposed loss terms based on either the sum or the determinant of the similarity matrix.  Based on experiments in several classification models on CIFAR10, CIFAR100, and ImageNet, models trained with this additional loss term achieve higher test accuracy, and also reduce the generalization gap between train and test. 

In this paper, the authors propose a regularization technique encourage the "activation diversity". Specifically, they design a within-layer loss that add penalty to the similar neurons with the same activation pattern. They also showed that encouraging the within-layer diversity can be used to control the Rademacher complexity of model.