Improving Neural Network Generalization via Promoting Within-Layer Diversity
Authors: Firas Laakom
Jenni Raitoharju
Alexandros Iosifidis
Moncef Gabbouj
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
regularization
overfitting
learning theory
neural network

Abstract: Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one. At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy. In this paper, we propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity. By penalizing similarities and promoting diversity, we encourage each unit within the layer to learn a distinctive representation and, thus, to enrich the data representation learned and to increase the total capacity of the model. We theoretically study how the within-layer activation diversity affects the generalization performance of a neural network and prove that increasing the diversity of hidden activations reduces the estimation error. In addition to the theoretical guarantees, we present an extensive empirical study confirming that the proposed approach enhances the performance of state-of-the-art neural network models and decreases the generalization gap in multiple tasks. 

Under review as a conference paper at ICLR 2022
IMPROVING NEURAL NETWORK GENERALIZATION VIA PROMOTING WITHIN-LAYER DIVERSITY
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks are composed of multiple layers arranged in a hierarchical struc- ture jointly trained with a gradient- based optimization, where the errors are back- propagated from the last layer back to the first one. At each optimization step. neurons at a given layer receive feedhack from neurons belonging to higher lay- ers of the hierarchy. In this paper, we propose to complement this traditional "between-layer" feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity. By penalizing similarities and promoting diversity, we encourage each unit within the layer to learn a distinctive representation and, thus, enrich the data representation learned and to increase the total capacity of the model. We derive novel generalization bounds for neural networks depend- ing on the wathin-layer activation diversity and prove that increasing the diversity of hidden activations reduces the estimation error. In addition to the theoretical guarantees, we present an extensive empirical study confirming that the proposed approach enhances the performance of state-of-the -fet--rrr--- neural network models and decreases the generalization gap in multiple tasks.
INTRODUCTION
Neural networks powerful class of non-linear function approximators that have been success- fully used to tackle a wide range of problems. They have enabled breakthroughs many tasks, such image classification (Krizhevsky al. 2012), speech recognition (Hinton et al. 2012aa and
anomaly detection (Golan & EI- Yaniv 2018). Formally, the output of a neural network consisting of P layers can be defined follows:
f(2: W) o (WP (W
(1)
where (²(.) is the element-wise activation function, e.g., ReLU Sigmoid, of the 5th layer and W = [Wi ..... W are the corresponding weights of the network. The parameters of f(;; W) optimized by minimizing the empirical loss:
i())
Σ ())(a):)))))
where I(-) is the loss function, and {x; V/s 1-=1 are the training samples and their associated ground- truth labels. The loss is minimized using the gradient decent-based optimization coupled with back- propagation.
However, neural networks are often over-parameterized, i.e., have more parameters than data. As a result, they tend to overfit to the training samples and not generalize well unseen examples
[Goodfellow et al., 2016). While research on double descent (Belkin et al.) 2019 Advani et al.
2020 Nakkiran et al. 2020 shows that pratettriiztnnoo does not necessarily lead to overfit- ting, avoiding overfitting has been extensively studied (Neyshabur et al. 2018 Nagarajan an & Kolter 2019 Poggio et al. 2017 Dziugaite & Roy 2017 Foret et 2020) and various approaches and strategies have been proposed, such as data augmentation (Goodfellow et al., 2016 Zhang et al.
2018), regularization (Kukačka et al. 2017 Bietti et al. 2019 Arora et al. 2019). and Dropout

---

Under review as a conference paper at ICLR 2022
(Hinton al. 2012b Wang al. 2019: Lee al. 2019 Li al. 2016), to close the gap between the empirical loss and the expected loss.
2012 Yu al. 2011) and, particularly in deep learning context, diversity of information extracted by the network viable way to improve generalization (Xie et al. 201 2015b) In most cases, these efforts have focused on making the set of weights more diverse (Yngg et al.) Malkin & Bilmes 2009). However, diversity of the activations has not received much attention. Here, we argue that due to the presence of non-linear activations, diverse weights do not guarantee diverse feature representation. Thus, we propose applying the diversity on top of feature mapping instead of the weights.
Diversity of learners is widely known to be important in ensemble learning (Li et al.
neurons has been recognized
To the best of our knowledge, only Cogswell et al. (2016) have considered diversity of the activations directly in the neural network context. They proposed an additional loss term using cross-covariance of hidden activations, which encourages the neurons to learn diverse or non-redundant representa- tions. Their proposed approach, known DeCov, was empirically proven to alleviate overfitting and to improve the generalization ability of neural network, yet a theoretical analysis to prove this has so far been lacking. Moreover, modeling diversity as the sum of the pairwise cross-covariance, it can capture only the pairwise diversity between components and is unable to capture the "higher-order diversity".
In this work, we start by theoretically showing that the within-layer activation diversity boosts the generalization ability of neural networks and reduces overfitting. Moreover, propose novel approach to encourage activation diversity within a layer. We propose complementing the 'between- layer' feedback with additional 'within-layer' feedback to penalize similarities between neurons on the same layer. Thus, we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer. We propose three variants for our approach that are based on different global diversity definitions.
Our contributions in this paper are as follows:
Theoretically, we derive novel generalization bounds for neural networks depending on the within-layer activation diversity. As shown in Section 2 express the upper- upper-bound of the estimation error as a function of the diversity factor. Thus, we provide theoretical evidence that the within-layer activation diversity helps reduce the generalization error.
Methodologically, we propose a new approach to encourage the 'diversification' of the layers' output feature maps in neural networks. The proposed approach has three variants. The main intuition is that by promoting the within-layer activation diversity, neurons within a layer learn distinct patterns and, thus, increase the overall capacity of the model.
Empirically, show that the proposed within-layer activation diversification boosts the performance of neural networks. Experimental results on several tasks show that the pro- posed approach outperforms competing methods.
2 GENERALIZATION ERROR ANALYSIS
In this section, we derive generalization bounds of neural networks depending on the within-layer activation diversity. Generalization theory (Zhang al., 2017 Kawaguchi et al. (017)) focuses on the relation between the empirical loss defined in equation and the expected risk, for any f in the hypothesis class F, defined as follows:
L(S) ((
(3)
where Q is the underlying distribution of the dataset. Let f" arg min L(J) be the expected risk minimizer and f f= arg min scF [(f) be the empirical risk minimizer. We are interested in the estimation error, i.c., L(f*) - L(f), defined as the gap in the loss between both minimizers (Barron 1994). The estimation error represents how well an algorithm can learn. It usually depends on the complexity of the hypothesis class and the number of training samples (Barron 1993 Zhai & Wang 2018).
In this work, we are interested in the effect of the within-layer activation diversity on the estimation emror. In order study this effect, given layer with M units, we assume that with high proba-
2

---

Under review as a conference paper at ICLR 2022
bility the average pairwise distance between the outputs of the units, 2M( Lizf Zitz(ón(a) (on(x)
M
om (z))), is lower bounded by dmin for any input æ. Intuitively, if two neurons n and m have similar outputs for many samples, the corresponding distance (#) - Im (xr)²² will be small. If the average distance is small, the lower bound dmnn is also small and the units within this layer are considered redundant and "not diverse". Otherwise, if the average distance between the different pairs is large, their corresponding dmin is large and they are considered "diverse". By studying how the lower bound dmai affects the generalization of the model, we can theoretically understand how diversity affects the performance of neural networks. To this end, we derive generalization bounds for neural networks using dmiin-
Several techniques have been used to quantify the estimation error, such as PAC learning (Shalev- Shwartz & Ben-David 2014; Valiant 1984), VC dimension (Sontag) 1998), and the Rademacher complexity "Shalev-Shwarde & Ben David 2014). The Rademacher complexity has been widely used as it usually leads to a tighter generalization error bound (Sokolic et al. 2016, Neyshabur et al.) 2018 Golowich et al. 2018). The formal definition of the empirical Rademacher complexity is given as follows:
Ben-David 2014 Bartlett Mendelson 2002 For a given dataset with N samples D = (x,, (i Vim1 generated by distribution Q and for model space F X R with a single dimensional output, the empirical Rademacher complexity RR (F) of the set F is
Definition 1. Shalev-Shward
defined as follows:
RN( Eg
sup LEET
Σ of ols(
N
where the Rademacher variables a = (01,-.. , oon } are independent uniform random variables in {-1,1}.
In this work, we rely on the Rademacher complexity to study diversity. Our analysis starts with the following lemma:
Lemma 1. (Bartlett & Mendelson 2002) With probability of least
2 log((2/))
T(f) I(f*) & (A)
N
where B 2 suP (x)())) y) and R (AA) is the Rademacher complexity of the loss set A.
upper-bounds the estimation eтror using the Rademacher complexity defined over the loss set and supz.p.f (x)) y)1. Our analysis continues by seeking tighter upper bound of this error and showing how the within-layer diversity, expressed with d affects the bound.
In this paper, derive such upper-bound for simple network with one hidden layer trained for a regression task. We show how to extend it for classification, general multi-layer network and for different losses in the Appendix. The proofs are provided supplementary material.
2.1 SINGLE HIDDEN-LAVER NETWORK
Here, consider simple neural network with one hidden-layer with M neurons and one- dimensional output trained for a regression task. The full characterization of the setup can be sum- marized in the following assumptions:
Assumptions 1.
The input vector a E R satisfies ||x||2 S Cn and the output scalar E R. satisfies yl S Cz. The activation function of the hidden layer, d6.), positive LapLipschitz continuous function.
The weight matrix W [wi, W2. ... wvl DxM connecting the input to the hidden layer satisfies |/wml 1 < Cz. The weight vector to E R.M connecting the hidden-layer to the output nearon satisfies | lxx 5 Ca.
The hypothesis class is F (fI((z) ΣM Una Pm =EM= Un ai ww x)))
3

---

Under review as a conference paper at ICLR 2022
Loss function set is A {11(f(x))))) 21f(2)) yl
With probability T, 2M(M-13 Enzm(bi 6m (m
de dnnnn
The main idea our proof is to find a diversity-dependant bound for both terms in Lemma I To this end, we derive novel bound for loss hypothesis class A depending dmin and use for both terms. The loss depends on the hypothesis class F. Thus, we start by deriving an upper bound for this class:
Lemma 2. Under Assumptions
/ with probability at least T, we have
[f(=) S VJ,
sup
where J = CM + ((M - 1) dmmin) and Cs = LaCiC's @
Note that in Lemma we have expressed the upper-bound of supz, f(x) in terms of dman- Using this bound, we can now find an upper-bound for suP (()())))) 3) in the following lemma: Lemma 3. Under Assumptions
7 with a probability at least t, we have
sup [(f(x),))) a.9.J
2 (VJ +2)2
Our main goal is to analyze the estimation error bound of the neural network and to how its upper-bound is linked to the diversity, expressed by dmins of the different neurons. The main result of the paper is presented in Theorem
Theorem 1. Under Assumptions r(1 (), 6), we have
? there exist constant A, such that with probability at least
A VN 2V(+++c))
+ 1 +C2)
2 2log(2/8)
-()-((F) (VJ+
N
where J = CMCC ((M 1) de, d. and C3 LaCiC3 + 6(0).
Theorem [ provides an upper-bound for the generalization gap. We that is decreasing function of dmnn- Thus, this suggests that higher dmin. i.e., more diverse activations, yields a lower estimation error bound. In other words, by promoting the within-layer diversity, we can reduce the generalization error of neural networks. It should be noted that in the last item of our assumptions, we considered a relaxed variant of the following assumption 'H There exists lower bound to the distance, valid for any input x'. H* is impractical especially if the intermediate layer has ReLu activations. Therefore, we considered a relaxed variant this assumption by introducing the probability This makes the theoretical findings useful in practice.
3 WITHIN-LAYER ACTIVATION DIVERSITY
As shown in the previous section, promoting diversity of activations within a layer can lead to tighter generalization bound and can theoretically decrease the gap between the empirical and the true risks. In this section, propose novel diversification strategy, where encourage neurons within layer to activate in a mutually different manner, i.c., to capture different patterns. To this end, we propose additional within-layer loss which penalizes the neurons that activate similarly. The loss function i(f) defined in equation is augmented follows:
(f) = i(f) + XJ',
where J expresses the overall similarity of the neurons within the jth layer and X is the penalty coefficient for the diversity loss. As in (Cogswell al. 20166) our proposed diversity loss can be applied to single layer or multiple layers in network. For simplicity, let us focus on single layer.
Let of, (xt) and di, (my) be the outputs of the th and mh neurons in the ith layer for the same input sample wz. The similarity 8am between the the th and m neurons be obtained as the
4

---

Under review as a conference paper at ICLR 2022
average similarity measure of their outputs for N input samples. We use the radial basis function to express the similarity:
N
Spm Σ exp( - >x))" o (x1))19),
(10)
]=1
where 7 is a hyper-parameter. The similarity 8am can be computed over the whole dataset or batch- wise. Intuitively, if two neurons and m have similar outputs for many samples, their corresponding similarity Sym will be high. Otherwise, their similarity Smn is small and they are considered "di- verse". Based on these pairwise similarities, we propose three variants for the overall similarity J in equation 9
Direct: J = Ln 8nm- In this variant, we model the global layer similarity directly as the sum of the pairwise similarities between the neurons. By minimizing their sum, we encourage the neurons to learn different representations.
Det: J = -S) where S is similarity matrix defined as Sam Symm This variant is inspired by the Determinantal Point Process (DPP) (Kulesza Taskar 2010 2012), as the determinant of S measures the global diversity of the set. Geometrically. det (S) is the vol- ume of the parallelepiped formed by vectors the feature space associated with s. Vectors that result in larger volume considered to be more "diverse". Thus, maximizing det(-) (minimizing -det(-) )) encourages the diversity of the learned features.
Logdet: J -oggeet(S] I This variant has the same motivation as the second one. We use logdet instead of det as logdet is convex function over the positive definite matrix space.
It should be noted here that the first proposed variant, i.e., direct, similar to Decov (Cogswell al., 2016), captures only the pairwise diversity between components and is unable to capture the higher- order "diversity", whereas the other two variants consider the global similarity and are able to mea- sure diversity in more global manner.
Our newly proposed loss function defined in equation has two terms. The first term is the classic loss function. It computes the loss with respect to the ground-truth. In the pack-propagation, this feedback is pack-propagated from the last layer the first layer of the network. Thus, it can be considered as between-layer feedback, whereas the second term is computed within a layer. From equation we can see that our proposed approach can be interpreted as regularization scheme. However, regularization in deep learning is usually applied directly on the parameters, i.e., weights (Goodfellow et al., 2016 Kukačka et al. 2017). while in our approach, similar to (Cogswell et al.) 2016), an additional term is defined over the output maps of the layers. For layer with C' neurons and a batch size of N, the additional computational cost is O(C + 1)) for direct variant and 3 + N)) for both the determinant and log of the determinant variants.
In connection to the bounds derived in Section 2 and to how diversity is theoretically defined in the last item in the Assumptions we note that our regularizer relies the pairwise RBF distance instead of the standard L2 distance. We note that minimizing the average RBF distance, i.e., our direct variant, is equivalent to maximizing the average L2 distance (exp(-x) is a decreasing function). Moreover, we note that empirically the RBF-based distance is less sensitive to noise (Savas & Dovis,
2019 Hay kin
2010).
RELATED WORK
Diversity promoting strategies have been widely used in ensemble learning (Li al. 2012 Yu et al. 2011), sampling (Derezinski et al. 2019: Bryrk et al. 2019; Gartrell et al. 2019), energy-
based models (Zhao al. 2017: Laakom et al. [2021). ranking Yang et al. Gan al. 2020). pruning by reducing redundancy (ondo & Yamauchi] 2014; He et al. 2019 Singh et al. 2020
Lee et al. 2020), and semi cup-ruupee learning (Zbontar et al., 2021). In the deep learning context, various approaches have used diversity as a direct regularizer on top of the weight parameters. Here,
'This is defined only if S is positive definite. It can be shown that in our S is positive semi-definite. Thus, in practice we use regularized version (S + eT) to ensure the positive definiteness.
5

---

Under review as a conference paper at ICLR 2022
present brief overview of these regularizers. Based on the way diversity is defined, we group these approaches into two categories. The first group considers the regularizers that are based on the pairwise dissimilarity of components, i.c,, the overall set of weights
diverse if every pair of weights are dissimilar. Given the weight vectors {wo 1M Yu et al. (2011) define the regularizer Emn (1 0mn)) where mm represents the cosine similarity between Wm. and wn-
Bao al. (2013) proposed incoherence score defined as log 11 Emn fomn/9 3|0mm| where
is a positve hyperparameter. Xie al. (2015al 2016 used mcan( var (ar(0m)) to regularize Boltzmann machines. They theoretically analyzed its effect on the generalization error bounds in
Xie al. 2015b) and extended it kernel space in (Xie et al.) (0177)). The second group of regularizers considers a more globalist view diversity. For example, in (Malkin & Bilmes] [209)] 2008 Xie et al. (0) weight regularization based on the determinant of the weights' covariance
is proposed based on determinantal point process in (Kulesza & Taskar 2012 2012).
Kwok & Adams
Unlike the aforementioned methods which promote diversity on the weight level and similar to our method, (Cogswell al. 2016) proposed to enforce dissimilarity on the feature map outputs, i.c., on the activations. To this end, they proposed an additional loss based on the pairwise covariance of the activation outputs. Their additional loss, L Decov is defined as the squared sum of the non-diagonal elements of the global covariance matrix C' of the activations: Deecov 2 (I|C) ||diag( C') 1l2), where ||-[|F p the Frobenius norm. Their approach, Decov, yielded superior empirical performance; however, it lacks theoretical proof. In this paper, we closed this gap and we showed theoretically how employing a diversity strategy on the network activations can indeed decrease the estimation error bound and improve the generalization of the model. Besides, proposed variants of our approach which consider a global view of diversity.
5
EXPERIMENTAL RESULTS
5.1 IMAGE CLASSIFICATION
We start by evaluating our proposed diversity approach on two image datasets: CIFARIO and CI- FAR100 (Krizhevsky et al., 2009). We use our approach on three tfae-of-thhe-an art CNNs: ResNext- 29-08-16: 16: we consider the standard ResNext Model (Xie al. (017c) with 29-layer architecture,
cardinality of 8, and width of 16. DenseNet- 12: we use DenseNet (Huang al. 2017) with the 40- laver architecture and a growth rate of 12. ResNet50:
consider the standard ResNet model (He cct al. 2016) with 50 layers. We compare against the standard networks as well as networks trained with DeCov diversity strategy (Cogswell al. 2016). For each approach the hyperparameters selected based on the validation set. The full experimental setup is presented the Appendix. We report the average performance three random seeds.
Table I: Classification errors of the different approaches on CIFARIO IO and CIFARI00 with three different models. Results are averaged over three random seeds.
DenseNet-12 12
ResNext-29-088
ResNet50
method
CIFARIO
CIFAR 100
CIFARIO
CIFAR 100
CIFARIO
CIFAR100
Standard
7.07
29.25
6.93
26.73
8.27
34.06
DeCov
7.18
29.17
6.84
26.70
8.03
32.26
Ours(direct)
6.95
29.16
6.74
26.54
7.86
32.15
Ours(det)
7.04
28.78
6.67
26.67
7.73
32.12
Ours(logdet) 6.96
29.15
6.70
26.67
7.91
32.20
Table reports the average top-1 errors of the different approaches with the three basis networks. We note that, compared to the standard approach, employing a diversity strategy generally boosts the results for all the three models and that our approach consistency outperforms both competing
For the standard approach, the only difference is not using an additional diversity loss. The remaining regularizers, data augmentation, weight decay etc, are all applied specified per-experiment.

---

Under review as a conference paper at ICLR 2022
Table 2: Performance of different models with different diversity strategies ImageNet dataset
ResNet50
Wide-ResNet50
Method
Top-1 Errors
Gap
Top-1 Errors
Gap
Standard
23.84
2.87
22.42
6.33
DeCov
23.62
2.70
22.68
6.34
Ours(direct)
23.75
2.73
22.39
6.22
Ours(det)
23.62
2.77
22.33
6.13
Ours(logdet) 23.64
1.07
22.27
6.03
methods (standard and DeCov) in all the experiments. For example with ResNet50, the three variants of our proposed approach significantly reduce the test errors over both datasets: 0.36% 0.54% improvement on CIFARIO and 1.86% 1.94% on CIFAR100.
ImageNet: To further demonstrate the effectiveness of our approach and its ability reduce the generalization gap in neural networks, we conduct additional image classification experiments on ImageNet-2012 classification dataset (Russakovsky et al., 2015) using the ResNet50 (He al.) 2016) and Wide-Res (Zagoruyko & Komodakis) (0166) models. The diversity term is applied on the last intermediate layer, i.c., the global average pooling layer for both DeCov and our method. For the hyperparameters, we use À 0.001 and 10. The full experimental setup is presented in the Appendix. Table reports the test errors of the different diversity strategies. To study the effect of diversity the generalization gap, also report the final training enrors and the generalization gap, i.e., training accuracy test accuracy. As can be seen, diversity (our approach and DeCov) reduces the test error of the model and yields better performance. We note that, in accordance with our theoretical findings in Section 2 using diversity indeed reduces overfitting and decreases the empirical generalisation gap neural networks. In fact, our logdet variant reduces the empirical generalization gap of the model by 1.8% compared to the standard approach. We note that our approach has small additional time cost. For example for ResNet50, our direct, det and logdet variants take only 0.29%, 0.39%, and 0.49% extra training time, respectively.
%
3.0
Stetery
24.4
2nd cl
Ceneraization Gap
24.2
Test Error
2.0
24.0
0.0001
0.005
0.0003
Figure 1: Sensitivity analysis of À and 7 on both the model accuracy and its generalization ability using ResNet50 trained on ImageNet.
Sensitivity analysis: To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: which is the RBF parameter used to measure the pairwise similarity between two units, and X, which controls the con- tribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure I and in supplementary material. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparame- ters values. Moreover, tighter generalization gaps compared to the non-globe direct approach. It is worth noting that Fig- ure I shows that there is a trade-off between the generalization gap and the final error. Emphasizing
note that global modeling of diversity, i.c., det and logdet variants, yield

---

Under review as a conference paper at ICLR 2022
Table 3: The compatibility of the proposed approach with Dropout. Test errors and generalization gap of different combinations on ImageNet dataset.
ResNet50
Wide-ResNet50
Method
Top-1 1 Errors
Gap
Top-1
Errors
Gap
Dropout
23.73
0.14
22.
3.68
Dropout + direct
23.72
0.11
22.00
1.46
Dropout + det
23.65
0.26
22.16
3.80
Dropout logdet
23.65
0.30
21.89
3.41
diversity and using high weight for the diversity term significantly decreases the generalization gap, but this damages the performance of the model compared the standard approach. For lower values of X, the model is able to significantly outperform the standard approach both the test error and the generalization gap.
In Section 2 we provided theoretical bounds for the generalization errors of neural networks, which are inversely proportional to the diversity term, dman- the lower bound of 2M[M-1] Eizs( (2)(2) - 6m ( To show that the proposed regularizer in- deed improves diversity, we track the empirical average of the aforementioned variable, i.e., o. (x:))² during the training. The results for both the standard
Connection to Theory:
M 2M(M-) Lnzimm (x,),(x,) approach and the logdet variant of our approach are reported in Figure 2 As it can be seen, our regularizer yields in higher diversity which reduces overfitting and leads to better generalization.
Qversity
Autional
Epocts
Figure 2: Tracking diversity in the training phase using ResNet50 trained on ImageNet.
Compatibility with Dropout: Dropout (Srivastava al. 2014) is popular regularization technique that, similarly to our approach, is applied on top of the layer output and has been known to improve generalization and reduce overfitting. Here, we evaluate the compatibility of approach with Dropout, We add a Dropout regularizer with 20% rate on top the last intermediate representation layer of both ResNet50 and a diversity regularizer alongside the Dropout consistently yields lower eπror rates compared to only Dropout except for the det variant Wide-ResNet50. For Wide-ResNet50,, the combination of our logdet variant and Dropout leads to 0.53%. 0.25%, and 0.38% improvement compared to standard approach, Dropout only, and our logdet variant only, respectively.
The results are reported in Table We note that adding
Beyond CNN models, we also evaluate the performance of our diversity strat- egy on modern attention-frre free, multi-layer perceptron (MLP) based models for image classification Tolstikkhin et al. 2021 Liu et al. 2021 Lee-Thorp al. 202T). Such models are known to exhibit high overfitting and require regularization. We evaluate how diversity affects the accuracy of such models on CIFAR1O 10.. In particular, we conduct simple experiment using two models: MLP-Mixer Tolstikhin et al. 2021), gMLP (Liu et al. 2021) with four blocks each. The full description of the experimental setup presented in the Appendix. The results in Table 4 show that employing diversity strategy can indeed improve the performance of these models thanks to its ability to learn rich and robust representation of the input.
MLP-based models:
Transfer learning: Beyond standard classification. the proposed approach can be useful, e.g.. in transfer learning, where the main goal is to 'transfer' previously learned representation solve
8

---

Under review as a conference paper at ICLR 2022
tasks. Thus, learning rich and diverse representation is beneficial and can lead to better transfer- ability. To demonstrate this, we conduct another experiment, where we use ImageNet-pretralte ResNet50 models with the different diversity approaches and finetune them to CIFARIO and CIFARIO0 The results are reported in Table 5 As it can be seen, employing diversity strategy helps in the transfer learning context and leads consistently lower rates. For example, the log variant of our approach leads to 0.94% and 1.27% gains on CIFARIO and CIFAR 100, respectively.
Table 4: Classification errors of modern MLP- based approaches on CIFARIO. Results are av- eraged over ten random seeds.
Table 5: Transfer learning performance CI- FAR10 and CIFARI00 ResNet50 models pre-trained ImageNet.
MLP-Mixer
gMLP
4 CIFARIO
4 CIFARI00
Standard
23.96
24.69
Standard
6.14
22.99
DeCov
23.68
24.17
DeCov
5.92
21,91
Ours(direct)
23.84
23.82
Ours(direct)
5.89
21.48
Ours(det)
23.62
24.08
Ours(det)
5,51
22.01
Ours(logdet)
23.85
24.15
Ours(logdet)
5.20
21.72
5.2 LEARNING THE PRESENCE LABEL NOISE
To further demonstrate the usefulness of promoting diversity, test the robustness of approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and generalize well to the test set. Enforcing diversity can lead better and richer representation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFAR10 and CIFAR100 using ResNet50. The results are reported in Table 6 As it can be seen, in the presence of noise, the gap between the standard approach and diversity (Decov and ours) increases. For example, our logdet variant boosts the results by 1.71% and 3.59% on CIFAR10 and CIFAR100 with 40% noise, respectively.
Table 6: Classification errors of ResNet50 using different diversity strategies CIFARIO and CI- FARI00 datasets with different label noise ratios. Results are averaged over three random seeds.
20% label noise
40% label noise
Method
CIFARIO
CIFARI00
CIFARIO
CIFARI00
Standard
14.38
45.11
19.40
51.88
DeCov
13.64
43.02
18.11
50.66
Ours(direct)
13.48
41.78
17.57
48.91
Ours(det)
13.77
41.49
17.87
48.50
Ours(logdet)
13.59
41.11
17.69
48.29
CONCLUSIONS
In this paper, proposed new approach encourage 'diversification' of the layer-wise feature map outputs in neural networks. The main motivation is that by promoting within-layer activation diversity, units within the same layer learn capture mutually distinct patterns. We proposed an ad- ditional loss term that can be added on top of any fully-connected layer. This term complements the traditional Thetween-layer feedhack with an additional 'within-layer' feedback encouraging diver- sity of the activations. We theoretically proved that the proposed approach decreases the estimation error bound and, thus, improves the generalization ability of neural networks. This analysis was further supported by extensive experimental results showing that such a strategy indeed improve the performance of different -tate-of-the-ar networks across different datasets and different tasks, ie., image classification, transfer learning, and label noise. We are confident that these results will spark further research in diversity-brat bassdd approaches to improve the generalization ability of neural networks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Nenworks, 132:428-446, 2020.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, pp. 7413-7424, 2019.
Yebo Bao, Hui Jiang, Lirong Dai, and Cong Liu. Incoherent training of deep neural networks to de- correlate bottleneck features for speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pp. 6980-6984, 2013.
Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, pp. 930-945, 1993.
Andrew R Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, pp. 113-133, 1994.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, pp. 463-482, 2002.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, (5843)):1549-15355)
2019.
Alberto Bietti, Grégoire Mialon, Dexiong Chen, and Julien Mairal. A kernel perspective for regu- larizing deep neural networks. In International Conference on Machine Learning, pp. 664-674, 2019.
Erdem BIyIk, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determi- nantal point processes. arXiv preprint 19066v:900..100 2019.
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. In International Conference on Learning Representations, 2016.
Michal Derezinski, Daniele Calandriello, and Michal Valko. Exact sampling of determinantal point processes with sublinear time preprocessing. In Advances in Neural Information Processing Sys- tems, pp. 1 11546-11558, 2019.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 1703v 1008, 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprini arXiv:2010.01412, 2020.
Lu Gan, Diana Nurbakova, Léa Laporte, and Sylvie Calabretto. Enhancing recommendation diver- sity using determinantal point processes on knowledge graphs. In Conference on Research and Development Information Retrieval, pp. 2001-2004 2020.
Mike Gartrell, Victor-Emmanue! Brunel, Elvis Dohmatob, and Syrine Krichene. Learning nonsym- metric determinantal point processes. In Advances in Neural Information Processing Systems, pp- 6718 1728, 2019.
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. Ad- vances in Neural Information Processing Systems, pp. 9758-9769, 2018.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pp. 297-299, 2018.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT Press, 2016.
Simon Haykin. Neural networks and learning machines, 3/E. Pearson Education India, 2010.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340-4349 2019.
Geoffrey Hinton, Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views four research groups. Signal processing magazine, 29(6):82 97, 2012a.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint frX.iv1207.0588. 2012b.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4705 2017.
Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio. Generalization in deep learning. arXiv preprint arXiv:1710.0544 2017.
Yusuke Kondo and Koichiro Yamauchi. dynamic pruning strategy incremental learning on budget. In International Conference on Neural Information Processing, pp. 295-303. Springer, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv preprint arXiv:1710. 2017.
Alex Kulesza and Ben Taskar. Structured determinantal point processes. In Advances in Neural Information Processing Systems, pp. 1171-1179, 2010.
Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. arXiv preprint arXiv: 1207.6083, 2012.
James Kwok and Ryan P Adams. Priors for diversity in generative latent variable models. In Advances in Neural Information Processing Systems, pp. 2996-3004, 2012.
Firas Laakom, Jenni Raitoharju, Alexandros losifidis, and Moncef Gabbouj. On feature diversity in energy-based models. In Energy Based Models Yorkshop-ICLP 2021, 2021.
Hae Beom Lee, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In International Conference on Learning Representations, 2019.
Seunghyun Lee, Byeongho Heo, Jung-Wor Woo Ha, and Byung Cheol Song. Filter pruning and re-
initialization via latent space clustering. IEEE Access, 8: 18989777-889997 2020.
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824 2021.
Nan Li, Yang Yu, and Hui-Hua Zhou. Diversity regularized ensemble pruning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 330-345, 2012.
Zhe Li, Boqing Gong, and Tianbao Yang. Improved dropout for shallow and deep learning. In Advances in Neural Information Processing Systems, pp. 2523-2531, 2016.
11

---

Under review as a conference paper at ICLR 2022
Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv preprint arXiv:210S.08050 2021.
Jonathan Malkin and Jeff Bilmes. Ratio semi-definite classifiers. In International Conference on Acoustics, Speech and Signal Processing, pp. 4113 4116, 2008.
Jonathan Malkin and Jeff Bilmes. Multi-layer ratio semi-definite classifiers. In International Con- ference on Acoustics, Speech and Signal Processing, pp. 4465-4468, 2009.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generaliza- tion in deep learning. In Advances in Neural Information Processing Systems, pp. 11615-11626, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learn- ing Representations, 2020. URL httth:tt////// /ttp:////////eeeveee/ee net/ www.oonnddlgaaaaa......
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of vveepparamettizzaioon in generalization of neural networks. In International Conference Learning Representations, 2018.
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning III: explaining the non-overfitting 2017.
puzzle. arXiv preprint
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 15(3)2211-252 2015.
Caner Savas and Fabio Dovis. The impact of different kernel functions on the performance of scintillation detection based on support vector machines. Sensors, 19(23):5219, 2019.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory algo-
rithms. Cambridge university press, 2014.
Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay Namboodiri. Leveraging filter corre- lations for deep model compression. In The IEEE Winter Conference on Applications of Computer Vision, pp. 835-844, 2020.
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. rademacher complexity for deep learning. 2016.
Lessons from the
Eduardo D Sontag. VC dimension of neural networks. NATO ASI Series F Computer and Systems Sciences, pp. 69-96, 1998.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 1958,, 2014.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01660 2021.
Valiant. A theory ofthe learnable. Commun. ofthe ACM, 27(1)) 1984.
Haotian Wang. Wenjing Yang, Zhenyu Zhao, Tingjin Luo, Ji Wang, and Yuhua Tang. Rademacher dropout: An adaptive dropout for deep neural network via optimizing generalization gap. Neuro- computing, pp- 177-187, 2019.
Bo Xie, Yingyu Liang, and Le Song. Diverse neural network learns true target functions. In Artificial Intelligence and Statistics, pp. 1216 1224, 2017a.
Pengtao Xie, Yuntian Deng, and Eric Xing. Diversifying restricted boltzmann machine for document modeling. In International Conference on Knowledge Discovery and Data Mining, pp. 1315- 1324, 2015a.
12

---

Under review as a conference paper at ICLR 2022
Pengtao Xie, Yuntian Deng, and Eric Xing. On the generalization enror bounds of neural networks under inducing mutual angular regularization. arXiv preprint 1rxiv:551..07100, 2015b.
Pengtao Xie, Jun Zhu, and Eric Xing, Diversity-promoting bayesian learning of latent variable models. In International Conference on Machine Learning, pp. 59-68, 2016.
Pengtao Xie, Aarti Singh, and Eric P Xing. Uncorrelation and evenness: a new diversity-promoting regularizer. In International Conference on Machine Learning, pp.
2017b
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017c
Ke Yang. Vasilis Gkatzelis, and Julia Stoyanovich. Balanced ranking with diversity constraints. In International Joint Conference on Artificial Intelligence, pp. 6035-6042.
Yang Yu, Yu-Feng Li, and Zhi-Hua Zhou. Diversity regularized machine. In International Joint Conference on Artificial Intelligence, 2011.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. 30.87. URL http::tt::///tctt : doi org/ 324.//.....244
doi: 10.5244/C.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint
2021.
Ke Zhai and Huan Wang. Adaptive dropout with rademacher complexity regularization. In Interna- tional Conference on Learning Representations, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations 2018, 2018.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. International Conference Learning Representations, 2017.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
The full characterization of the setup can summarized the following assumptions: Assumptions 2.
The activation function of the hidden layer, ((), is a positive Lip-schttt continuous function.
The input vector x: E
IR. satisfies ||||| S C₁.
The oulput scalar E R satisfies y S Cz.
The weight matrix W (wi, W2:- wMm E RRDxM connecting the input to the hidden layer satisfies [| 12 S Cs.
The weight vector vo E IR connecting the hidden-layer to the output neuron satisfies ||0|/> Ca.
The hypothesis class is F = =fff(x)) = -EM= Um 2m (x) = Em= Um "mo(w x)).
Loss function set (11(((()).)) = =11((2))) 3
With probability T, 2M(M-I) Enym (0,, (2x) - 6m (x²) d2 dma
We recall the following two lemmas related to the estimation error and the Rademacher complexity:
Lemma 4.
Bartlett & Mendelson 2002) For F E R* assume that R R is Lg Lipschitz
continuous function and A ig
J Then we have
(x) S (FF).
(11)
Lemma 5.
(Xie et al. 2015b) Under Assumptions
! the Rademacher complexity R, RN(F) of the Emi Um ", can be upper-bounded
hypothesis class (flf(x) Em= Um 2m (x) as follows:
2LoC13IM Cal2()JMM
Rx(F) <
(12)
VN
VN
where C134 C3C3 and ((0) is the output of the activation function at the origin.
A.1 PROOF OF LEMMA 2
Lemma Under Assumptions with probability at least t, we have
sup ff(x) [[x) vJ, 2.
(13)
where J = (MC + M(M-MMMMM)) - 1) - d d and Cs = Ts
Proof.
2
2
P(x)) =
Σ
Fm Am (I))
<
|| Pm (am))a)))))))))) (ar)
= 1101133
Σ am (x))
S C2
Σ
Am (xz)
= c²
Σ
Dm (2z) (x))
= =c²
Σ Dmm (xr) +
Σ
2n (2()))mm
(14)
msn
We have sup Om (2) 5 sup(Lal wo ((0)) because Lgg-iipchiiz. Thus, 110| S L&CiCs + Ø(0) = Cs. For the first term in equation 14 we have Σm @m(x) <
M(L.CiCs + 2(0)) = MC3. The second term, using the identity (m (x) + (x)² (dm(x) - (n()))?) can be rewritten as
Σ (z)))
Σ (x)) 0x()))² (0m - (()()))))())) m
(15)
14

---

Under review as a conference paper at ICLR 2022
In addition, have with probability T, 1/2 Emzto (0m(x)) )n(e)))² M(M 1)d for Thus, we have with a probability at least 7:
Σ Om (o)
1 Σ (203) M(M 1)dmin = M(M 1)(C3 deainn) 1 2
(16)
By putting everything back to equation
14 have probability T.
((x)
S ci
MC3 + M(M - 1)(C3 - demin) =
(17)
Thus, with probability 7,
sup [f()) & a.f
V Voi
sup f(x) & vJ.
(18)
A.2
PROOF OF LEMMA 3
Lemma Under Assumptions 7 with probability at least T, we have
1
sup [((f(z), 4)1 - [VJ + C2) a.y.f
(19)
2
Proof. We have supmy. ff()) yl s sup (1f(() + lvl)
VJ + C2.
Thus
supz.y. [(f(x), v)1 s (VJ + +2)².
A.3 PROOF OF THROREM 1
Theorem 1 Under Assumptions
7 there exist constant A, such that with probability at least
r(1- ([1-)), 6), we have
- L(F)) ( (VJ+C,)
A VN + 22( + C
2 2lo(2//8)
(20)
N
where J = CMCC + M(M - 1) - d and C3 LaCiO3 +3
Proof. Given that ((-) is K K-Lipschitz with constant K
supa (s(x) ya S VJ + C2, and
using Lemma 4 we can show that R2 (AA) & KR (F) use the bound Found in Lemma 5 Using Lemmas I and we have
(VJ C2 (F). For Rr (F), we
M s (VV + (2L (2) Clo(01))
21og(2/8)
L())
+ [VJ + C2)²
VN
N
(21)
where C131 = C,C3C,, J = CM + M(M () d) and C3 = LsC
Thus, setting A = (1 Calo(0))) M completes the proof.
B BINARY CLASSIFICATION
We now extend our analysis of the effect of the within layer diversity on the generalization error in the case of a binary classification task, i.e., y E (1. The extensions of Theorem in the case of hinge loss and a logistic loss are presented in Cheorems@aande respectively.
Theorem 2. Using the hinge loss, there exist a constant such that with probability at least r(1 5). we have
2 (2g
UÍ T(S*) A/VN V
(22)
N
where = CMCMC + M(M M(M-1)(C d and Cs T. + Φ(0).
15

---

Under review as a conference paper at ICLR 2022
Theorem 3. Using the logistic loss [2f()), y) log(1 + e-)n(=)), there exist constant such that, with probability at least T(1 - 8), we have
uń) - - < (1+ [1+vv-JWN + log(1 + +ev3)) /
A
2lg((2/6)
(23)
N
where: CMC + MMM- () dminn) and C2 LoC 0
As we can see, also for the binary classification task, the error bounds of the estimation error for the hinge and logistic losses are decreasing with respect to dmii- Thus, employing diversity strategy can improve the generalization also for the binary classification task.
B. PROOFS OF THEOREMS 2 AND
Similar to the proofs of Lemmas 7 and 8 in Xie et al. (0115), we can show the following two lemmas:
Lemma 6. Using the hinge loss, we have with probability at least T(1 - 6)
I(f) I(1*) s 4(21aC1s4 Calo(01) VN
M
(1 + VJN
2 (2o822/)) (24)
N
where C131 C,0sOs, J CR(MC3 + MMM 1) - demin)), and C3 = LoCiss 6(0).
Lemma 7, Using the logistic loss ((f(x).)) log(1 + e e-as(x)), we have with probability at least r(1 (1--)
M
2 2log(2/8)
L(f) L(f)) + (2L +c]o((0))
VN log(1 eV >v
(25)
where C134 C,C3C4, J CG(MC + M(M 1) ). and C3- LaCiCs + 6(0).
Taking A= 4=4(21001ss + Calá(0))) M in Lemma 6 and Lemma completes the proofs.
C MULTI-LAYER NETWORKS
Here, we extend our result for networks with P (> 1) hidden layers. We assume that the pairwise distances between the activations within layer p nower-bounded by d(r) with probability f())_ In this case, the hypothesis class can be defined recursively. In addition, we replace the fourth assumption in Assumptions with: || loo s c(e) for every ((p), i... the weight matrix of
the p-th layer. this case, the main theorem is extended as follows:
Theorem 4. There exist constant A such that, with probability of at least T (T(0)) 8), we have
L - I()) & (VJP+Ca) A 1
VN + (VJP
2log(2/6)
(26)
N
where JP is defined recursively using the following identities: Jo = C and J(r)
Mvr) (M + 20 M(M 1)d(r) for I...,,
In Theorem 4 we see that JP is decreasing with respect to (P) Thus, we see that maximizing the within-layer diversity, we can reduce the estimation error of a multi-layer neural network.
C.1 PROOF OF THEOREM
Theorem 4 There exist constant A such that, with probability of least 11 (- 8),
have
-() (VTT +
1 [VJ++c;)²
2log(2/8)
(27)
N
16

---

Under review as a conference paper at ICLR 2022
where JP is defined recursively using the following identities: Jo C3C₁ and J(m) 2 (MM 1)d )), for p I.....P..
CV2 (MP2
6(0)) M(M
2015b) provides an upper-bound for the hypothesis class. We denote by ((P) the outputs of the yd
Proof. Lemma in Xie et al.
hidden layer before applying the activation function:
0 iw1
(28)
M
MP-1
Σ
W (p)
1.l (1)
w(() h.ri
(29)
j=1
(1)) = |wm
(b())
(p)
(b(n)},
(30)
Mimi
where ((()
(1 Mp-)) }}. We have
M
||0(00
Σ
(D)
(31)
m
m=1
and wm (p)
Φip)
S
c(P)
Σn on
Thus,
3
M
S
(p) 3
Σ o(P))² = M Ed 21
0g
Σ
((P) ((P) m pn
(32)
m==1
ma
(p) We use the same decomposition trick of oin on as in the proof of Lemma 2. We need to bound sup, g(P):
sup r(P) < sup(Lolu> ((0)) < 1 12 (0))
(33)
Thus, have
|| S CF2 (M²
o(0)) M(M 1)dmms)) =J..
(34)
We found recursive bound for (|0(e) note that for p O, we have ||00 112 S C&C = Jo. Thus,
||
110
C1
S
sup PEP
[(()]]
sup PEER
1 JTP
(35)
By replacing the variables in Lemma
we have
(2C0) P-
16(0) P-
P-1
L(f)-((F) < 4(V
II p=0
VM
Σ (2L P-1-p
II
VMC
VN
VN
2=0
j=p
+ 2 ( [VJr + C²)²
2log(2/0)
N ) P- -P VMCC
Taking A
4 4 ((2L)) p=0
VMWC + 10(0) (2Lx)
IT
completes the proof.
D MULTIPLE OUTPUTS
Finally, we consider the case of neural network with multi-dimensional output, i.e., y E RD, In this case, we can extend Theorem with the following two theorems:
Theorem 5. For multivariate regression trained with the squared error, there exist constant A such that, with probability at least 7(1 - 5), we have
D PIVJ + C2)2
2 2log(2/8)
I(f) - -() s (VJ +C) VN +
(36)
N
where = CMCM M(M- d and Cs CiCs + ((0).
17

---

Under review as a conference paper at ICLR 2022
Theorem 6. For multi-class classification task using the cross-entropy loss, there exist constant such that, with probability at least T(1 - 6), we have
1 (())
DD - 1 -2JJNN +log (D
A
2 2log(2/0)
(37)
where CMC M(M-M(MMMMMM 1)(C3 demin) and Cs La (0))
Theorems 5 and 6 extend our result for the multi-dimensions regression and classification tasks, respectively. Both bounds are inversely proportional to the diversity factor dman. We note that for the classification task, the upper-bound is exponentially decreasing with respect to dmin. This shows that increasing diversity within the layer yields a tighter generalization gap and, thus, theoretically guarantees stronger generalization performance.
D.1 PROOF OF THEOREM 5
Proof. The squared loss 1 [() 2lا2 can be decomposed into D terms {f(())) -ya)². Using
Theorem we can derive the bound for each term and thus, we have:
VN ( 21og(2/0) (38)
M D
N
where C131
= C + M(M 1) denin),, and Co LoC1C' +(0).
Taking A (D(21 (alooo)) M completes the proof.
D.2 PROOF OF THEOREM 6
Proof. Using Lemma 9 Xie et al. (015b), we have sup fæv,y / log - -D-1)e23) and / D-1
D-1te- Lipschitz. Thus, using the decomposition property of the Rademacher complexity, we have
(A) < 1D(D - D-1+e 2vJ (2L,C134 + Calo(0))) M
(39)
VN
Taking A 4D(D 1) C134 Calo(0))) M completes the proof.
E EXPERIMENTAL RESULTS
Here, report the experimental setup of the different experiments done in the paper along with more detailed analysis the results.
E CIFARIO & CIFAR100
We start by evaluating our proposed diversity approach on two image datasets: CIFARIO and CI- FARI00 100 (Krizhevsky et al. 2009). They contain 60,000 (50,000 train/10,000 test) 32 x 32 images grouped into 10 and 100 distinct categories, respectively. We split the original training set (50,000) into two sets: we use the first 40,000 images as the main training set and the last 10,000 as a vali- dation set for hyperparameters optimization. We use our approach on three state-of-the-ar CNNs: ResNext-29-08-
we consider the standard ResNext Model (Xie et al.) 2017c) with 29-layer architecture, a cardinality of and a width of 16. DenseNet-12: we use DenseNet (Fuann et al., 2017) with the 40-layer architecture and growth rate of 12. ResNet50: we consider the standard ResNet model (He et al. 2016) with 50 layers. We compare against the standard networks as well networks trained with DeCov diversity strategy (Cogswell et al. 2016).
All the models are trained using stochastic gradient descent (SGD) with a momentum of 0.9, weight decay of 0.0001, and batch size of 128 for 200 epochs. The initial learning rate is set to 0.1 and is then decreased by a factor of after 60, 120, and 160 epochs, respectively. We also adopt standard data augmentation scheme that widely used for these datasets
18

---

Under review as a conference paper at ICLR 2022
(He et al.] 2016; Huang al., 2017). For all models, the additional diversity term is applied on top the last intermediate layer. For the hyperparameters: The loss weight is chosen from {0.00001, 0.00005, 0,0001, 0.0005, 0.001, 0.005, 0.01} for both our approach and Decov and y in the radial basis function is chosen from {0.01, 0.1.1, 10, 50, 100). For each approach, the model with the best validation performance is used in the test phase. We report the average performance over three random seeds.
E.2 IMAGENET AND SENSITIVITY ANALYSIS
To further demonstrate the effectiveness of our approach and its ability to reduce the generalization gap in neural networks, we conduct additional image classification experiments on the ImageNet- 2012 classification dataset (Russakovsky et al. 2013) using the ResNet50 model (He et al. 2016). The diversity term is applied on the last intermediate layer, i.e. the global average pooling layer for both DeCov and our method. For the hyperparameters, we use 7 10 and À 0.0001 for all the different approaches. We use the standard augmentation practice for this dataset as in (Zhang et al.) 2018 Huang et al. [017 Cogswell et al. 2016). All the models trained with batch size of 256 for 100 epoch using SGD with Nesterov Momentum of 0.9. The learning rate is initially set to 0.1 and decreases at epochs 30, 60, 90 by factor of 10.
To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: 7. which is the RBF parameter used to measure the pairwise similarity between two units, and X, which controls the contribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure D
In Table 7 we report the same results of the main figure of the paper in a tabular form along with extra results. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparameters values. Moreover, we note that global modeling of diversity, i.e., det and logdet variants, yield tighter generalization gaps between the train and test errors compared to the non-global direct approach. In fact, while direct variant decreases the generalization gap compared to the standard approach, it decreases it only by 0.5% for most hyperparameter values, whereas, for the more global approaches, i.c., det and logdet, the generalization gap is less than 1.1% in multiple cases compared the gaps 2.87% and 2.50% achieved by the standard approach and the direct variant, respectively.
For the direct variant (the curves in blue), we note that the performance of the method is not sensi- tive to the hyperparameters, and the method achieves its best performance for low values of X and y. For the det variant (the curves in orange), we note that it significantly improves the generalization ability of the model. However, there is a trade-off between the generalization gap and the final error. Emphasizing diversity and using high weight for the diversity term significantly decreases the gen- eralization gap, but this damages the performance of the model compared to the standard approach. For example, with = 0.01 and 7 10, the generalization gap of the model is 0.9% compared to 2.87% of the standard. However, the test error for this model gets up to 24.42% compared to 23.87% for the standard. For lower values of X, the model is able to significantly outperform the standard approach on both the test error and the generalization gap. For the logdet variant (green curves), we note that, in terms of generalization gap. the approach consistently outperforms the standard approach. Using small value for X, the model yields lower error rates than the standard approach. For high values of X, the error rates become similar to the standard approach but with a lower generalization gap. This variant is not sensitive to the hyperparameter
E.3 ALL-MLP MODELS
Here, we evaluated the performance of our diversity strategy on modern attention-fres free, multi-layer perceptron (MLP) based models for image classification on CIFAR10. We conduct simple exper- iment using two models: MLP-Mixer (Tolstikhin et al. 2021). gMLP (Liu et al.] (022)) with four blocks each. For diversity strategies, i.e., ours and Decov, similar to our other experiments, the additional loss has been added on top of the last intermediate layer. The input image are resized to 72 x 72. We used patch size of 8 8 and an embedding dimension of 256. all models has been trained for 100 epochs using Adam with learning rate of 0.002, weight decay with rate 0.0001, batch size 256. Standard data augmentation, i.e., Random horizontal Flip and random zoom with factor
19

---

Under review as a conference paper at ICLR 2022
Table 7: Performance of ResNet50 with different diversity strategies ImageNet dataset with
different hyper-paramters
Method
Top-1 Test Errors
Generalization Gap
Standard
23.84
2.87
Ours d 10, À 0.0001)
23.63
2.55
Ours direct(7 10, 0.001)
23.75
2.73
Ours direct(7 = 10. À = 0.005)
23.83
2.88
Ours direct(? 10.
0.01)
23.58
2.70
Ours det(( = 10, À = 0.0001)
23.48
2.78
Ours det(? = 10, À = 0.001)
23.58
2.77
Ours det( 10, À 0.005)
23.67
2.70
Ours det(? = 10, À = 0.01)
24.47
0.92
Ours logdet(? 10, À 0.0001)
23.52
2.58
Ours !ogdet(> = 10, À = 0.001)
23.64
1.07
Ours ?ogdet(? 10, À 0.005)
23.88
2.70
Ours logdet(> 10, À 0.01)
23.79
2.64
Ours d = 1, À = 0.001)
23.66
2.86
Ours direct(? 50, À 0.001)
23.85
2.67
Ours det(yy = 1, A 0.01)
23.77
2.88
Ours det( = 50, À 0.01)
23.97
2.97
Ours logdet(? 1, A 0.001)
23.65
2.70
Ours ?ogdet(? = 50, À = 0.001)
23.70
2.65
Ours logdet(? 50, À = 0.005)
23.57
2.78
20% has been used. We use 10% of the training data for validation. We also reduce the learning rate by factor of 2 if the validation loss does improve for 5 epochs and use early stopping when the validation loss does not improve for 10 epochs. All experiments has been repeated over 10 random seeds and the average results are reported.
E.4 TRANSFER LEARNING
For the transfer learning experiment, we use ResNet50 models pre-trained on ImageNet and we finetune them on CIFAR10 and CIFAR100. Diversity strategy has been applied in both phases. The models are trained for 20 epoch using Adam optimizer with learning rate equal 0.0001 and standard data augmentation is applied. The original images of CIFAR are preprocessed and resized to (96,96.3) in order to be adequate for ResNet50 trained on ImageNet. As it can be seen, employing a diversity strategy helps in the transfer learning context and Icads consistently to lower error rates on both datasets. For example, the log variant of approach leads to 0.94% and 1.27% gains CIFAR10 and CIFAR100, respectively.
E.5 LEARNING THE PRESENCE OF LABEL NOISE
To further demonstrate the usefulness of promoting diversity, test the robustness of our approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and not generalize well the test set. Enforcing diversity can lead to better and richer rep- resentation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFARIO and CIFARION using ResNet50. We use the same training protocol used for the original CIFAR10 10 and CIFAR 100: all models are trained using stochastic gra- dient descent (SGD) with a momentum 0.9, weight decay of 0.0001, and a batch size of 128 for 200 epochs. The initial learning rate is set to 0.1 and is then decreased by factor of 5 after 60, 120, and 160 epochs, respectively. We also adopt a standard data augmentation scheme that widely used for these two datasets (He et al. 2016; Huang et al.) 2017). For all models, the additional diver- sity term is applied on top the last intermediate layer. For the hyperparameters: The loss weight is chosen from (0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01] for both approach and De-
20

---

Under review as a conference paper at ICLR 2022
cov and in the radial basis function is chosen from /0.01,0.1.3) 10, 50, 100). For each approach, the model with the best validation performance used in the test phase. The average errors over three random seed are reported.
E.6 WEIGHT VS FEATURE DIVERSITY
Weight diversity is an active field of research which applies a diversity regularizer on top of the weights instead of the representation. Here, we compare the performance of two weight diversity approaches, i.e., (Yu et al. 2011 Xie et al. 2015b), against activation diversity approaches. Results are reported in As can be seen, activation-basee diversity leads superior results compared the weight-based.
Table 8: Performance of ResNet50 with different diversity strategies on ImageNet dataset
Method
Top-1 Errors
Weight diversity
Yu al.
2011)
25.08
Xie ct al.
2015b)
25.24
DeCov
23.62
Activation diversity
Ours(direct)
23.75
Ours(det)
23.62
Ours(logdet)
23.64
21


Reviews:
[{"review_id": "s2vrTUf87jx", "reviewer": "Reviewer_WnGv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My review on this paper is mixed. The main concern is on the theory side. \n\nThe Assumptions 1 seem way too strong so that the conclusion of Theorem 1 is not interesting any more. In particular there may be some issues with the last line of assumption in assumptions 1. \n\n\nFirst that line of assumption is not stated clearly. \n\na) There is a minor notation issue on the indexing i and j. They should be m and n instead?\n\n\nb) Is this an argument for all w or for a fixed w or for a randomized w? This needs to be explicitly stated.\n\nc) From the proof I turn to believe the authors are assuming that with probability \\tau the inequality holds for all ws in the last line of assumption. I would say, for a fixed w, such assumptions may make some sense. Now to prove the theorem 1 we need that assumption to hold for all ws with probability \\tau. And we have an infinite number of ws. That assumption is almost as strong as the original Rademacher bound on the model generalization. As a consequence the theorem 1 is not so informative to me.\n\n\nOn the other hand, the empirical side does show that diversifying the within-layer output may help improve models’ performance. This is an interesting empirical observation even though I do not quite buy the authors' claim on the connection to the Rademacher bound.\n", "summary_of_the_paper": "The paper proposes an interesting angle on the generalization of neural networks. Through proving a new generalization bound that has an internal distance term, the authors argue the neural networks’ generalization performance is related to the internal diversity of the neurons within each layer. As such, some regularization terms are introduced to encourage the internal ``diversity” within each layer. The effectiveness is verified empirically through experiments on a series of data sets including CIFAR and ImageNet. \n", "summary_of_the_review": "Overall I feel this is a borderline paper with a mixed feedback from me. It has some interesting thoughts and empirical results with come concerns on the theoretical justifications.\n\n\n#######Review after reading the authors' feedback########\n\n\nI would like to thank the authors for their explanation and the revised draft. Still I feel the assumption in the theoretical justification is kind of strong even though similar assumptions have been used in some previous work. I would keep the ratings unchanged. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rzF4q4tq3zs", "reviewer": "Reviewer_LXoK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This work has both a theoretical analysis of the proposed diversity regularizer and an empirical study of the effectiveness of the regularizer on existing models.\n\nWeaknesses:\n1. This work appears similar to existing diversity promoting regularizers, especially Xie et al 2015b (https://arxiv.org/pdf/1511.07110.pdf). While the diversity regularizer proposed here is applied to the neural activations instead of the weights, there seems to be no intuition/justification for why a diversity regularizer over activations is preferred over a diversity regularizer over weights, other than this sentence \"... Decov, yielded superior empirical performance; however, it lacks theoretical proof. In this paper, we closed this gap ...\" (page 7, first paragraph). I think this work would be stronger if it can either make a theoretical argument of why diversity regularization over activations is better, or make an empirical argument by comparing to Xie et al 2015b as well as other diversity-promoting works.\n\n2. This work aims at regularizing neural networks, so more broadly it would be nice to compare to other regularization methods such as dropout and MC dropout.\n\n3. Theoretically, I am not convinced that a better generalization bound always leads to a better performance, for several reasons: first, while the generalization error decreases with regularization, the approximation ability gets worse (i.e., the model cannot fit the training data as well), and this part is not analyzed in this paper. Second, the generalization bound seems to be very loose upon inspecting the constants: for example, $C_5$ in Eq. (8) is on the order of $C_1C_3$, with $C_1$ being the max norm of the input and $C_3$ being the max norm of the weight of the hidden layer. Lastly, a lower generalization error doesn't justify a regularizer. For example, if we flip the direction of the regularizer, i.e., regularize the activations to be less diverse instead of more diverse, the Rademacher complexity also decreases since the function family gets more constrained, but intuitively we don't want to use such a regularizer.\n\n4. Empirically, this work led to \"0.36%−0.54% improvement on CIFAR10 and1.86%−1.94% on CIFAR100\" (page 7, under table 1), which does not seem very significant. Besides, this work proposed multiple versions of diversity regularizer and presented results for all of them, while a more reasonable thing to do is to select a version on the validation set and then present the test result of that one only. Besides, from Fig. 1, the accuracy seems to be sensitive to hyperparameters.\n\n5. Computationally, the proposed regularizer seems expensive. For both the determinant and log determinant variants, the complexity of $O(C^3)$ with $C$ being the hidden size. I wonder if it's practical to apply this regularizer to larger networks with more hidden units.\n\nMinor Issues:\n1. \"was empirically been proven to\" (page 2, second paragraph)\n2. Eq. (14), missing sum w.r.t. j?\n3. Fig. 1, the legend appears small.\n\nSuggestions:\n1. It would be interesting to see the effectiveness of the proposed regularizer under different amounts of training data to better understand its behavior.", "summary_of_the_paper": "This paper proposes to use a regularizer to encourage the diversity of activations within each layer of deep neural networks. The contribution of this paper is two-fold: theoretically, the authors proved that more diverse activations correspond to a lower generalization error bound due to a lower Rademacher complexity; practically, this work showed that the proposed diversity regularizer led to better performance on several image classification tasks.", "summary_of_the_review": "Overall, this is well-structured work with both theoretical analyses and empirical results. However, this work did not provide a justification of why diversifying activations is better than diversifying weights, and it did not compare to commonly used regularization techniques such as dropout and other diversity-promoting works. Therefore, I am not recommending the acceptance of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iplCeK4Lm4Q", "reviewer": "Reviewer_LPWQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*The purpose of the bound*\n\nThe authors derive an upper bound on Rademacher complexity. However, as a reader I am not convinced of the importance/utility of these bounds:\nThe bounds depend on upper bounds on norms of the weights. The norms are distribution dependent. Further, such norms (in unregularized SGD and SGD with weight decay) have been shown to grow with the amount of data (Nagarajan and Kolter ‘19). Thus it is not even clear if the bound decreases with the amount of data when considering classifiers that are obtained with the proposed algorithm. Do the norms grow with the data? If so, how fast relative to the inter-layer activation diversity term?\nDemonstrating that the bound on Rademacher is tight for some distributions would be another way to demonstrate that the bounds are interesting.\n\nThe authors talk about studying generalization starting in the abstract and throughout the text, e.g.: “..we analyse the effect of the within-layer activation diversity on the generalization error bound of neural network.”.\n\nI see multiple inaccuracies with this and related statements: the authors provide a bound, rather than study an effect of within-layer activation diversity -- the effect would be totally different on a different upper bound. And these are just upper bounds, after all.\n\nI also wanted to note that what is actually provided in the paper is an estimation error bound, not a generalization error bound. I am guessing the authors arrived at the estimation error by starting from a Rademacher-based risk bound. But there is no mention of that. Why not state a generalization bound instead?\n\nHere is another statement (out of several) that is very misleading and incorrect (Section 3, first sentence):\n\n“As shown in the previous section, promoting diversity of activations within a layer can lead to tighter generalization bound and can theoretically decrease the gap between the empirical and the true risks.”\n\nWhere are the results showing that promoting diversity of activations decreases the gap between the risk and empirical risk? Also, the bounds are tighter than what? There is no comparison to any other bounds.\n\nJust after Eq (3), f* is defined as argmin_f L(f). What class is the minimization performed over?\n\n\n*How does activation diversity connect to the rest of the literature*\n\nThe authors discuss other approaches to increasing weight, activation, etc., diversity. However, since they argue that increased diversity improves generalization, it would be interesting to see a comparison to other properties that are linked to improvements in generalization. Probably one of the most recently discussed properties is the flatness of the minima. Are these complementary properties? Do they relate in any way? (see, e.g., work on Sharpness Aware minimization by Foret et al, Computing Nonvacuous Generalization Bounds by Dziugaite et al, Sharp minima work by Keskar et al., and many others)\n\nWhat exactly is “Standard Training” in Section 5? Is it regularized in any way? How would the performance of the proposed algorithm compare to various other regularization methods? \n\n*The efficiency*\n\nIn my opinion the real contribution of the paper is the algorithm, not the bound on estimation error. I do not see any discussion around the computational costs or training times of the algorithm.\n\n*Hyperparameter settings*\n\nThe algorithm requires tuning regularization-related hyperparameters. How are those chosen exactly? Do the authors have a held-out set? Based on Fig 1, it seems that the wrong values of these hyperparameters may wipe out all the improvements in generalization presented in Table 1.\n\nAlso, the discussion of Figure 1 and more generally section 5 once again suggest that one may expect to see a smaller generalization error gap when optimizing for the inter-layer activation diversity. There is no such theoretical connection made in the paper.\n\nHow about other hyperparameters? How are those set? Are they all standard or have they been changed at all?\n\n*Ensembling* \n\nThe authors several times relate activation diversity to functional diversity (introduction, section 4). I do not quite understand the connection. I can easily construct cases how improved activation diversity in individual classifiers would result in a poor ensemble. Can the authors explain the connection?\n\n\nMinor: The loss set  in Lemma 1 is undefined.\n", "summary_of_the_paper": "The authors prove an upper bound on the estimation error that depends on pairwise distances between hidden unit values per layer. The bound itself is quite likely very loose in standard deep learning settings due to the dependence on various norms of the weights. Loosely inspired by the bound, the authors then propose an optimization algorithm with a particular regularizer that encourages within-layer activation value diversity. The empirical results suggest that the proposed algorithm boosts the performance in most regimes studied (CIFAR 10, CIFAR100 trained on ResNets, Dense nets). The boost is minimal for standard datasets, but it increases when the labels are randomized. Compared to standard empirical risk minimization, a small boost in test accuracy on transfer learning tasks is also reported.", "summary_of_the_review": "To reconsider my score, I would like to see the language around implications of the bound clarified throughout the text (in response to my comments under “the purpose of the bound”). Currently there is a huge disconnect between what has been shown and what the claims are.\n\nAlso, since the main contribution is the algorithm, I would like to see more evidence towards the algorithm potentially being adopted in practice. What are the computational costs? How do they compare? How does the algorithm compare to other recently proposed algorithms that were derived to improve generalization (like sharpness aware minimization)? How does the algorithm compare to other regularization techniques?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MAdvbBTgu3o", "reviewer": "Reviewer_vc2s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Definition:\n\n- In the definition of the diversity, there is a $\\tau$. What does that mean? What is the probability space? Does it mean when you sample a point from the input distribution, with probability $\\tau$ you have the lower bound? But you should also quantify based on the weights. Am I missing something? For instance when you have a multilayer, the events are not independent so the statement of Theorem 4 is difficult to understand for me. \n\n- It would be nice if the authors can provide some sufficient conditions under which we can have the desired lower bound on the diversity. \n\nMotivation behind the diversity method:\n- In the definition of the diversity we want that the output of activations are not close to each other. However, is it sufficient? what if the weight vectors of the next layer basically \"kill\" this diversity? In general, what is the impact of the weights on the diversity?\n\nComparison with Dropout method:\n- Empirical comparison: My understanding of the dropout is that it also has a very similar impact as your method on learning. I appreciate it if you could compare the performance of your method with the dropout method.\n-Computation Comparison: \nWhat is the difference of the dropout and your method in terms of computational cost?\n\nTheorem 4: \n\nWhat is the intuition behind the Tau^p? what is the dependence of your bound on depth and width and how does it compare with Golowich et al paper?\n\nA Note regarding the references:\n\nIn many places in the manuscript, the citations do not match with the content. \nFor instance on Page 2, before definition 1, almost all of the citations are wrong. PAC learning is not from Hanneke'16. VC dimension is not from those papers you cited, and the same for Rademcher complexity.\n\n", "summary_of_the_paper": "In this paper, the authors propose a regularization technique encourage the \"activation diversity\". Specifically, they design a within-layer loss that add penalty to the similar neurons with the same activation pattern. They also showed that encouraging the within-layer diversity can be used to control the Rademacher complexity of model. ", "summary_of_the_review": "Very Interesting idea but the presentation can be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DLOVJ3vb7T", "reviewer": "Reviewer_dHr7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Theoretically-motivated and simple (in a good way) technique to encourage activation diversity.\n* Empirically evaluated a wide set of benchmarks and tasks.\n* The significant generalization gap reduction in Table 2 (from 2.77 -> 1.07) is intruiging and worth additional study.\n\nWeaknesses\n\nThe mathematical section was difficult to follow for this reviewer, because the theorems and lemmas are presented with minimal description or intuition of how they were derived and connected. The paper could be improved by only describing one of the cases (e.g. single hidden-layer network) in the main text, but in greater detail, and leaving the rest to the appendix. As an example in Theorem 3, its difficult to reason about the effect of d_min on the bounds, as J appears in two terms. The explanation would benefit from a figure showing how the bounds change with d_min.\n\nTo provide a more direct link between the theory, loss term, and empirical results, the paper could be improved by visualizing the activation diversity across layers -- given that the diversity term is only applied in one layer, what is its broader effect?\n\nEmpirically, the results are mixed. On CIFAR10/CIFAR100, the apporach improves over DeCov. For ImageNet, the approach does not yield better results than previous work (DeCov) on one model, but improves on a different model. With MLP-mixer, some methods improve over DeConv, some do not. It should be noted that the DeCov results are not cited from the original paper, but the authors own implementations (because DeCov was only evaluated on AlexNet at the time). \n\nMethodological questions:\n* how is the author applying ResNet-50 on CIFAR10 and CIFAR100? The original paper (He et al, 2016) uses a slightly modified architecture in ResNet-56.\n* given that DeCov results are not from the original paper, how were its hyperparameters tuned? e.g. in Table 7 in the Appendix, I see tuning of the hyperparameters for the author's method, but not for DeCov.", "summary_of_the_paper": "The authors propose a theoretically-grounded technique to improve neural network generalization. The loss is augmented with a term that encourages units to have a more diversity in its activations. More specifically, the authors use a radial basis function to measure pairwise similarity between activations and proposed loss terms based on either the sum or the determinant of the similarity matrix.\n\nBased on experiments in several classification models on CIFAR10, CIFAR100, and ImageNet, models trained with this additional loss term achieve higher test accuracy, and also reduce the generalization gap between train and test. ", "summary_of_the_review": "The proposed method is appealingly simple, but yields mixed results. The paper's presentation and analysis does not present a strong link between the theory and the empirical section, which leaves me to recommend not accepting the paper in its current form. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes an approach to encourage within-layer diversity in neuron activations, and derive a generalization bound meant to motivate their approach.

None of the reviewers support the acceptance of this work, despite the authors' detailed rebuttals, with the majority of reviewers confirming their preference for rejection following the author response. Many raised concerns regarding the value of the accompanying theory. The empirical results demonstrated by the proposed regularizer were also not judged to be sufficiently compelling to compensate for the shortfall on the theory side.

I unfortunately could not find a good reason to dissent from the reviewers majority opinion, and therefore also recommend rejection at this time.
Decision: Reject