Question: How does activation diversity connect to the rest of the literature？
Evidence: Neural networks are a powerful class of non-linear function approximators that have been successfully used to tackle a wide range of problems. They have enabled breakthroughs in many tasks, such as image classification, speech recognition, and anomaly detection. Formally, the output of a neural network consisting of P layers can be defined as follows:
where φ i (.) is the element-wise activation function, e.g., ReLU or Sigmoid, of the i th layer and W = {W 1 , . . . , W P } are the corresponding weights of the network. The parameters of f (x; W) are optimized by minimizing the empirical loss:
where l(•) is the loss function, and {x i , y i } N i=1 are the training samples and their associated groundtruth labels. The loss is minimized using the gradient decent-based optimization coupled with backpropagation.
However, neural networks are often over-parameterized, i.e., have more parameters than data. As a result, they tend to overfit to the training samples and not generalize well on unseen examples. While research on double descent shows that over-parameterization does not necessarily lead to overfitting, avoiding overfitting has been extensively studied and various approaches and strategies have been proposed, such as data augmentation, regularization, and Dropout, to close the gap between the empirical loss and the expected loss. Diversity of learners is widely known to be important in ensemble learning and, particularly in deep learning context, diversity of information extracted by the network neurons has been recognized as a viable way to improve generalization. In most cases, these efforts have focused on making the set of weights more diverse. However, diversity of the activations has not received much attention. Here, we argue that due to the presence of non-linear activations, diverse weights do not guarantee diverse feature representation. Thus, we propose applying the diversity on top of feature mapping instead of the weights.
To the best of our knowledge, only have considered diversity of the activations directly in the neural network context. They proposed an additional loss term using cross-covariance of hidden activations, which encourages the neurons to learn diverse or non-redundant representations. Their proposed approach, known as DeCov, was empirically proven to alleviate overfitting and to improve the generalization ability of neural network, yet a theoretical analysis to prove this has so far been lacking. Moreover, modeling diversity as the sum of the pairwise cross-covariance, it can capture only the pairwise diversity between components and is unable to capture the "higher-order diversity".
In this work, we start by theoretically showing that the within-layer activation diversity boosts the generalization ability of neural networks and reduces overfitting. Moreover, we propose a novel approach to encourage activation diversity within a layer. We propose complementing the 'betweenlayer' feedback with additional 'within-layer' feedback to penalize similarities between neurons on the same layer. Thus, we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer. We propose three variants for our approach that are based on different global diversity definitions.
Our contributions in this paper are as follows:
• Theoretically, we derive novel generalization bounds for neural networks depending on the within-layer activation diversity. As shown in Section 2, we express the upper-bound of the estimation error as a function of the diversity factor. Thus, we provide theoretical evidence that the within-layer activation diversity helps reduce the generalization error. • Methodologically, we propose a new approach to encourage the 'diversification' of the layers' output feature maps in neural networks. The proposed approach has three variants.
The main intuition is that by promoting the within-layer activation diversity, neurons within a layer learn distinct patterns and, thus, increase the overall capacity of the model. • Empirically, we show that the proposed within-layer activation diversification boosts the performance of neural networks. Experimental results on several tasks show that the proposed approach outperforms competing methods.
Diversity of learners is widely known to be important in ensemble learning (Li et al., 2012; Yu et al., 2011) and, particularly in deep learning context, diversity of information extracted by the network neurons has been recognized as a viable way to improve generalization (Xie et al., 2017a; 2015b). In most cases, these efforts have focused on making the set of weights more diverse (Yang et al.; Malkin & Bilmes, 2009). However, diversity of the activations has not received much attention. Here, we argue that due to the presence of non-linear activations, diverse weights do not guarantee diverse feature representation. Thus, we propose applying the diversity on top of feature mapping instead of the weights. To the best of our knowledge, only Cogswell et al. (2016) have considered diversity of the activations directly in the neural network context. They proposed an additional loss term using cross-covariance of hidden activations, which encourages the neurons to learn diverse or non-redundant representations. Their proposed approach, known as DeCov, was empirically proven to alleviate overfitting and to improve the generalization ability of neural network, yet a theoretical analysis to prove this has so far been lacking. Moreover, modeling diversity as the sum of the pairwise cross-covariance, it can capture only the pairwise diversity between components and is unable to capture the ”higher-order diversity”. In this work, we start by theoretically showing that the within-layer activation diversity boosts the generalization ability of neural networks and reduces overfitting. Moreover, we propose a novel approach to encourage activation diversity within a layer. We propose complementing the ’betweenlayer’ feedback with additional ’within-layer’ feedback to penalize similarities between neurons on the same layer. Thus, we encourage each neuron to learn a distinctive representation and to enrich the data representation learned within each layer. We propose three variants for our approach that are based on different global diversity definitions.
Answer: The connections to related literature has been described in the introduction and the related work section. We added the references mentioned by the reviewer. As the proposed regularizer is an activation-based diversity approach, we focused on our experimental comparisons with prior activation-based diversity approaches, i.e., 'Decov'. We also now included in the updated manuscript dropout experiments and weight-based diversity.