Question: Where are the results showing that promoting diversity of activations decreases the gap between the risk and empirical risk? Also, the bounds are tighter than what?
Evidence: We start by evaluating our proposed diversity approach on two image datasets: CIFAR10 and CI-FAR100. We use our approach on three state-of-the-art CNNs: ResNext-29-08-16: we consider the standard ResNext Model with a 29-layer architecture, a cardinality of 8, and a width of 16. DenseNet-12: we use DenseNet with the 40layer architecture and a growth rate of 12. ResNet50: we consider the standard ResNet model with 50 layers. We compare against the standard networks 2 as well as networks trained with DeCov diversity strategy. For each approach the hyperparameters are selected based on the validation set. The full experimental setup is presented in the Appendix. We report the average performance over three random seeds. We note that, compared to the standard approach, employing a diversity strategy generally boosts the results for all the three models and that our approach consistency outperforms both competing For the hyperparameters, we use λ = 0.001 and γ = 10. The full experimental setup is presented in the Appendix. Table reports the test errors of the different diversity strategies. To study the effect of diversity on the generalization gap, we also report the final training errors and the generalization gap, i.e., training accuracy -test accuracy. As it can be seen, diversity (our approach and DeCov) reduces the test error of the model and yields a better performance. We note that, in accordance with our theoretical findings in Section 2, using diversity indeed reduces overfitting and decreases the empirical generalisation gap of neural networks. In fact, our logdet variant reduces the empirical generalization gap of the model by 1.8% compared to the standard approach. We note that our approach has a small additional time cost. For example for ResNet50, our direct, det and logdet variants take only 0.29%, 0.39%, and 0.49% extra training time, respectively.
Figure: Sensitivity analysis of λ and γ on both the model accuracy and its generalization ability using ResNet50 trained on ImageNet.
Sensitivity analysis: To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: γ, which is the RBF parameter used to measure the pairwise similarity between two units, and λ, which controls the contribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure and in supplementary material. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparameters values. Moreover, we note that global modeling of diversity, i.e., det and logdet variants, yield tighter generalization gaps compared to the non-global direct approach. It is worth noting that Figure shows that there is a trade-off between the generalization gap and the final error. Emphasizing diversity and using a high weight for the diversity term significantly decreases the generalization gap, but this damages the performance of the model compared to the standard approach. For lower values of λ, the model is able to significantly outperform the standard approach on both the test error and the generalization gap.
Connection to Theory: In Section 2, we provided theoretical bounds for the generalization errors of neural networks, which are inversely proportional to the diversity term, d min , the lower bound of
To show that the proposed regularizer indeed improves diversity, we track the empirical average of the aforementioned variable, i.e.,
2 during the training. The results for both the standard approach and the logdet variant of our approach are reported in Figure. As it can be seen, our regularizer yields in higher diversity which reduces overfitting and leads to better generalization. Compatibility with Dropout: Dropout) is a popular regularization technique that, similarly to our approach, is applied on top of the layer output and has been known to improve generalization and reduce overfitting. Here, we evaluate the compatibility of our approach with Dropout. We add a Dropout regularizer with 20% rate on top the last intermediate representation layer of both ResNet50 and Wide-ResNet50. The results are reported in Table. We note that adding a diversity regularizer alongside the Dropout consistently yields lower error rates compared to only Dropout except for the det variant on Wide-ResNet50. For Wide-ResNet50, the combination of our logdet variant and Dropout leads to 0.53%, 0.25%, and 0.38% improvement compared to standard approach, Dropout only, and our logdet variant only, respectively.
Beyond CNN models, we also evaluate the performance of our diversity strategy on modern attention-free, multi-layer perceptron (MLP) based models for image classification. Such models are known to exhibit high overfitting and require regularization. We evaluate how diversity affects the accuracy of such models on CIFAR10. In particular, we conduct a simple experiment using two models: MLP-Mixer, gMLP with four blocks each. The full description of the experimental setup is presented in the Appendix. The results in Table show that employing a diversity strategy can indeed improve the performance of these models thanks to its ability to learn rich and robust representation of the input.
Transfer learning: Beyond standard classification, the proposed approach can be useful, e.g., in transfer learning, where the main goal is to 'transfer' previously learned representation to solve new tasks. Thus, learning a rich and diverse representation is beneficial and can lead to better transferability. To demonstrate this, we conduct another experiment, where we use ImageNet-pretrained ResNet50 models with the different diversity approaches and we finetune them to CIFAR10 and CIFAR100. The results are reported in Table. As it can be seen, employing a diversity strategy helps in the transfer learning context and leads consistently to lower error rates. For example, the log variant of our approach leads to 0.94% and 1.27% gains on CIFAR10 and CIFAR100, respectively. To further demonstrate the usefulness of promoting diversity, we test the robustness of our approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and not generalize well to the test set. Enforcing diversity can lead to better and richer representation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFAR10 and CIFAR100 using ResNet50. The results are reported in Table. As it can be seen, in the presence of noise, the gap between the standard approach and diversity (Decov and ours) increases. For example, our logdet variant boosts the results by 1.71% and 3.59% on CIFAR10 and CIFAR100 with 40% noise, respectively.
We start by evaluating our proposed diversity approach on two image datasets: CIFAR10 and CI-FAR100. We use our approach on three state-of-the-art CNNs: ResNext-29-08-16: we consider the standard ResNext Model with a 29-layer architecture, a cardinality of 8, and a width of 16. DenseNet-12: we use DenseNet with the 40layer architecture and a growth rate of 12. ResNet50: we consider the standard ResNet model with 50 layers. We compare against the standard networks 2 as well as networks trained with DeCov diversity strategy. For each approach the hyperparameters are selected based on the validation set. The full experimental setup is presented in the Appendix. We report the average performance over three random seeds. We note that, compared to the standard approach, employing a diversity strategy generally boosts the results for all the three models and that our approach consistency outperforms both competing For the hyperparameters, we use λ = 0.001 and γ = 10. The full experimental setup is presented in the Appendix. Table reports the test errors of the different diversity strategies. To study the effect of diversity on the generalization gap, we also report the final training errors and the generalization gap, i.e., training accuracy -test accuracy. As it can be seen, diversity (our approach and DeCov) reduces the test error of the model and yields a better performance. We note that, in accordance with our theoretical findings in Section 2, using diversity indeed reduces overfitting and decreases the empirical generalisation gap of neural networks. In fact, our logdet variant reduces the empirical generalization gap of the model by 1.8% compared to the standard approach. We note that our approach has a small additional time cost. For example for ResNet50, our direct, det and logdet variants take only 0.29%, 0.39%, and 0.49% extra training time, respectively.
Figure: Sensitivity analysis of λ and γ on both the model accuracy and its generalization ability using ResNet50 trained on ImageNet.
Sensitivity analysis: To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: γ, which is the RBF parameter used to measure the pairwise similarity between two units, and λ, which controls the contribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure and in supplementary material. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparameters values. Moreover, we note that global modeling of diversity, i.e., det and logdet variants, yield tighter generalization gaps compared to the non-global direct approach. It is worth noting that Figure shows that there is a trade-off between the generalization gap and the final error. Emphasizing diversity and using a high weight for the diversity term significantly decreases the generalization gap, but this damages the performance of the model compared to the standard approach. For lower values of λ, the model is able to significantly outperform the standard approach on both the test error and the generalization gap.
Connection to Theory: In Section 2, we provided theoretical bounds for the generalization errors of neural networks, which are inversely proportional to the diversity term, d min , the lower bound of
To show that the proposed regularizer indeed improves diversity, we track the empirical average of the aforementioned variable, i.e.,
2 during the training. The results for both the standard approach and the logdet variant of our approach are reported in Figure. As it can be seen, our regularizer yields in higher diversity which reduces overfitting and leads to better generalization. Compatibility with Dropout: Dropout) is a popular regularization technique that, similarly to our approach, is applied on top of the layer output and has been known to improve generalization and reduce overfitting. Here, we evaluate the compatibility of our approach with Dropout. We add a Dropout regularizer with 20% rate on top the last intermediate representation layer of both ResNet50 and Wide-ResNet50. The results are reported in Table. We note that adding a diversity regularizer alongside the Dropout consistently yields lower error rates compared to only Dropout except for the det variant on Wide-ResNet50. For Wide-ResNet50, the combination of our logdet variant and Dropout leads to 0.53%, 0.25%, and 0.38% improvement compared to standard approach, Dropout only, and our logdet variant only, respectively.
Beyond CNN models, we also evaluate the performance of our diversity strategy on modern attention-free, multi-layer perceptron (MLP) based models for image classification. Such models are known to exhibit high overfitting and require regularization. We evaluate how diversity affects the accuracy of such models on CIFAR10. In particular, we conduct a simple experiment using two models: MLP-Mixer, gMLP with four blocks each. The full description of the experimental setup is presented in the Appendix. The results in Table show that employing a diversity strategy can indeed improve the performance of these models thanks to its ability to learn rich and robust representation of the input.
Transfer learning: Beyond standard classification, the proposed approach can be useful, e.g., in transfer learning, where the main goal is to 'transfer' previously learned representation to solve new tasks. Thus, learning a rich and diverse representation is beneficial and can lead to better transferability. To demonstrate this, we conduct another experiment, where we use ImageNet-pretrained ResNet50 models with the different diversity approaches and we finetune them to CIFAR10 and CIFAR100. The results are reported in Table. As it can be seen, employing a diversity strategy helps in the transfer learning context and leads consistently to lower error rates. For example, the log variant of our approach leads to 0.94% and 1.27% gains on CIFAR10 and CIFAR100, respectively. To further demonstrate the usefulness of promoting diversity, we test the robustness of our approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and not generalize well to the test set. Enforcing diversity can lead to better and richer representation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFAR10 and CIFAR100 using ResNet50. The results are reported in Table. As it can be seen, in the presence of noise, the gap between the standard approach and diversity (Decov and ours) increases. For example, our logdet variant boosts the results by 1.71% and 3.59% on CIFAR10 and CIFAR100 with 40% noise, respectively.
Table 2: Performance of different models with different diversity strategies on ImageNet dataset
****Table 2 reports the test errors of the different diversity strategies
Answer: This is shown theoretically in Theorem 1 and empirically in Table 2. The main idea of this paper is that by promoting the diversity and penalizing redundancy between the neurons within the same layer, we can improve the generalization. The appearance of the term related to the layer output diversity, i.e.,d_{min} , in the generalization bound in Theorems 1-4 provides theoretical support for the proposed idea. Note that we are not comparing our bound against prior bounds. We just derived a novel upperbound depending on the diversity. For example in Theorem 1, we derived a bound for the generalization gap, i.e.,   L(\hat{f}) - L(f^*) , which is inversely proportional to d_{min}. This means when d_{min} is higher, the bound gets smaller (tighter) and as the result the gap gets smaller. Such theoretical reasoning has been applied before, see for example [1-4].
 Xie, P., Deng, Y. and Xing, E., 2015. On the generalization error bounds of neural networks under diversity-inducing mutual angular regularization. arXiv preprint arXiv:1511.07110.
 Gouk, H., Hospedales, T.M. and Pontil, M., 2020. Distance-Based Regularisation of Deep Networks for Fine-Tuning. ICLR 2021
 Cortes, C., Mohri, M. and Storcheus, D., 2019. Regularized gradient boosting. Advances in Neural Information Processing Systems.
 Wan, L., Zeiler, M., Zhang, S., Le Cun, Y. and Fergus, R., 2013, May. Regularization of neural networks using dropconnect. In International conference on machine learning (pp. 1058-1066). PMLR.