Question: What exactly is “Standard Training” in Section 5?
Evidence: We start by evaluating our proposed diversity approach on two image datasets: CIFAR10 and CI-FAR100. We use our approach on three state-of-the-art CNNs: ResNext-29-08-16: we consider the standard ResNext Model with a 29-layer architecture, a cardinality of 8, and a width of 16. DenseNet-12: we use DenseNet with the 40layer architecture and a growth rate of 12. ResNet50: we consider the standard ResNet model with 50 layers. We compare against the standard networks 2 as well as networks trained with DeCov diversity strategy. For each approach the hyperparameters are selected based on the validation set. The full experimental setup is presented in the Appendix. We report the average performance over three random seeds. We note that, compared to the standard approach, employing a diversity strategy generally boosts the results for all the three models and that our approach consistency outperforms both competing For the hyperparameters, we use λ = 0.001 and γ = 10. The full experimental setup is presented in the Appendix. Table reports the test errors of the different diversity strategies. To study the effect of diversity on the generalization gap, we also report the final training errors and the generalization gap, i.e., training accuracy -test accuracy. As it can be seen, diversity (our approach and DeCov) reduces the test error of the model and yields a better performance. We note that, in accordance with our theoretical findings in Section 2, using diversity indeed reduces overfitting and decreases the empirical generalisation gap of neural networks. In fact, our logdet variant reduces the empirical generalization gap of the model by 1.8% compared to the standard approach. We note that our approach has a small additional time cost. For example for ResNet50, our direct, det and logdet variants take only 0.29%, 0.39%, and 0.49% extra training time, respectively.
Figure: Sensitivity analysis of λ and γ on both the model accuracy and its generalization ability using ResNet50 trained on ImageNet.
Sensitivity analysis: To further investigate the effect of the proposed diversity strategy, we conduct a sensitivity analysis using ImageNet on the hyperparameters of our methods: γ, which is the RBF parameter used to measure the pairwise similarity between two units, and λ, which controls the contribution of the global diversity term to the global loss. We analyse the effect of the two parameters on both the final performance of the models and its generalization ability. The analysis is presented in Figure and in supplementary material. As it can be seen, promoting the within-layer diversity consistently reduces overfitting and decreases the generalization gap for most of the hyperparameters values. Moreover, we note that global modeling of diversity, i.e., det and logdet variants, yield tighter generalization gaps compared to the non-global direct approach. It is worth noting that Figure shows that there is a trade-off between the generalization gap and the final error. Emphasizing diversity and using a high weight for the diversity term significantly decreases the generalization gap, but this damages the performance of the model compared to the standard approach. For lower values of λ, the model is able to significantly outperform the standard approach on both the test error and the generalization gap.
Connection to Theory: In Section 2, we provided theoretical bounds for the generalization errors of neural networks, which are inversely proportional to the diversity term, d min , the lower bound of
To show that the proposed regularizer indeed improves diversity, we track the empirical average of the aforementioned variable, i.e.,
2 during the training. The results for both the standard approach and the logdet variant of our approach are reported in Figure. As it can be seen, our regularizer yields in higher diversity which reduces overfitting and leads to better generalization. Compatibility with Dropout: Dropout) is a popular regularization technique that, similarly to our approach, is applied on top of the layer output and has been known to improve generalization and reduce overfitting. Here, we evaluate the compatibility of our approach with Dropout. We add a Dropout regularizer with 20% rate on top the last intermediate representation layer of both ResNet50 and Wide-ResNet50. The results are reported in Table. We note that adding a diversity regularizer alongside the Dropout consistently yields lower error rates compared to only Dropout except for the det variant on Wide-ResNet50. For Wide-ResNet50, the combination of our logdet variant and Dropout leads to 0.53%, 0.25%, and 0.38% improvement compared to standard approach, Dropout only, and our logdet variant only, respectively.
Beyond CNN models, we also evaluate the performance of our diversity strategy on modern attention-free, multi-layer perceptron (MLP) based models for image classification. Such models are known to exhibit high overfitting and require regularization. We evaluate how diversity affects the accuracy of such models on CIFAR10. In particular, we conduct a simple experiment using two models: MLP-Mixer, gMLP with four blocks each. The full description of the experimental setup is presented in the Appendix. The results in Table show that employing a diversity strategy can indeed improve the performance of these models thanks to its ability to learn rich and robust representation of the input.
Transfer learning: Beyond standard classification, the proposed approach can be useful, e.g., in transfer learning, where the main goal is to 'transfer' previously learned representation to solve new tasks. Thus, learning a rich and diverse representation is beneficial and can lead to better transferability. To demonstrate this, we conduct another experiment, where we use ImageNet-pretrained ResNet50 models with the different diversity approaches and we finetune them to CIFAR10 and CIFAR100. The results are reported in Table. As it can be seen, employing a diversity strategy helps in the transfer learning context and leads consistently to lower error rates. For example, the log variant of our approach leads to 0.94% and 1.27% gains on CIFAR10 and CIFAR100, respectively. To further demonstrate the usefulness of promoting diversity, we test the robustness of our approach in the presence of label noise. In such situations, standard neural network tend to overfit to the noisy sample and not generalize well to the test set. Enforcing diversity can lead to better and richer representation attenuating the effect of noise. To show this, we performed additional experiments with label noise (20% and 40%) on CIFAR10 and CIFAR100 using ResNet50. The results are reported in Table. As it can be seen, in the presence of noise, the gap between the standard approach and diversity (Decov and ours) increases. For example, our logdet variant boosts the results by 1.71% and 3.59% on CIFAR10 and CIFAR100 with 40% noise, respectively.
We use our approach on three state-of-the-art CNNs: ResNext29-08-16: we consider the standard ResNext Model (Xie et al., 2017c) with a 29-layer architecture, a cardinality of 8, and a width of 16. DenseNet-12: we use DenseNet (Huang et al., 2017) with the 40- layer architecture and a growth rate of 12. ResNet50: we consider the standard ResNet model (He et al., 2016) with 50 layers. We compare against the standard networks2 as well as networks trained with DeCov diversity strategy (Cogswell et al., 2016).
Answer: For standard approach, the only difference is not using an additional diversity loss. The remaining regularizers, e.g., data augmentation, weight decay etc, are all applied as specified per-experiment. we clarified this in the updated manuscript.