Intraclass clustering: an implicit learning ability that regularizes DNNs
Authors: Simon Carbonnelle
Christophe De Vleeschouwer
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning
generalization
implicit regularization

Abstract: Several works have shown that the regularization mechanisms underlying deep neural networks' generalization performances are still poorly understood. In this paper, we hypothesize that deep neural networks are regularized through their ability to extract meaningful clusters among the samples of a class. This constitutes an implicit form of regularization, as no explicit training mechanisms or supervision target such behaviour. To support our hypothesis, we design four different measures of intraclass clustering, based on the neuron- and layer-level representations of the training data. We then show that these measures constitute accurate predictors of generalization performance across variations of a large set of hyperparameters (learning rate, batch size, optimizer, weight decay, dropout rate, data augmentation, network depth and width).

Published as a conference paper at ICLR 2021
INTRACLASS CLUSTERING: AN IMPLICIT LEARNING ABILITY THAT REGULARIZES DNNs
Simon Carbonnelle, Christophe De Vleeschouwer
FNRS research fellows
ICTEAM, Universit catholique de Louvain
Louvain-La-Neuve, Belgium
simon. www.o.ccrrooneele@@ma... com, abt.attooh...eeeeeecccoeerrrbbbb
ABSTRACT
Several works have shown that the regularization mechanisms underlying deep neural networks' generalization performances are still poorly understood (Neyshabur et al.) 2015; Zhang et al.) 2017). In this paper, hypothesize that deep neural networks are regularized through their ability to extract meaningful clusters among the samples of a class. This constitutes implicit form of regu- larization, as no explicit training mechanisms or supervision target such behaviour. design four different measures of intraclass cluster- ing, based on the neuron- and layer-level representations of the training data. We then show that these measures constitute accurate predictors of generalization per- formance across variations of large set of hyperparameters (learning rate, batch size, optimizer, weight decay, dropout rate, data augmentation, network depth and width).
To support our hypothesis,
INTRODUCTION
MNIST 11"
CIFARIO "airplane" "
1
1
Figure 1: standard image classification datasets, classes are typically composed multiple clus- ters of similarly looking images. We call intraclass clustering model's ability to differentiate such clusters despite their association to identical labels.
The generalization ability of deep neural networks remains largely unexplained. In particular, the traditional view that explicit forms of regularization (e.g. dropout, Ly-regularization, data augmen- tation) are the sole factors for generalization performance of state of the art neural networks has been experimentally invalidated (Neyshabur et al.) 2015 Zhang et al. 2017). Today's conventional wisdom rather conjectures the presence implicit forms regularization, emerging from the teractions between neural network architectures, optimization, and the inherent structure of the data itself (Arpit al. 2017).
One structural component that seems occur in most image classification datasets is the presence of multiple clusters amongst the samples of a class (or intraclass clusters, efr. Fi. The extraction of such structure in the context of supervised learning is not self-evident, today's standard training algorithms are designed to group samples from a class together, without any considerations for eventual intraclass clusters.

---

Published as a conference paper at ICLR 2021
This paper hypothesizes the identification of intraclass clusters emerges during supervised train- ing of deep neural networks, despite the absence of supervision or explicit training mechanisms targeting this behaviour. Moreover, our study suggests that this phenomenon improves the general- ization ability of deep neural networks, hence constituting an implicit form of regularization.
To verify our hypotheses, define four measures of intraclass clustering and inspect the correlation between those measures and a network's s generalization performance. These measures are designed capture intraclass clustering from four different perspectives, defined by the representation level (neuron vs. layer) and the amount of knowledge about the data's inherent structure (datasets with or without hierarchical labels). To evaluate these measures' predictive power, we train more than 500 models, varying standard hyperparameters in a principled way in order to generate a wide range of generalization performances. The measures are then evaluated qualitatively through visual in- spection of their relationship with generalization and quantitatively through the granulated Kendall rank-correlation coefficient introduced by [aing et (2020). Both evaluations reveal tight connec- tion between intraclass clustering measures and generalization ability, providing important evidence to support this work's hypotheses.
2 MEASURING INTRACLASS CLUSTERING IN INTERNAL REPRESENTATIONS
challenge of our work resides in measuring model's ability to differentiate intraclass clusters, without knowing which mechanisms underlie such ability. In such context, the design of multiple complementary measures (i) offers different perspectives that will help better characterize intraclass clustering mechanisms and (ii) reduces the risk that their potential correlation with generalization is induced by other phenomena independent of intraclass clustering. This section thus describes four measures of intraclass clustering, differing in terms of representation level (neuron layer) and the amount of knowledge about the data's inherent structure (datasets with or without hierarchical labels).
2.1 TERMINOLOGY AND NOTATIONS
D denotes the training dataset. Let / be the number of classes in the dataset D, denote the set of samples from class by C₁ with i E I = {1, 1..,,,,, /}. In the case of hierarchical labels, C; denotes the samples from subclass and S.(0) the samples from the superelass containing subclass We denote by N (1, ...,.N and -= 2, ..., L} the indexes of the N neurons and L layers of network respectively. Neurons are considered across all the layers of a network, not a specific layer. The methodology by which indexes are assigned neurons or layers does not matter. We further denote by meanjeur and mediançez the mean and median operations over the index j respectively. Moreover, meanfez corresponds to the mean of the top-k highest values, over the index j.
We call pre-activations (and activations) the values preceding (respectively following) the applica- tion of the ReLU activation function (Nair & Hinton 2010). In our experiments, batch normalization [loff & Szegedy 2015) applied before the ReLU, and pre-activation values are collected after batch normalization. In convolutional layers, a neuron refers to an entire feature map. The spatial dimensions of such neuron's srrevvactvsssonns are reduced through a global max pooling operation before applying our measures.
2.2 MEASURES BASED ON LABEL HIERARCHIES
The first two measures take advantage of datasets that include a hierarchy of labels. For example, CIFAR 100 organized into 20 superclasses (e.g. flowers) each comprising 5 subclasses (e.g. or- chids, poppies, roses, sunflowers, tulips). We hypothesize that these hierarchical labels reflect an inherent structure the data. In particular, we expect the subclasses to approximately correspond to different clusters amongst the samples of a superclass. Hence, measuring the extent by which a network differentiates subclasses when being trained on superclasses should reflect its ability to extract intraclass clusters during training.
2

---

Published as a conference paper at ICLR 2021
2.2.1 NEURON-LEVEL SUBCLASS SELECTIVITY
The first measure quantifies how selective individual neurons are for given subclass C, with respect to the other samples of the associated superclass Sa(i)- Here, strong selectivity means that the subclass C, can be reliably discriminated from the other samples of Sx(i) based on the neuron's pre- activation Let Pn,E and R be the mean and standard deviation of a neuron n's s's pre-activation values taken over the samples of set E. The measure is defined follows:
C1 me mean Crn,C, - Mn,S.co
Gn.C, + On,Scco
Since cannot expect all neurons of network to be selective for given subclass, we only con- sider the top-k most selective neurons. The measure thus relies on k neurons to capture the overall network's 's ability to differentiate each subclass. We noticed slightly better results when using the median operation instead of the mean to aggregate over the subclasses. We suspect this arises from outlier behaviour certain subclasses that we observe Section 4.5
2.2.2 LAYER-LEVEL SILHOUETTE SCORE
The second measure quantifies to what extent the samples of subclass close together relative to the other samples from the associated superclass in the space induced by layer's activations. In other words, measure to what degree different subclasses can be associated different clusters in the intermediate representations of network. We quantify this by computing the pairwise cosine distance on the samples of a superclass and applying the Silhouette score (Kaufman & Rousseeuw 2009) to assess the clustered structure of the subclasses. Let silhouette Sait), C:) be the mean silhouette score of subelass C, based on the activations d; of superclass Sa(o) in layer I, the measure is then defined as:
C2 = mediante meantec silhouette(ay, S,)) Ci)
2.3 MEASURES BASED ON VARIANCE
To establish the generality of our results, also design measures that can be applied in ab- sence of hierarchical labels. We hypothesize that the discrimination of intraclass clusters should be reflected by high variance in the representations associated class. If all the samples of class are mapped to close-by points in the neuron- or layer-level representations, it is likely that the neuron/layer did not identify intraclass clusters.
2.3.1 VARIANCE IN THE NEURON-LEVED REPRESENTATIONS OF THE DATA
The first variance measure is based standard deviations of neuron's pre-activations. If the standard deviation computed over the samples of a class is high compared to the standard deviation computed over the entire dataset, infer that the neuron has learned features that differentiate samples belonging to this class. The measure defined as:
C3 mean,ez mean & On,C
(3)
On.D
visual representation the measure is provided in Figure
2.3.2 VARIANCE IN THE LAYER-LEVEL REPRESENTATIONS OF THE DATA
The fourth measure transfers the neuron-level variance approach to layers by computing the stan- dard deviations over the pairwise cosine distances calculated in the space induced by the layer's s activations. Let Σ, be the standard deviation of the pairwise cosine distances between the samples of set E in the space induced by layer 1. The measure is defined as:
C4 mean,ez mean & Σn,C
(4)
M1,D
In other words, we are interested evaluating whether the linear projection implemented by the neuron has heen effective in isolating given subclass. Using cosine distances provided slightly better results than euclidean distances.
3

---

Published as a conference paper at ICLR 2021
To compute this measure, found helpful to standardize the representations of different neurons. More precisely, we normalize each neuron's ' pre-activations to have zero mean and unit variance, then apply bias and ReLU activation function such that 25% of the samples activated] This makes the measure invariant to rescaling and translation of each neuron's preactivations.
Neuron
600
Dataset D Class C)
samples
400
On,Cr O D,D
200
2.5
5.0 Preactivation value
Figure 2: Our simplest measure (denoted 0ạ) quantifies intraclass clustering through the ratio of standard deviations J.C. and Dn associated to the class C, and the entire dataset D respectively. Intuitively, a high ratio means that the neuron relies on features that differentiate samples from C₁ although they belong to the same class. Despite its simplicity, our results in Section suggest a remarkably strong connection between cg and generalization performance. This illustration of measure c3 is based neuron from experimental study, and the associated ratio is 2.47.
EXPERIMENTAL SETUP
The purpose of our experimental endeavour is to assess the predictive power of the proposed intra- class clustering measures with respect to the generalization performance. To this end, we reproduce the methodology introduced by [ et al. (2020). First of all, this methodology puts emphasis on the scale of the experiments improve the generality of the observations. Second, it tries to go be- yond standard measures of correlation, and puts extra care to detect causal relationships between the measures and generalization performance. This achieved through a systematic variation of multi- ple hyperparameters when building the set of models to be studied, combined with the application of principled correlation measures.
3.1 BUILDING A SET OF MODELS WITH VARYING HYPERPARAMETE
Our experiments are conducted on three datasets and network architectures. The datasets are CIFAR10, CIFAR 100 and the coarse version of CIFAR100 with 20 superclasses (Krizhesky & Hinton 2009). The two network architectures are Wide ResNets (He et al. 2016: Zagoruyko & Komodakis 2016) (applied on CIFARION 100 datasets) and VGG variants (Simonyan & Zisserman, 2014) (applied on CIFARIO dataset). Both architectures use batch normalization layers (loffe & Szegedy 2015) since they greatly facilitate the training procedure.
In order to build a set of models with wide range of generalization performances, we vary hy- perparameters that are known to be critical. Since varying multiple hyperparameters improves the identification causal relationships, we vary 8 different hyperparameters: learning rate, batch size, optimizer (SGD or Adam (Kingma & Ba 2015)). weight decay, dropout rate (Srivastava et al., 2014), data augmentation, network depth and width. In total, we elaborated set of 192 hyperparameter configurations for each dataset-architeecure pai More information about the hyperparameter con- figurations, training procedures and the resulting model performances are provided in Appendix worth noting that most models achieve close to 100% training accu- racy. Hence, models with poor test accuracy largely overfit.
A.1 A.2 A.3 and A.4).
AAttvatting 25% of the samples was arbitrary choice that did seek optimize. Studying the influence of this hyperparameter on the results is left as future work.
'The final set of VGG variants trained CIFAR10 results from only 144 hyperparameter the network could not reach 100% training accuracy with high dropout prrerpprrmmtterccffguaatiiion configurations
4

---

Published as a conference paper at ICLR 2021
3.2 EVALUATING THE MEASURES' PREDICTIVE POWER
Jiang et al. 2020) provides multiple criteria to evaluate generalization measures. We opted for the granulated Kendall coefficient for its simplicity and intuitiveness. This coefficient compares two rankings of the models, respectively provided by (i) the generalization measure of interest and (ii) their actual generalization performance. The Kendall coefficient is computed across variations of each hyperparameter independently. The average over all hyperparameters is then computed for the final score. The goal this approach to better capture causal relationships by not overvaluing measures that correlate with generalization only when specific hyperparameters are tuned.
We compare our intraclass clustering measures to sharpness- bbssed measures, The latter con- stituted the most promising measure family from the large-scale study presented in Jiang et al. (2020). Among the many different sharpness measures, we leverage the magnitude-awan that measure sharpness through random and worst-case perturbations of the weights (denoted by 1 and L, respectivelly in Jiang et al. ). We also include the application of these measures with perturbations applied on kernels only (I.e. not on biases and batch normalization weights) with batch normalization layers in batch statistics mode (i.e. not in inference mode). We observed that these alternate versions often provided better estimations of generalization performance. We denote these measures by 6 and a
versions
4 RESULTS
This section starts with a thorough evaluation of the relationship between the four proposed measures and the generalization performance, using the setup described in 3 Then, it presents series of experiments to better characterize intraclass clustering, the phenomenon by the measures. These experiments include (i) an analysis of the measures' evolution across layers and training iterations, (ii) study of the neuron-level measures' sensitivity to A in the mean top-k operation, as well as (iii) visualizations of subclass extraction in individual neurons.
expect to be captured
4.1 EVALUATING THE MEASURES' RELATIONSHIPS WITH GENERALIZATION
We compute all four measures on the models trained on the CIFARI00 superclasses, and the two vareance-based measures on the models trained on standard CIFAR 100 and CIFAR10. We k 30 for the neuron-level measures, meaning that 30 neurons subclass (for e1) or class (for e3) are used capture intraclass clustering. For the layer-level measures, we set networks and k for VGG networks.
for residual
We start our assessment of the measures' predictive power by visualizing their relationship with generalization performance in Figure We observe clear trend across datasets, network architec- tures and measures, suggesting a tight connection between intraclass clustering and generalization. To further support the conclusions of our visualizations, through the granulated Kendall coefficient (cfr. Section 3.2). Tables and 3 present the granu- lated Kendall measures, for the three dataset-architecture pairs (Tables and 3 are in Appendix A.5).
evaluate the measures' predictive power
coefficients associated with intraclass clustering and sharpness-based
The Kendall coefficients further confirm the observations in Figure by revealing strong correlations between intraclass clustering measures and generalization performance across all hyperparameters. In terms of overall score, intraclass clustering measures surpass the sharpness- baase measures vari- ants by a large margin across all dataset-architecture pairs. On some specific hyperparameters, sharpness-base based measures outperform intraclass clustering measures. In particular, performs remarkably well when the batch size parameter is varied, which is coherent with previous work (Keskar al. 2017).
4.2 INFLUENCE OF & ON THE KENDALL COEFFICIENTS OF NEURON-LEVEL MEASURES
In our evaluation the measures in Section 4.1 the parameter, which controls the number of highest values considered in the mean over top-k operations. was fixed quite arbitrarily. Figure shows how the Kendall coefficient of c and ca changes with this parameter. We observe a relatively low sensitivity of the measures' predictive power with respect to k. In particular, in the case of
5

---

Published as a conference paper at ICLR 2021
Based on label hierarchies
Based on variance
Neuron-level
Layer-level
Neuron-level
Layer-level
0.50
1.6
1.5
CIFARION
Resnet
0.40
14
0.35
1.3
122
0.25
X-axes:
CIFAR100
144
Test accuracy
Resnet
1.5
Y-axes:
1.3
1.4
Metric values
CIFARIO
1.5
VGG
1.3
1.4
12
Figure 3: Visualization of the relationship between the four proposed intraclass clustering measures and generalization performance, across datasets and network architectures. The four columns respond to Cı, C2, C3 and C4 measures respectively. All measures display a tight connection with generalization performance, suggesting crucial role for intraclass clustering the implicit regu- larization of deep neural networks.
Table 1: Kendall coefficients for resnets trained CIFARI00 superclasses. Overall, the intraclass clustering measures surpass the sharpness-based measures by a large margin.
learning rate
batch size
weight dccay
dropout rate
data
total
optim.
width
depth
augm.
score
C1
0.88
0.31
0.38
0.67
0.96
1.0
0.81
0.69
0.71
Intraclass clustering
C2
0.86
0.5
0.67
0.58
0.38
0.62
C3
0.88
0.6
0.46
0.62
0.81
1.0
0.81
0.66
0.73
0.89
0.69
0.62
0.65
0.86
1.0
0.44
0.69
0.73
1/ 0.81
0.51
0.31
0.69
0.28
-0.58
0.67
0.61
0.41
l/a"
0.86
0.58
0.17
0.4
-0.05
0.42
0.69
0.72
0.47
Sharpness
1/a²
0.88
0.94
0.29
0.26
0.6
0.08
-0.03
-0.09
0.37
1/a"
0.85
0.8
0.48
0.71
0.16
-0.08
0.08
0.34
0.42
Resnets trained on CIFAR100 the Kendall coefficient associated with seems to stay above 0.7 for any in the range [1, 900). The optimal i value changes with the considered dataset and architecture. We leave the study of this dependency as future work.
Observing the influence of also confers insights about the phenomenon captured by the measures. Figure&revenn that very small & values work remarkably well. Using a single neuron per subclass (k in Equation [ confers Kendall coefficient of 0.69 to ei. Using a single neuron per class confers Kendall coefficient of 0.78 to in the case of VGGs trained suggest that individual neurons play a crucial role in the extraction of intraclass clusters during training. The fact that the Kendall coefficients monotonically decrease after some k value suggests that the extraction of given intraclass cluster takes place a sub part of the network, indicating some form of specialization.
CIFARIO. These results

---

Published as a conference paper at ICLR 2021
CIFAR100
CIFAR100 Resnet
CIFARLO VGG
Resnet
Kendall coeff
0.75
0.70
0.79
0.65
0.76
0.5
0.60
0.73
0.4
0.70
500
750
750
k 730
k
k
Figure 4: Plots showing how the Kendall coefficients of C1 and C3 change with parameter & (cfr. Equations and 3). The k parameter associated to c1 is multiplied by 5 in the plots, enable comparison with Cs (there are subclasses in each of CIFAR 100's superclasses). The total number of neurons varies from 1920 to 2880 in Resnets and from 960 to 1440 in VGGs. The plots reveal that generalization performance can be quite accurately estimated using the representations of a surprisingly small set of neurons (k: 1, i.e. single neuron per class, suffices in some cases).
4.3 EVOLUTIOO OF THE MEASURES ACROSS LAYERS
We pursue experimental endeavour with analysis of the proposed measures' evolution across layers. For each dataset-archine aaisstttrrc pair, we select 64 models which have the same depth hyperpa- rameter value. We then compute the four measures of each layer for the neuron-level measures) and average the resulting values over the 64 models. Figure deepicts how the average value of each measure evolves across layers for Resnets trained on CIFART00 superclasses. The results associated with the two other dataset-architecture pairs are in Appendix A.5 Figures
layer-level basis (we use the top-5 neurons
and TO
We observe two interesting trends. First, all four measures tend to increase with layer depth. This suggests that intraclass clustering also occurs in the deepest representations of neural networks, and not merely in the first layers, which are commonly assumed to capture generic or class-independe features. Second, the variance based measures (C3 and c4) decrease drastically in the penultimate layer. We suspect this reflects the grouping of samples of class in tight clusters in preparation for the final classification layer (such behaviour has been studied in Kamnitssa et al. 20:8); |uüller et al.) 2219)). The measures ei and are robust to this phenomenon as they rely relative distances inside single class, irrespectively of the representations of the rest of the dataset.
4m
L25
1,4
0.10
-0.0
13 Layer
25
L33
13 Layer
Layer
Layer
Figure 5: Evolution of each measure (after averaging over models) across layers. The overall in- crease of the measures with layer depth suggests that intraclass clustering occurs even in the deepest representations of neural networks.
4.4 EVOLUTION OF THE MEASURES OVER THE COURSE TRAINING
In this section, we provide small step towards the understanding of the dynamics of the phe- nomenon captured by the measures. We visualize in Figure evolution the measures over the course of training of three Resnet models. The first interesting observation comes from the com- parison of models with high and low generalization performances. It appears that their differences in terms of intraclass clustering measures arise essentially during the early phase of training. The second observation is that significant increases in intraclass clustering measures systematically coin- cide with significant increases of the training accuracy (in the few first epochs and around epoch 150,
7

---

Published as a conference paper at ICLR 2021
where the learning rate is reduced). This suggests that supervised training could act necessary driver for intraclass clustering ability, despite not explicitly targeting such behaviour.
0.45
My -am 0.05
Train acc
5
-0.01
1.15
1:15
250
250
Epoch
accuracy
Figure 6: Evolution of the intraclass clustering measures over the course of training for three models with different generalization performances. We observe that the differences between models with high and low generalization performance arise essentially in the early phase of training.
4.5 VISUALIZATION OF SUBCLASS EXTRACTION IN INDIVIDUAL NEURONS
We have in Section 4.2 that the measure 61 reaches Kendall coefficient of 0.69 when sidering single neuron per subclass (k = in Eq. D. Visualizing the training dynamics in this specific neuron should enable us to directly observe the phenomenon captured by c1. We study Resnet model trained on CIFAR100 100 superclasses with high generalization performance (82.31% test accuracy). For each of the 100 subclasses, we compute the selectivity value and the index of the most selective neuron based on the part of Eq. I which the median operation is applied. We then rank the subclasses by their selectivity value, and display the training dynamics of the neurons associated to the subclasses with maximum and median selectivity values in Figure) 7
The evolution of the neurons' preactivation distributions along training reveals that the 'Rocket' sub- class, which has the highest selectivity value, is progressively distinguished from its corresponding superclass during training. The neuron behaves like was trained to identify this specific subclass although supervision explicit training mechanisms were implemented to target this behaviour: The same phenomenon occurs to a lesser extent with the 'Ray' subclass, which has the median selectivity value. We observed that very few subclasses reached selectivity values high as the 'Rocket' subclass (the distribution of selectivity values is provided in Figure [] in Appendix AA.5). We suspect that the occurrence such outliers explain why the median operation outperformed the mean in the definition of C₁ and C2.
Initialization
Epoch
Epoch
Epoch 20
Epoch 250
Subciass 'Rockett
sce
200
10
Preactivation
Subciass RRyy
zamples
Preactivation 1.0
3.0
Superclass
Figure 7: Evolution along training of the preactivation distributions associated with the neurons that are the most selective (cfr. Eq. I for 'Rocket' and 'Ray' subclasses. The neurons behave like they were trained identify these specific subclasses although no supervision or explicit training mechanisms were implemented to target this behaviour.
8

---

Published as a conference paper at ICLR 2021
4.6 DISCUSSION
Our results show that all four measures (i) strongly correlate with generalization, (ii) tend to increase with layer depth and (iii) change mostly in the early phase of training. These similarities suggest that the four measures capture one unique phenomenon. Since all measures quantify to what extent neural network differentiates samples from the same class, the captured phenomenon presumably consists in the identification of intraclass clusters by neural networks. This hypothesis is further sup- ported by the neuron-level visualizations provided in Section 4.5 Overall, our results thus provide empirical evidence for this works hypotheses, i.. that intraclass clustering emerges during standard neural network training and constitutes an implicit form of regularization.
The mechanisms that induce intraclass clustering in deep neural networks remain undisclosed, leav- ing lot of room for future work. In particular, Section 4.2 reveals that understanding the local nature of intraclass clustering could lead to better estimation of the optimal & parameter and in- crease the measures' predictive power. Additionally, the neuron-level visualizations in Figures 2 and 7 suggest that quantifying the bimodal shape of neuron's preactivations for each class might provide an alternative measure of intraclass clustering that doesn't 't require hierarchical labels.
5 RELATED WORK
This work follows the line of inquiry initiated by Neyshabur et al. (2015) ;: Zhang et al. (2017). and tries unveil the phenomena underlying the generalization ability of deep neural networks. We follow empirical approach based the design of measures that correlate with generalization performance. This approach has been extensively used in recent years, leading to measures such sharpness of minima (Keskar et al.) 2017). sensitivity (Novak et al., 2018), reliance single directions (Morcos et al.] 2018) and stiffness (Fort et al. 2019). To our knowledge, we are the first work to empirically study the relationship between intraclass clustering measures and generalization.
Many observations made in our paper are coherent with previous work. In the context of transfer learning, Huuh et al. (2016) shows that representations that discriminate ImageNet classes naturally emerge when training on their corresponding superclasses, suggesting the occurrence of intraclass clustering. Sections 4.2 and 4.5 suggest a key role for individual neurons in the extraction of in- traclass clusters. This is coherent with the large body of work that studied the emergence of in- terpretable features in the hidden neurons (or feature maps) of deep nets (Zeiler & Fergus 2014 Simonyan et al.) 2014: Yosinski et al. 2015 Zhou et al., 2015 Bau et al., 2017). In Section 4.4 we notice that intraclass clustering occurs mostly in the early phase of training. Previous works have also highlighted the criticality of this phase of training with respect to regularization (Golatkar et al.) 2019), optimization trajectories (Jastrzebski et al. 2020) Fort et al. 2020), Hessian eigenspectra (Gur-Ari et al. 2018), training data perturbations (Achille et al. 2019) and weight rewinding (Fran- kle et al. 20200a b). Morcos al. (2018); Leavitt [ & Morcos (2020) have shown that class-selective neurons are not necessary and might be detrimental for performance. This is coherent with our observation that neurons that differentiate samples from the same class improve performance.
6 CONCLUSION
Our work starts from the observation that classes of samples from standard image classification datasets are usually structured into multiple clusters. Since learning structure in the data com- monly associated to generalization, we ask if state-of-the-an neural network extract such clusters. While no supervision or training mechanisms are explicitly programmed into deep neural networks for targeting this behaviour, it is possible that this learning ability implicitly emerges during train- ing. Such a hypothesis is especially compelling as recent work on generalization conjectured the emergence of implicit forms of regularization during deep neural network training.
Our work provides four different attempts at quantifying intraclass clustering. We then conduct a large-scale experimental study to characterize intraclass clustering and its potential relationship with generalization performance. Overall, the results suggest a crucial role for intraclass clustering in the regularization of deep neural networks. We hope our work will stimulate further investigations of intraclass clustering and ultimately lead to a better understanding of the generalization properties of deep neural networks.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
Thanks to the reviewers for suggesting several experiments that were added during the rebuttal phase and increased the quality of this paper.
Thanks to Amirafshar Moshtaghpour, Anne-Sophie Collin, Antoine Vanderschueren, Victor Joos de ter Beerst, Tahani Madmad, Pierre Carbonnelle, Vincent Francois, Gilles Peiffer for proofreading and helpful feedhack.
Thanks to the Reddit r/machinelearning community for keeping us up to date with our fast-moving field.
Simon and Christophe are Research Fellows of the Fonds de la Recherche Scientique FNRS, which provided funding for this work.
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical Learning Periods in Deep Neural Networks. In ICLR, pp. 1-14, 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In ICML, 2017.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection:
Quantifying Interpretability Deep Visual Representations. In CVPR, 2017.
Simon Carbonnelle and Christophe De Vleeschouwer. Layer rotation: surprisingly powerful in-
dicator of generalization in deep networks? In ICML. workshop "Identifying and Understanding Deep Learning Phenomena", 2020.
Stanislav Fort, Pawel Krzysztof Nowak. Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A
New Perspective Generalization in Neural Networks. arXiv 1901.09491, 2019.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning empirical study of loss landscape geometry and the time evolution the Neural Tangent Kernel. In NeurIPS, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. In ICML, 2020a.
Jonathan Frankle, David J Schwab, and Ari S Morcos. The Early Phase of Neural Network Training. In ICLR, 2020b.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Time Matters in Regularizing Deep Net- works Weight Decay and Data Augmentation Affect Early Learning. NeurlPS, 2019.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient Descent Happens in a Tiny Subspace. 1rXiv:1812.04754. 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, 2016.
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes ImageNet good for transfer learning? arXiv:1608 08 2016.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML, pp. 448 456, 2015. ISSN 0717-6163. doi: 10. 1111007/113388-811-017-----
03167
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The Break-Even Point on Optimization Trajectories of Deep Neural Networks. In ICLR, 2020.
10

---

Published as a conference paper at ICLR 2021
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where to Find Them. In ICLR, 2020.
Konstantinos Kamnitsas, Daniel C Castro, Loic Le Folgoe, Walker lan, Tanno Ryutaro, Daniel Rueckert, Ben Glocker, Antonio Criminisi, and Aditya Nori. Semi-Supervised Learning via Com- pact Latent Space Clustering. In ICML, 2018.
Leonard Kaufman and Peter Rousseeuw. Finding groups in data: an introduction to cluster anal- ysis. John Wiley & Sons, 2009.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On arge-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR, 2017.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.
Matthew L Leavitt and Ari Morcos, Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs. arXiv:2003.01262, 2020.
Ari Morcos, David G T Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In ICLR, 2018.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When Does Label Smoothing Help ? In NeurlPS, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML, pp. 807 -81481144 844, 2010.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. ICLR, 2015.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Sensitivity and generalization in neural networks: an empirical study. ICLR, 2018.
K. Simonyan, A. Vedaldi, and A Zisserman. Deep Inside Convolutional Networks Visualising Image Classification Models and Saliency Maps, ICLR, 2014.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv: /409. 1556, 2014.
N. Srivastava, G. E. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15 1911929-1958, 2014.
Twan van Laarhoven. I.2 Regularization versus Batch and Weight Normalization.
(rXiv:/706005355 2017. URL/pttpp.//axx...orr///bbb
00350)
Ashia Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, and Benjamin Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. In NIPS, pp. 4151-4161, 2017.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding Neural Networks Through Deep Visualization. In Deep Learning Workshop at ICML, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. ECCV, pp. 818-833, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires re-thinking generalization. In ICLRR 2017.
11

---

Published as a conference paper at ICLR 2021
Guodong Zhang, Chaoqi Wang. Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In ICLR, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in Deep Scene CNNs. In ICLR, 2015.
A APPENDIX
A.1 HYPERPARAMETER CONFIGURATIONS
We define here the hyperparameter configurations used to build our set models (cfr. Section 3.1 3.T.. A total of 8 hyperparameters
tuned: learning rate, batch size, optimizer, weight decay, dropout rate, data augmentation, network depth and width. A straightforward way to generate configurations specify values for each hyperparameter independently and then generate all possible combina- tions. However, given the amount of hyperparameters, this quickly leads to unrealistic amounts of models to be trained.
To deal with this, we decided to remove co-variations of hyperparameters whose influence on train- ing and generalization is suspected be related. More precisely, we use weight decay only in combination with the highest learning rate value, as recent works demonstrated a relation between weight decay and learning rate (van Laarhoven 2017 Zhang et al. 2019). We also don't combine dropout and data augmentation, as the effect of dropout is drastically reduced when data augmen- tation is used, Finally, we do not jointly increase width and depth, to avoid very large models that would slow down our experiments.
The resulting hyperparameter values are as follows:
1. (Learning rate, Weight decay): ((0.01, (0.32, 0,,,,,0,,,,,,,,,,,000,0 10-
2. Batch size: (100,300}
3. Optimizer: {SGD, Adam)
4. (Dropout rate, Data augm.): (O0,,
(0.2,
5. (Width factor, Depth factor): {(x1., x1.), (x1.5, x x1.), (x1., x1.5))}
We generate all possible combinations of these hyperparameter values (or pairs of values), leading to 192 configurations. Since dropout rates of 0.4 lead poor training performance on VGG variants, only 144 configurations are used in these cases.
A.2 FURTHER INFORMATION ON THE HYPERPARAMETERY
It has been shown that different optimizers may require different learning rates for optimal perfor- mance (Wilson et al. 2017). In our experiments, we divide the learning rate by 100 when using Adam improve performance (the same approach used in Jiang al. (0200). When using SGD we apply a momentum of 01.9.
We use the following data augmentations: horizontal and vertical shifts of up to 4 pixels, and hor- izontal flips. We fill the resulting image through reflection. This data augmentation procedure is widely used for the CIFAR datasets.
We use two types of network architectures: VGG and Wide ResNets. We build these architectures using four stages, separated by 3 pooling layers. This enabled a wider range of generalization performances, as the standard three stage versions of these architectures tend to be less capable of overfitting. Width factors of x1. and x1.5 correspond to widths of 32 and 48 respectively in the layers of the first stage. The width of layers is doubled after every pooling layer. Depth factors of 1. and x x1.5 correspond to 9 and 13 layers for VGG variants, and 18 and 26 layers for Wide ResNet variants.
A.3 FURTHER INFORMATION ON THE TRAINING PROCEDURE
We train models for 250 epochs, and reduce the learning rate by a factor 0.2 at epochs 150, 230, 240. Training stops prematurely if the training loss gets smaller than 10-1
12

---

Published as a conference paper at ICLR 2021
A.4
MODEL PERFORMANCES
Cfr. Figure g
Dcc
OCC
100
CFAR100
12
025
220
CIFAR100
1.2
100
02
CIFAR1
3.00
0.00
0.10
0.990
0.900
1.000
3.0
3.5
07
2.0
06
Figure Histogram of performances of the set of models used in our experiments.
A.5 SUPPLEMENTARY RESULTS
Table 2: Kendall coefficients for resnets trained on CIFARI00.
learning rate
batch size
weight decay
dropout rate
data augm.
total
optim.
width
depth
score
Intraclass clustering
C3
0.94
0.65
0.62
0.58
1.0
1.0
1.0
0.78
0.82
C4
0.93
0.62
0.62
0.21
0.91
0.78
0.76
1/ 0.88
0.68
0.17
0.8
0.4
-0.62
0.94
0.61
0.48
l/o"
0.92
0.61
0.12
0.35
-0.06
0.31
0.94
0.53
0.47
Sharpness
1/a
0.96
0.96
0.17
0.25
0.54
0.15
-0.16
-0,23
0.33
1/a"
0.96
0.91
0.42
0.64
0.12
-0.25
0.17
0.14
0.39
Table 3: Kendall coefficients for VGG networks trained CIFARIO.
learning
batch size
weight decay
dropout
data augm.
total
optim.
width
depth
Intraclass
C3
0.92
0.83
0.67
0.51
0.92
1.0
1.0
0.88
0.84
clustering
0.86
0.75
0.33
0.29
0.92
1.0
0.54
0.92
0.7
1/d
0.86
0.62
-0.25
0.6
-0.04
-0.27
1.0
0.85
0.42
1/
0.9
0.67
0.11
0.69
0.92
0.19
1.0
0.94
0.68
Sharpness
1/
0.94
0.89
0.61
0.53
0.67
0.77
0.15
0.06
0.58
1/a"
0.93
0.67
0.36
0.54
0.69
-0.02
0.15
-0.15
0.4
13

---

Published as a conference paper at ICLR 2021
1.25
1.4
1.15
1.2
1.10
1
13 Layer
25
13 Layer
25
Figure 9: Evolution of the measures across layers for Resnets trained on CIFAR100
1.3
1.2
1.2
1.1-
1.1
1.0-
0.9
10
Layer
Layer
Figure 10: Evolution of the measures across layers for VGGs trained on CIFARI0
12.5
10.0
5.0
2.5
0.0
0.5
1.0
1.5
Subclass selectivity
Figure 11: Distribution of neural subclass selectivity values (cfr. measure ej over the 100 subclasses of CIFARI00. For each subclass, neural subclass selectivity computed based the most selective neuron in the neural network (i.e. = 1). We observe that 1) only few subclasses reach high selectivity values and 2) the selectivity values vary much across subclasses. We suspect that the outliers with exceptionally high selectivity values cause the median operation to outperform the mean in the measures ei and C2.
14


Reviews:
[{"review_id": "m_B3KzThbW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### 1. Brief summary\nThe authors note that in classification tasks there typically exist within-class groups of similar images that are not explicitly encoded in the coarse class label -- they call this intraclass clustering. They hypothesize that the ability of DNNs to recognize these intraclass clusters without being explicitly told about that could correlate with generalization. They then proceed to verify this on a range of networks, architectures, and a large number of hyperparameter configurations. They take care to establish causality where possible. In addition, they show that the intraclass clustering can be detected with simple variance-based methods, and that it emerges early in training.\n\n### 2. Strengths\n* This is a super interesting question and I really like the paper overall.\n* I appreciate that you looked at a large hypercube of hyperparameters to establish correlation with generalization\n* I also like the care you put into establishing causality\n* The fact that you tried a simple variance based measure is also really good, especially given that it is very predictive!\n\n### 3. Weaknesses\nI think this paper is really good, I have nothing much to point out here. Possible a large range of architectures and scaling up to ImageNet would be useful to establish that this scales all the way to very large data, but it is very good as is!\n\n### 4. Related papers that you might like\n[1] You cite Stiffness: A New Perspective on Generalization in Neural Networks by Stanislav Fort, Paweł Krzysztof Nowak, Stanislaw Jastrzebski, Srini Narayanan (https://arxiv.org/abs/1901.09491) as measuring an amount of class-specific clustering. In that paper in Figure 9 they show that stiffness is aware of the super-classes of CIFAR-10 and even their super-super-classes (animals, etc.), which seems relevant here too. Though there it goes the other way round -- the training is on subclasses, generating awareness of superclasses. Your results are I'd stay stronger than that.\n\n[2] In talking about the sensitivity to early stages of training The Break-Even Point on Optimization Trajectories of Deep Neural Networks by Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, Krzysztof Geras (https://arxiv.org/abs/2002.09572 and ICLR 2020) might be relevant, where they also establish a very strong effects of the early stages of training.\n\n[3] Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel by Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel Roy, Surya Ganguli (https://arxiv.org/abs/2010.15110 and NeurIPS 2020) also shows a very strong effect of the early stages of training on a large number of DNN measures.\n\n### 5. Summary\nThis paper is really good! It starts with a strong hypothesis, verifies it on a large number of experiments, is mindful of causality and generates a potentially practically useful insight into generalization. Well done!", "extra_scores": {}, "extra_text": {}}, {"review_id": "TPLk6id30xS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work investigates whether intra-class separation in the latent space of a neural network correlates with its generalization capabilities. To perform this, the authors design 4 measures that attempt to capture whether separate sub-clusters are formed for different sub-classes of a (super-)class. All measures take into account all activations in all layers of network. 2 measures operate at neuron level, 2 at layer level. 2 are designed for cases where we do have explicit sub-class labels, and 2 are for the case that no such labels are given. The work performs multiple experiments on Cifar10 & Cifar100 & Cifar100(superclasses) where multiple models are trained with varying hyper-parameter configurations, each leading to different performance. Then, it is shown that the 4 measures correlate positively with performance, which according to the authors suggests that intra-class separation within the network is happening and is important for generalization. \n\n------------------------------------------------------------\nReasons for score: \nI am very borderline on this. On one hand it is interesting to see such a strong correlation of the intra-class separation with generalization, but on the other hand there are many questions that the study leaves unanswered regarding whether we are measuring what we think we are measuring, whether the results are sensitive or not to choice of hyperparameters for the measures, etc. In general, I am currently left quite unconfident in drawing strong conclusions. Perhaps the rebuttal will help.\n\n------------------------------------------------------------\n\nPros: \n\n1) The main research questions, whether the networks have an inductive bias to learn features that perform intra-class separation, without a training loss that demands it, and whether this separation actually correlates with generalization, are interesting questions.\n\n2) The results suggest a positive correlation of intra-class separability with generalization. This may be an interesting finding to parts of the community. For example there is work that attempts the opposite to improve generalization, introducing losses for learning a single tight cluster per class (losing intra-class separability) and show that this improves generalization. Contradicting evidence could spark interesting discussion / research in the community. (see references [1, 2] further down in my comments)\n\n3) There is a very significant number of experiments performed (500+), which may offer significant information to the community. \n\n------------------------------------------------------------\n\nCons:  \n\n \n1) The work presents no analysis of the behaviour of the measures with respect to design choices or the hyperparameters of the measures. As a result, I think we do not get much insight in “exactly” what is being measured. For example, are the measures capturing intra-classs eparation at 1st or last layer? Are the results robust to hyperparameters? Insights like this are important to understand the value of the results. The empirical investigation is unfortunately limited to simply reporting a seemingly high correlation of the presented metrics with the generalization, without further insights. This limited insight is particularly troublesome when coupled with several questions I have about the design of the measures and their configuration, which leave me unconfident about the rigor of the study, and whether we are indeed measuring intra-class separation in meaningful ways. (see below “Questions for rebuttal” for more details)\n\n2) There are design choices with respect to the measures and their application on the net that seem adhoc, without appropriate explanation. (why use k-top values and not all? Why normalize the pre-activations and take 25% top percent? And so on so forth. See detailed questions below)\n\n3) Similarly, there are configuration parameters that the work merely mentions the value chosen (e.g. k=5), without any explanation how this value was chosen, nor any sensitivity study of whether results would change for different values. This adds the unconfidence I have about the rigorousness of the study.\n\n4) Reproducibility seems low. Code is not available, there are no details on the values explored for the hyperparameters of the networks in the main experiments, such as learning rates, etc, and it seems challenging for another party to re-run 500+ experiments to reproduce the results. It is especially in such studies that rigorous analysis for convincing results is extra important, which I think leaves a lot to be desired here.\n\n5) Experiments are performed only on CIFAR 10 / 100. I understand that performing all these experiments is expensive, but on the other hand, results are limited only to 1 type of data. Do we know if conclusions generalize?\n\n------------------------------------------------------------\n\nQuestions to address during rebuttal period:\n\nIn Sec 3.1, please clarify explicitly whether “pre-activations” are before or after normalization by batch-norm. (this will inter-play with statistics used in the study such as sigma_n,D).\n\nSec. 3.2.2 and Sec.3.3.2: Why use the cosine distance instead of Euclidean? If I am note mistaken, Euclidean would also evaluate whether a cluster is tight (concentrated closer into a point), rather than only the angle that the cosine measures? I don’t see a reason why the magnitude of activations wouldn’t matter for cluster/class discrimination in an arbitrary deep net. Please clarify in text why you prefer the use of cosine and whether you believe things would be similar with other distances (e.g. Euclidean)\n\nWhy is Eq 1 and Eq 3 done on the “pre-activations”, while Eq 2 and Eq 4 are done on the “activations” (after ReLU)? Have Eq 2 & 4 been tried on pre-activations and vice versa? Perhaps this also relates to why Eq2&4 are on cosine distance instead of Euclidean. If so, state reasons explicitly, and if you ve tried things differently.\n\nWhy do Eq. 1 and Eq 2 use median_i and Eq 3, 4 use mean_i ? Any theoretical or empirical support to this? Does the choice influence results? Please explicitly state in the paper too.\n\nWhy do Eq 1 and Eq 2 use max_n and max_l respectively, while Eq 3 use mean_n and mean_l ? Any theoretical or empirical support to this? Does the choice influence results? Please explicitly state in the paper too.\n\nThe max_n and max_l operators, as well as the mean_^k_n and mean_^k_l operators, choose a single n/l or k-top n/l neurons/layers with separability from the whole network. It could be the very first layer, or the last. Do you have any insights on which layers are usually chosen? I think this would add great insights if we knew. If first or last layers are chosen, does it make a difference on the value of this intra-class separability? For example, intra-cluster separability at last layers will say much about classification itself. At the first layer, it says that there are filters that merely identify multiple colors for the same class (both white and black cats). It would be nice to know what actually is being measured. Perhaps adding a discussion, or a plot showing which layers are usually chosen would be very useful, and it would add confidence to the reader about exactly what is being measured and how to interpret the results/conclusions. This is needed especially because some measures are not “actually” measuring intra-class clustering (e.g. c3 & c4) but something else, that we “assume” indicates intra-class clutsering, hence makes interpretation ambiguous.\n\nHow was the choice of values (k=5 for resnet, k=1 for VGG) done in Sec. 3.3.2? Do these values influence the behaviour? Would they influence the conclusions? It would be nice to have a sensitivity study on such values?\n\nSec 3.3.2: “we found it helpful…25% of the samples are activated”: Please elaborate more on how this was found, why do you think it’s important, how was the value 25% defined, and how the results change if this is not performed? It would be nice to have empirical analysis of how this influences how the measure behaves and whether it changes any results.\n\nFig 2: This is helpful in understanding why c3 and c4 may indicate indirectly some intra-class separation in some cases. But, what if a class presents uni-modal distribution with large std? (and not bimodal as in the example?). Can you think of a way to provide more supporting evidence that c3,c4 capture intra-class separation? (here is where a more extensive analysis would have also been useful)\n\nSec. 5.2: How do the results with sharpness based measures you get (Tabl1,2,3) compare to those reported by Jiang 2020? With a quick look of mine, they seem quite close. Please confirm and discuss in your paper. If the results are similar, it adds confidence to the reader that your experimental setup is correct, replicating theirs.\n\nThe “sharpness” measures do not require the class labels, right? If this is correct, then their capability of predicting generalization should not be directly comparable with the measures in this study, which do require the class labels. The study does not do this directly, but I think an explicit comment should be added for the reader, so that they are aware of this significant difference.\n\nFig.3: Can you discuss whether you think the values returned by the measures (y-axis) suggest the existence of actual intra-class clustering, and support the main assumption of the paper? In fact, the values 0.01-0.08 of the silhouette score seem very low (Silhouette is -1 to +1, with 0 meaning overlapping clusters), right? Similarly, the 0.1-0.7 returned by c1 also seems small. E.g. imagine 2 gaussians with same std for the 2 clusters. If they wouldn’t overlap, the distance of their centers would be 6+ stds, right? Rough estimate. 0.1-0.7 suggests quite a strong overlap I think. Perhaps adding a discussion on this would be nice. If you indeed agree there is no strong supporting evidence of clear overlap, perhaps ensure there are no strong statements in the paper about it, and mostly emphasize on the correlation to generalization instead.\n\nSec 5.3 “strong increases of the training accuracy”: Can you explicitly state to which points (e.g. epoch) you are refering to? Moreover, what happened at epoch ~140 and caused the spike in training accuracy? Lowering learning rate? I would advise you explicitly state it.\n\nFig.4: How do you group all the models/experiments in Good/medium/low performance groups? Clarify in text. What exactly are these numbers (82.31%...) in the caption? State explicitly. The reader should not have to guess about anything.\n\nWhat does Kendall Coefficient == 1.0 mean, when data-augmentation changes? That all measures predict absolutely perfectly generalization? Isn’t this weird? How do authors interpret this? Perhaps a short discussion would add to the reader’s understanding, insights, and confidence in the interpretation of the results.\n\nThere is a body of literature that designs losses for learning 1 single tight cluster per class, losing intra-class separation *in the last layers*, and shows this improves generalization of classifiers. The current study seems to contradict it, suggesting intra-class clusters help generalization. I think this should be discussed. The issue is that, because of no analysis, we don’t know whether the current measures find intra-class separation in early or late layers, to help us understand how the results connect exactly with the rest of the literature (contradicting it, or complementing it if only earlier layers are chosen by the max_l operators). This is one of the reasons that I think an analysis with further insights on how exactly the proposed measures compute is important (as I said previously), and add confidence to the reader by pointing out how they relate to existing literature.\nIf you would agree, here a couple of references that do tight clustering: A recent such work on deep nets I know is [1], which shows tight clustering improves generalization (the work is mainly on semi-supervision, but in the end they also have experiments for standard fully-supervised learning). Similarly, [2] show that label smoothing promotes tight-clustering and improves generalization (supervised learning).\n\n[1] Kamnitsas et al, Semi-Supervised Learning via Compact Latent Space Clustering, ICML 2018\n\n[2] Muller et al, When Does Label Smoothing Help? NeurIPS 2019.\n\n\nTo help with reproducibility: Is it possible to provide implementation of the measures (even if not the whole code is possible to release), and details about exactly what values have been explored for the network/training hyperparameters in the main experiments in Table1 etc? (e.g. in an appendix).\n\n------------------------------------------------------------\n\nMinor comments & some additional feedback for improving the work (which are not necessary to discuss within the rebuttal, but do address as many as possible):\n\nI think Eq.2 is not well descriptive of what is being described in Sec. 3.2.2. Specifically, it shows that silhouete(a_l, C_i) is computed only on the set of a subclass C_i. If I understand correctly from the text, instead, you actually compute the mean silhoute score of all the samples in a SuperClass. So, I think it should be silhouette(a_l, S_s(i)) ?\nMoreover, a_l is undefined. Is it supposed to be the cosine distances? Please clarify and update the text to make things clear. Perhaps give the equation for the silhouette score explicitly if this helps.\n\nSec 1. The current statement “could account for … invalidated” I think is too strong. There are many studies that showed that these help generalization (e.g. I think the positive effect of augmentation is unquestionable). I would suggest rephrasing it to something more tactful, like “not the sole factors for generalization performance” or something similar.\n\nIn Sec 3.3.1, I think that the range of appropriate values for k is dependent on the number of classes in the database. Perhaps you would like to state this explicitly. Consider if you could also derive a rule of thumb (likely empirically) of what is a good k with respect to number of classes. \n\nSec 4.2 “ranking of models…performance.”: I think this sentence descriing Kendall coeff could use a slight rephrase to be more intuitive.\n\nSec. 4.2: “by penalizing measures”: I am not sure that the approach “penalizes”. Rather, it seems because it takes average over loads of values, it merely is robust to outlier cases (measure + hyperparam tuned). Or, averages-away the influence of the hyperparam? Am I right? Perhaps a slight rephrase would avoid this confusion.\n\n“is designed to better capture causality”: I would suggest this argument to be rephrased a bit to avoid interpretation that it accurately captures causality. It is merely robust (due to averaging across hyperparams) to the case of only 1 hyperparam correlating with generalization. Notice that Jiang et al 2020 also take special care to not over-state the capabilities of this measure to identify causal factors (see Sec 2.2.2 in their paper).\n\n\n**============  Summary of improvements and revisiting reviewer’s score after rebuttal ==============**\n\nSummary of main points of improvement related to my comments after the revision:\n\n- The authors have significantly extended the analysis within the paper with more experiments to investigate the proposed measures and the deep nets’ behaviour in question. This adds a lot of value to the paper, offering more insights and support for the main claims.\n\n- The authors have added many in-text clarifications about certain design choices (and improved some, e.g. by average k-max instead of max), which makes the study much clearer and adds confidence to the reader about interpretating the investigation and its results.\n\n- The authors have added a study of the influence of the k parameter in the measures, which simultaneously offers confidence in the conclusions (conclusions hold for a significant range of values for k) and offers new insights about how separability happens (1 neuron vs multiple).\n\n- Reproducibility is greatly improved, both by improved clarifications of the experimental settings within the text, and by providing the code in the supplementary.\n\n- The authors improved discussions about the results with respect to related literature, taking into account and linking them also to papers that at first glance seem to be contradicting (e.g. references [1,2] I provided above that show tight clustering improves generalization). These links and discussions further improve my confidence in the conclusions.\n\nOverall, the authors have addressed sufficiently most of my concerns, significantly improving the manuscript. The updated manuscript provides significantly more insights, which should be of interest to a part of the community. The extra analysis provides better support of the study’s hypothesis and claims. Remaining weaknesses of the paper include the CIFAR-only experimentation (we do not know if conclusions hold beyond it). I am happy to increase my review’s score from 5 (marginally bellow acceptance) to 7 (Good paper, accept).\n\n= Minor =\n\nIn case the paper gets accepted, I would suggest the authors to try to complete Fig 4 with the layer-wise investigation, which they said was too expensive for the rebuttal period (I cannot estimate how expensive that can be. Hopefully it is possible in longer time period, and would complete nicely Fig 4).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7_lLnBCutOE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\n\nThis paper introduces a notion coined “intraclass clustering” that describes a deep neural network’s implicit ability to cluster within data. 4 different quantities that measure the networks’ clustering ability are proposed, and large scale experiments show that they are highly effective at predicting the models’ generalization ability across diverse types of hyperparameters. \n\n**Pros**:\n1. The paper presents an interesting idea that also seems to be practically highly relevant based on the experiments. Notably, it performs *nearly perfectly* on several interesting hyperparameters that are challenging for previous measures. I believe that there is definitely something unusual in this idea and it is worth future research.\n2. The experiments on the evolution of the coefficients is extremely interesting, since this may indicate that we do not need to train the models to convergence to verify how good the models are.\nThe quantity is simple and seems to be efficient to compute (?)\n\n**Cons**:\n1. I think this paper is not very well-written and needs much more work. For example, why are there 4 different coefficients when two of them clearly outperform the previous two based on label hierarchies, which are nonetheless the motivations for the work and supposedly model the actual intraclass clustering ability? The paper claims that they are complementary, but the experiments, in my opinion, suggests otherwise. No further explanation is offered for this phenomenon, and I am just left wondering. I think this warrants much more investigation and explanation.\n2. What is the exact definition of a neuron and a layer? The activation tensor is of shape NxHxWxC. Is the neuron a single scalar or single channel (NxHxWx1)? Likewise, is the layer the whole tensor of a single channel? This is not very clear in the paper, and it would help to spell out exactly how the variance or means are computed (using formula).\n3. It would be nice to have a couple more comparisons instead of just sharpness.\n4. It’s not immediately clear to me how this phenomenon can be converted to a learning theoretical argument about generalization, but this is minor.\n5. I must admit that it is not clear to me why this phenomenon should capture generalization. I assume the authors had some conjectures or intuitions, so it would be nice to include that in the paper. Specifically, what is special about “intraclass”? Does this mean good models are just memorizing prototypes?\n6. In 3.3.1, “If the standard deviation computed over the samples of a class is high compared to the standard deviation computed over the entire dataset, we infer that the neuron has learned features that differentiate samples belonging to this class”, why does standard deviation capture this? I think an justification would be nice. The figure in some sense does it but I don't think that's thorough enough.\n7. It would be good to have analysis of which layer is usually picked when doing max or top k.\n\n**Comments**:\n1. Can intraclass clustering be viewed as some kind of specialization of different parts of the model or some kind of compression? I think more interesting things can be said about this.\n2. What is a_l in eq 2?\n3. For figure 4, is there adversarial noise for the green curve?\n4. It would be interesting to see the evolution of the ranking instead of the coefficients as table 1-3 but this is quite expensive and I think the existing materials are already interesting enough.\n5. It would be nice to see the mutual information based metric from Jiang et al, although this is not high on priority. If the authors are interested, they could use the data and code released by PGDL competition at NeurIPS which includes the implementation of mutual information metric.\n6. Are all the models interpolating? Would be nice to see a visualization of training loss or accuracy.\n\n\n**Conclusion**:\n\nThis paper presents something that I personally believe is very interesting, if not exciting. But, the paper needs quite a lot of works before it can be published. I want to emphasize that I believe the idea is extremely promising, but the presentation, the execution and thoroughness of analysis unfortunately just miss the mark. As such, I am inclined to reject the paper for now. I will increase my scores if the authors can address my concerns.\n\n**================================== Update after rebuttal ==================================**\n\nI appreciate the reviewers' hard work for adding this many new materials in a short period of time. A large number of my concerns have been addressed and the quality of paper has improved significantly. Some newly added experiments give a lot more insights than the original draft. As such, I am in favor of accepting the paper and have increased my scores from 4 to 6.\n\nHowever, I still have a few lingering questions in the light of the rebuttal and would love to see them addressed should the paper be accepted:\n1. Re: figure 4, the question is why the performance of the green curve is so bad. I was wondering if some of the labels are wrongly labeled to induce this effect.\n2. For reducing the feature map to a single quantity, why is the max operation chosen? Intuitively, it makes more sense to use mean or even order statisitc to make the metric more robust.\n\nI also understand that many of the experiments cannot be added in a short period of time and hope that the authors add them as promised in the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0TqlJnMAS-Q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the intraclass clustering ability of neural networks trained in supervised learning and found that networks show intraclass clustering ability despite not explicitly enforced by the label to do so. And criterions based on those correlate well with model generalization performance. \n\nHowever, I think the current state of the paper is still a bit below the standard of ICLR, and there are a few ways that this paper could be potentially improved.\n\n1. I think observation 1 (clustering is learned without explicit constraints) is more interesting than observation 2 (correlate well with generalization performance), because it is intuitive that if you can classify a finer class assignment well, then you should also do well on more coarse classes. Maybe one thing that can be added to the experiments is to study whether the intraclass clusterability could also overfit. In other words, do you see a big generalization gap if you measure your criterions on the training set and test set separately?\n\n2. As mentioned in the previous point, I think it is probably worth allocating more space to study how / why the intra-class clustering ability is learned during training despite supervised with corse label assignments. I find it especially intriguing from Fig.4 that all the clusterability metrics are actually very low at the beginning of the training where the network weights are random. My intuition was that at the beginning of training, there are probably some neurons that is lucky and separate the intra-class clustering well. But it seems that my intuition is wrong. It would be great if the paper has more experiments focusing on discovering what happens during the training that helps intraclass clustering. For example, are those intraclass clusters actually pushed away? Or they stay the same distances to each other, but benefits from by-product of supervised learning that shrinks all the clusters? Other experiments could be to study what kind of regularization / training techniques affect clusterability, and can we explicitly encourage clusterability to improve generalization during training, etc.\n\n3. I'm a bit uncomfortable with the 'max' operation taken in the definition of clusterability criterions. It essentially searches over all the neurons looking for patterns that you want to see. Given that there are a lot of randomness in the neural network weights, this might not be depicting the 'typical' behavior of neurons. For example, it might be that there is a random neuron that happen to correlate very well with the intraclass clusterability. Maybe one can replace the max with the top_k averaging with a relatively large k like in the case of variances, or even with something like 99 percentile. A baseline can also be provided to see how much the randomness comes into play -- one can define a random clustering assignments, and compute the same criterion and see what number could be achieved. \n\n4. Could you please include details in the experimental setup showing how do you get the intra-class clustering assignments that is needed to compute the criterions? I imagine for the 20-class CIFAR-100 you can just use the original labels as clustering assignment, but I did not find how you do it for other datasets. \n\n5. Similar to the previous point, in practice, one usually do not have the intra-class cluster assignment available to compute those criterions. In contrast, the sharpness criterion can be computed without extra information. Can you add some materials to show what could be done in this case? Maybe show some way to approximately estimate the proposed metrics and study the estimation accuracies?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes intra-class clustering as an indicator of generalization performance and validates this by extensive empirical evaluation. All reviewers have found this connection highly interesting. The author response has also duly addressed most of the reviewers' concerns. Given the importance of studying generalization performance of overparameterized deep models, the paper will potentially generate interesting discussion at the conference. 
Decision: Accept (Poster)