Intraclass clustering: an implicit learning ability that regularizes DNNs

1 INTRODUCTION . The generalization ability of deep neural networks remains largely unexplained . In particular , the traditional view that explicit forms of regularization ( e.g . dropout , L2-regularization , data augmentation ) are the sole factors for generalization performance of state of the art neural networks has been experimentally invalidated ( Neyshabur et al. , 2015 ; Zhang et al. , 2017 ) . Today ’ s conventional wisdom rather conjectures the presence of implicit forms of regularization , emerging from the interactions between neural network architectures , optimization , and the inherent structure of the data itself ( Arpit et al. , 2017 ) . One structural component that seems to occur in most image classification datasets is the presence of multiple clusters amongst the samples of a class ( or intraclass clusters , cfr . Figure 1 ) . The extraction of such structure in the context of supervised learning is not self-evident , as today ’ s standard training algorithms are designed to group samples from a class together , without any considerations for eventual intraclass clusters . This paper hypothesizes that the identification of intraclass clusters emerges during supervised training of deep neural networks , despite the absence of supervision or explicit training mechanisms targeting this behaviour . Moreover , our study suggests that this phenomenon improves the generalization ability of deep neural networks , hence constituting an implicit form of regularization . To verify our hypotheses , we define four measures of intraclass clustering and inspect the correlation between those measures and a network ’ s generalization performance . These measures are designed to capture intraclass clustering from four different perspectives , defined by the representation level ( neuron vs. layer ) and the amount of knowledge about the data ’ s inherent structure ( datasets with or without hierarchical labels ) . To evaluate these measures ’ predictive power , we train more than 500 models , varying standard hyperparameters in a principled way in order to generate a wide range of generalization performances . The measures are then evaluated qualitatively through visual inspection of their relationship with generalization and quantitatively through the granulated Kendall rank-correlation coefficient introduced by Jiang et al . ( 2020 ) . Both evaluations reveal a tight connection between intraclass clustering measures and generalization ability , providing important evidence to support this work ’ s hypotheses . 2 MEASURING INTRACLASS CLUSTERING IN INTERNAL REPRESENTATIONS . A challenge of our work resides in measuring a model ’ s ability to differentiate intraclass clusters , without knowing which mechanisms underlie such ability . In such context , the design of multiple complementary measures ( i ) offers different perspectives that will help better characterize intraclass clustering mechanisms and ( ii ) reduces the risk that their potential correlation with generalization is induced by other phenomena independent of intraclass clustering . This section thus describes four measures of intraclass clustering , differing in terms of representation level ( neuron vs. layer ) and the amount of knowledge about the data ’ s inherent structure ( datasets with or without hierarchical labels ) . 2.1 TERMINOLOGY AND NOTATIONS . D denotes the training dataset . Let I be the number of classes in the dataset D , we denote the set of samples from class i by Ci with i ∈ I = { 1 , 2 , ... , I } . In the case of hierarchical labels , Ci denotes the samples from subclass i and Ss ( i ) the samples from the superclass containing subclass i . We denote by N = { 1 , 2 , ... , N } and L = { 1 , 2 , ... , L } the indexes of the N neurons and L layers of a network respectively . Neurons are considered across all the layers of a network , not a specific layer . The methodology by which indexes are assigned to neurons or layers does not matter . We further denote by meanj∈J and medianj∈J the mean and median operations over the index j respectively . Moreover , meankj∈J corresponds to the mean of the top-k highest values , over the index j . We call pre-activations ( and activations ) the values preceding ( respectively following ) the application of the ReLU activation function ( Nair & Hinton , 2010 ) . In our experiments , batch normalization ( Ioffe & Szegedy , 2015 ) is applied before the ReLU , and pre-activation values are collected after batch normalization . In convolutional layers , a neuron refers to an entire feature map . The spatial dimensions of such a neuron ’ s ( pre- ) activations are reduced through a global max pooling operation before applying our measures . 2.2 MEASURES BASED ON LABEL HIERARCHIES . The first two measures take advantage of datasets that include a hierarchy of labels . For example , CIFAR100 is organized into 20 superclasses ( e.g . flowers ) each comprising 5 subclasses ( e.g . orchids , poppies , roses , sunflowers , tulips ) . We hypothesize that these hierarchical labels reflect an inherent structure of the data . In particular , we expect the subclasses to approximately correspond to different clusters amongst the samples of a superclass . Hence , measuring the extent by which a network differentiates subclasses when being trained on superclasses should reflect its ability to extract intraclass clusters during training . 2.2.1 NEURON-LEVEL SUBCLASS SELECTIVITY . The first measure quantifies how selective individual neurons are for a given subclassCi with respect to the other samples of the associated superclass Ss ( i ) . Here , strong selectivity means that the subclass Ci can be reliably discriminated from the other samples of Ss ( i ) based on the neuron ’ s preactivations1 . Let µn , E and σn , E be the mean and standard deviation of a neuron n ’ s pre-activation values taken over the samples of set E. The measure is defined as follows : c1 = mediani∈I meankn∈N µn , Ci − µn , Ss ( i ) \Ci σn , Ci + σn , Ss ( i ) \Ci ( 1 ) Since we can not expect all neurons of a network to be selective for a given subclass , we only consider the top-k most selective neurons . The measure thus relies on k neurons to capture the overall network ’ s ability to differentiate each subclass . We noticed slightly better results when using the median operation instead of the mean to aggregate over the subclasses . We suspect this arises from the outlier behaviour of certain subclasses that we observe in Section 4.5 . 2.2.2 LAYER-LEVEL SILHOUETTE SCORE . The second measure quantifies to what extent the samples of a subclass are close together relative to the other samples from the associated superclass in the space induced by a layer ’ s activations . In other words , we measure to what degree different subclasses can be associated to different clusters in the intermediate representations of a network . We quantify this by computing the pairwise cosine distances2 on the samples of a superclass and applying the Silhouette score ( Kaufman & Rousseeuw , 2009 ) to assess the clustered structure of the subclasses . Let silhouette ( al , Ss ( i ) , Ci ) be the mean silhouette score of subclass Ci based on the activations al of superclass Ss ( i ) in layer l , the measure is then defined as : c2 = mediani∈I meankl∈L silhouette ( al , Ss ( i ) , Ci ) ( 2 ) 2.3 MEASURES BASED ON VARIANCE . To establish the generality of our results , we also design two measures that can be applied in absence of hierarchical labels . We hypothesize that the discrimination of intraclass clusters should be reflected by a high variance in the representations associated to a class . If all the samples of a class are mapped to close-by points in the neuron- or layer-level representations , it is likely that the neuron/layer did not identify intraclass clusters . 2.3.1 VARIANCE IN THE NEURON-LEVEL REPRESENTATIONS OF THE DATA . The first variance measure is based on standard deviations of a neuron ’ s pre-activations . If the standard deviation computed over the samples of a class is high compared to the standard deviation computed over the entire dataset , we infer that the neuron has learned features that differentiate samples belonging to this class . The measure is defined as : c3 = meani∈I meankn∈N σn , Ci σn , D ( 3 ) A visual representation of the measure is provided in Figure 2 . 2.3.2 VARIANCE IN THE LAYER-LEVEL REPRESENTATIONS OF THE DATA . The fourth measure transfers the neuron-level variance approach to layers by computing the standard deviations over the pairwise cosine distances calculated in the space induced by the layer ’ s activations . Let Σl , E be the standard deviation of the pairwise cosine distances between the samples of set E in the space induced by layer l. The measure is defined as : c4 = meani∈I meankl∈L Σl , Ci Σl , D ( 4 ) 1In other words , we are interested in evaluating whether the linear projection implemented by the neuron has been effective in isolating a given subclass . 2Using cosine distances provided slightly better results than euclidean distances . To compute this measure , we found it helpful to standardize the representations of different neurons . More precisely , we normalize each neuron ’ s pre-activations to have zero mean and unit variance , then apply a bias and ReLU activation function such that 25 % of the samples are activated3 . This makes the measure invariant to rescaling and translation of each neuron ’ s preactivations . 3 EXPERIMENTAL SETUP . The purpose of our experimental endeavour is to assess the predictive power of the proposed intraclass clustering measures with respect to the generalization performance . To this end , we reproduce the methodology introduced by Jiang et al . ( 2020 ) . First of all , this methodology puts emphasis on the scale of the experiments to improve the generality of the observations . Second , it tries to go beyond standard measures of correlation , and puts extra care to detect causal relationships between the measures and generalization performance . This is achieved through a systematic variation of multiple hyperparameters when building the set of models to be studied , combined with the application of principled correlation measures . 3.1 BUILDING A SET OF MODELS WITH VARYING HYPERPARAMETERS . Our experiments are conducted on three datasets and two network architectures . The datasets are CIFAR10 , CIFAR100 and the coarse version of CIFAR100 with 20 superclasses ( Krizhevsky & Hinton , 2009 ) . The two network architectures are Wide ResNets ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ) ( applied on CIFAR100 datasets ) and VGG variants ( Simonyan & Zisserman , 2014 ) ( applied on CIFAR10 dataset ) . Both architectures use batch normalization layers ( Ioffe & Szegedy , 2015 ) since they greatly facilitate the training procedure . In order to build a set of models with a wide range of generalization performances , we vary hyperparameters that are known to be critical . Since varying multiple hyperparameters improves the identification of causal relationships , we vary 8 different hyperparameters : learning rate , batch size , optimizer ( SGD or Adam ( Kingma & Ba , 2015 ) ) , weight decay , dropout rate ( Srivastava et al. , 2014 ) , data augmentation , network depth and width . In total , we elaborated a set of 192 hyperparameter configurations for each dataset-architecture pair4 . More information about the hyperparameter configurations , training procedures and the resulting model performances are provided in Appendix ( A.1 , A.2 , A.3 and A.4 ) . It is worth noting that most models achieve close to 100 % training accuracy . Hence , models with poor test accuracy largely overfit . 3Activating 25 % of the samples was an arbitrary choice that we did not seek to optimize . Studying the influence of this hyperparameter on the results is left as future work . 4The final set of VGG variants trained on CIFAR10 results from only 144 hyperparameter configurations as the network could not reach ∼ 100 % training accuracy with high dropout rates .

Summary: The authors note that in classification tasks there typically exist within-class groups of similar images that are not explicitly encoded in the coarse class label -- they call this intraclass clustering. They hypothesize that the ability of DNNs to recognize these intraclass clusters without being explicitly told about that could correlate with generalization. They then proceed to verify this on a range of networks, architectures, and a large number of hyperparameter configurations. They take care to establish causality where possible. In addition, they show that the intraclass clustering can be detected with simple variance-based methods, and that it emerges early in training.