Intraclass clustering: an implicit learning ability that regularizes DNNs

1 INTRODUCTION . The generalization ability of deep neural networks remains largely unexplained . In particular , the traditional view that explicit forms of regularization ( e.g . dropout , L2-regularization , data augmentation ) are the sole factors for generalization performance of state of the art neural networks has been experimentally invalidated ( Neyshabur et al. , 2015 ; Zhang et al. , 2017 ) . Today ’ s conventional wisdom rather conjectures the presence of implicit forms of regularization , emerging from the interactions between neural network architectures , optimization , and the inherent structure of the data itself ( Arpit et al. , 2017 ) . One structural component that seems to occur in most image classification datasets is the presence of multiple clusters amongst the samples of a class ( or intraclass clusters , cfr . Figure 1 ) . The extraction of such structure in the context of supervised learning is not self-evident , as today ’ s standard training algorithms are designed to group samples from a class together , without any considerations for eventual intraclass clusters . This paper hypothesizes that the identification of intraclass clusters emerges during supervised training of deep neural networks , despite the absence of supervision or explicit training mechanisms targeting this behaviour . Moreover , our study suggests that this phenomenon improves the generalization ability of deep neural networks , hence constituting an implicit form of regularization . To verify our hypotheses , we define four measures of intraclass clustering and inspect the correlation between those measures and a network ’ s generalization performance . These measures are designed to capture intraclass clustering from four different perspectives , defined by the representation level ( neuron vs. layer ) and the amount of knowledge about the data ’ s inherent structure ( datasets with or without hierarchical labels ) . To evaluate these measures ’ predictive power , we train more than 500 models , varying standard hyperparameters in a principled way in order to generate a wide range of generalization performances . The measures are then evaluated qualitatively through visual inspection of their relationship with generalization and quantitatively through the granulated Kendall rank-correlation coefficient introduced by Jiang et al . ( 2020 ) . Both evaluations reveal a tight connection between intraclass clustering measures and generalization ability , providing important evidence to support this work ’ s hypotheses . 2 MEASURING INTRACLASS CLUSTERING IN INTERNAL REPRESENTATIONS . A challenge of our work resides in measuring a model ’ s ability to differentiate intraclass clusters , without knowing which mechanisms underlie such ability . In such context , the design of multiple complementary measures ( i ) offers different perspectives that will help better characterize intraclass clustering mechanisms and ( ii ) reduces the risk that their potential correlation with generalization is induced by other phenomena independent of intraclass clustering . This section thus describes four measures of intraclass clustering , differing in terms of representation level ( neuron vs. layer ) and the amount of knowledge about the data ’ s inherent structure ( datasets with or without hierarchical labels ) . 2.1 TERMINOLOGY AND NOTATIONS . D denotes the training dataset . Let I be the number of classes in the dataset D , we denote the set of samples from class i by Ci with i ∈ I = { 1 , 2 , ... , I } . In the case of hierarchical labels , Ci denotes the samples from subclass i and Ss ( i ) the samples from the superclass containing subclass i . We denote by N = { 1 , 2 , ... , N } and L = { 1 , 2 , ... , L } the indexes of the N neurons and L layers of a network respectively . Neurons are considered across all the layers of a network , not a specific layer . The methodology by which indexes are assigned to neurons or layers does not matter . We further denote by meanj∈J and medianj∈J the mean and median operations over the index j respectively . Moreover , meankj∈J corresponds to the mean of the top-k highest values , over the index j . We call pre-activations ( and activations ) the values preceding ( respectively following ) the application of the ReLU activation function ( Nair & Hinton , 2010 ) . In our experiments , batch normalization ( Ioffe & Szegedy , 2015 ) is applied before the ReLU , and pre-activation values are collected after batch normalization . In convolutional layers , a neuron refers to an entire feature map . The spatial dimensions of such a neuron ’ s ( pre- ) activations are reduced through a global max pooling operation before applying our measures . 2.2 MEASURES BASED ON LABEL HIERARCHIES . The first two measures take advantage of datasets that include a hierarchy of labels . For example , CIFAR100 is organized into 20 superclasses ( e.g . flowers ) each comprising 5 subclasses ( e.g . orchids , poppies , roses , sunflowers , tulips ) . We hypothesize that these hierarchical labels reflect an inherent structure of the data . In particular , we expect the subclasses to approximately correspond to different clusters amongst the samples of a superclass . Hence , measuring the extent by which a network differentiates subclasses when being trained on superclasses should reflect its ability to extract intraclass clusters during training . 2.2.1 NEURON-LEVEL SUBCLASS SELECTIVITY . The first measure quantifies how selective individual neurons are for a given subclassCi with respect to the other samples of the associated superclass Ss ( i ) . Here , strong selectivity means that the subclass Ci can be reliably discriminated from the other samples of Ss ( i ) based on the neuron ’ s preactivations1 . Let µn , E and σn , E be the mean and standard deviation of a neuron n ’ s pre-activation values taken over the samples of set E. The measure is defined as follows : c1 = mediani∈I meankn∈N µn , Ci − µn , Ss ( i ) \Ci σn , Ci + σn , Ss ( i ) \Ci ( 1 ) Since we can not expect all neurons of a network to be selective for a given subclass , we only consider the top-k most selective neurons . The measure thus relies on k neurons to capture the overall network ’ s ability to differentiate each subclass . We noticed slightly better results when using the median operation instead of the mean to aggregate over the subclasses . We suspect this arises from the outlier behaviour of certain subclasses that we observe in Section 4.5 . 2.2.2 LAYER-LEVEL SILHOUETTE SCORE . The second measure quantifies to what extent the samples of a subclass are close together relative to the other samples from the associated superclass in the space induced by a layer ’ s activations . In other words , we measure to what degree different subclasses can be associated to different clusters in the intermediate representations of a network . We quantify this by computing the pairwise cosine distances2 on the samples of a superclass and applying the Silhouette score ( Kaufman & Rousseeuw , 2009 ) to assess the clustered structure of the subclasses . Let silhouette ( al , Ss ( i ) , Ci ) be the mean silhouette score of subclass Ci based on the activations al of superclass Ss ( i ) in layer l , the measure is then defined as : c2 = mediani∈I meankl∈L silhouette ( al , Ss ( i ) , Ci ) ( 2 ) 2.3 MEASURES BASED ON VARIANCE . To establish the generality of our results , we also design two measures that can be applied in absence of hierarchical labels . We hypothesize that the discrimination of intraclass clusters should be reflected by a high variance in the representations associated to a class . If all the samples of a class are mapped to close-by points in the neuron- or layer-level representations , it is likely that the neuron/layer did not identify intraclass clusters . 2.3.1 VARIANCE IN THE NEURON-LEVEL REPRESENTATIONS OF THE DATA . The first variance measure is based on standard deviations of a neuron ’ s pre-activations . If the standard deviation computed over the samples of a class is high compared to the standard deviation computed over the entire dataset , we infer that the neuron has learned features that differentiate samples belonging to this class . The measure is defined as : c3 = meani∈I meankn∈N σn , Ci σn , D ( 3 ) A visual representation of the measure is provided in Figure 2 . 2.3.2 VARIANCE IN THE LAYER-LEVEL REPRESENTATIONS OF THE DATA . The fourth measure transfers the neuron-level variance approach to layers by computing the standard deviations over the pairwise cosine distances calculated in the space induced by the layer ’ s activations . Let Σl , E be the standard deviation of the pairwise cosine distances between the samples of set E in the space induced by layer l. The measure is defined as : c4 = meani∈I meankl∈L Σl , Ci Σl , D ( 4 ) 1In other words , we are interested in evaluating whether the linear projection implemented by the neuron has been effective in isolating a given subclass . 2Using cosine distances provided slightly better results than euclidean distances . To compute this measure , we found it helpful to standardize the representations of different neurons . More precisely , we normalize each neuron ’ s pre-activations to have zero mean and unit variance , then apply a bias and ReLU activation function such that 25 % of the samples are activated3 . This makes the measure invariant to rescaling and translation of each neuron ’ s preactivations . 3 EXPERIMENTAL SETUP . The purpose of our experimental endeavour is to assess the predictive power of the proposed intraclass clustering measures with respect to the generalization performance . To this end , we reproduce the methodology introduced by Jiang et al . ( 2020 ) . First of all , this methodology puts emphasis on the scale of the experiments to improve the generality of the observations . Second , it tries to go beyond standard measures of correlation , and puts extra care to detect causal relationships between the measures and generalization performance . This is achieved through a systematic variation of multiple hyperparameters when building the set of models to be studied , combined with the application of principled correlation measures . 3.1 BUILDING A SET OF MODELS WITH VARYING HYPERPARAMETERS . Our experiments are conducted on three datasets and two network architectures . The datasets are CIFAR10 , CIFAR100 and the coarse version of CIFAR100 with 20 superclasses ( Krizhevsky & Hinton , 2009 ) . The two network architectures are Wide ResNets ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ) ( applied on CIFAR100 datasets ) and VGG variants ( Simonyan & Zisserman , 2014 ) ( applied on CIFAR10 dataset ) . Both architectures use batch normalization layers ( Ioffe & Szegedy , 2015 ) since they greatly facilitate the training procedure . In order to build a set of models with a wide range of generalization performances , we vary hyperparameters that are known to be critical . Since varying multiple hyperparameters improves the identification of causal relationships , we vary 8 different hyperparameters : learning rate , batch size , optimizer ( SGD or Adam ( Kingma & Ba , 2015 ) ) , weight decay , dropout rate ( Srivastava et al. , 2014 ) , data augmentation , network depth and width . In total , we elaborated a set of 192 hyperparameter configurations for each dataset-architecture pair4 . More information about the hyperparameter configurations , training procedures and the resulting model performances are provided in Appendix ( A.1 , A.2 , A.3 and A.4 ) . It is worth noting that most models achieve close to 100 % training accuracy . Hence , models with poor test accuracy largely overfit . 3Activating 25 % of the samples was an arbitrary choice that we did not seek to optimize . Studying the influence of this hyperparameter on the results is left as future work . 4The final set of VGG variants trained on CIFAR10 results from only 144 hyperparameter configurations as the network could not reach ∼ 100 % training accuracy with high dropout rates .

Summary: This work investigates whether intra-class separation in the latent space of a neural network correlates with its generalization capabilities. To perform this, the authors design 4 measures that attempt to capture whether separate sub-clusters are formed for different sub-classes of a (super-)class. All measures take into account all activations in all layers of network. 2 measures operate at neuron level, 2 at layer level. 2 are designed for cases where we do have explicit sub-class labels, and 2 are for the case that no such labels are given. The work performs multiple experiments on Cifar10 & Cifar100 & Cifar100(superclasses) where multiple models are trained with varying hyper-parameter configurations, each leading to different performance. Then, it is shown that the 4 measures correlate positively with performance, which according to the authors suggests that intra-class separation within the network is happening and is important for generalization.