Divisive Feature Normalization Improves Image Recognition Performance in AlexNet

1 INTRODUCTION . Neural networks ( NN ’ s ) in general and convolutional NN ’ s ( CNN ’ s ) in particular were originally inspired by the brain . However , only the barest sketch of brain function has been incorporated into NN ’ s . Conversely , studies of brain-like function in NN ’ s have only begun to impact neuroscience . Here we consider a biological form of “ divisive normalization '' ( DN ) , which is postulated to be a canonical computation of at least sensory cortex ( Carandini & Heeger , 2012 ) . We show that it can enhance the image classification performance of AlexNet ( Krizhevsky et al. , 2012 ) , and study how it alters representations in the context of this architecture and task . Divisive normalization is a phenomenological description ( Geisler & Albrecht , 1992 ; Heeger , 1992 ) of nonlinear neuronal response properties observed throughout sensory cortex : when multiple stimuli are simultaneously presented , either within a neuron ’ s receptive field ( RF ; the region of sensory space in which appropriate stimuli drive a given neuron ’ s response ) or both inside the RF ( in the “ center ” ) and outside of it ( “ surround ” ) , then ( 1 ) responses tend to be less than the sum of the responses to the stimuli shown individually , that is , summation is sublinear ; but ( 2 ) when stimuli are weak , summation becomes more linear or even supralinear ( more than the sum of the responses to the individual stimuli ) . The phenomenological description posits that a neuron ’ s response is its unnormalized response , divided by a function of a constant plus a sum over the unnormalized responses ( perhaps raised to a power ) of all the other surrounding neurons in a “ normalization pool ” . Thus , anything that adds to the collective response of the population also suppresses ( effectively inhibits ) each individual neuron ’ s response . However , the divisive function reduces to the constant for weak unnormalized responses , thus removing the effects of normalization for weak stimuli . The unnormalized response is often modeled as an expansive function , e.g . a rectified quadratic ; then the response to multiple weak stimuli can show supralinear summation . There are other standard forms of normalization being used in neural networks ( Ren et al. , 2016 ) , which we will call `` canonical '' . These include ( but are not limited to ) batch ( Ioffe & Szegedy , 2015 ) , layer ( Ba et al. , 2016 ) , instance ( Ulyanov et al. , 2016 ) , and group ( Wu & He , 2018 ) normalization . These all standardize ( zero mean , unit variance ) and then affinely transform sets of activations in a given layer ; they differ in the sets of channels and images over which standardization is performed ( for all , the sets include all of space ) . These normalizations prevent or reduce covariate shifts and can have other advantages . The set includes one image for all but batch normalization , which uses all images in a batch . The first three do not lead to competition between channels , as the set either includes a single channel ( batch or instance ) or all channels ( layer ) , and the same operations are being applied to all channels in the set . However , in group normalization , the channels are divided into non-overlapping groups , with standardization performed separately over each group . Thus there is competition within the group – one channel ’ s strong activity can suppress the activity of other channels in the group , relative to the activities of channels in other groups . This is closest to DN , which is competitive . In our formulation of DN , the group with which a neuron is competing changes continuously with the neuron , constituting some local region around each neuron . We take this local region to be a single point of space and a learnable span of channels about a given channel ( more precisely , the channels are topologically arranged on a line ; the contribution of nearby channels is weighted according to the distance between it and the channel being normalized , by a decaying exponential with a learnable length constant ) . DN does not prevent covariate shifts , so we will find it useful to combine DN with one of the other normalizations . Our contributions in this paper are , for the first time ( to our knowledge ) , characterizing how a canonical biological operation , DN , learned along with the CNN filters , affects ImageNet and CIFAR100 performance and learned representations in a CNN ( AlexNet ) with and without `` canonical '' normalizations . In particular , we show : • Addition of DN improves performance for image recognition in AlexNet models with or without canonical normalizations , and the best performance is found by combining both types of normalization ; • DN increases the large or medium ( depending on presence and type of canonical normalization ) wavelength Fourier modes in the first layer receptive fields . • Both canonical and divisive normalizations reduce the network ’ s manifold capacity and correspondingly change associated geometric measures at interior layers , leading to improved manifold capacity and associated changes in geometric measures at the final level , corresponding well to improvements in performance . • DN consistently increases the sparsity of activations ( Gini index ) at each normalization step and in the output layer . We also find preliminary evidence suggesting that DN can improve out-of-distribution ( OOD ) performance . This work should be of interest both to the ML and neuroscience communities , and warrants further study , for example , to understand why DN produces the associated changes in representations , whether and how these changes are related to the improvements in performance , and how performance with DN can be optimized . 2 RELATED WORK . In recent work a neural circuit model was found that produces the neural responses that had been phenomenologically described by DN , along with a number of other biological response properties ( Ahmadian et al. , 2013 ; Rubin et al. , 2015 ) . This has raised interest in understanding the possible functions of this normalization , for which there are many hypotheses , of which we mention only a few . It has been postulated to keep activations within an appropriate dynamic range for the neurons ( Carandini & Heeger , 2012 ) . It has been shown to remove higher-order statistical dependencies in responses to auditory or visual stimuli ( Schwartz & Simoncelli , 2001 ) , and more generally to minimize redundancy , maximize information , or efficiently or optimally encode ( Malo & Laparra , 2010 ; Gomez-Villa et al. , 2020 ; Malo , 2020 ; Ballé et al. , 2016 ) . It has also been shown to arise from statistical inference of the reflectances underlying a model of the statistics of natural scenes , the Gaussian scale mixture model ( GSM ) ( Coen-Cagli et al. , 2012 ; 2015 ; Echeveste et al. , 2020 ) . The original AlexNet ( Krizhevsky et al. , 2012 ) included local response normalization ( LRN ) , much like ours ( Eq . 1 ) but with a linear numerator and the sum in the denominator over ±2 neighbors without exponential weighting . Parameter values were hyperparameters set using a validation set ; the equivalent of our parameter kα/λ was 10−4 , with k = 2 , making it difficult to understand how LRN could have had much impact . Nonetheless it improved performance , though this was disputed by Simonyan & Zisserman ( 2015 ) , but in our hands by less than DN ( see Table 1 ) . Ren et al . ( 2016 ) developed a unified mathematical framework for slightly modified batch , layer , and DN , combined it with an L1 regularizer , and showed that various forms of this ( learned ) regularized normalizer improved performance on CIFAR-10 and CIFAR-100 in a network with 3 convolutional and 2 fully connected layers , with the best performance by a modified batch norm . Their DN included in a unit ’ s normalization pool all channels in a local spatial region about the unit . Giraldo & Schwartz ( 2019 ) explored a flexible , stimulus-dependent form of DN across space , based on the GSM , with learned parameters , that was added to the 2nd layer of a pretrained Alexnet to model contextual modulation in V1 . Others have examined effects of DN on tasks in various biologically-motivated architectures ( Coen-Cagli & Schwartz , 2013 ; Bertalmío et al. , 2020 ) . Burg et al . ( 2021 ) implemented a learnable form of DN in a model trained end-to-end to replicate spike counts of V1 neurons . The model included a single convolutional layer of 32 filters with batch normalization , followed by DN and a readout layer . The filters developed with no topology , that is , normalization weights were learned between each directed pair of channels . The work most similar to our own was done independently by Pan et al . ( 2021 ) . They considered a form of DN in which channels were partitioned into groups of 8 , which normalized one another , followed by an affine transformation . They also considered adding a spatially local normalization pool restricted to a unit ’ s own channel . For every unit , the affine transformations and the weights from every member of its normalization pool were learned . They also considered the DN Ren et al . ( 2016 ) . They found that , compared to canonical normalizations , their channel normalization , but not the additional spatial normalization nor DN , improved performance on CIFAR-10 in shallow convolutional nets but not in deeper ones ( 4-5 or more layers ) . They attributed the improved performance on shallow networks to their channel normalization making activity distributions in early layers more Gaussian , Their channel normalization also showed some improvement over canonical normalizations for AlexNet on ImageNet . Our normalization pool size is learned ( determined by the space constant of an exponential kernel ) , we examine pairings of divisive and canonical normalization which we find important to avoid failures to learn and improve performance relative to divisive alone , and we examine several properties – receptive fields and their Fourier power , manifold capacity , sparsity – that characterize ways in which the normalizations change representations . 3 METHODS . Architecture . We studied 8 models , each a variant of AlexNet ( 5 convolutional layers and 3 fully connected layers ; Krizhevsky et al. , 2012 ) with different normalization layers , with a Kaiming He initialization ( He et al. , 2015 ) and without pytorch local response normalization ( LRN ) . The filters are 11x11 for ImageNet and 3x3 for CIFAR-100 in the first layer , 3x3 in all subsequent layers . The order of operations in each convolutional layer is ReLU , then DN if used , then canonical normalization if used ( Divisive , Batch , Group , Layer ) . Normalization Formalisms . For DN , the channels in a given layer develop topologically arranged on a line . Given n channels in a layer , numbered from 1 to n , we let ac ( x ) be the rectified output of the convolution with the filter of channel c at 2D spatial position x . We take the unnormalized activation of this channel to be ac ( x ) 2 . We then divisively normalize , using as a “ normalization pool '' an exponentially weighted sum of the unnormalized activations of nearby channels at the same spatial position , to yield the unit ’ s normalized activity bc ( x ) : bc ( x ) = ac ( x ) 2 ( k ( 1 + αλ ∑4λ j=−4λ ac+j ( x ) 2e−|j|/λ ) ) β ( 1 ) Here , β , α , k and λ are all learnable parameters , learned independently for each convolutional layer . We also considered models in which each divisive normalization was followed by a `` canonical '' normalization : either batch , group , or layer . In all three , the normalization is of the form : z̃n , j = γ zn , j−E [ zn ] √ V ar [ zn ] + + β . Here γ and β are learnable parameters . The subscript n denotes the set that is normalized together . For example , in batch normalization , for an input of dimension N × c × H ×W , in which c is the number of features , H and W the spatial dimensions and N the number of images in a batch . The mean and variance in this equation are calculated for a given feature across all of space and the whole batch . For layer normalization , the mean and variance are calculated across the spatial and feature dimensions for each image . For group normalization , the feature dimension is divided into 4 equal-sized groups in each layer , and the normalization is done within each group across space for each image . Hyperparameters . Unless otherwise specified , the learning rate used in the models was .01 . Batch sizes were 128 . The initial normalization parameters were λ = 10. , α = .1 , β = 1. , k = 10 , except for the Divisive model with no other normalizations , for which initial λ = 1. and k = 0.5 to make learning reliable ( further discussed in Results ) . The Weight initialization method followed that of He et al . ( 2015 ) in which weights are initialized with the same statistics for differing seeds . Specifically , the He formulation for ReLU activation functions is meant to keep the expected activation variances constant across layers . We used the same principle for our networks with ReLU plus DN and arrived at the same weight initialization . We then used the same initialization for combined divisive/canonical models . See Appendix A and B for more information . The CIFAR training and validation images were resized to 32 × 32 × 3 and horizontally flipped ; Imagenet training images resized to 224 × 224 × 3 and horizontally flipped ; Imagenet validation images resized to 256 × 256 × 3 and center cropped . In both data sets , each color channel was standardized .

Summary: The authors study the effect of divisive normalization on AlexNet. They show that, when combined with standard normalization schemes, it increases performance. They also investigate the filter shapes, manifold capacity and (adversarial) robustness of the learned representations.  Following the authors' response I have increased my score form 5 to 6.