Divisive Feature Normalization Improves Image Recognition Performance in AlexNet
Authors: Michelle Miller
SueYeon Chung
Kenneth D. Miller
Venue: ICLR 2022 Poster (2022)
Keywords: divisive normalization
AlexNet
ImageNet
CIFAR-100
manifold capacity
sparsity
receptive fields
Batch Normalization
Group Normalization
Layer Normalization

Abstract: Local divisive normalization provides a phenomenological description of many nonlinear response properties of neurons across visual cortical areas. To gain insight into the utility of this operation, we studied the effects on AlexNet of a local divisive normalization between features, with learned parameters. Developing features were arranged in a line topology, with the influence between features determined by an exponential function of the distance between them. We compared an AlexNet model with no normalization or with canonical normalizations (Batch, Group, Layer) to the same models with divisive normalization added. Divisive normalization always improved performance for models with batch or group or no normalization, generally by 1-2 percentage points, on both the CIFAR-100 and ImageNet databases. To gain insight into mechanisms underlying the improved performance, we examined several aspects of network representations. In the early layers both canonical and divisive normalizations reduced manifold capacities and increased average dimension of the individual categorical manifolds. In later layers the capacity was higher and manifold dimension lower for models roughly in order of their performance improvement. Examining the sparsity of activations across a given layer, divisive normalization layers increased sparsity, while the canonical normalization layers decreased it. Nonetheless, in the final layer, the sparseness of activity increased in the order of no normalization, divisive, com- bined, and canonical. We also investigated how the receptive fields (RFs) in the first convolutional layer (where RFs are most interpretable) change with normalization. Divisive normalization enhanced RF Fourier power at low wavelengths, while divisive+canonical enhanced power at mid (batch, group) or low (layer) wavelengths, compared to canonical alone or no normalization. In conclusion, divisive normalization enhances image recognition performance, most strongly when combined with canonical normalization, and in doing so it reduces manifold capacity and sparsity in early layers while increasing them in final layers, and increases low- or mid-wavelength power in the first-layer receptive fields.

Published as a conference paper at ICLR 2022
DIVISIVE FEATURE NORMALIZATION IMPROVES IM- AGE RECOGNITION PERFORMANCE IN ALEXNET
Michelle Miller', Yeon Chung!.23 Ken D. Miller
1 Center for Theoretical Neuroscience, Columbia University, 2 Center for Neural Science, New York University, Flatiron Institute, Simons Foundation, 4 Swartz Program in Theoretical Neuroscience, Kavli Institute for Brain Science, Department of Neuroscience, College of Physicians and Surgeons, Zuckerman Mind Brain Behavior Institute, Columbia University
ABSTRACT
Local divisive normalization provides phenomenological description of many nonlinear response properties of neurons across visual cortical areas. To gain insight into the utility of this operation, studied the effects AlexNet of local divisive normalization between features, with learned parameters. Developing features were arranged in a line topology, with the influence between features determined by an exponential function of the distance between them. We compared AlexNet model with no normalization with canonical normalizations (Batch, Group. Layer) to the same models with divisive normalization added. Divisive normalization always improved performance for models with batch or group or no normalization, generally by 1-2 percentage points, on both the CIFAR-100 and ImageNet databases. To gain insight into mechanisms underlying the improved performance, examined several aspects of network representations. In the early layers both canonical and divisive normalizations reduced manifold capacities and increased average dimension of the individual categorical manifolds. In later layers the capacity was higher and manifold dimension lower for models roughly in order of their performance improvement. Examining the sparsity of activations across a given layer, divisive normalization layers increased sparsity, while the canonical normalization layers decreased it. Nonetheless, in the final layer, the sparseness of activity increased in the order of no normalization, divisive, com- bined, and canonical. We also investigated how the receptive fields (RFs) in the first convolutional layer (where RFs are most interpretable) change with normal- ization. Divisive normalization enhanced RF Fourier power low wavelengths, while divisive+canonical enhanced power at mid (batch, group) or low (layer) wavelengths, compared to canonical alone or no normalization.
conclusion,
divisive normalization enhances image recognition performance, most strongly when combined with canonical normalization, and in doing so it reduces manifold capacity and sparsity in early layers while increasing them in final layers, and increases low- or mid-wavelength power in the first-layer receptive fields.
INTRODUCTION
Neural networks (NN's) in general and convolutional NN's (CNN's) in particular were originally inspired by the brain. However, only the barest sketch of brain function has been incorporated into NN's. Conversely, studies of brain-like function NN's have only begun to impact neuroscience. Here we consider a biological form of "divisive normalization" (DN), which is postulated to be canonical computation of at least sensory cortex (Carandini & Heeger 2012). We show that it can enhance the image classification performance of AlexNet (Krizhevsky et al. 2012), and study how it alters representations in the context of this architecture and task.
Divisive normalization is phenomenological description (Geisler & Albrechi 1992 Heeger 1992
of nonlinear neuronal response properties observed throughout sensory cortex: when multiple stimuli simultaneously presented, either within a neuron's receptive field (RF; the region of sensory space in which appropriate stimuli drive given neuron's 's response) or both inside the RF (in the "center") and outside of it ("surround"", then (1) responses tend to be less than the of the responses the

---

Published as a conference paper at ICLR 2022
stimuli shown individually, that is, summation is sublinear; but (2) when stimuli weak, summation becomes more linear or even supralinear (more than the sum of the responses to the individual stimuli). The phenomenological description posits that a neuron's 's response is its unnormalized response, divided by a function of a constant plus a sum over the unnormalized responses (perhaps raised to power) of all the other surrounding neurons in a "normalization pool". Thus, anything that adds to the collective response of the population also suppresses (effectively inhibits) each individual neuron's 's response. However, the divisive function reduces to the constant for weak unnormalized responses, thus removing the effects of normalization for weak stimuli. The unnormalized response is often modeled as an expansive function, e.g. rectified quadratie; then the response to multiple weak stimuli can show supralinear summation.
There are other standard forms of normalization being used in neural networks (Ren et al., 2016).
which we will call "canonical". These include (but are not limited to) batch (Toffe & Szegedy 2015), layer (Ba al. 2016), instance (Ulyanov et al. 2016), and group (Wu & He| 2018)) normalization. These all standardize (zero mean, unit variance) and then affinely transform sets of activations in given layer; they differ in the sets of channels and images over which standardization is performed (for all, the sets include all of space). These normalizations prevent or reduce covariate shifts and can have other advantages. The set includes one image for all but batch normalization, which uses all images in batch. The first three do not lead to competition between channels, the either includes a single channel (batch or instance) or all channels (layer), and the same operations are being applied to all channels in the set. However, in group normalization, the channels are divided into non-overlapping groups, with standardization performed separately over each group. Thus there competition within the group - one channel's strong activity can suppress the activity of other channels in the group, relative to the activities of channels in other groups. This is closest to DN, which is competitive.
In our formulation of DN, the group with which neuron is competing changes continuously with the neuron, constituting some local region around each neuron. We take this local region be a single point of space and learnable span of channels about given channel. More precisely, the channels are topologically arranged on a line; the contribution of nearby channels is weighted according to the distance between it and the channel being normalized, by a decaying exponential with a learnable length constant. DN does not prevent covariate shifts, so we will find useful to combine DN with one of the other normalizations.
Our contributions in this paper are, for the first time (to our knowledge), characterizing how a canonical biological operation, DN, learned along with the CNN filters, affects ImageNet and CIFAR- 100 performance and learned representations in a CNN (AlexNet) with and without "canonical" normalizations. In partieular, show:
Addition of DN improves performance for image recognition in AlexNet models with or without canonical normalizations, and the best performance is found by combining both types of normalization;
DN increases the large or medium (depending on presence and type of canonical normaliza- tion) wavelength Fourier modes in the first layer receptive fields.
Both canonical and divisive normalizations reduce the network's manifold capacity and correspondingly change associated geometric measures at interior layers, leading to im- proved manifold capacity and associated changes in geometric measures at the final level, corresponding well to improvements in performance.
DN consistently increases the sparsity of activations (Gini index) each normalization step and in the output layer.
We also find preliminary evidence suggesting that DN can improve out-of-distrrbuutin (OOD) performance. This work should be of interest both to the ML and neuroscience communities, and warrants further study, for example, to understand why DN produces the associated changes in representations, whether and how these changes are related to the improvements in performance, and how performance with DN can be optimized.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
In recent work a neural circuit model was found that produces the neural responses that had been phenomenologically described by DN, along with number of other biological response properties (Ahmadian et al.] 2013: Rubin et al.] 2015). This has raised interest in understanding the possible functions of this Normalization for which there are many hypotheses, of which we mention only a few. It has been postulated to keep activations within an appropriate dynamic range for the neurons (Carandini & Heeger 2012). has been shown to remove higher-order statistical dependencies in responses to auditory visual stimuli (Schwartz & Simoncelli] 2001), and more generally to minimize redundancy. maximize information, or efficiently or optimally encode (Malo & Laparra 2010: Gomez- et al. 2020; Malo 2020; Ballé et al. 2016). It has also been shown arise from statistical inference of the reflectances underlying a model of the statistics of natural scenes, the
Gaussian scale mixture model (GSM) (Coen-Cagli et al. 2012 2015 Echeveste al. 2020).
The original AlexNet (Krizhevsky et al. 2012) included local response normalization (LRN), much like ours (Eq. D but with a linear numerator and the sum in the denominator over 422 neighbors without exponential weighting. Parameter values were hyperparameters set using validation set; the equivalent of our parameter ka/A was 10-4 with k 2, making difficult to understand how LRN could have had much impact. Nonetheless it improved performance, though this was disputed by Simonyan & Zisserman (2015), but in our hands by less than DN (see Table 1). Ren et al. (2016) developed a unified mathematical framework for slightly modified batch, layer, and DN,, combined it with an LI regularizer, and showed that various forms of this (learned) regularized normalizer improved performance on CIFAR- 10 and CIFAR- 100 in network with 3 convolutional and 2 fully connected layers, with the best performance by a modified batch norm. Their DN included in a unit's normalization pool all channels in local spatial region about the unit. Giraldo & Schwartz (2019) explored a flexible, ciimulus-dependee learned parameters, that was added the 2nd layer of pretrained Alexnet model contextual modulation in VI. Others have examined effects of DN on tasks in various biologically-motivaee architectures "Coen-Cagl & Schwartz 2013 Bertalmío et al. 2020). Burg et al. (202I) implemented a learnable form of DN in model trained end-to-end to replicate spike counts of VI neurons. The model included a single convolutional layer of 32 filters with batch normalization, followed by DN and a readout layer. The filters developed with no topology, that is, normalization weights were learned between each directed pair of channels.
form of DN across space, based on the GSM, with
The work most similar to our own was done independently by Pan et al. (2021). They considered a form of DN in which channels were partitioned into groups of 8, which normalized one another, followed by affine transformation. They also considered adding a spatially local normalization pool restricted to a unit's 's own channel. For every unit, the affine transformations and the weights from every member of its normalization pool were learned. They also considered the RNn et al. (2016). They found that, compared to canonical normalizations, their channel normalization, but not the additional spatial normalization nor DN, improved performance on CIFAR-10 10 in shallow convolutional nets but not in deeper ones (4-5 or more layers). They attributed the improved performance on shallow networks to their channel normalization making activity distributions in early layers more Gaussian, Their channel normalization also showed some improvement over canonical normalizations for AlexNet on ImageNet. Our normalization pool size is learned (determined by the space constant of an exponential kernel), we examine pairings of divisive and canonical normalization which we find important to avoid failures to learn and improve performance relative to divisive alone, and we examine several properties- receptive fields and their Fourier power, manifold capacity, sparsity- that characterize ways in which the normalizations change representations.
METHODS
Architecture. We studied 8 models, each a variant of AlexNet (5 convolutional layers and fully connected layers: Krizhevsky et al. 2012) with different normalization layers, with a Kaiming He initialization (He et al. (015) and without pytorch local response normalization (LRN). The filters are 11x11 for ImageNet and 3x3 for CIFAR- 100 in the first layer, 3x3 in all subsequent layers. The order of operations in each convolutional layer is ReLU, then DN if used, then canonical normalization used (Divisive, Batch, Group, Layer).
3

---

Published as a conference paper at ICLR 2022
Normalization Formalisms. For DN, the channels in given layer develop topologically arranged on a line, while normalizing each other over a learnable distance specified by an exponential kernel. Thus the channels, while learning appropriate filters, are also learning appropriate linear arrangements that determine who normalizes whom. Given n. channels in a layer, numbered from 1 to n, we let aa(a)) be the rectified output of the convolution with the filter of channel at 2D spatial position x. We take the unnormalized activation of this channel to be ao(x)². We then divisively normalize, using as "normalization pool" an exponentially weighted sum of the unnormalized activations of nearby channels at the same spatial position, to yield the unit's 's normalized activity be(z))
Ge(x))
be(x))
k (1+
LA 2 - ac (ct)(())e-
Here, B. a, and are all learnable parameters, learned independently for each convolutional layer.
We also considered models in which each divisive normalization was followed by "canonical" normalization: either batch, group, layer. In all three, the normalization of the form: Zn.j = Fn.. Riz.] + B. Here ry and |3 are learnable parameters. The subscript n denotes the
/Vrr(E.)-c
set that is normalized together. For example, in batch normalization, for an input of dimension N x H W, in which e is the number of features, H and W the spatial dimensions and N the number of images in batch. The mean and variance in this equation are calculated for given feature across all of space and the whole batch. For layer normalization, the mean and variance are calculated across the spatial and feature dimensions for each image. For group normalization, the feature dimension is divided into 4 equal-sized groups in each layer, and the normalization is done within each group across space for each image.
Hyperparameters. Unless otherwise specified, the learning rate used in the models was ,01. Batch sizes were 128. The initial normalization parameters were À 10., a 3 1., k = 10, except for the Divisive model with no other normalizations, for which initial X = 1. and 1: = 0.5 to make learning reliable (further discussed in Results). The Weight initialization method followed that He et al. 2015) in which weights are initialized with the same statistics for differing seeds. Specifically, the He formulation for ReLU activation functions is meant keep the expected activation variances constant across layers. We used the same principle for our networks with ReLU plus DN and arrived the same weight initialization. We then used the same initialization for combined divisive/canonical models. See Appendix A and B for more information.
The CIFAR training and validation images were resized to 32 x 32 and horizontally flipped: Imagenet training images resized to 224 x 224 x 3 and horizontally flipped: Imagenet validation images resized to 256 x 256 x 3 and center cropped. Each color channel was always standardized.
4 RESULTS
Top-1 Accuracy. Tables show top-1 accuracies at epoch 90 for the various models on ImageNet and CIFAR-100. The models are named by the type(s) of normalization used. For name with two normalizations, the second normalization was applied immediately after the first, for example DivisiveBatch means divisive and then batch normalization were applied after each convolutional layer and ReLU. NoNorm indicates that none of the normalizations were used, simply a vanilla AlexNet. For ImageNet, these numbers are based on a single seed per model, as the final performance differed by no more than .1 - .2 points across seeds and it was not worth computational resources to do 30 seeds. The CIFAR- 100 results show isteer across 30 trials per model. Performance curves for CIFAR-100 (Fig. I show that performance differences were relatively stable from the earliest epochs.
For NoNorm and each canonical normalization, addition of DN almost always improved performance (exception: layer normalization for CIFAR-100) DivisiveBatch showed the best performance. Con- versely, adding canonical normalization almost always improved performance for DN (exception: Divisive and DivisiveLayer were statistically tied for CIFAR- and also improved reliability of learning. The Divisive model (without canonical normalization) was less able to learn than the others; a parameter search on the DN parameters found that it learned reliably only for small values of X and & (see Eq. []. Applying a canonical normalization to standardize the activations after the DN
4

---

Published as a conference paper at ICLR 2022
Model
Validation Ace
Training Acc
Model
Validation Acc
Training Acc
NoNorm
56.49
53.16
NoNorm
49.45 + .072
56.00 .04
NoNorm LRN
57,55
54.98
Batch
53.96 +
65.79 +
Batch
59.58
56.14
Group
51.55 + .07
63.62 = .05
Group
58.57
57.35
Layer
52.06 + .07
65,00 = .05
Layer
59.87
58.32
Divisive
50.11 .21
58.46 = .32
Divisive
59,39
55.66
DivisiveBatch
54.88 .07
66.35 .03
DivisiveBatch
61.33
59.27
DivisiveGroup
52.75 .15
63.92 = .05
DivisiveGroup
60.35
59.61
DivisiveLayer
50.18 +.11
61.74 - .10
Dive
58.32
Table 2: CIFAR-100 Top-1 Accuracies
Table 1: ImageNet Top-1 Accuracies
CIFAR-100 Performance
Validation Accuracy
50
Training Accuracy
50-
DivisiveLayer
30-
DivisiveGroup
Divisive
20-
Batch
29-
DivisiveBatch
Group
10
Layer
NoNorm
Epoch
Epoch
Figure 1: Performance curves for validation (left) and training (right) accuracy for each model across 30 trials on the CIFAR-100 100 dataset. Standard errors are plotted but are too small to see. Dashed curves are the models with single type of normalization (canonical or divisive), black is the vanilla AlexNet (NoNorm), and solid curves are models with DN followed by a canonical normalization. Performance order is stable across epochs. Every 30 epochs, the learning rate is reduced by factor of 10, hence the jumps in the curves.
made learning reliable. Additionally, across the DivisiveBatch, DivisiveGroupp and DivisivelLayer models, the divisive parameters quickly evolved to their final values, and were similar between the three different models for a given initial condition (Appendix C). While the models tended to perform better with larger À and & when we fixed these parameters (Appendix B), we also found that, when these parameters were learned, the initial divisive parameters chosen did not greatly change performance (Appendix C).
Receptive Fields, Fourier Power, and Orientation Selectivity. DN is a biologically motivated throughout the visual cortical stream. Therefore, better understand how it improves image classification performance, it is informative to study the receptive fields (RFs; i.e., filters) learned by the models. We focus on the 11x11 filters learned for each channel in the first layer of the ImageNet-trained models, which are the filters that are most interpretable and most easily compared those seen VI. The RFs of the models with DN show increased long-wavelength (Divisive and DivisiveLayer) or -waillun-ttaeeeggthhh (DivisiveBatch and DivisiveGroup) Fourier power compared other models (Fig. 2 This can be seen in comparing the first of the 96 RFs for the Divisive model (Fig. 3 to those of the NoNorm model (Fig. 4 (RFs for the other models are shown the Appendix G). Compared to the Divisive model, the NoNorm model RFs have many more irregular and small scale structures, and fewer (and the ceatttatioo-sslectiii simple cells found
computation,
grating-like RFs resembling (e.g., Hubel & Wiesel 1962). Consistent with these results, found that addition of DN increased the median orientation selectivity of RFs in models with Batch or Group normalization (& no significant change in other models; Fig. 31 Appendix K).
Feature Correlation. Figs. illustrate which filters along which features are arranged) and thus have developed while normalizing one another. In Fig. 2 Appendix examine this in another way, plotting the layer-wise 1D correlations between feature
another (in the line topology
5

---

Published as a conference paper at ICLR 2022
Fourier Transform Average Radial Profile
Power
0.1
0.4
Frequency
Power
Frequency
Power
Frequency
Figure 2: The radial Fourier power (t.e., integrated over @ in polar coordinates) averaged over the 96 receptive fields (filters) in layer 1 of AlexNet for each architecture. Columns are red, green, and blue channels respectively. Rows show models with batch (blue lines), group (green lines) and layer (orange lines) respectively; darker color is divisive++anonicall lighter is canonical alone. The purple (Divisive) and black (NoNorm) lines are identical in all three rows of given column, repeated for comparison with the various divisive+canonical and canonical models. Shading shows standard errors. While the small wavelength (high frequency) power is relatively similar from model to model, the long wavelength power is increased for the Divisive and DivisiveLayer models, while the mid- mai-lwweeengtt power increased for the DivisiveGrouu and the DivisiveBatch models.
weights as function of feature separation. DN induces de- or anti-correlation of nearby features, and in some cases induces a reduction correlation among all features regardless of separation.
Replica-based Mean Field Manifold Geometry. When an image classification model is trained, the activity patterns of N neurons for the images in class form manifold corresponding that class in the N-dimensional space of activities. DDCarloo et al. (2012) argued that, in a feed-forwar CNN, these object manifolds are hierarchically untangled such that, at higher layers, the individual classes are more linearly separable. A well performing model should produce object manifolds that are readily separable and therefore distinguishable. The system load of a model, a = f. where is the number of object manifolds and N is the number of neurons (Gardner. 1988; Chung et al. (2018) can measure the potential for model to distinguish P manifolds. The classification capacity of a neural network is the maximum load for which most dichotomies of manifolds are linearly separable. Thus, model that has higher performance on specific dataset is expected to have higher classification capacity of object manifolds. This has been shown in recent studies (Stephenson et al. 2019 Chung S (0220..
Using mean field theory methods developed by Chung et al. 2018). we measured the capacity of object manifolds and their geometric properties This depends on three quantities:
Radius: The average size of a neural manifold across the classes considered.
Dimension: The average embedding dimension of an individual object manifold from the perspective of linear classification of manifold dichotomies.
Correlation: The average correlation between the centers of the individual class neural manifold representations in state space. A lower correlation means that the individual neural manifolds would be more distinguishable (less entangled).
The manifold capacity - the ability to separate the object manifolds is larger for smaller radius, dimension, and correlation.
In Figure 5 we plot the manifold capacity and properties for the NoNorm, Divisive, BavisiveBatch, and Batch models (results involving group and layer normalization, and changes in manifold parame-
'The analysis code used in this work is from https:: wtw..ccomm/cuungg3////////////// manifolés_rege caM'T from prior work by [Stephenson et al. (2019)

---

Published as a conference paper at ICLR 2022
Receptive Fields for Divisive
Figure 3: The receptive fields (RFs) for each the channels for the first features along topological line of 96 features in layer one for the Divisive model show many wide-set Gabor-like filters. (#'s 1-16 are arranged in English reading order, i.e. row has #'s 1-4.)
Receptive Relds for NoNarm
C22
Ch
Ch-2
Co
Cha
011
022
C13
200
Figure 4: The receptive fields for each of the 3 channels for the first 16 features (as in Fig. 3 in layer one for the NoNorm model show a wider variety of filters with more irregularity and more small-scale features. In Figs. the 16 appear fully representative, space precludes showing all 96.
ters after each individual normalization convolutional layer, presented in Appendix D). In the early layers, models using batch and/or divisive normalization, and particularly the Batch model, tend have lower capacities and higher manifold dimension, radii, and correlation compared to NoNorm. The most notable change is the pronounced jump in correlation between manifold centers at interior ReLU layers. By the fully connected layers, the situation has reversed, with the two models involving DN having largest manifold capacity and smallest radius and dimensions, followed by the Batch model, with NoNorm last (for correlation, the three normalized models have the same ordering, but NoNorm is smallest). It is intriguing that somehow, by creating early representations with lower capacity and higher-dimensional object manifolds, the network with these normalizations can arrive in the end at higher-capacity representation with lower-dimensional object manifolds.
Sparsity. Firing sensory cortex relatively sparse, with long-tailed activity distributions (Shafi et al., 2007: Hromadka et al. 2008 O'Connor al. 2010 Barth & Poulet 2012), particularly for responses to natural stimuli (Froudarakis et al. 2014). Both higher areas visual cortex and higher layers of Alex and other deep nets trained for visual object recognition respond better stimuli with naturalistic statistics than those without them (Zhuang et al. 2017). and in the deep higher layers this property is strongly correlated with sparsity of firing and with any other tested property (Zhuang et al. 2017). Sparseness of neural firing has been postulated to have many functions, including improving discrimination, learning, memory capacity, disentangling and linear readout, and other properties by reducing overlap between activity patterns for different stimuli (e.g., Olshausen Field 2004 Ganguli & Sompolinsky/ 2012). More generally, by various criteria there an optimal level of sparsity, such as optimizing a tradeoff between discrimination and generalization (Barak et al. 2013) or optimizing the dimensionality of representation (Litwin-Kumi Kumar et al. 2017).
While there are various measures of sparsity, we use the Gini index, which is designed to capture the kind of sparsity represented by heavy-tailed distributions. The Gini index is a general metric for measuring inequality, typically across countries. It is defined in terms of the Lorenz curve: graph of the cumulative activations each layer (or income) as a function of the cumulative of activations (or incomes) counted, where the activations the most extreme inequality, the curve would be at zero as the x-axis moves from 0 to 100%, and then jump to 1 at 100%, meaning all of the activation is in single unit. In contrast, the diagonal
arranged from smallest to largest. For

---

Published as a conference paper at ICLR 2022
ImageNet Batch Models Manifoold Metrics Radius
Capacity
Dimensions
Correlation
hohorm
0.42
hotkom
Botch
Bitch
D DisiiveBatch
Blei-veBatth
D stavv
0.33
Divsive
0.0250
1.14
Dimensions
Correlation
Capacity
4.6925
Radius
cC-
6.32
2.3
25-
0.25
0.950
Noboom
Nchorrs
0.023
DartkeeBacch Dhrikve
NishuBatchh
0.5
necos
roners
Axto
roweus
roweu
Nects
Retp
rokeut
rehect
Aetu
neci
necs
nect
raneut
rokeup
Figure 5: The mean field theory manifold capacity is calculated on the 50 least correlated classes CChhe et al. 2020) for the ImageNet training data, using responses to 100 images randomly selected for each class. Results are shown for the ReLU layers of the NoNorm, DivisiveBatch, Batch, and Divisive models. Vertical grey lines indicate ReLU layers. Error bars denote the standard error across five different seeds of these 100 randomly selected images each class.
Gini Index for CIFAR-100 1-00 Models
Gini Across Layers
Gini Layers
Gini Across Layers
0.71
97
5.7
0.6
0.0
0.0
Figure 6: The sparsity, as measured by the Gini index most sparse, 0 all activations equal), for the validation set of images at epoch 90, for CIFAR- models. Results are show for canonical (e.g., Batch; dashed color) and combined divisive-cance canonical (e.g.. Divisive Batch, solid color) models involving the batch norm (blue, left panel), the group norm (green, middle), and the layer norm (orange, right), as well as NoNorm (black) and Divisive (purple) models (repeated in all 3 panels, for comparisons). The vertical purple lines indicate where the divisive normalization occurs for models that have it. Similarly the vertical blue, green, and orange lines indicate where a canonical normalization, e.g. batch, group, layer norm respectively, occurs.
represents equality of all activations. The Gini index is defined as the area between the diagonal and the Lorenz curve, divided by the area under the diagonal. Thus, it is I for the case of most extreme inequality or sparseness, and 0 for complete equality.
We examined the Gini index in each layer for the models trained on CIFAR-100, (Distributions of probabilities of activation are in Appendix F.) The Gini index is slightly increased (i.e., sparsity is increased) by divisive normalization layer, while canonical normalization layers strongly reduce the Gini index. In the first four layers, after all normalizations, the Divisive and NoNorm models are most sparse, while canonical and combined divisive-canonical models are less so. By the final, fully-connected layers, the canonical models are most sparse (mean probability activation ~ 10%, Figs. 17 19 Appendix F). followed by combined cavisive-canonical and Divisive (probability ~ 20%), and NoNorm model (probability ~ 30%), with the exception that for layer norm, the DivisiveLayer model is equally sparse the Layer model. Thus divisive normalization strongly impacts sparsity of activations, generally fixing sparsity of the final representation at an intermediate level (~ 20%) but producing stronger sparsity when combined with layer norm.
Out of Distribution (OOD) Images. We investigated whether DN might improve OOD performance in two ways. First, we considered a black box adversarial attack of adding Gaussian noise images
8

---

Published as a conference paper at ICLR 2022
Appendix H). DivisiveGroup performed best against modderate-strnnne attacks, while DivisiveBatch and Divisive Layer performed best against the strongest attacks we tried, providing preliminary evidence that DN can improve OOD performance. Second, given the bias of CNNs to recognize texture while humans are biased toward shape, we considered images in which the texture of one category was combined with the shape of another (Geirhos et al.. 2018) (Fig. 27] Appendix I). Among images for which the network chose either the texture or the shape category, DN led to an increase in the % of shape choices, but the networks with DN also chose other categories more often. This gives some hope of DN improving shape sensitivity, but does not allow clear conclusions.
(Fig. 26
5 DISCUSSION
Divisive normalization a phenomenological model that captures nonlinear response properties seen ubiquitously in sensory cortical areas (Carandini & Heeger 2012). While many functions have been proposed for these properties, why cortical neurons have them remains an open question. This paper has been first exploration of what a learnable DN may achieve computationally in the context of AlexNet and canonical CNN normalizations. We examined how it changed both performance and the learned representations in object recognition task.
Channels were learned in a line topology, with the normalization pool of each unit determined by the channels to either side of it weighted by learnable exponential kernel. DN improved performance on ImageNet and CIFAR- 100 by a few percent, but it was prone to failing to learn. We therefore studied combined models, in which DN was followed by batch, group, or layer norm, which learned much more reliably. A similar stabilizing role biologically might be played by forms of homeostatic plasticity, which keep neural activities in desired operating range (Turrigiano) (01T). (It could be interesting to substitute homeostatic plasticity for canonical normalizations in future studies (Shen) al., 2).T)). For batch, group, and no normalization, addition of DN always improved performance. with the DivisiveBatch model performing best of all studied models.
As measures of how representations are changed by the normalizations, we studied the receptive fields (RFs) of the models' first layer, calculated the sparsity in each layer, and analyzed the neural manifold geometry in each layer. The Divisive model's first layer RFs have increased power at large wavelengths and the combined models similarly have increased power at large and/or mid-scale wavelengths. For DivisiveBatch and DivisiveGroup, the RFs have increased orientation selectivity. Under DN, nearby RFs, which can normalize one another, develop to be de- or anti-correlated relative to other RFs. and in some cases all RFs are de-correlated under DN compared to RFs in models without DN. We found that DN can improve performance OOD images formed by adding Gaussian noise, and by one measure (but not another) can increase bias for recognizing objects by their shape rather than their texture. Models with DN generally led to sparsity of final layer activations intermediate between those of purely "canonical" normalizations (more sparse) and unnormalized models (less sparse). Surprisingly, in earlier layers, the normalized models (except purely divisive) are less sparse than unnormalized ones, which somehow leads to more sparse final representation. similar reversal is seen in manifold capacity and its associated geometric quantities. Normalized models produce representations in earlier layers with lower manifold capacity, yet produce increased capacity in the final layer, with DN models having the highest capacity.
This study raises many questions. Why does DN produce the observed representational changes, and are these correlates of improved performance also causal? How is it that decrease in sparsity or manifold capacity in intermediate layers can yield increase in the final layer? Under what conditions is it best for DN to come from more dissimilar rather than more similar RFs? Might DN have advantages that would be better revealed by other tasks, such as object recognition in more cluttered scenes, object segmentation, or analysis of videos? How will divisively normalizing across space as well channels, and considering 2- or 3-D topologies, alter results?
Here, we have explored how DN alters computations and representations in a simple context, learning object recognition in a CNN, possibly along with other canonical CNN normalizations. Ultimately we would like to understand visual systems that replicate both neural (e.g., See.mmpi et al.) 2018) and psychophysical behaviors of biological systems. This will require learning with more complex and richer contexts, constraints, and combinations of mechanisms including DN, which may alter the function of DN. Here we begin by learning how DN functions in simpler systems that can learn visual tasks. we hope this contributes to fruitful interaction between studies of deep networks and of neuroscience that advances both.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY
All software used in this project will be deposited in a publiely accessible github repository no later than the time of the 2022 ICLR meeting.
ACKNOWLEDGEMENTS
Funded by NSF grants ISS4770938 and DBI-I707399 and the Gatsby Charitable Foundation.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Y. Ahmadian, D. B. Rubin, and K. D. Miller. Analysis of the stabilized supralinear network. Neural Computation, 25:1994-2037, 2013.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. End-to-end optimized image compression. 11 2016.
O. Barak, M. Rigotti, and S. Fusi. The sparseness of mixed selectivity neurons controls the seneeraizaaiionddieriimiaatiii discrimination trade-off. Neurosci, 33:3844-388 3856, 2013.
A. L. Barth and J. F. Poulet. Experimental evidence for sparse firing in the neocortex. Trends Neurosci., 35:345-355, 2012.
M. Bertalmío, A Gomez-Villa A Martín, J Vazquez-Corral D Kane, and Malo. Evidence for the intrinsically nonlinear nature of receptive fields in vision. Scientific Reports, 10, 2020.
Max F. Burg, Santiago A. Cadena, George H. Denfield, Edgar Walker, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Learning divisive normalization in primary visual cor- tex. bioRxiv, 2021. doi: 10. 10.1101/767285. URL htttt::t///weii http///wwwwwoxxx.oooooooo wwtp://tt.oooootttttt faxz1/2222703306/75555
M. Carandini and D. Heeger. Normalization as canonical neural computation. Nat. Rev. Neurosci, 13:51-62, 2012.
Yue Chung, Daniel D. Lee, and Haim Sompolinsky. Classification and geometry general perceptual manifolds. Phys. Rev. X, 8:031003, Jul 2018. doi: 100//hhsseexx...0000000 URL. https:://inkk aps ortps/////////////////////////// 1103 1ttp////e.....////.eeeee..x 8, 031 031003
Cohen U DiCarlo JJ Sompolinksy H Chung Dapello Separable manifold geometry in macaque ventral stream and denns. In Computational and Systems Neuroscience, 2020.
R. Coen-Cagli, P. Dayan, and O. Schwartz. Cortical Surround Interactions and Perceptual Salience via Natural Scene Statistics. PLoS Comput Biol, 8:10002455, 2012.
R. Coen-Cagli, A. Kohn, and O. Schwartz. Flexible gating contextual influences natural vision. Nat Neurosci, 18: 86:1648-666 1655, 2015.
Ruben Coen-Cagli and Odelia Schwartz. The impact on midlevel vision of statistically optimal divisive normalization in vl. Journal of vision, 13 8, 2013.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
J. DiCarlo, D. Zoccolan, and N. Rust. How does the brain solve visual object recognition? Neuron, 73:415-434, 2012.
R. Echeveste, L. Aitchison, G. Hennequin, and M. Lengyel. Cortical-like dynamics in recurrent bassed circuits optimized for sampling- based probabilistic inference. Nat Neurosci, 23:1 138-114449 149, 2020.
E. Froudarakis, P. Berens, A. S. Ecker, J. Cotton, F. H. Sinz, D. Yatsenko, P. Saggau, M. Bethge, and A. S. Tolias. Population code in mouse VI facilitates readout of natural scenes through increased sparseness. Nat Neurosci, 17:851-857, 2014.
S. Ganguli and H. Sompolinsky. Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. Annu Rev Neurosci, 35:485-508, 2012.
E Gardner. The space of interactions in neural network models. Journal of Physics A: Mathematical and General, 21(1) jan 1988. doi: htte:100.003//305544/////111111
URL https:://///....... //doi

---

Published as a conference paper at ICLR 2022
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. CoRR, 11s/1811.12231, 2018. URL
1811. 12231
W. S. Geisler and D. G. Albrecht. Cortical neurons: isolation of contrast gain control. Vis. Res., 32: 1409-1410, 1992.
L. G. S. Giraldo and O. Schwartz. Integrating Flexible Normalization into Midlevel Representations of Deep Convolutional Neural Networks. Neural Comput, 31:2138-21 2019.
Alexander Gomez- Villa, Marcelo Bertalmío, and Jesus Malo. Visual information flow in wil- son-cowan networks. Journal of Neurophysiology, 123(6):2249-2265 2020. doi: 10.1152/jm. PMID: 32159407.
00487.2019. URL http://:://oo..oo//....../. 1152 1152/ jn. 00480.488800 111448.2011 2019
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR, abs/1502.01152, 2015. URL http:
7/arxiv. //rxxiv.org/abss/55000
01852
D. J. Heeger. Normalization of cell responses in cat striate cortex. Vis. Neurosci, 9:181-198, 1992.
T. Hromadka, M. R. Deweese, and A. M. Zador. Sparse representation of sounds in the unanesthetized auditory cortex. PLoS Biol., 6(1):e16, 2008.
D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Physiol., 160:106-154, 1962.
S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing [cs. 2015.
internal covariate shift.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
A. Litwin-Kunn Kumma, K. D. Harris, R. Axel, H. Sompolinsky, and L. F. Abbott. Optimal Degrees of Synaptic Connectivity. Neuron, 2017.
Jesus Malo. Spatio-chromatic information available from different neural layers via gaussianization, 2020.
Jesús Malo and Valero Laparra. Psychophysically Tuned Divisive Normalization Approximately Factorizes the PDF of Natural Images. Neural Computation, 22(12):179-32266 2010. ISSN 0899-7667. doi: 10. 10.110 1021111/NECCCCC a 00046. UtL/tttp:://doo /ttttpp://ddi..og/100..1. 11t.:////00....1//1111.. 00046
D. H. O' O'Connor, S. P. Peron, D. Huber, and K. Svoboda. Neural activity in barrel cortex underlying vibrissa-based object localization in mice. Neuron, 67:1048-1061, 2010.
B. A. Olshausen and D. J. Field. Sparse coding of sensory inputs. Curr Opin Neurobiol, 14:481-487, 2004.
Xu Pan, Luis Gonzalo Sánchez Giraldo, Elif Kartal, and Odelia Schwartz. Brain-inspired weighted normalization for enn image classification. bioRxiv, 2021. doi: 10.1101/2021.05.20. 445029. URL https: htmpp:://wwww h/tp://wwww//ot biorx org http://coonteeeeerrr///////////0
05.20 20. 445029
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark
the robustness of machine learning models. In Reliable Machine Learnnng in the Wild Workshop. 34th International Conference on Machine Learning, 2017. URL http::/ //rovi///xv... org abbs/
1707. 04131
12

---

Published as a conference paper at ICLR 2022
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax. Journal of Open Source Software. 5(53):2607, 2020. doi: 10.21 105/jos000s00... URL :tteps///oo wrggrrg/ggo / C. 20101/111101 joss 02607
Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, and Richard S. Zemel. Normalizing the normalizers: Comparing and extending network normalization schemes. CoRR, abs/161
2016. http:
org/ abs 1611.11100 C4520
D. B. Rubin, S. D. Van Hooser, and K. D.. Miller. The stabilized supralinear network: A unifying circuit motif underlying multi-input integration sensory cortex. Neuron, 85:402-417, 2015.
Martin Schrimpf. Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Daniel L. K. Yamins, and James J.
DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain- like? bioRxiv, 2018 doi: 10. 10.1100/477007. URL https: //www//wwwwww..www www.:.boox.....//coooetee.......
caz11/2015/09/05/00000
O. Schwartz and E. P. Simoncelli. Natural signal statistics and sensory gain control. Nat Neurosci, 4: 819-825, 2001.
M. Shafi, Y. Zhou, Quintana, C. Chow, J. Fuster, and M. Bodner. Variability in neuronal activity in primate cortex during working memory tasks. Neuroscience, 146:1082-11 1108, 2007.
Yang Shen, Julia Wang, and Saket Navlakha. A Correspondence Between Normalization Strategies in Artificial and Biological Neural Networks. Neural Computation, 33(12)33199-300) 11 2021. ISSN 0899-7667. doi: 10. 10.1162/neco__ 01439. URL http::://ti..ooooooo http:////oo...011..1111111 C1439
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Cory Stephenson, Jenelle Feather, Suchismita Padhy, Oguz Elibol, Hanlin Tang, Josh McDermott, and Yeon Chung. Untangling in invariant speech recognition. In NeurlPS, 2019.
G. Turrigiano. Too many cooks? Intrinsic and synaptic homeostatic mechanisms in cortical circuit refinement. Annu. Rev. Neurosci, 14:89-103, 2011.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, fbbs/1607.00022 2016. URL UtL/tttpp://rrxi..orr////
1607. 08022
Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018.
C. Zhuang, Y. Wang, D. Yamins, and X. Hu. Deep Learning Predicts Correlation between Functional Signature of Higher Visual Areas and Sparse Firing of Neurons. Front Comput Neurosci. 11:100, 2017.
13

---

Published as a conference paper at ICLR 2022
CONTENTS
Part
Appendix
Table of Contents
Initialization Weights and Normalization Parameters
B Divisive Normalization Parameter Choices
18
C Divisive Normalization Parameters Learned
20
Neural Manifold Analysis with Group and Layer norm models
22
E Comparison between Theoretical Empirical Manifold Capacity
24
F Sparsity of Activation
25
G Receptive Fields
28
Adversarial Attacks
30
Shape vs Texture Bias
Correlations
35
K Orientation Selectivity
39
14

---

Published as a conference paper at ICLR 2022
A INITIALIZATION OF WEIGHTS AND NORMALIZATION PARAMETERS
We utilize a Kaiming He initialization (He et al. 2013). The traditional Kaiming He initialization is formulated for ReLU activation functions by calculating the variance of the output for layer and then obtaining values for the initial weights such that the network will remain stable. In the original formulation, He utilizes the variance of the response in each layer:
y1 Wix + by
Similar to the original He paper, we take by to be 0 for all B/s. Additionally, wy and 2y are independent of one another and the yt, wi and æq variables random variables.
Then they pass Y: through a ReLU function denoted as:
X f(y
where in the initial He formulation, f(-) is the rectifying function, mar(0. yi-1). In our formulation, instead of ReLU, the function goes through the divisive normalization after the ReLU.
mar(0, y1-1)
X = f(y--1)
((11+ & Σ maa (0
The original He derivation does the following: It takes the variance of yn, taking w and wi to be mean zero.
Var(yi ny V ar or[wwz]]
Var [wx] ni (E [ff x²1 )
Var [w]] = nu (E[
Here we can rewrite and substitute [[x7]) = Var [mx] + R[z]]² and E[w?] = Var[uo]] + [R[uz]2
Var [uo] = n=((Var(z)
(Vaor 3 B/wn)) 2(
Var [m]] ([[ar(s)]Varr(wn Var[w]]B[rr) + Var (mr] [[ur]] B[ar] [[nr]] B[wers (3)
Var [yv] vnn[Valww:/Varll Varlun]B[x]] + Var(z,
Var [yr] nu(Var(wn) (B[a3) [E(x)]?)
Var (ar[ww]BBB B(r]])
Var[y]] (v [v
Above we used the independence of the variables and wy to say E[wyzy] [[u]]"[wu] Additionally, E[[n = as the weights are initialized with mean In the initial He form, æn mar(0, W-11). which symmetry the ReLU function, we can see that E[z]] = 2 Var[yt-1).
ReLU does not have mean 0, but yt-1 does have mean zero. Using
By still modifying the variance of y to consider our the added stages of going through the divisive normalization function
Var [yi] = n, V ar [izm
Var [yn] = ni V ar Varln)
mar (1-
Varlyi] = ny nnVarluu)Vor
k8 (1 2 E max((, y1-1) 1c-51/1)
The purpose this formulation is ultimately to get an expression for Var[wy] such that the weights will not blow up or die out across subsequent layers. Thus, need hone in and simplify the
15

---

Published as a conference paper at ICLR 2022
variance of the quotient. Recalling that the variance of quotient of two random variables and is the form Var[z/y] -Va[(z] [x]] + Yarll [y] 2V ar [2] ar (arlylllooorry corr(z, y) such that
max (0, y1-1)²
Varl kB
B 11)e--)))----113 -
1 +
max yr- 11-1)3 Σ
N
Var(mar000,y11133 Varlk
1+
mar Y1-1)3 1o-51/1 (
]-
j=1
N
(0 x Varlh 1+ a
Σ
max (0. -le--111 ] x corr(;,))
For our purposes, we will assume the correlation between the numerator and the denominator is small enough to be negligible. (This is still possibly a big assumption because the denominator does have "self" term, but we are assuming this self term is small relative to the sum all of the other terms to make the math tenable. This makes little more sense when we assume large X.
max y1-1)²
Varl k (1 ( Σ max (0, yI- 100-
8
m0 y0- 10--11/ )
-Var(mar(O, Varlhe
1+ a 7
(ar Varlk3
1
max yı- 1)38 1k-511-11/3 51/2
Trying actually take the variance of the second term when 3, A, and l are all variable parameters, the math becomes intractable. Particularly because any given power of value 3 will give an intractable binomial expansion. Since this calculation only depends on the initial values of the parameters, initialize B
Var Var[(ut-18) + Var(k: 1 m0x 10-e--1--100 1--5//1 51/A
2
Now in order to make the rest of this tractable, we hone in the second term. The term is calculating the variance of the convolution of the ReLU output with an exponential kernel along the feature dimension.
we notice that lc--l//> will always be bounded by 1, then we can see there will be an upper bound for the value within the parentheses when considering some maximum activation value for given layer. The input images have dimensions H W x N, where is the height, W the width and N. Further, for exponential probability distribution, 98% of the distribution falls within 4) of the center. Therefore, it is a reasonable assumption to approximate an upper bound of the variance to be in terms of 4A neighbors.
Var(k (
N
max y1-1)3e -1- o1/2
Is Varlk (1 (0.
#=1
< Varlk (1+ 4amax (0, Y1-1 )
< Varsk (1
a
*Nmar (0nar(0,,0) 0.y111111 ) )a
x
This upper bound corresponds to using a uniform kernel rather than an exponential kernel which each of the neighbors would be considered equally important up to N neighbors or realistically,
16

---

Published as a conference paper at ICLR 2022
4A neighbors. If the quantity 4a is sufficiently small then this term will drop out when caleulating the variance. This bounds the variance of our expression by zero, and since the variance can only be nonzero term, it is approximately zero. Further noting that Var x] Var and V Yarl Far[ka] = [aVar( where is a constant, we can rewrite the variance
Varlk (
N
a 1
maz v1-133 -e--111 - = Varl(( + 4amax((0 yı )mmx)
A
1=1
I k2 V arll (4amax y1-1)
S 16h a2Varl( v1-1 mma))
I 16k² a² Var Varl(yi-
2
s Sk a ar Varl(y)-1mmm
Thus we can rewrite an approximate upper bound for the variance the divisive normalization term below:
Varl KD (1+ 1 E mar(0, yr- (11
man(0,y1-11)
8)
1
Var Var[()(-122] Varlk
mar(0.yt-1133-eer
2
1 ar ( 2 2 (yr
2
Empirically, early in training that the activations after the ReLUs layers for the NoNor,, Divisive, DivisiveBatch and the Batch models are all on the order of 1. Thus, it is reasonable to assume that if we use small enough and i upon initialization, then the second term will become negligible relative to the first term so we can approximate the overall V or (y]]
Varlyn] miV mnVarlwj 1 ar ar((((n
1 [T| m mn arrwu))
Varlyz] Var[(u)) )a]
Using similar arguments made in the Kaiming He initialization, we want to avoid the magnitudes of the input signals from growing unboundedly. Thus, we can approximately set a sufficient condition for stable learning using the Kaiming He initialization bet requiring:
ni [Varrwu]] 1,
This allows us to initialize the weights of the Divisive model be zero mean with a standard deviation of V
Through this section of the appendix, we have found set of values for the Divisive normalization parameter such that can Kaiming He initialization for the weights when implementing Divisive normalization after each ReLU in AlexNet use the Kaiming He. If we initialize 3=1 and have and & to be reasonably small, we can address the vanishin//exlloingg gradient problem. Namely, these values for a and k are .1 and .5 respectively. However, despite addressing vanishing/exploding gradient problem, we found the Divisive model does not reliably learn isolation when placed in all five layers. This helped motivate including the more canonical normalizations such Batch norm, Group norm and Layer norm after the Divisive normalization layers. The next section disucsses our parameter exploration.
17

---

Published as a conference paper at ICLR 2022
B DIVISIVE NORMALIZATION PARAMETER CHOICES
In the prior section we determined a stable initialization for the divisive parameters to avoid the exploding/vanishigg gradient problem by leaning on the Kaiming He initialization. While this derivation required we fix 3 to be 1.0 and keep and k small, it left À free to vary at initialization. Because this, we explored the parameter space for the Divisive normalization on the CIFAR-100 dataset to determine whether the initial values of these parameters impact learning performance. In particular, we considered initial values for À 1, 2, 5, and and for of .1, .5, 1 and 10. Unfortunately, the Divisive model failed to learn reliably when varying these two parameters, so for the Divisive model we used initial values of 1.0 for À and 0.5 for k. Varying values beyond these usually did not produce Divisive models that could learn. However, the combined models, in which Divisive normalization was paired with Batch, Layer, or Group normalization, learned for wide range of initial values. Figures 8 9 show the validation accuracies of these models for each different parameter pairing of and k. Typically, for the DivisiveBatch model, the initial values did not drastically impact performance. However, for the DivisiveGroup model, there seemed to be improved performance for larger and À values. Further, the DivisiveLayer model performed few points better when the initial À values much of an impact on the DivisiveLayer model. Because of this, all models other than Divisive in the main paper have the same initial values. We chose to initialize k be 10 and À to be 10 both on the CIFAR- 100 dataset and on ImageNet.
larger. The initial value did not seem to have
Performance Varying Divisive Parameters DivisiveBatch
50
A:1.00 k: 0.10
1.00 k: 0.50
1:1.00 k: 10.00
Accuracy
2:2.00 k: 0.10
2: 0.50
2: 10.00
30
5:5.00 0.10
5.500 0.50
A:
A: 10,00
A: 10.00 1.00
10,00 lc: 10.00
0
20
40
60
80
Epoch
Figure 7: Dependence performance on initial normalization parameters À and ls for the Divisive- Batch model. Red, orange, blue, and purple indicate initial À values of 2, 5, and 10 respectively. The solid, dashed, dashdot, and dotted lines correspond to l values of 11 .3, and 10 respectively. The initial value of A does not obviously impact performance. There appears to be very modest improvement with larger & values.
18

---

Published as a conference paper at ICLR 2022
Performance for Varying Divisive Parameters DivisiveGroup
A: 1: k: 0.10
1:000 k: 0.50
A:1.00 k: 1.00
A:1.00 k: 10.00
2:2.00 k: 0.10
Accuracy
2:00 k: 0.50
2:00 k: 1.00
2:2000 k: 10.00
5:5,00 k: 0,50
5:5,00 1.00
20
5:5.00 10.00
10
A:
A:
A: 10.00
0
20
40
60
80
Epoch
Figure 8: Dependence of performance initial normalization parameters and k for the Divi- siveGroup model. Larger À values result in modest performance improvement, as does larger k.
Performance for Varying Divisive Parameters DivisiveLayer
1.00 k: 0.10
A: 1:000 k: 0.50
A:1.00 k: 1.00
1.00 k: 10.00
2:2.00 k: 0.10
Accuracy
A:2.00 k: 0.50
30
2:2.00 k: 1.00
2.00 k: 10.00
5.5.00 k: 0.50
5.5.0 k: 1.00
5.5.00 k: 10.00
A: l:: 0.10
10,00 kc 0.50
A: 10,00 1.00
10.00 10.00
20
40
60
Epoch
Figure 9: Dependence of performance normalization parameters and i for the DivisiveLayer model. There appears to be clear preference in the DivisiveLayer model to initialize with a larger and k. This improvement is more pronounced relative the DivisiveGroup model in which larger only gave modest improvement
19

---

Published as a conference paper at ICLR 2022
C DIVISIVE NORMALIZATION PARAMETERS LEARNED
Using the parameter initializations explored in the prior section we plot the resulting normalization parameters that develop. All of the models learn slightly different divisive normalization parameters depending on the initial conditions. The Divisive model did not learn reliably enough for such a parameter search be informative. However, there are some common trends across the combined models. For example, generally À and & are relatively stable throughout learning. In contrast, the and | values change
initially and settle to some steady state after about 20 epochs.
In the Bivcs model, the learned value tends to get larger with larger initial À and k. Such trend less clear in the DivisiveLayer model. the DivisiveGroup model, it less clear. For example, in layers and 5, it appears larger À will give a smaller a.
Across all of the models in all of the layers, the 3 values range from to 2. the DivisiveBatch and DivisiveGroup models, they tend to be around I or less except in the last layer where 3 gets much larger. the DivisiveLayer model, the learned / parameter steadily increases across the layers. Mcanwhile, the learned steadily decreases across the layers and then increases again in layer 5. We see similar behavior in the DivisiveBatch model, in which a tends to get smaller regardless of the other parameter initializations but then begins increase again in layers 4 and 5. For the DivisiveGroup model, the a values tend to be between 0 and 1 throughout all five layers.
Difrrrer Die Repmeeee
20
65
20
2
20
6
6
25-
27
16
65
20
25
20
Figure 10: The different normalization parameters learned for the DivisiveBatch model over epochs. Each curve represents a slightly different set of initial conditions. Darker lines correspond to larger initial X. It is clear that the values and À values do not vary much.
20

---

Published as a conference paper at ICLR 2022
kpd
byar2
lyya
13
204
160
300
1207
LOoch
60477
10500
Figure 11: The different normalization parameters learned for the DivisiveBatch model over epochs. Each curve represents slightly different set of initial conditions. Similar to the DivisiveBatch model, parameters roughly approach a steady state after about 20 epochs.
1psr2
epp
kyye
kpp
364
58
304
160
35
1
304
45 85
169
35
26
100P
Eooch
Escch
Figure 12: The different normalization parameters learned for the DivisiveLayer model over epochs. Each curve represents slightly different set of initial conditions. In layer 3, B is larger for smaller X and k but layer and 5, modestly gets smaller.
21

---

Published as a conference paper at ICLR 2022
D NEURAL MANIFOLD ANALYSIS WITH GROUP AND LAYER NORM MODELS
While we only discussed a few of the normalized models for the Replica-based mean field geometry analysis in section 4 (primarily Batch, Divisive, and DivisiveBatch models), the other normalized models exhibit similar behavior. We also include the same results as Figure 3 but showing results after each normalization and convolutional layer rather than only for the ReLU layers. In the combined models, at given layer, the divisive normalization tends to decrease the capacity or leave unchanged.
The correlations between the manifolds are much more complex when looking at all of the layers. When only looking the ReLUs,, the normalized models have much higher correlations the interior layers. However, when considering the convolutional and normalization layers. the normalized models have lower correlations. The convolutional layers increase the correlations between the manifolds, while ReLUs decrease the correlations layer-wise. For normalized models, after going through a convolutional ReLL pair, the variation in the correlations is still more narrow than the variation in the NoNorm model. In other words, the correlation after convolutional layer is higher in a NoNorm model relative to a normalized model, but the correlation after the ReLU is lower relative to that in normalized model.
Imagalitt Batch Models Manifold Motrics
Aatfura
voloclace
2.00
3 3.4
180
Banth
190
Figure 13: The mean field theory manifold capacity is calculated on the 50 least correlated classes (Cohen et al. 2020), using responses to 100 images randomly selected for each class, for the ReLU layers of the NoNorm, DivisiveBatch, Batch, and Divisive models. Vertical purple lines indicate where divisive normalization would occur, vertical blue lines indicate where Batch normalization would occur, and vertical grey lines indicate ReLU or convolutional layers. Error bars denote the standard error across five different samples of these 100 randomly selected images in each class. For the models with Batch normalization, there is a consistent jump in number of dimensions that describe the neural manifolds in the early layers. There is also a reduction in capacity in the early layers, more prominently in the Batch model. After each divisive normalization layer in the Divisive model, the correlation is consistently reduced. In the DivisiveBatch model, the divisive normalization layer decreases the correlation in layers 2, 4, and 5. Batch normalization consistently decreases the correlation between the neural manifolds in each layer, while in the Batch model, it consistently increases the correlation.
22

---

Published as a conference paper at ICLR 2022
Imagelet Models Manifold Netrics
Copouity
Brrensane
496
The mean field theory manifold capacity is calculated the 50 least correlated classes Cohen et al. (2020) with 100 images randomly selected for each class for the ReLU layers of the NoNorm, DivisiveGroup. Group, and Divisive models. Vertical purple lines indicate where divisive normalization would be present, vertical green lines indicate where Group normalization would be present, and vertical grey lines indicate ReLU convolutional layers. Error bars denote the standard error across five different samples of these 100 randomly selected images in each class.
Figure 14:
ImageNNtt Layer Modeels Manifoold Metrics
Capacity
Radis
Armendone
Cerraonn
5438
CONC
400
Figure 15: The mean field theory manifold capacity is calculated the 50 least correlated classes Cohen et al. (2020) with 100 images randomly selected for each class for the ReLU layers of the NoNorm, DiyesiveLayer, Layer, and Divisive models. Vertical purple lines indicate where divisive normalization would present, vertical orange lines indicate where Layer normalization would be present, and vertical grey lines indicate ReLU convolutional layers. Error bars denote the standard error across five different samples of these 100 randomly selected images in each class.
23

---

Published as a conference paper at ICLR 2022
E COMPARISON BETWEEN THEORETICAL AND EMPIRICAL MANIFOLD CAPACITY
The manifold capacity reported in section 4 is that calculated using a replica-based mean field theory. The manifold capacity can be also measured empirically with a bisection search to find a critical number of features such that the fraction of linearly separably manifold dichotomies close 1/2 Chung et al. (2018); Cohen et al. (0200). As a sanity check, Figure I6 plots the mean field theory capacity calculations relative the empirically calculated manifold capacity.
NFT Capacity 5 miiio Capacity
Capacit
Smidation Caparity
6.890
0.300
B0001
0.3312
Batth Diratebezt
0.375
0.821
0.00%
0.004
0.300
6.82/
0.82
0.8250
5.82
4.
0.033
0,225
0.30
0.00
0.022
6.044
0.2E
C.08
Siilltten Counity
Figure 16: Left shows the Mean Field Theory (MFT) capacity plotted relative to the simulation capacity values layer by layer. The black line is the unity line. The second plot shows layer-wise MFT capacities with the standard error plotted for each ReLU in the network. The third plot shows the layer-wise simulation capacities plotted for each ReLU. Deviations from the simulation capacity in the MFT capacity plot could indicate real changes in the underlying neural manifold geometry.
24

---

Published as a conference paper at ICLR 2022
F SPARSITY OF ACTIVATION
In the main text we presented the Gini index as a measure of sparsity of the distribution activations. Here we present the full activity distributions across layers for models involving batch (figure (7). group (figure 18 and layer (figure [9) normalization. In the main text we defined the Gini index in terms of the Lorenz curve. It is worth noting that an equivalent definition of the Gini index for different activities ag = 1..... n in a given layer is given by:
04 azl
G
2n j= Dj
Mean Probability of Activation
conv1
Betch
Deosiweaath
Divisive
67
conv2
0.8
62
0.4
1.0
coTv4
0.0
conv5
7.5
3.-
0.8
FC1
25
0.0
FC2
75.
0.
26
Figure 17: Distribution of activities for the validation set images at epoch 90, for CIFAR- 100 models involving batch normalization, along with the divisive and NoNorm models: Batch (lighter blue), DivisiveBatch (darker blue), Divisive (purple), NoNorm (black).
25

---

Published as a conference paper at ICLR 2022
Mean Probability of Activation
Nolom
Group
Divisverrow
conv2
Group
Diesiiv
conv3
NoNorm
DivisvcGroup
Divisive
62
64
066
10.0
15
Nahorm
Group
Dirtivetrroup
conv5
Uroopp
DivisivGrooup
255
Divilive
FC1
155
Natarm
Group
5.0
0.0
02 64
0.6
2.0
10.0
NoNor
Group
DivisivGrrog
Divisive
60
Figure 18: Distribution of activities for the validation set of images at epoch 90, for CIFAR-100 models involving group normalization, along with the divisive and NoNorm models: Group (lighter green), DivisiveGroup (darker green), Divisive (purple), NoNorm (black).
26

---

Published as a conference paper at ICLR 2022
Mean Probability of Activation
NoNorm
Divisivelaye
ConV2
NoNorm
Layer
Divisivel syer
Divisive
conv3
NoNorm
Layer
Divilvolaayer
DMvisive
60
69
04
06
0.
10.0
15
NoMorm
Layer
conv5s
Layer
DivisiveLayer
25
Divitive
FC1
155
NoNorm
5.0
000
62
Q4
08
0.6
100
10.0
NoNorm
Layer
Divisivel
Divistve
Figure 19: Distribution of activities for the validation set of images at epoch 90, for CIFAR-100 models involving layer normalization, along with the divisive and NoNorm models: Layer (lighter orange), DivisiveLayer (darker orange), Divisive (purple), NoNorm (black).
27

---

Published as a conference paper at ICLR 2022
For each neuron, calculate the average probability of activation for all images in the validation set for CIFAR-100. We then plot the distribution of these probabilities for each layer. This is plotted for the activations corresponding to the last of: the ReLU; the divisive normalization; and the canonical (hatch, group, layer) normalization.
G RECEPTIVE FIELDS
We plot the first 25 features and each model to have a closer look at the characteristics of the receptive fields. There is slightly wider range of intensity for the receptive fields in the Divisive model relative to the NoNorm model. For the form, the values of the receptive fields rang in values from -96 to 1.18, whereas the latter range from -.79 to +.94. Visually, is clear the Gabor-like receptive fields wider field and have lower frequency oscillations than those of the NoNorm model. There are fewer small-scale structures in the Divisive model receptive fields.
Perhaps unsurprisingly from the Fourier analysis, there are still small scale fluctuations and structure from the Divhs and the Batch models their receptive fields. Recall the Fourier modes were larger for mid to small scale structure in the combined and canonical models. Interestingly, the Batch model appears have more amorphous and Gaussian- receptive fields as opposed to Gabor-like. While there still some small-scale structures, there are more small-scale structures in DivisiveBatch mode receptive fields.
Receptive Finlds for Divisive
01
01
Ch
Ch2
C1
Ch3
C13
Figure 20: The receptive fields for the first 25 features of the Divisive model. They are clearly wide field Gabor-like receptive fields of various orientations in most channels. Some are more amorphous
Receptive for
20
Figure 21: For the first 25 features of the receptive fields the NoNorm model, there are more small scale fluctuations that are not as prevalent in the receptive fields of the normalized models.
28

---

Published as a conference paper at ICLR 2022
Roceptive
30
Figure 22: For the first 25 features the receptive fields the DivisiveBatch model, there are more small scale fluctuations than those seen in the Divisive model. This could mean that the DivisiveBatch model, while it performs better accuracy, may not be filtering out small scale fluctuations as well.
Receptive Fields for Batch
3.59
Figure 23: The first 25 features of the receptive fields in the Batch model. There are more rounded receptive fields that appear Gaussian-like The larger structures could more large-scale edge detectors.
29

---

Published as a conference paper at ICLR 2022
H ADVERSARIAL ATTACKS
Neural network performance can be drastically reduced with very simple modifications to an image known as adversarial examples. Such examples can take two different forms: white box and black box. White box "attacks" are images which have some information (such as the gradients) about the network that is used to create adversarial examples that disrupt the performance. Black box "attacks" have no information about the model. For example, additive Gaussian noise is a black box attack since the noise added to the images is randomly generated regardless of any of the weights or other parameters of the network.
As a beginning exploration of the performance of the various normalizations against adversarial attacks, we tested the models against three different adversarial attacks: two white box attacks, Projective Gradient Descent (PGD) and Fast Gradient Sign Method (FGSM) Goodfellow et al. (2015); and one black box attack, L2 Additive Gaussian Noise (Figs. 24 26).. We did this using the FoolBox
package developed by Rauber et al. (2020 2017). The L2 Additive Gaussian noise adds Gaussian noise with given L2 size (t.e., standard deviation) to the input. FGSM adds noise to the input with given 1.2 size that corresponds to the direction the gradient of the loss function with respect to the data. The PGD method is similar in that it aims to find perturbation to the pixel that maxmimizes the loss.
While it is hard generalize from these limited studies, a few results are notable. The Layer model is the most robust of the models in the sense that it is the only model that outperforms the NoNorm model for all three models for (almost) all values of attack strengths (all but the strongest Gaussian noise attack). The models with divisive normalization (except DivisiveGroup) are the most robust against the strongest Gaussian noise. They generally do not perform well against white box models.
In the main text, we discuss performance on Out of Distribution (OOD) images. Because the white box attacks design images targeted specific network, we suggest they are not good test of ability to perform well OOD. A black box attack, being simply a class of OOD images without knowledge of the particular network and its weaknesses, provide better test, and that is why we discuss the black box attack in the section on OOD performance.
network's weaknesses based on knowledge of the
30

---

Published as a conference paper at ICLR 2022
FoolBox Adversarial Image Attacks Attack Type: Projective Gradient Descent With Running Avg Window 10
Divisive
NoNorm
Divisivelaatch
DivisiveLayer
DivisiveGroup
0.4
Layer
Group
Accuracy on Adversarial Images
0.3
0.802
0.004
0.006
0.008
0.010
0.012
Epsilon for PGD Attack: ImageNet
Figure 24: Projective gradient descent (PGD) attacks: Using batch sizes of 64, the inputs were given perturbations of L2 norm in the direction of the gradient of the loss function with respect to the input. The Layer model is the most robust. Models with divisive normalization tend to be the least robust.
31

---

Published as a conference paper at ICLR 2022
FoolBox Adversarial Image Attacks Attack Type: Projective Gradient Descent With Running Avg Window 30
Divisive
NoNorm
DivisiveB
DivisiveLayer
DivisiveGroup
Batch
Layer
Group
Accuracy on Adversarial Images
0.0
0.002
0.004
0.006
0.008
0.010
0.014
Epsilon for FGSM Attack: ImageNet
Figure 25: Fast Gradient Sign Method (FGSM) attacks: Similarly to the PGD attack, measures the strength of the attack. Models with divisive normalization perform poorly relative to NoNorm, except for the DivisiveGroup model for range of stronger attacks. The most robust models Layer (weaker attacks) and Batch (stronger attacks).
32

---

Published as a conference paper at ICLR 2022
FoolBox Adversarial Image Attacks Attack Type: LeAddditveeaussiinniisis With Running Avg Window 30
0.8
Divisive
NoNorm
DivisiveB
DivisiveLayer
DivisiveGroup
0.7
Batch
Layer
Group
Accuracy on Adversaria Images
0.3
0.2
0.1
0.0
25
125
Epsilon for LzAddiiiveGausaanNoise Attack: ImageNet
Figure 26: L2 Additive Gaussian Noise attack: Using batch sizes of 64, the inputs were perturbed by random Gaussian noise of standard deviation e. For the largest perturbations, most of the models with Divisive normalization (Divisive, DivisiveBatch. DivisiveLayer) are more robust than the other models. For somewhat weaker perturbations, DivisiveGroup is the most robust of the models.
33

---

Published as a conference paper at ICLR 2022
I SHAPE VS TEXTURE BIAS
We studied shape vs texture bias using the texture-vs-shape package on github developed by (Geirhoss al., 2018). This dataset has 16 shape classes, each with 80 photos. These are subset of the textures and images used in the stylized ImageNet dataset. They use style transfer method to impose different textures onto each image. Each shape class has 80 textures for total dataset of 1280 images. Each image thus has a shape label and a texture label. When model's 's inferred category matched category either the shape or the texture used in the decision, this was counted as shape or texture decision, respectively. The shape bias is the number of shape decisions, divided by the sum of shape and texture decisions. We found that the batch-norm model had a stronger shape bias than no-norm, and, most importantly, that addition of divisive norm to either of these models increased the shape bias (see figure; vertical lines show the mean model represented with corresponding color).
Fraction of 'shape' decisions
0.9
0.8
0.7
0.6
0.5
0.4 0.3
0.2 0.1 0
Shape categories
Divisive
DivisiveBatch
Batch NoNorm human lavg)
0.1
0.2
0.3
0.4
0.5
0.6 0.7
0.8
0.9
1
F Fraction of 'texture' decisions
Figure 27:
However, many of the models' decisions were "other" a category corresponding to neither the shape the texture and the models with divisive normalization had increased "other" decisions (66.8% for Divisive vs. 61.1 % for NoNorm: 72.67% for DivisiveBatch vs. 63.75% for Batch). Thus, the models with divisive normalization had increased shape bias, yet had the same number of or fewer shape decisions out of all categorizations. This gives some hope of divisive normalization and its increased low- or moderate--patiill-frqeeee power improving shape sensitivity, but does allow clear conclusions.
34

---

Published as a conference paper at ICLR 2022
J CORRELATIONS
For the models trained on ImageNet, we examined the correlations (correlation coefficient; cosine of the angle between the vectors) between features as a function of the distance between them along the feature dimension (Figs. 28 30) This can provide insight into the scale of the impact that Divisive normalization has on developing features, and also reveals some more global effects of the different normalizations. We saw similar behavior for the models trained on CIFAR- --F0R-100 (not shown).
In any of the models with divisive normalization, there tends be an anti-correlation induced for nearby neighbors in the first four layers, generally most strongly for the combined divisive/canonical models. That is, features that suppress one another through normalization tend to develop to be anticorrelated. In Layer 5, the DivisiveBatch model shows a weakening, and the DivisiveGroup and DivisiveLayer models strengthening, of positive correlations between nearby features.
In addition, the combined divisive/canonical models tend to globally decorrelate features, so that correlation is across all distances than for the other models. This true most layers all the models. For example, for the batch-norm models (figure 28), the DivisiveBatch model has globally weaker correlations than the other models in all layers except layer 1. Furthermore, the models with the canonical normalizations (Batch, Group, Layer) tend to globally have the most positive correlations (for Batch and Group, particularly for layers 2 and 5; for Layer, for layer 5).
35

---

Published as a conference paper at ICLR 2022
1D Correlations
0.2
0.1
0.0
-0.1
Features
Layer
0.10
Fetura
0.05
0.00
-0.05
50
100
150
200
250
Reatune
0.05
0.00
-0.05
100
150
200
250
300
350
400
Features Layer
0.10
Fuabures
0.05
0.00
-0.05
100
150
200
250
300
350
400
Layer
0.15
Ratunes
0.10
0.05
0.00
50
100
150
200
250
Layer
The mean ID pairwise correlations between features, as function of the distance between them in the feature dimension, each convolutional layer, for DivisiveBatch Dashed), Batch (Blue). Divisive (Purple), and NoNorm (Black) models. Shaded regions indicate the standard error of the pairwise correlations. Fluctuations the correlations grow significantly for further distances the sample size decreases. This is because many of these layers only have 256 384 features, so for correlation distances of 250+ there are very few pairs.
Figure 28:
36

---

Published as a conference paper at ICLR 2022
1D Correlations
0.2
0.1
Group
DirsiveSrop
Divsive
0.0
-0.1
20
40
60
80
Faiturs
0.05
0.00
-0.05
100
150
200
250
Features Layer
0.10
Feturea
0.05
0.00
100
150
200
250
300
400
Layer
0.10
Featunes
0.05
0.00
-0.05
0
50
100
150
200
250
300
350
400
0.15
Falturs
0.10
0.05
0.00
Features
Layer
Figure 29: The mean 1D pairwise correlations between features, function of the distance between them in the feature dimension, for each convolutional layer, for the DivisiveGroup (Green,Dashed)) Group (Green), Divisive (Purple), and NoNorm (Black) models. Otherwise in figure 28
37

---

Published as a conference paper at ICLR 2022
1D Correlations
0.2
Nokerm
0.1
Layer
LasteeLaeer
0.0
-0.1
20
40
60
80
Faiture
0.05
0.00
-0.05
100
150
200
250
Features Layer
0.10
Feture
0.05
0.00
100
150
200
250
300
350
400
Layer
0.10
Featunes
0.05
0.00
-0.05
0
50
100
150
200
250
300
350
400
0.15
Falturs
0.10
0.05
0.00
Features
Layer
Figure 30: The mean 1D pairwise correlations between features, function of the distance between them in the feature dimension, for each convolutional layer, for the Divisivel.ayer Dtrange,Dashed), Layer (Orange), Divisive (Purple), and NoNorm (Black) models.
38

---

Published as a conference paper at ICLR 2022
K ORIENTATION SELECTIVITY
We measured the orientation selectivity of each filter in each color channel in the first layer of the models. We compute orientation tuning curve by taking the Fourier transform of each 11x11 filter, dividing the 2-D wavenumbers k into 12 orientation bins (e.g., if the vector k points between and 15deg or between 180 and 195deg, goes into the bin of orientation 0 15 deg), and calling the response to a given orientation bin the maximum of the amplitude of the Fourier transform for k's within that bin. This is equivalent to the maximum response full-RF sinusoidal grating of any spatial frequency and phase within the given orientation bin. We can then calculate the circular variance (CV), global measure of a tuning curves shape:
F1
CV
FO
where Fl and FO are the amplitudes of the first harmonic and DC of the orientation tuning curve. CV = represents no orientation selectivity, while CV maximum orientation selectivity (nonzero response only to single orientation).
Distribution of CVs
NoNorm
Divisive
Batch
DivisiveBatch
0.8
8,8
8,8
8,8
0.7
0.7
0.7
0.7
0.6
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
011
01
0.0
0.0
0.0
0.0
Figure 31: Violin plots of the NoNorm, Divisive, Batch and DivisiveBatch models indicate a modest drop in the medians and interquartile ranges of the divisively normalized distributions. This difference is significant for DivisiveBatch vs. Batch but not for Divisive vs. NoNorm, see discussion of statistical tests in this Appendix.
To verify whether the CV distributions for the NoNorm vs. Divisive and the Batch vs. DivisiveBatch models are in fact different, we conduct two sample Solmogorov-Smirnoy (KS) test. The KS test tests whether two different sets of data come from the same distribution. While we do not calculate statistically significant p- p-vllue for NoNorm vs Divisive (p- = .151), for Batch vs BavisiveBatch, we calculate that the CV distributions are in fact different (p- = .000197). We also calculate statistically significant value for Group vs DivisiveGroup (p- = .012).
Next we use the Mann- Whitney test, which determines whether samples from one distribution are significantly larger than samples from the other distribution. Again, Batch vs. Bavhh and Group vs. Divisive Group are significantly different. Since in both cases the model with
39

---

Published as a conference paper at ICLR 2022
Divisive normalization has the lower median (medians: Batch, .364; DivisiveBatch, .332; Group, .369; DivisiveGroup. .336), we take this to mean that in these two cases the addition of Divisive normalization increases orientation selectivity.
Finally, we calculate the skews of the CV distributions. A negative skew means there is more bulk on the right and long tail on the left, positive value implies the opposite (right-skewed). Thus a positive skew suggests a greater bulk of units with low CV (higher orientation selectivity). For Batch and Group, addition of Divisive normalization converted negative skew a positive skew, again consistent with Divisive normalization causing lowering of CV's for these two models. That the skew for Batch (-.0499) shifts to being positive for DivisiveBatch (.2064) and the skew for Group (-.099 shifts to positive for DivisiveGroup (.304). Thus, there is shift of the CV values be smaller with more positive skew in the divisively normalized models, leading to increased orientation selectivity.
40


Reviews:
[{"review_id": "rAuiDV1VD2d", "reviewer": "Reviewer_Qotw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The incorporation of biological nonlinearities in artificial networks is interesting both to enrich architectures in machine learning and to better study computational principles in neuroscience. Therefore, studies like this are (and will be) of interest for the ICLR audience.\n\nI advocate for the publication of this work in the conference when the authors address/comment on the following concerns:\n\nA. LIMITATIONS OF THE CONSIDERED NORMALIZATION AND TASK OBSCURE THE MESSAGE: clarify discussion!\n\nWhile I trust the reported results (given the use of current-and-standard analysis tools, e.g. Chung18), I feel the 2%-improvement message (as it is written now) can be misleading given the restricted nature of the normalization and the specific training task. The same is true about the evolution of sparsity or the decorrelation between the responses: all depends on the details of the selected interactions in the normalization. The modest results may give the impression that biological normalization is not that different from the conventional normalizations already in use, and hence it may not be worth pursuing that path.\n \nI am persuaded that (1) using more general (and \"explainable\") divisive normalization schemes and (2) considering visual tasks that require the specific adaptation that can be obtained from \"explainable\" relations between \"explainable\" features, would certainly make a big quantitative difference. And the results would be qualitatively \"explainable\".\n\nRegarding (1), including interaction between neurons tuned to different spatial locations (which is not considered in this work) is crucial to compute local illumination and local contrast, and these are fundamental to make proper classification decisions in scenarios with uncontrolled illumination, shadows or fog. On the other hand, interaction between neurons tuned to similar textures (similar in spatial frequency and orientation) is crucial to increase the statistical independence among responses. This will lead to better information transmission and eventually better classifications. \nNormalization between sensors tuned to similar features should also happen at higher abstraction levels(e.g. normalization of similar mid-level primitives or similar objects). This certainly happens in the human visual system (see Webster JoV 2011 on adaptation and masking). In this regard, a major problem of the considered divisive normalization is that it takes a limited line topography in the features. By doing so, in order to get an effective normalization, the evolving features should organize themselves along that line in certain order to let the Gaussian neighborhoods of evolving width do their job. If this order is not enforced in anyway, it is unlikely that it happens leading to a suboptimal normalization.\nA possible way to solve this could use an analysis of the emerging receptive fields and a reconfiguration of the Gaussian neighborhoods in the normalization depending on the observed similarities of the receptive fields.\n \nRegarding (2), instead of using generic databases, one could control the training conditions to enforce controlled invariances (color constancy, contrast invariance, invariances to simple geometrical transforms, etc.). These tasks would favour specific masking interactions between specific sensors whenever these sensors are identified as stated above. As a result, (i) the appropriate neighborhoods could be easier to learn, and (ii) the function of the final interaction kernel would be easier to understand.  \n\nThe limitations of the considered interaction kernel in the Div.Norm. and the limitations of the training sets (or tasks) should be more explicitly stated in the discussion. It has to be acknowledged that results could be more compelling and explainable if these limitations are solved. I am not suggesting to solve them now, since there is no time in the reviewing process, but this should be included in the discussion since it may inspire further work.\n\nB. IS THE FINAL DIVISIVE NORMALIZATION BIOLOGICALLY SENSIBLE?\n\nBiological plausibility is central for cross fertilization from machine learning to neuroscience. The considered normalization is certainly inspired by the biological models, but, in the end, do the final results make biological sense?. This is not checked in the current work in any way. \n\nThis question is not independent from previous comment A: for instance, Martinez et al. Front. Neurosci. 2019 showed that normalization models with limited architecture (i.e. simplistic neighborhoods missing relevant interactions) may lead to good performance in perceptually sensible tasks, but they may fail to reproduce classical psychophysics. In this mutidisciplinary (machine learning-and-neuroscience) context, it is relevant to check the biological plausibility of the models that emerge after training. \n\nA possibility would be reproducing human image quality opinion from the final stage of the network before the dense classification layers (following Zhang IEEE CVPR 2018 or Hepburn IEEE ICIP 2020). Do the models with Divisive Normalization provide a more accurate reproduction of human opinion?. \nAnother option (following Martinez Front. Neurosci. 19) is checking classical psychophysics such as the Contrast Sensitivity Function or the nonlinear responses to luminance and contrast.\n\nAt least, the need of these safety checks has to be included in the discussion because optimization on a visual task does not guarantee biological plausibility.\n\nC. OTHER ANALYSIS\n\nPrevious comments imply that other analysis could have been performed in the final model:\n\n- What are the features that fall within the same neighborhood?. Can you take one receptive field and show the receptive fields that fall in its neighborhood (or \\lambda distance)?. Are these receptive fields similar? \n\n- Can you give explicit evidences of the biological meaningfulness of the models with Divisive Normalization?\n\n- The consideration of extra computations as Div. Norm. increases the number of paramters. Therefore, does the increase in the performance comes from the function of the new layer or just from the increased flexibility of the model?. Assessment of the accuracy of the model independent of the number of parameters should be performed.\n\nD. LITERATURE (missing citations)\n\n* There is literature on advantages of biological Divisive Normalization (with DN tuned according to biological goals). This literature, focused on biological interpretation of the parameters, assumes certain linear receptive fields (such as Gabor/Wavelet/DCT basis) and studies the advantages of representations with divisive normalization. In this context, the first paper to analyze the impact of Divisive Normalization on image classification (with fixed V1-like Gabors) was Coen-Cagli & Schwartz JoV 2013. Other papers focus on measuring the redundancy reduction ability of biological Divisive Normalization (e.g. Malo & Laparra Neural Computation 2010, Gomez-Villa et al. Journal of Neurophysiol. 2019). Even more meaningful than the redundancy is looking at the transmitted information from the input up to the internal representation (Malo J. Math. Neurosci. 2020). In all these cases, psychophysically meaningful divisive normalizations including Gaussian pooling over space and different frequency channels substantially reduce redundancy and maximize the transmitted information wrt versions with non-biological parameters.\n\n* Other literature optimizes Divisive Normalization in deep learning context (as in this ICLR submission): e.g. DN for optimal rate-distortion image coding in Balle et al. ICLR 2017. Other examples that use a biological nonlinearity instead of or on top of the conventional nonlinearities in deep learning include Bertalmio et al. Sci. Rep. 2020 that generalize Wilson-Cowan layers (which are equivalent to Div.Norm.) and used these nonlinear layers in artificial architectures in a number of applications (classification, adversarial attacks, etc.).\n\nE. MINOR \n\n- The authors show the \"first\" receptive fields at some layer with and without normalization. However, how do they order the filters? what does it mean \"first\"?\n\n- Analysis of RGB channels (as in Figs. 2-4) is not meaningful. In fact, quite soon in the pathway, opponent chromatic channels should appear and hence, achromatic, red-green and yellow-blue channels are more meaningful than RGB.\n\n\nREFERENCES\n\n[Balle17]  Johannes Ballé, Valero Laparra, Eero P. Simoncelli. End-to-end Optimized Image Compression. Int'l Conf on Learning Representations (ICLR), Apr 2017 2017\n\n[Bertalmio20] Bertalmío, M., Gomez-Villa, A., Martín, A. et al. Evidence for the intrinsically nonlinear nature of receptive fields in vision. Sci Rep 10, 16277 (2020). https://doi.org/10.1038/s41598-020-73113-0\n\n[Coen-Cagli13] Ruben Coen-Cagli, Odelia Schwartz; The impact on midlevel vision of statistically optimal divisive normalization in V1. Journal of Vision 2013;13(8):13. doi: https://doi.org/10.1167/13.8.13.\n\n[Gomez-Villa19] Alexander Gomez-Villa, Marcelo Bertalmío, and Jesus Malo. Visual information flow in Wilson–Cowan networks. Journal of Neurophysiology 2020 123:6, 2249-2268\n\n[Malo10] Jesús Malo, Valero Laparra; Psychophysically Tuned Divisive Normalization Approximately Factorizes the PDF of Natural Images. Neural Comput 2010; 22 (12): 3179–3206. doi: https://doi.org/10.1162/NECO_a_00046\n\n[Malo20] J. Malo. Spatio-chromatic information available from different neural layers via Gaussianization. The Journal of Mathematical Neuroscience; Vol. 10, Iss. 1, (Dec 2020). DOI:10.1186/s13408-020-00095-8\n\n[Martinez19] Martinez-Garcia M, Bertalmío M and Malo J (2019) In Praise of Artifice Reloaded: Caution With Natural Image Databases in Modeling Vision. Front. Neurosci. 13:8. doi: 10.3389/fnins.2019.0000\n\n[Hepburn20] A. Hepburn, V. Laparra, J. Malo, R. McConville and R. Santos, \"Perceptnet: A Human Visual System Inspired Neural Network For Estimating Perceptual Distance,\" 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 121-125, doi: 10.1109/ICIP40778.2020.9190691.\n\n[Webster11] Michael A. Webster; Adaptation and visual coding. Journal of Vision 2011;11(5):3. doi: https://doi.org/10.1167/11.5.3\n\n[Zhang18] R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE CVPR, 2018, pp. 586–595", "summary_of_the_paper": "The manuscript 4523 studies the impact of a simplified Divisive Normalization in Alexnet in standard image classification tasks. \nIn particular the authors analyze the performance, the nature of the receptive fields, the geometry of the image class manifolds, and the sparsity of the responses.\n", "summary_of_the_review": "The incorporation of biological nonlinearities in artificial networks is interesting both to enrich architectures in machine learning and to better study computational principles in neuroscience. Therefore, studies like this are (and will be) of interest for the ICLR audience.\nI advocate for the publication of this work in the conference when the authors address/comment on the following concerns:\nA. LIMITATIONS OF THE CONSIDERED NORMALIZATION AND TASK OBSCURE THE MESSAGE: clarify discussion!\nB. IS THE FINAL DIVISIVE NORMALIZATION BIOLOGICALLY SENSIBLE?\nC. OTHER ANALYSIS\nD. LITERATURE (missing citations)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CRULY7oRF-2", "reviewer": "Reviewer_nq4A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, the figure labels are too small and it is difficult to extract the information that they are intended to show.\n\nI think the idea of arranging filters is neat. But this should really be motivated better (e.g. discuss topographic ICA or recent work on topographic CNNs). Also, it would be interesting to see the learned topography.\n\nThe brief conclusion section about robustness is interesting but feels ad-hoc.\n\nThe appendix looks like a giant dump of plots and additional information. While there is no constrant on appendix length, it would still be nice to arrange and filters this better, so that a reader looking for specific details could find them more easily. It is also unreasonable to expect that reviewers go through >20 pages of appendix to verify its correctness.\n\nComments:\n- Why study only AlexNet and no newer CNN architecture?\n- The contributions list is rather a list of findings\n- I do not understand the first sentence of section 2\n- I think Balle & Simoncelli's GDN (ICLR, 2017) model might fit well into the related work\n- What performance do you see with the original AlexNet, i.e. keeping LRN?\n- Other works (e.g. Burg et al.) have shown that the power in eq. should also be learned. Why did you fix it to 2?\n- Section 4: by 'trial', do you mean 'seed' on Cifar100?\n- 'Receptive fields and Fourier power': The second sentence starts with 'therefore', this seems like a non sequitur.\n- Figure 2 is illegible, there is too much information and it is not at all clear what to look at or what this is supposed to show.\n- You write that a model with higher performance also has higher capacity, this seems wrong in the general case.\n- In the sparsity section you write 'respond better', what does that mean? Higher response value given fixed contrast or something like that?\n- The Gini index is a little confusing, why don't you just measure entropy? This would seem more easy to follow for an ML audience.\n- Fig5 How would these curves look like for Cifar100 across seeds? I.e. is this a significant observation. Why did you use only 100 images and what do you mean by least correlated classes? Also, more generally, what have we learned from looking at these metrics?\n- Error bars for Fig6?\n\nSmall suggestions:\nNN's -> NNs", "summary_of_the_paper": "The authors study the effect of divisive normalization on AlexNet. They show that, when combined with standard normalization schemes, it increases performance. They also investigate the filter shapes, manifold capacity and (adversarial) robustness of the learned representations.\n\nFollowing the authors' response I have increased my score form 5 to 6.", "summary_of_the_review": "The paper explores an interesting but not novel idea. The analysis is extensive, but one is left wondering what we have learned from the study. More generally, there is a mixture of ideas and analyses which are interesting but somehow not well connected in this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BsKySFV684H", "reviewer": "Reviewer_h6d9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe results show on the one hand a slight increase in performance. Interestingly and surely more significant in terms of the scope of this paper are the structural changes that are made in the network and in particular the increase in the level of sparsity of the activity in the different layers as well as on the shape of the receptive fields. \n\nIn general the paper is very well presented and the results are very convincing. I have some minor points that I detail below\n\n- The abstract in particular is too long and should be cut in half.\n\n- In each layer, the channels are arranged topologically along a line. Some computational neuroscience models especially those representing orientations are arranged according to a circular typology. Have you tested this option?\n\n- In figure 1, what happens during learning around epoch 30?\n\n- In figures 2, 5, 6 the characters describing the X and Y axes are too small to be readable.\n\n- In figure 4 and four the receptive fields could be shown in color in order to show all the receptive fields higher.\n\n- Is the choice of the AlexNet model important for the generality of the results or does this also apply to other networks such as VGG?\n\n- Finally, the weights of the normalization model are learned during the training. On the other hand, you do not analyze the weights that are obtained at convergence and whether they can be interpreted in terms of the relations between the receptive fields, as can be done in neuroscientific models like those of Schwartz and Simoncelli.\n\n", "summary_of_the_paper": "In this paper, the authors propose to study the influence of a feature of biological neural networks on a classical deep learning network. In particular, the authors are interested in including a divisive normalization like the one characterized in the primary visual cortical era and to see the influence of this mechanism on the performance of the AlexNet network.\n", "summary_of_the_review": "These findings could help explain a largely unanswered question in neuroscience, which is to understand why divisible normalization, which is first and foremost a phenomenological heuristic, exists in biological networks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9J8hm0MgtS", "reviewer": "Reviewer_1XvT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Divisive normalization is a crucial computation extensively studied by the neuroscience community; it is an important research question to compare and contrast this computation with the more recently proposed normalization techniques (batch / layer / group normalization) that work well with deep networks (DNNs).\n- I find the writing to be clear and readable overall, I appreciate the clarity in describing the various normalization computations in Sec. 3. \n- The authors show that adding divisive normalization in addition to another normalization technique (batch normalization) produces the best performing image classification model on two datasets (ImageNet, CIFAR-100)\n- The authors perform an extensive set of visualization, representational, and Fourier analyses to understand why divisive normalization gives the above performance boost on image classification.\n- It is interesting that as shown in Fig.2, the divisive normalization computation generally increases power in lower frequency bands. I believe divisive normalization performs a kind of smoothing operation with the exponential weighting that leads to increased sensitivity to lower frequency information.\n\nClarification requested:\n(i) I would have been able to more confidently trust the ImageNet analyses had there been replication of the increased accuracy over multiple random seeds / initializations. In case this is time consuming on ImageNet, the authors could have used the ImageNet-100 dataset standardized in [1] which is smaller in size and faster to prototype.\n(ii) Were the AlexNet parameters of all the 8 models compared in Fig.1 initialized to the same values? I could not find this information in the text, it would be good to add this information as it is important to ensure fair comparison of these architectures without any differences caused by a better AlexNet initialization that is independent of normalization.\n\nWeaknesses:\n- I don't think it is fair to conclude that divisive normalization produces first convolution layer kernels that look more like oriented Gabors while only comparing the first 16 kernels. Why did the authors only compare the first 16 kernels as opposed to looking at all 96 kernels? Despite this, I think the right comparison would be to quantitatively measure the orientation selectivity of the divisive and baseline models using sinusoidal grating stimuli for more concreteness in this analysis.\n- The analyses conducted that suggest increased power in lower frequency spectrum, increased sparsity and manifold capacity are interesting, however, I am unable to coherently connect them to functional advantage (or disadvantage). The authors must better attempt to address why each of these findings are important and how they relate to the improved classification performance.\n- Suggestion: Since the authors have ImageNet-trained models, it would be straightforward to evaluate whether the models with divisive normalization also perform better on out-of-distribution images (e.g. in [2]) to correlate some of the findings about low-frequency sensitivity to better generalization or shape-bias.\n- Minor change: I would suggest using different markers for the various models in Figure 1 and 2 for better readability and accessibility.\n\nReferences:\n1. Tian, Yonglong, Dilip Krishnan, and Phillip Isola. \"Contrastive multiview coding.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer International Publishing, 2020.\n2. Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., & Wichmann, F. A. (2018). Generalisation in humans and deep neural networks. arXiv preprint arXiv:1808.08750.", "summary_of_the_paper": "The authors study the role of the biologically realistic divisive normalization computation in the context of deep learning models trained to perform image classification tasks. The authors compare divisive normalization (scaling neuronal response by exponentially weighted sum of its neighbors) with those normalization techniques commonly used in machine learning models (such as batch normalization, layer normalization etc) and find that their implementation of divisive normalization provides improved image recognition performance. The authors perform a range of analyses to representationally understand why divisive normalization provides the above-mentioned increase in classification accuracy.", "summary_of_the_review": "I find that this paper presents promising evidence suggesting that divisive normalization in combination with batch normalization can improve classification performance. The authors attempt to explain the performance gain by analyzing the representation of divisive normalization model in comparison to baseline models without normalization (or with other types of normalization). The findings are interesting but I am finding it difficult to grasp the main takeaway from the paper, especially since the gains produced by divisive normalization are quite small. I do not recommend acceptance of this paper at the current stage, but I am willing to change my decision if my comments above are addressed by the authors during the rebuttal phase.\n\nKey factors I am concerned about that can influence my score:\n1. Answers to the clarification requested in the main review above.\n2. More clarity on connecting the representational analyses of divisive normalization models and other baselines with functional advantages (such as the performance gain on ImageNet and CIFAR-100).\n3. I would be very impressed if the divisive (or divisive batch) models perform significantly better than other baselines on the OOD generalization benchmark referred in my main review (or any other comparable OOD generalization task that the authors find relevant).\n\n======== UPDATE AFTER AUTHOR RESPONSE ==========\nI am increasing my score to 6 and am recommending acceptance of this paper. I am updating my initial opinion of this paper, in my opinion the paper performs interesting experiments and analyses on OOD generalization and feature selectivity benefits produced by divisive normalized networks. I am not giving the paper an even higher score since I feel that these benefits / gains are quite small compared to just using batch normalization.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores addition of a version of divisive normalization to AlexNets and
compares performance and other measures of these networks to those
with more commmonly used normalization schemes (batch, group, and
layer norm).  Various tests are performed to explore the effect of
their divisive normalization.

Scores were initially mixed but after clarifications for design and
experiment decisions, and experiments run in response to comments by
the reviewers the paper improved significantly.  While reviewers still
had several suggestions for further improvements, after the authors' 
revisions reviewers were in favor of acceptance which I support.
Decision: Accept (Poster)