Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study the role of the biologically realistic divisive normalization computation in the context of deep learning models trained to perform image classification tasks. The authors compare divisive normalization (scaling neuronal response by exponentially weighted sum of its neighbors) with those normalization techniques commonly used in machine learning models (such as batch normalization, layer normalization etc) and find that their implementation of divisive normalization provides improved image recognition performance. The authors perform a range of analyses to representationally understand why divisive normalization provides the above-mentioned increase in classification accuracy.
main_review: Strengths:
- Divisive normalization is a crucial computation extensively studied by the neuroscience community; it is an important research question to compare and contrast this computation with the more recently proposed normalization techniques (batch / layer / group normalization) that work well with deep networks (DNNs).
- I find the writing to be clear and readable overall, I appreciate the clarity in describing the various normalization computations in Sec. 3. 
- The authors show that adding divisive normalization in addition to another normalization technique (batch normalization) produces the best performing image classification model on two datasets (ImageNet, CIFAR-100)
- The authors perform an extensive set of visualization, representational, and Fourier analyses to understand why divisive normalization gives the above performance boost on image classification.
- It is interesting that as shown in Fig.2, the divisive normalization computation generally increases power in lower frequency bands. I believe divisive normalization performs a kind of smoothing operation with the exponential weighting that leads to increased sensitivity to lower frequency information.

Clarification requested:
(i) I would have been able to more confidently trust the ImageNet analyses had there been replication of the increased accuracy over multiple random seeds / initializations. In case this is time consuming on ImageNet, the authors could have used the ImageNet-100 dataset standardized in [1] which is smaller in size and faster to prototype.
(ii) Were the AlexNet parameters of all the 8 models compared in Fig.1 initialized to the same values? I could not find this information in the text, it would be good to add this information as it is important to ensure fair comparison of these architectures without any differences caused by a better AlexNet initialization that is independent of normalization.

Weaknesses:
- I don't think it is fair to conclude that divisive normalization produces first convolution layer kernels that look more like oriented Gabors while only comparing the first 16 kernels. Why did the authors only compare the first 16 kernels as opposed to looking at all 96 kernels? Despite this, I think the right comparison would be to quantitatively measure the orientation selectivity of the divisive and baseline models using sinusoidal grating stimuli for more concreteness in this analysis.
- The analyses conducted that suggest increased power in lower frequency spectrum, increased sparsity and manifold capacity are interesting, however, I am unable to coherently connect them to functional advantage (or disadvantage). The authors must better attempt to address why each of these findings are important and how they relate to the improved classification performance.
- Suggestion: Since the authors have ImageNet-trained models, it would be straightforward to evaluate whether the models with divisive normalization also perform better on out-of-distribution images (e.g. in [2]) to correlate some of the findings about low-frequency sensitivity to better generalization or shape-bias.
- Minor change: I would suggest using different markers for the various models in Figure 1 and 2 for better readability and accessibility.

References:
1. Tian, Yonglong, Dilip Krishnan, and Phillip Isola. "Contrastive multiview coding." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. Springer International Publishing, 2020.
2. Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., & Wichmann, F. A. (2018). Generalisation in humans and deep neural networks. arXiv preprint arXiv:1808.08750.
summary_of_the_review: I find that this paper presents promising evidence suggesting that divisive normalization in combination with batch normalization can improve classification performance. The authors attempt to explain the performance gain by analyzing the representation of divisive normalization model in comparison to baseline models without normalization (or with other types of normalization). The findings are interesting but I am finding it difficult to grasp the main takeaway from the paper, especially since the gains produced by divisive normalization are quite small. I do not recommend acceptance of this paper at the current stage, but I am willing to change my decision if my comments above are addressed by the authors during the rebuttal phase.

Key factors I am concerned about that can influence my score:
1. Answers to the clarification requested in the main review above.
2. More clarity on connecting the representational analyses of divisive normalization models and other baselines with functional advantages (such as the performance gain on ImageNet and CIFAR-100).
3. I would be very impressed if the divisive (or divisive batch) models perform significantly better than other baselines on the OOD generalization benchmark referred in my main review (or any other comparable OOD generalization task that the authors find relevant).

======== UPDATE AFTER AUTHOR RESPONSE ==========
I am increasing my score to 6 and am recommending acceptance of this paper. I am updating my initial opinion of this paper, in my opinion the paper performs interesting experiments and analyses on OOD generalization and feature selectivity benefits produced by divisive normalized networks. I am not giving the paper an even higher score since I feel that these benefits / gains are quite small compared to just using batch normalization.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript 4523 studies the impact of a simplified Divisive Normalization in Alexnet in standard image classification tasks. 
In particular the authors analyze the performance, the nature of the receptive fields, the geometry of the image class manifolds, and the sparsity of the responses.

main_review: The incorporation of biological nonlinearities in artificial networks is interesting both to enrich architectures in machine learning and to better study computational principles in neuroscience. Therefore, studies like this are (and will be) of interest for the ICLR audience.

I advocate for the publication of this work in the conference when the authors address/comment on the following concerns:

A. LIMITATIONS OF THE CONSIDERED NORMALIZATION AND TASK OBSCURE THE MESSAGE: clarify discussion!

While I trust the reported results (given the use of current-and-standard analysis tools, e.g. Chung18), I feel the 2%-improvement message (as it is written now) can be misleading given the restricted nature of the normalization and the specific training task. The same is true about the evolution of sparsity or the decorrelation between the responses: all depends on the details of the selected interactions in the normalization. The modest results may give the impression that biological normalization is not that different from the conventional normalizations already in use, and hence it may not be worth pursuing that path.
 
I am persuaded that (1) using more general (and "explainable") divisive normalization schemes and (2) considering visual tasks that require the specific adaptation that can be obtained from "explainable" relations between "explainable" features, would certainly make a big quantitative difference. And the results would be qualitatively "explainable".

Regarding (1), including interaction between neurons tuned to different spatial locations (which is not considered in this work) is crucial to compute local illumination and local contrast, and these are fundamental to make proper classification decisions in scenarios with uncontrolled illumination, shadows or fog. On the other hand, interaction between neurons tuned to similar textures (similar in spatial frequency and orientation) is crucial to increase the statistical independence among responses. This will lead to better information transmission and eventually better classifications. 
Normalization between sensors tuned to similar features should also happen at higher abstraction levels(e.g. normalization of similar mid-level primitives or similar objects). This certainly happens in the human visual system (see Webster JoV 2011 on adaptation and masking). In this regard, a major problem of the considered divisive normalization is that it takes a limited line topography in the features. By doing so, in order to get an effective normalization, the evolving features should organize themselves along that line in certain order to let the Gaussian neighborhoods of evolving width do their job. If this order is not enforced in anyway, it is unlikely that it happens leading to a suboptimal normalization.
A possible way to solve this could use an analysis of the emerging receptive fields and a reconfiguration of the Gaussian neighborhoods in the normalization depending on the observed similarities of the receptive fields.
 
Regarding (2), instead of using generic databases, one could control the training conditions to enforce controlled invariances (color constancy, contrast invariance, invariances to simple geometrical transforms, etc.). These tasks would favour specific masking interactions between specific sensors whenever these sensors are identified as stated above. As a result, (i) the appropriate neighborhoods could be easier to learn, and (ii) the function of the final interaction kernel would be easier to understand.  

The limitations of the considered interaction kernel in the Div.Norm. and the limitations of the training sets (or tasks) should be more explicitly stated in the discussion. It has to be acknowledged that results could be more compelling and explainable if these limitations are solved. I am not suggesting to solve them now, since there is no time in the reviewing process, but this should be included in the discussion since it may inspire further work.

B. IS THE FINAL DIVISIVE NORMALIZATION BIOLOGICALLY SENSIBLE?

Biological plausibility is central for cross fertilization from machine learning to neuroscience. The considered normalization is certainly inspired by the biological models, but, in the end, do the final results make biological sense?. This is not checked in the current work in any way. 

This question is not independent from previous comment A: for instance, Martinez et al. Front. Neurosci. 2019 showed that normalization models with limited architecture (i.e. simplistic neighborhoods missing relevant interactions) may lead to good performance in perceptually sensible tasks, but they may fail to reproduce classical psychophysics. In this mutidisciplinary (machine learning-and-neuroscience) context, it is relevant to check the biological plausibility of the models that emerge after training. 

A possibility would be reproducing human image quality opinion from the final stage of the network before the dense classification layers (following Zhang IEEE CVPR 2018 or Hepburn IEEE ICIP 2020). Do the models with Divisive Normalization provide a more accurate reproduction of human opinion?. 
Another option (following Martinez Front. Neurosci. 19) is checking classical psychophysics such as the Contrast Sensitivity Function or the nonlinear responses to luminance and contrast.

At least, the need of these safety checks has to be included in the discussion because optimization on a visual task does not guarantee biological plausibility.

C. OTHER ANALYSIS

Previous comments imply that other analysis could have been performed in the final model:

- What are the features that fall within the same neighborhood?. Can you take one receptive field and show the receptive fields that fall in its neighborhood (or \lambda distance)?. Are these receptive fields similar? 

- Can you give explicit evidences of the biological meaningfulness of the models with Divisive Normalization?

- The consideration of extra computations as Div. Norm. increases the number of paramters. Therefore, does the increase in the performance comes from the function of the new layer or just from the increased flexibility of the model?. Assessment of the accuracy of the model independent of the number of parameters should be performed.

D. LITERATURE (missing citations)

* There is literature on advantages of biological Divisive Normalization (with DN tuned according to biological goals). This literature, focused on biological interpretation of the parameters, assumes certain linear receptive fields (such as Gabor/Wavelet/DCT basis) and studies the advantages of representations with divisive normalization. In this context, the first paper to analyze the impact of Divisive Normalization on image classification (with fixed V1-like Gabors) was Coen-Cagli & Schwartz JoV 2013. Other papers focus on measuring the redundancy reduction ability of biological Divisive Normalization (e.g. Malo & Laparra Neural Computation 2010, Gomez-Villa et al. Journal of Neurophysiol. 2019). Even more meaningful than the redundancy is looking at the transmitted information from the input up to the internal representation (Malo J. Math. Neurosci. 2020). In all these cases, psychophysically meaningful divisive normalizations including Gaussian pooling over space and different frequency channels substantially reduce redundancy and maximize the transmitted information wrt versions with non-biological parameters.

* Other literature optimizes Divisive Normalization in deep learning context (as in this ICLR submission): e.g. DN for optimal rate-distortion image coding in Balle et al. ICLR 2017. Other examples that use a biological nonlinearity instead of or on top of the conventional nonlinearities in deep learning include Bertalmio et al. Sci. Rep. 2020 that generalize Wilson-Cowan layers (which are equivalent to Div.Norm.) and used these nonlinear layers in artificial architectures in a number of applications (classification, adversarial attacks, etc.).

E. MINOR 

- The authors show the "first" receptive fields at some layer with and without normalization. However, how do they order the filters? what does it mean "first"?

- Analysis of RGB channels (as in Figs. 2-4) is not meaningful. In fact, quite soon in the pathway, opponent chromatic channels should appear and hence, achromatic, red-green and yellow-blue channels are more meaningful than RGB.


REFERENCES

[Balle17]  Johannes Ballé, Valero Laparra, Eero P. Simoncelli. End-to-end Optimized Image Compression. Int'l Conf on Learning Representations (ICLR), Apr 2017 2017

[Bertalmio20] Bertalmío, M., Gomez-Villa, A., Martín, A. et al. Evidence for the intrinsically nonlinear nature of receptive fields in vision. Sci Rep 10, 16277 (2020). https://doi.org/10.1038/s41598-020-73113-0

[Coen-Cagli13] Ruben Coen-Cagli, Odelia Schwartz; The impact on midlevel vision of statistically optimal divisive normalization in V1. Journal of Vision 2013;13(8):13. doi: https://doi.org/10.1167/13.8.13.

[Gomez-Villa19] Alexander Gomez-Villa, Marcelo Bertalmío, and Jesus Malo. Visual information flow in Wilson–Cowan networks. Journal of Neurophysiology 2020 123:6, 2249-2268

[Malo10] Jesús Malo, Valero Laparra; Psychophysically Tuned Divisive Normalization Approximately Factorizes the PDF of Natural Images. Neural Comput 2010; 22 (12): 3179–3206. doi: https://doi.org/10.1162/NECO_a_00046

[Malo20] J. Malo. Spatio-chromatic information available from different neural layers via Gaussianization. The Journal of Mathematical Neuroscience; Vol. 10, Iss. 1, (Dec 2020). DOI:10.1186/s13408-020-00095-8

[Martinez19] Martinez-Garcia M, Bertalmío M and Malo J (2019) In Praise of Artifice Reloaded: Caution With Natural Image Databases in Modeling Vision. Front. Neurosci. 13:8. doi: 10.3389/fnins.2019.0000

[Hepburn20] A. Hepburn, V. Laparra, J. Malo, R. McConville and R. Santos, "Perceptnet: A Human Visual System Inspired Neural Network For Estimating Perceptual Distance," 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 121-125, doi: 10.1109/ICIP40778.2020.9190691.

[Webster11] Michael A. Webster; Adaptation and visual coding. Journal of Vision 2011;11(5):3. doi: https://doi.org/10.1167/11.5.3

[Zhang18] R. Zhang, P. Isola, A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE CVPR, 2018, pp. 586–595
summary_of_the_review: The incorporation of biological nonlinearities in artificial networks is interesting both to enrich architectures in machine learning and to better study computational principles in neuroscience. Therefore, studies like this are (and will be) of interest for the ICLR audience.
I advocate for the publication of this work in the conference when the authors address/comment on the following concerns:
A. LIMITATIONS OF THE CONSIDERED NORMALIZATION AND TASK OBSCURE THE MESSAGE: clarify discussion!
B. IS THE FINAL DIVISIVE NORMALIZATION BIOLOGICALLY SENSIBLE?
C. OTHER ANALYSIS
D. LITERATURE (missing citations)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose to study the influence of a feature of biological neural networks on a classical deep learning network. In particular, the authors are interested in including a divisive normalization like the one characterized in the primary visual cortical era and to see the influence of this mechanism on the performance of the AlexNet network.

main_review: 
The results show on the one hand a slight increase in performance. Interestingly and surely more significant in terms of the scope of this paper are the structural changes that are made in the network and in particular the increase in the level of sparsity of the activity in the different layers as well as on the shape of the receptive fields. 

In general the paper is very well presented and the results are very convincing. I have some minor points that I detail below

- The abstract in particular is too long and should be cut in half.

- In each layer, the channels are arranged topologically along a line. Some computational neuroscience models especially those representing orientations are arranged according to a circular typology. Have you tested this option?

- In figure 1, what happens during learning around epoch 30?

- In figures 2, 5, 6 the characters describing the X and Y axes are too small to be readable.

- In figure 4 and four the receptive fields could be shown in color in order to show all the receptive fields higher.

- Is the choice of the AlexNet model important for the generality of the results or does this also apply to other networks such as VGG?

- Finally, the weights of the normalization model are learned during the training. On the other hand, you do not analyze the weights that are obtained at convergence and whether they can be interpreted in terms of the relations between the receptive fields, as can be done in neuroscientific models like those of Schwartz and Simoncelli.


summary_of_the_review: These findings could help explain a largely unanswered question in neuroscience, which is to understand why divisible normalization, which is first and foremost a phenomenological heuristic, exists in biological networks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study the effect of divisive normalization on AlexNet. They show that, when combined with standard normalization schemes, it increases performance. They also investigate the filter shapes, manifold capacity and (adversarial) robustness of the learned representations.

Following the authors' response I have increased my score form 5 to 6.
main_review: Generally, the figure labels are too small and it is difficult to extract the information that they are intended to show.

I think the idea of arranging filters is neat. But this should really be motivated better (e.g. discuss topographic ICA or recent work on topographic CNNs). Also, it would be interesting to see the learned topography.

The brief conclusion section about robustness is interesting but feels ad-hoc.

The appendix looks like a giant dump of plots and additional information. While there is no constrant on appendix length, it would still be nice to arrange and filters this better, so that a reader looking for specific details could find them more easily. It is also unreasonable to expect that reviewers go through >20 pages of appendix to verify its correctness.

Comments:
- Why study only AlexNet and no newer CNN architecture?
- The contributions list is rather a list of findings
- I do not understand the first sentence of section 2
- I think Balle & Simoncelli's GDN (ICLR, 2017) model might fit well into the related work
- What performance do you see with the original AlexNet, i.e. keeping LRN?
- Other works (e.g. Burg et al.) have shown that the power in eq. should also be learned. Why did you fix it to 2?
- Section 4: by 'trial', do you mean 'seed' on Cifar100?
- 'Receptive fields and Fourier power': The second sentence starts with 'therefore', this seems like a non sequitur.
- Figure 2 is illegible, there is too much information and it is not at all clear what to look at or what this is supposed to show.
- You write that a model with higher performance also has higher capacity, this seems wrong in the general case.
- In the sparsity section you write 'respond better', what does that mean? Higher response value given fixed contrast or something like that?
- The Gini index is a little confusing, why don't you just measure entropy? This would seem more easy to follow for an ML audience.
- Fig5 How would these curves look like for Cifar100 across seeds? I.e. is this a significant observation. Why did you use only 100 images and what do you mean by least correlated classes? Also, more generally, what have we learned from looking at these metrics?
- Error bars for Fig6?

Small suggestions:
NN's -> NNs
summary_of_the_review: The paper explores an interesting but not novel idea. The analysis is extensive, but one is left wondering what we have learned from the study. More generally, there is a mixture of ideas and analyses which are interesting but somehow not well connected in this work.
review_title: null

Meta-review: This paper explores addition of a version of divisive normalization to AlexNets and
compares performance and other measures of these networks to those
with more commmonly used normalization schemes (batch, group, and
layer norm).  Various tests are performed to explore the effect of
their divisive normalization.

Scores were initially mixed but after clarifications for design and
experiment decisions, and experiments run in response to comments by
the reviewers the paper improved significantly.  While reviewers still
had several suggestions for further improvements, after the authors' 
revisions reviewers were in favor of acceptance which I support.
Decision: Accept (Poster)