Venue: NeurIPS 2021 Workshop SVRHM

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This submission devises a way to fit a neural inspired normalization model (DivNormEI) to large datasets. The authors proposed a learnable divisive normalization layer, which could be embedded in deep neural networks. The authors performed several experiments. One result showed that the resulting network exhibited contrast invariant orientation tuning after normalization, while a pre-trained VGG network on ImageNet dataset did show contrast invariance. The authors also showed DivNormEI could lead to some improvements on object recognition tasks, although the generality of these results are unclear. 

Overall, this reviewer considers the ideas presented in this submission are interesting and the preliminary results are promising (although not conclusive). This could potentially be a useful contribution both for computational neuroscience and machine learning. 

One suggestion: The contrast invariance is difficult to see and compare in Fig 1A,B,C. It would be useful if the authors could showed the scaled-version or directly quantity the amount of the contrast invariance. 
review_title: interesting idea and promising preliminary results 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I enjoyed reading this manuscript. It's easy to read and follow the authors' thread of thinking.
I also think the idea of incorporating the biologically-inspired modules into deep networks is a very interesting line of research, (i) it can offer a performance advantage, (ii) it allows us to better understand the biological brain.

There are several suggestions that I would like to make. I understand some of those might not be implementable for the current workshop given the page limit. Nevertheless, I think they're important and can be considered for an extension of this work.

## The fundamental conclusion
Based on the results presented in Figure 1, I don't agree with this conclusion "This property of contrast invariant tuning is similar to that of primary visual cortical neurons."
To me, inserts A and B in Figure 1 are closer to the tuning curve of the biological neurons (insert D) in comparison to the post-normalisation (C). In D, we can see a large gap between the firing rate of the highest and lowest level of contrast. This is also present in A and B. However in C, the gap is much smaller.
This point requires further investigation and I strongly suggest a quantitative metric to compare these tuning carves.

## Performance advantage
The reported "1.8%" performance advantage in ImageNet-100 could be of significant value, but it should be supported with further evidence/experiments/considerations: (i) the number of instances should be increased to ten or so; two is not enough to ensure it holds for other instances. (ii) the number of epochs should be increased as well; judging based on Figure 2, the curves haven't fully plateaued. (iii) other architectures such as ResNet must be evaluated; it can be equally a shallow one; (iv) If I understand correctly the DivNorm contains more learnable parameters than the Baseline, this should be reported and taken into account for evaluation.
It would also be nice to see the results of the super-resolution experiment described in Section 3.1. Is the performance of DivNorm systematically better there as well?

## Relevant articles
Authors might be interested in the following works regarding divisive normalisation and deep networks: 
 * "Image Compression With Recurrent Neural Network and Generalized Divisive Normalization" CVPR-Workshops 2021. 
 * "Normalizing the normalizers: Comparing and extending network normalization schemes" ICLR 2017.

Should authors be interested in making m and n in Eq. 2 dynamic, these works might be of interest to them showing the benefit of dynamic DoG kernels in classical computer vision: 
 * "Colour constancy beyond the classical receptive field" PAMI 2017.
 * "Feedback and surround modulated boundary detection" IJCV 2018.

Finally, authors might also be interested in previous works of deep networks and image contrasts:
 * "Generalisation in humans and deep neural network" NeurIPS 2018.
 * "Deciphering image contrast in object classification deep networks" Vision Research 2020.

## Minor comments
Variables of the equations can be better explained to facilitate their understandings, for example:
 * "y" in Eq. 1 is introduced later on.
 * The same goes for "c" in R^{h,w,c,c}.

ANN encompasses a larger group of networks, in this study only CNNs are studied. Thus, it might be more appropriate to use the term CNN, or convolutional autoencoders when authors refer to their experiments/results/study.

ResNet is used in the method section to give an example of the number of features in Eq. 1. However, this architecture is never used after. It would be better to make the example for VGG that is used in the manuscript.

Related works are often described before the method section. Furthermore, it would be nicer to have it as a coherent story that links those works to the presented work by stating the similarities and differences rather than a bullet point style "our work is related to X" ... 
review_title: A promising line of investigation for future of deep networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: Overall, the authors present an interesting implementation for theoretically improving large scale VGG nets in image classification and segmentation tasks while suggesting lateral inhibitory and excitatory input, in addition to divisive normalization, factor into this improvement vis-a-vis a known computational mechanism observed in biological organisms-"contrast invariant tuning." 

Quality: 
Pros-
The authors provide a nice experimental manipulation in their work by developing tuning curves for individual neurons in the DNN before and after implementing the DivNormEI module to show that it improves (Fig. 1a-1c) contrast invariant tuning relative to a base VGG network for Gabor-like stimuli (Fig-1e).

Cons- 
It would be nice to see additional analysis, specifically comparisons of the contrast invariant tuning under all the permutations of the DivNormEI module. That is, just divisive normalization, divisive normalization + excitatory input, divisive normalization and inhibitory input, etc. to really demonstrate that the whole of DivNormEI is greater than the sum of its parts. 

Clarity:
Pros-
The authors do a good job of contextualizing their work with ample references to previous implementations of divisive normalization algorithms and lateral connections to motivate what has been done for the interested reader. Additionally, the equations in section 2.1 are described succinctly and easy to understand. 

Cons-
There are a few nit-picky details in the methods section that I believe the authors should address to fully maximize the communicability of this conference proceeding paper. 1) In section 3.1, the authors mention 2 spatial frequencies in units of cycles per degree. This is confusing because the cycles per degree typically means cycles per degree of visual angle, which requires a human observer sitting in front of a monitor a specific viewing distance. 2) Sinusoidal gratings are different than Gabors (a windowed sinusoidal grating), yet the authors refer to the stimuli presented to the DNN as being sinusoidal in the text, but show a picture of Gabor patches in the Fig 1e. 3) The authors do not mention the contrasts of the stimuli used in experiment 3.1 to produce the tuning curves. It is hard evaluate the impressiveness of their finding if we do not have actual numbers set to the different colored tuning curves in Figures 1a-1c. 

Originality and significance of the work:
Experiment 3.2 is the crux of the significance of this work, in my opinion. As pointed out in the clarity section (pros), there is previous work that is similar to DivNormEI but implemented at a smaller scale. Thus, the thrust of this paper should be how DivNormEI improves the quality of the DNN for important metrics in the computer vision community. They show a modest improvement in classification accuracy (1% at 75% baseline). I believe that they would benefit from implementing this model in a large full-scale network (they do point out that there were resource limitations) and see how it does in the wild compared to other DNN architectures as this is what we are most interested in as readers. A more forceful demonstration of the importance of incorporating DivNormEI into CNN architectures would provide evidence to support their hypothesis spelled out in the introduction section: "We hypothesize that the hierarchical incorporation of our DivNormEI (out of the scope of this paper) will promote development of invariance to more stimulus dimensions." 
review_title: Relevant and interesting idea, but would like to see ideas implemented in a larger network architecture.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The present work presents the incorporation of a biologically inspired neural computation, divisive normalization and contextual modulation, into deep neural networks. This computation is an important and well-studied biological process suggested to be a canonical computation in the brain. As such, it makes an interesting target to incorporate into deep neural networks, for exploring possible improvements to this kind of model, and for better understanding its possible functions in the brain. Although previous work also trained networks with this computation on natural images, to my knowledge the authors are the first to do so with an object recognition task in the context of deep neural networks, a promising venue. Therefore, the aim of this work is very interesting in general, and in line with the topic of the workshop.

The experiments proposed in this work are interesting, showing both how such a computational module may explain some V1 behavior, and how it may improve performance at a supervised classification task as compared to the network without the computation. However, despite the merits mentioned above which give this work good potential, I think that some aspects make it feel like a work in progress needs some extra steps for its contributions to be clearer, or alternatively, the authors could better lay out the advances that this work represents, which I may be missing.

For example, in the first experiment, divisive normalization is trained over a set of Gabor filters, using an unsupervised encoding-decoding task. Although interesting, it is unclear to me precisely what this result contributes, besides showing that their implementation can work. For example, I do not see any discussion about what this computation adds to this task, does divisive normalization change anything performance-related? Also, I am not sure what to extract from the controls used. It is unsurprising that Gabor filters and conv1 filters do not show contrast invariance given that they are both linear operations on the image. On the other hand, one could expect that adding (close to) any divisive normalization would produce some contrast invariance. So, I am left with the question, what does the method proposed by the authors add to some “silly” divisive normalization? As far as I can tell, it is possible that the neurons in the network show similar contrast invariance pre- and post-training. In such a case, what would be the the contribution of this new way of training divisive normalization? Showing contrast invariance of conv2 neurons could also be a nice control, since that's a non-linear operation on the image. Thus, although the idea of the experiment and comparison to data is interesting, I would like to see more about what their contribution here is (which need not be any groundbreaking result, but at least what the method may add in the future). Related to the points above, if their method represents any substantive advance or difference over previous studies training divisive normalization on natural images with unsupervised tasks (i.e. increasing independence between responses, or similar from Odelia Schwartz and Ruben Coen-Cagli), it would be good to have it laid out. As an extra note, I think that this experiment would be considerably more interesting if a couple more comparisons to neural data were shown. Some examples that come to mind are the phenomena of tilt-illusion, end-stopping, and pop-out, but there are probably other alternatives.

Then, regarding the second experiment, I am also left with a couple of hanging questions, although they may be somewhat related to my non-mathematical background. Mainly, I am not sure in what way the lateral inhibition-excitation is different from just adding another convolutional layer. If I understand correctly, after the normalization step, the EI step just computes a linear combination of the local neuron activations and adds it linearly to the neuron response. So I wonder, isn’t this equivalent to just the linear combination of the neighborhood activations that could be obtained using an appropriate convolutional filter? If I am right here, I am failing to see what benefit the current version of lateral inhibition-excitation would have over existing feedforward-convolutional models. Some recurrence in this layer or some non-linearity would seemingly need to be added to turn these lateral interactions into a really novel computation, DNN-wise. If I am wrong, I think that a short explanation of how this layer is different from a convolutional layer would be good. Related to this, the control to which performance is compared in this experiment has fewer parameters or one less layer, it would be useful to see how their normalization, EI lateral interactions compare to a network where something else is added (i.e. another convolutional layer), rather than nothing.

In sum, I find the research direction proposed in this work extremely interesting, and I look forward to seeing how this develops. I also think that this topic would be of interest to the public in the workshop. However, because of the issues laid above, I think that more solid results/analysis would be required to allow for exchanges that go to the heart of the issue, rather than to technical quibbles. Or if I am wrong in my concerns, I think that these are issues that should be clearer.
review_title: Very interesting topic, but some method/analysis concerns

Decision: Accept (Poster)