Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The submission investigates the role of response stochasticity, a feature of biological neurons, in robustness to adversarial attacks in deep neural networks.  They use manifold analysis to compare the neural population geometry of standard-trained, adversarial-trained and stochastic networks and find geometrical signatures of the different conditions. They show that the robustness achieved through adversarial training is qualitatively different than that achieved by the stochastic network. They characterize how increasing noise affects the resulting manifolds. They observe a tradeoff between adversarial robustness and classification performance and can explain this tradeoff in terms of geometric features of the representations. These results (on images) are replicated in the auditory domain.
main_review: The submission employs recently developed manifold analysis techniques and sheds new light on a previously documented phenomenon. The demonstration of the value of stochasticity in auditory models is novel.

MFTMA is an exciting analysis technique which was proposed a few years ago but has yet to be widely employed. I am pleased to see it used here where is has helped to shed light on an interesting phenomenon. The work is complete and the authors clearly identify avenues for future work which are not addressed in the present submission.

In general, the submission is clearly written. However, there may be room for improvement in the introduction + related work section (often two sections, but here combined into one). In particular, I found the logical jump from paragraph 2 to 3 unclear. The particular experimental design and methodology does not feel clearly motivated in the text (although I believe that it is motivated). I imagine the authors have tried to condense this first section to make room for the rest of the paper but it feels slightly too condensed.  

minor points: 
- In the first paragraph the authors write "establishing them as leading candidate models of human perception". "Human perception" denotes a much broader set of capacities than those for which we have validated neural network models. Consider scaling back to "visual object recognition".
- line 269: "We include investigation how the level of noise...". Are there some missing words here?

The main conclusions are novel. Some of the results replicate previous findings. The work constitutes progress towards understanding the role of stochasticity in neural computation, which has potential implications for both neuroscience and machine learning. It also constitutes a step towards biologically-inspired deep learning.
limitations_and_societal_impact: The authors clearly acknowledge the limitations of their analysis. They do not mention any potential negative societal impact but none comes to mind.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper compares networks that are adversarially trained with networks that have biologically inspired stochasticity in the first layer, including previous visual object-recognition networks and a new auditory network. The analysis is based on manifolds of classes and exemplars in representation space, and considers measures of manifold size and overlap. Robustness to adversarial attacks is also considered. The analysis highlights different responses of the stochastic and adversarially trained networks to adversarial attacks.
main_review: Originality: 
This is a new kind of analysis, and I don’t think it has been used to study adversarial robustness before. It makes sense to do so. 

Quality: 
The work is technically sound and thorough. 

In comparing visual networks, the paper varies two things at once. The paper focuses on the fact that VOneResNet50 has stochastic responses while ATResNet50 has adversarial training. But another difference is that VOneResNet50 has biologically inspired first-layer filters, while ATResNet50 has standard first-layer filters. It seems cleaner to either address the first difference in isolation, or to test all combinations. The authors leave this for future work (line 343) but it may be a limitation. 

Clarity: 
The writing is clear and engaging. 

Significance: 
Here I am uncertain. The results with undefended and adversarially trained networks are unsurprising. It is interesting that stochasticity widens exemplar manifolds in early layers, contrary to adversarial training, but I am not sure what else I would have expected. I may need more guidance from the authors about what to take from the study.  

The last section of the paper shows that exemplar manifolds due to stochasticity and adversarial perturbations overlap, less so at high attack strengths, and more so with high stochasticity. Here again, I am not sure what else might have been expected. 

Minor: 
“basis of this stochasticity and its implications for information processing are open questions in neuroscience [19]” The reference is good but not recent. 

In Figure 2B, I am not clear about the significance of the decreasing manifold width in ATResNet50, versus the fact that for other networks, width is higher at the output than the pixels. I don’t think it matters, but maybe the authors could discuss this in more detail, or add a plot of un-perturbed manifold widths for additional context. 

Another helpful point of context would be the accuracy vs. attack strength added to Figure 1C,D. 

In Figure 4C, the black line is easy to miss. Lines may not be the best representation in this plot because the ordinate isn’t continuous, and multiple levels are skipped. 

I missed which layers are analyzed in Figure 5. 

limitations_and_societal_impact: The work has a minor societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: It is well known that a deep learning model is fragile to adversarial perturbation, whereas a biological system is not. This study investigate the geometry of internal representations (especially, in the penultimate later) of a DL model, that trained using adversarial examples, and that with biologically-inspired component. The basic approach is based on what the authors call MFTMA.

main_review: Roughly speaking, the reviewer failed in finding a significance in the results and discussions.

(1) The relationship with references [20,22] are unclear. The word "MFTMA" does not appear in the references. The supplementary material 2.1 suggests that it is a synonym fot the method in [8]. Please discuss the relationship with references.

(2) The references [20,22] demonstrated the manifold size is related to the accuracy. Then, the contribution of this study is unclear. All experiments can be done using the accuracy as the measure instead of the manifold size. The only thing that the reviewer considers as the contribution is the suggestion that the manifold analysis is applicable to adversarial perturbation. If so, the introduction is inappropriate.

(3) According to Figure 1, the top-1 accuracy is proportional to the manifold capacity. A vanilla ResNet loses its accuracy and manifold capacity with a strong attack, whereas a ResNet trained with adversarial perturbation does not. VOneResMet is more robust to attack than the vanilla. The similar results are found in auditory data recognition in Section 4.

Several previous works such as Miyato et al. demonstrated that all the dropout, random perturbation, and adversarial training smooth the score function. While dropout and random perturbation are stochastic and inefficient compared to adversarial training, their contribution is similar to adversarial training. Hence, it is unsurprising that "there exists an overlap between the stochastic representations of clean and adversarially perturbed stimuli".

Even through the very first paper by Goodfellow et al. suggested that the dropout is insufficient for robustness to the adversarial perturbation, many previous studies such as Dhillon et al. have attempted to improve the robustness by stochastic components. Hence, it is unsurprising that "stochastic representations provide a defense against adversarial perturbations."

- Miyato et al., "Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning," IEEE TPAMI, 2019.
- Goodfellow et al., "Explaining and Harnessing Adversarial Examples," ICLR 2015.
- Dhillon et al., "Stochastic Activation Pruning for Robust Adversarial Defense," ICLR2018.

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors note that stochasticity is an important component of recent attempts to improve network adversarial robustness by emulating known biological mechanisms. They also demonstrate their own similar result for a network trained on auditory signals. Motivated by this, they analyze the representations formed by a set of example networks to understand the relationship between representation geometry and robustness. Their analysis reveals differences among networks that underwent standard training, adversarial training, and those with stochasticity added. They claim that this analysis improves our understanding of mechanisms of robust perception as well as our understanding of the role stochasticity plays in biological computation.
main_review: *Review summary:*

The MFTMA analysis method reveals clear differences among the networks tested. Specifically, I find it compelling how clearly the authors demonstrated that two strategies for achieving robustness can show opposing behavior in terms of their analysis. I note a bunch of nitpick concerns in my review below, but overall I found the paper to be well written and the results interesting. I am recommending the paper for acceptance, and will further increase my score if my concerns are addressed.

*Strengths:*


The paper is well written and has helpful extended explanations in the appendix. Their analysis provides a very interesting perspective on how two different methods for achieving robustness (adversarial training vs noise training) can lead to alternative mechanisms. Their controls, both within the main text and in the appendix, are thorough and convincing of the overall result.

*Weaknesses:*
1) Claims:
1.a) I think the paper falls short of the high-level contributions claimed in the last sentence of the abstract. As the authors note in the background section, there are a number of published works that demonstrate the tradeoffs between clean accuracy, training with noise perturbations, and adversarial robustness. Many of these, especially Dapello et al., note the relevance with respect to stochasticity in the brain. I do not see how their additional analysis sheds new light on the mechanisms of robust perception or provides a better understanding of the role stochasticity plays in biological computation. To be clear - I think the paper is certainly worthy of publication and makes notable contributions. Just not all of the ones claimed in that sentence.

1.b) The authors note on lines 241-243 that “the two geometric properties show a similar dependence for the auditory (Figure 4A) and visual (Figure 4B) networks when varying the eps-sized perturbations used to construct the class manifolds.” I do not see this from the plots. I would agree that there is a shared general upward trend, but I do not agree that 4A and 4B show “similar dependence” between the variables measured. If nothing else, the authors should be more precise when describing the similarities.

2) Clarifications:
2.a) The authors say on lines 80-82 that the center correlation was not insightful for discriminating model defenses, but then use that metric in figure 4 A&B. I’m wondering why they found it useful here and not elsewhere? Or what they meant by the statement on lines 80-82.

2.b) On lines 182-183 the authors note measuring manifold capacity for unperturbed images, i.e. clean exemplar manifolds. Earlier they state that the exemplar manifolds are constructed using either adversarial perturbations or from stochasticity of the network. So I’m wondering how one constructs images for a clean exemplar manifold for a non-stochastic network? Or put another way, how is the denominator of figure 2.c computed for the ResNet50 & ATResNet50 networks?

2.c) The authors report mean capacity and width in figure 2. I think this is the mean across examples as well as across seeds. Is the STD also computed across examples and seeds? The figure caption says it is only computed across seeds. Is there a lot of variability across examples?

2.d) I am unsure why there would be a gap between the orange and blue/green lines at the minimum strength perturbation for the avgpool subplot in figure 2.c. At the minimum strength perturbation, by definition, the vertical axis should have a value of 1, right? And indeed in earlier layers at this same perturbation strength the capacities are equal. So why does the ResNet50 lose so much capacity for the same perturbation size from conv1 to avgpool? It would also be helpful if the authors commented on the switch in ordering for ATResNet and the stochastic networks between the middle and right subplots.

3) General curiosities (low priority):
3.a) What sort of variability is there in the results with the chosen random projection matrix? I think one could construct pathological projection matrices that skews the MFTMA capacity and width scores. These are probably unlikely with random projections, but it would still be helpful to see resilience of the metric to the choice of random projection. I might have missed this in the appendix, though.

3.b) There appears to be a pretty big difference in the overall trends of the networks when computing the class manifolds vs exemplar manifolds. Specifically, I think the claims made on lines 191-192 are much better supported by Figure 1 than Figure 2. I would be interested to hear what the authors think in general (i.e. at a high/discussion level) about how we should interpret the class vs exemplar manifold experiments.

4) Nitpick, typos (lowest priority):
4.a) The authors note on line 208 that “Unlike VOneNets, the architecture maintains the conv-relu-maxpool before the first residual block, on the grounds that the cochleagram models the ear rather than the primary auditory cortex.” I do not understand this justification. Any network transforming input signals (auditory or visual) would have to model an entire sensory pathway, from raw input signal to classification. I understand that VOneNets ignore all of the visual processing that occurs before V1. I do not see how this justifies adding the extra layer to the auditory network.

4.b) It is not clear why the authors chose a line plot in figure 4c. Is the trend as one increases depth actually linear? From the plot it appears as though the capacity was only measured at the ‘waveform’ and ‘avgpool’ depths; were there intermediate points measured as well? It would be helpful if they clarified this, or used a scatter/bar plot if there were indeed only two points measured per network type.

4.c) I am curious why there was a switch to reporting SEM instead of STD for figures 5 & 6.

4.c) I found typos on lines 104, 169, and the fig 5 caption (“10 image and”).
limitations_and_societal_impact: yes
review_title: null

Meta-review: This is an excellent paper with good consensus and engagement among the reviewers for the acceptance. It derives novel insights for mechanisms of robust perception, and will likely inspire researchers in the fields of adversarial robustness, biologically-inspired neural networks, and computational neuroscience, all important and active sub-fields in NeurIPS conference. 
Decision: Accept (Poster)