Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception
Authors: Joel Dapello
Jenelle Feather
Hang Le
Tiago Marques
David Daniel Cox
Josh Mcdermott
James J. DiCarlo
SueYeon Chung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representational Geometry
Biologically inspired vision models
Biologically inspired auditory models
Stochasticity
Adversarial robustness
Robust perception

Abstract: Adversarial examples are often cited by neuroscientists and machine learning researchers as an example of how computational models diverge from biological sensory systems. Recent work has proposed adding biologically-inspired components to visual neural networks as a way to improve their adversarial robustness. One surprisingly effective component for reducing adversarial vulnerability is response stochasticity, like that exhibited by biological neurons. Here, using recently developed geometrical techniques from computational neuroscience, we investigate how adversarial perturbations influence the internal representations of standard, adversarially trained, and biologically-inspired stochastic networks. We find distinct geometric signatures for each type of network, revealing different mechanisms for achieving robust representations. Next, we generalize these results to the auditory domain, showing that neural stochasticity also makes auditory models more robust to adversarial perturbations. Geometric analysis of the stochastic networks reveals overlap between representations of clean and adversarially perturbed stimuli, and quantitatively demonstrate that competing geometric effects of stochasticity mediate a tradeoff between adversarial and clean performance. Our results shed light on the strategies of robust perception utilized by adversarially trained and stochastic networks, and help explain how stochasticity may be beneficial to machine and biological computation.


Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception
Joel Dapello" Soppello.1..20 1,2.3, Jenelle Feather*:1.2.4 Hang Le Tiago Marques 1.2.4 David D. Cox Josh H. McDermott 22,46 James J. DiCarlo SueYeon Chung1.7.
"Equal contribution, ordered alphabetically
DDepartment of Brain and Cognitive Sciences, Massachusetts Institute of Technology MccGovern Institute for Brain Research, Massachusetts Institute of Technology School of Engineering and Applied Sciences, Harvard University
CCntee for Brains, Minds and Machines, Massachusetts Institute of Technology
MIT-IBM Watson Al Lab
Speech and Hearing Bioscience and Technology, Harvard University
Center for Theoretical Neuroscience, Columbia University Zuckerman Institute, Columbia Univeristy
Abstract
Adversarial examples are often cited by neuroscientists and machine learning re- searchers example of how computational models diverge from biological sensory systems. Recent work has proposed adding biologically-inspree compo- nents to visual neural networks as a way to improve their adversarial robustness. One surprisingly effective component for reducing adversarial vulnerability is re- sponse stochasticity, like that exhibited by biological neurons. Here, using recently developed geometrical techniques from computational neuroscience, we investigate how adversarial perturbations influence the internal representations of standard, ad- versarially trained, and biologically geometric signatures for each type of network, revealing different mechanisms for achieving robust representations. Next, we generalize these results to the auditory domain, showing that neural stochasticity also makes auditory models more robust to adversarial perturbations. Geometric analysis of the stochastic networks reveals overlap between representations of clean and adversarially perturbed stimuli, and quantitatively demonstrates that competing geometric effects of stochasticity medi- ate a tradeoff between adversarial and clean performance. Our results shed light on the strategies of robust perception utilized by adversarially trained and stochastic networks, and help explain how stochasticity may be beneficial to machine and biological computation
stochastic networks. We find distinct
Introduction
In recent years, artificial neural networks (ANNs) have come to dominate both visual object recog-
nition and auditory recognition tasks II
3 establishing them as leading candidate models for
several domains of human perception 4
5
6 7l However, they still exhibit many non-human-like
traits 8 9I. One such failure is the existence of adversarial perturbations small changes to
stimuli explicitly crafted to fool model that remain imperceptible to humans [10 1 demonstrate the fragility of some ANNs as models of biological perception.
which
Soe https://github..oo/chhut
htw.aaaareaarrialaaaaiiiiii for accompanying code.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Recently, Dapello. Marques et al. discovered that one such method, known as adversarial training [3], not only reduces the network's adversarial vulnerability but also yields network representations that are more similar to those in the primate primary visual cortex [14]. Motivated by this result, the authors developed VOneNets, class of networks that simulate the primate primary visual cortex at the front of a convolutional neural network, and show improved robustness to adversarial attacks with no adversarial training. However, a number of questions remain unanswered - in particular, while both adversarially trained networks and VOneNets have improved adversarial robustness and also greater similarity to the primate primary visual cortex, it is unclear how VOneNets achieve robustness, and in particular if the mechanism of robustness is similar to that induced by adversarial training.
key component of robustness in VOneNets is the inclusion of stochastic representations during both training and inference, feature inspired by biological sensory neurons which exhibit trial-to-trial variability across presentations of the same stimulus [1]. The implications of this stochasticity for information processing are open questions in neuroscience [16] 17 [8] 19]. Pinpointing how this representational stochasticity contributes to robustness in VOneNets could drive further developments
in the mechanisms of robust perception.
Here we use recently developed manifold analysis techniques from computational neuroscience [20] to look beyond accuracy and investigate the internal neural population geometry [21] of stan- dard, adversarially trained, and celoogicall-iispiree stochastic networks in response to clean and adversarially perturbed examples in both visual and auditory domains. We present several key findings:
Using manifold analysis, we demonstrate that standard, adversarially trained, and stochastic networks each have distinct geometric signatures in response clean and adversarially perturbed stimuli, shedding light on varied robustness mechanisms.
We demonstrate the generality of our findings by translating the results to a novel biologically-insprred auditory AÑN, StochCochResNet500 that includes stochastic responses. Stochasticity makes auditory networks more robust to adversarial perturbations, and the underlying neural population geometry is largely consistent with that in vision networks.
Analysis of stochastic networks reveals a protective overlap between the representations of adversarial examples and clean stimuli. We quantitatively survey the stochasticity conditions leading to the overlap, and map competing geometric effect mediating trade-off between clean and adversarial performance.
2 Related work
Previous work in machine learning has reported that additive noise can improve the adversarial robustness of model: Liu al. (22) used random noise applied to pixels and intermediate layers to improve adversarial robustness, and Cohen et al. [23] demonstrated a method to transform a model with Gaussian noise in the pixel space to one with certified robustness to attacks of given strength. Unlike VOneNets, both [22] and 23 rely on using multiple inference passes with different noise samples to improve robustness. Further, while both of these works suggest new defenses, none analyze the properties of stochastic representations that lead to robustness.
The need understand internal mechanisms of biological and artificial neural networks gave rise to the field of neural population geometry 24 21, an line of work exploring geometric properties of high-dimensional neural representations. To capture the complexity of neural representations in ANNs, various geometric analyses have been proposed, including representation similarity analysis 25]. geodesics (26), curvature (2). and intrinsic dimensionality 288. Another popular approach is through supervised linear probes 29 i.e., linear classifiers trained on top of these representations. However, recent work has discussed how such analyses with linear classifiers are limited 300, and that more structural studies are needed for investigating the internal layers of networks [1 A recent
theoretical development based on replica theory in statistical physics (3 20 33 provides a solution
to this, by formally connecting the geometry and linearly decodable information embedded in neural populations. By using this approach, our study situates the analysis of robustness mechanisms into
the developing field of neural population geometry [24 21

---

3 Methods and experimental setup
3.1 Replica-based manifold analysis
In this paper, use replica mean-field theoretic manifold analysis (MFTMA, hereafter) 20 34 33] which formally connects the linear decodability of object manifolds, defined as a set of stimulus- evoked representations in neural state space [35], their geometrical properties. Here we provide a brief description of the key quantities (See SM and [20] for a complete treatment of MFTMA). Using this framework, we analyze the size, shape, and distribution of object manifolds as they are transformed throughout the network, gaining insight into the neural population geometry underlying performance of the classification task. Specifically,, given P object manifolds (measured as feature point clouds) in N dimensions, MFTMA returns capacity (a measure of linear separability) and the manifold dimensions, radii, and center correlations associated with capacity estimation.
Manifold capacity (a = P/N) refers to the maximum number of object manifolds (Pmas) that can be linearly separated given N features, and characterizes the linearly-decodable object information per feature dimension. MFTMA estimates a through measures of the manifold dimension (Dwr,, manifold radius (Rxr). and manifold center correlation, which refer to dimensionality, size, and distribution of object manifolds relevant for the linear classification. In our analysis, we combine the manifold dimension and radii together into a single manifold width measure, defined as Ras VD which captures the width of the convex hull of a manifold. Manifold width formally links the linear separability of object manifolds with their underlying geometric structure. Specifically, small values of manifold width yield more linearly separable manifold geometry. The correlation between locations of the object manifolds also plays a role in determining manifold capacity, as more correlated manifolds are less separable. For our analysis, center correlation is calculated as the average of the absolute value of the cosine similarity between pairs of object manifold centroids.
In this work, we characterize two types of manifolds:
High capacity Low capacity Exemplar Manifolds Exemplar Manfolds
1) Class manifolds: each manifold is defined by the activa- tions evoked by multiple examples drawn from a specific class of visual or auditory stimuli (i.c., object or word identity). The variability within the manifold can come from different exemplars within the class, but may also be influenced by adversarial perturbations, and/or a layer with stochastic activations.
High capacity for Class Manifolds
Low capacity for Class Manifolds
each manifold is defined by the activations evoked by multiple instances of a single exem- plar (i.c., image or utterance) with the manifold variability reflecting either norm-bounded adversarial perturbations" and/or the influence of stochasticity.
2) Exemplar manifolds:
Class Manifold
Exempiar Manifold
Figure 1: Insights from class mani- folds and exemplar manifolds. Class manifolds have variability due to differ- ences in exemplars in the class in addi- tion to the stochasticity or adversarial perturbations, while exemplar manifolds only have variability due to stochasticity adversarial perturbations.
Although the class manifold analysis is most tightly cou- pled to the accuracy of the network on a classification task, analyzing the exemplar manifolds gives insight into how the class manifolds are constructed. Specifically, multiple types of exemplar manifold geometries can lead to the same class manifold capacity and width (Figure
3.2 Quantifying object manifold overlap
The MFTMA framework provides a geometric description of exemplar manifolds evoked by ad- versarial and clean stimuli in the case of multiple distributed object manifolds. However, we also characterize the overlap between adversarial and clean exemplar manifolds generated from the same stimuli (on the grounds that overlap should produce robustness to the adversarial perturbations in question). To do so, we use a notion of object manifold overlap defined as the generalization error of a linear Support Vector Machine (SVM) fit to separate two sets of representations, and report the
error rate on the held-out data. A normalized error rate of 1 means that two
We replicate many experiments for non adversarial but random perturbations within the e-sized ball to directly compare model geometry measured the same stimuli. Results similar and provided in SM 6.
3

---

representations are completely overlapped, and 0 means that they are completely separable and thus not overlapped. We use the scikit-lears SVM implementation with train/test split of 80/20.
3.3 Adversarial attacks
For performing adversarial attacks, use either the single-step fast gradient sign method (FGSM) with a random starting location or multi-step projected gradient descent (PGD). Unless otherwise specified, we use Lee norm constrained attacks. All attacks are untargeted and performed -ymdeel basis. When the goal is to evaluate adversarial accuracy in newly created networks with stochastic internal representations, we ensemble-PGD. where each step is in the mean direction of k samples of the noisy gradient to ensure a useful signal [36]. For more details, see SM 2.
4 Manifold analysis of robustness in ImageNetttrainedd networks
We first investigate the properties of A. ImageNet-trainee models, aiming to compare the geometric signatures of standard, adversarially trained, and stochastic networks.
Class
Class
Standard ANN layers
Nocy VOnc
Class 2
Manitold analysis
4.1 Models and Dataset
B.
Accuracy vs Capacity
Capacity vs Strength
D.
Width vs Strength
Last Layer
Last Layer
We use images sampled from the Ima- geNet (37) test set. For class manifold analysis, the clean stimulus set con- sists of 50 classes, with each class containing 50 unique exemplar im- ages for total of 2500 unique im- ages. For exemplar manifold analy- sis, 100 unique images are sampled from the ImageNet test set and each perturbed with FGSM from a ran- dom starting location 50 times for 5000 unique images. We analyzed three publicly available ImageNet models, including ResNet50 with stan- dard training. ResNet50 adversarially trained with an Too = 4/255 penalty (ATResNet50), and Nen5R, ResNe150-basee model with the first conv-relu-maxpon layers replaced model
top-1 Aocyaay
fog Gepegity
Mgli Weath
VDnelesNe550
G0DreResNee50
ATResNar50
14
ResNet50
r 89
0.0
3.0
0.10 0.12 0.14 0.15 MFTMA Capacity
0.0
1/1020
1/255
3/3020
12255
Strength (8)
Strength (c)
Figure 2: Geometry and capacity of clean and adversar- ially perturbed class manifolds in ImageNet models (A) Clean or adversarially perturbed stimuli grouped into class manifolds are provided as input to model, internal repre- sentations are extracted, and MFTMA analysis is applied. (B) MFTMA capacity from the final model layer reflects accuracy from clean and adversarial images. (C) Manifold capacity and (D) manifold width for the last model layer is plotted against attack strength, revealing enhanced represen- tational invariance in ATResNet50 (green) and to lesser degree VOneResNet50 (dark blue) and GeOneResNet50 (light blue), while undefended ResNet50 (orange) is least invariant. Error bars are standard deviation (STD) across random projection (RP) and MFTMA seeds.
by a peteear-onninnarPPoiii front-end (called the VOneBlock)
with Gabor filters and noise fitted to primate neuronal data. addition, we analyze a novel variant of Net50 (ee0neResNet50) with additive Gaussian noise scaled to match the mean over all units in the output of the VOneBlock in response to a set of reference stimuli. Ree performs similarly to the original VOet in clean and adversarial conditions (see SM 3.3 for more details). While the first two models are deterministic, VOn5ResNet50 and Ne55 have stochastic representations at the output of the VOneBlock. For additional details see SM 3.
4.2 MFTMA reveals unique robustness strategies for VOneNet and adversarial training
Our analysis begins by observing the effects of norm- bounde,d gradient- gradient-based adversarial attacks on the class manifold geometry of representations in ResNet50, ATResNe150, Vee5R, and NeO5eResNet5O. For all experiments, images perturbed with PGD Lix constraints of e [0, 1/1020, 1/255) are shown to model and intermediate representations are extracted, randomly projected to 5000 features [38] 39. and analyzed with MFTMA (Figure A Much like previously

---

A. Imagete
Figure 3: Geometry and capacity of adver- sarially perturbed exemplar manifolds in neural networks (A) Each manifold con- sists of a set of adversarially perturbed ex- amples within an e-sized ball around a sin- gle exemplar image. (B) Mean capacity and manifold width as function of layer depth; ATResNet50 maintains the size of e-sized ball manifolds, while undefended ResNet50,, NeeeRe and GVOneResNet50 ex- pand the size. (C) Adversarial representa- tion capacity normalized by clean represen- tation capacity is shown for pixels, first conv layer (the VOneBlock for VOneNets), and the last average pooling layer. VOneRes- Net50, GVOneResNet50, and ATResNet50 show less of a capacity decrease for low- perturbed image manifolds compared to the standard ResNet50. Error bars represent STD across RP and MFTMA seeds in all plots; unnormalized capacity detailed in SM 3.6.
Standard ANN byers
paDg IADA Sate
Maniidddanalys
8.
Examplar Manitoldds va Layers FGSM Lgp C 8/255
Menifold
Capacity
VonsitesNert
Manifold
GVDreBestee500
Wilth
2
Alllssaats
Reshiet0
pieele
KChattook
yonaBlock
bottimaak
blook
block
Blok
blok
oeggoel
Comy
Ade Manifotd Capacity
Cleen Manitold Capacty
pikels
VOneBlock/ oonvt
avgpool
0.8
0.8
0.4
10
10
10 10
10"
10
reported with MFTMA 33 models develop separable representations in later layers, with capacity peaking in the final layer before classification (see SM 3.5).
First we focus on this penultimate layer, where class manifold representations have become linearly separable and where capacity thus peaks. For the full layer-wise results, see SM 3.5. We empirically confirm that capacity of the penultimate layer is predictive of the top-1 accuracy across models and attack strengths (Figure B), indicating that MFTMA is sensitive to the attacks we are investigating. MFTMA exposes how capacity (Figure (C) and class manifold width (Figure 2D) vary in the four models as a function of perturbation strength. Adversarial attacks cause the width of class manifolds to grow most rapidly in ResNet50, to lesser degree in VOneNets, and least of all in ATResNec50.
Next, to investigate how the models represent adversarial perturbations around individual im- ages, we introduce another MFTMA-base approach. Instead of class manifolds, we con- sider exemplar manifolds consisting of points sampled using FGSM with L constraints of [0, 1 1/1020, 1/255, 4/255, 8/255] from a random starting location in the epsilon-sized ball around 100 exemplar images, with the goal of tracing how these c-sized ball exemplar manifolds develop as they travel through subsequent layers of our networks of interest (Figure BA).
Restricting our analysis to the layer-wise trajectory of e = 8/255 sized ball exemplar manifolds (Figure 3B), the most salient feature is how distinct the trajectories are for the ATResNe150 and VOneNets. Our results indicate the defense mechanism induced through adversarial training generally stabilizes the width of the e-sized ball around an exemplar as it propagates through the network, effectively mapping small perturbations around the clean image to small regions in later layers of the network. By contrast, the width of the e-sized ball increases in VOncNets and the standard ResNet50, In fact, at the VOneBlock output, the exemplar manifolds become highly entangled, suggesting different mechanisms of robustness for adversarially trained and stochastic models.
How then do stochastic responses improve robustness? Figure BC demonstrates that when the capacity of the adversarially induced exemplar manifolds is normalized by the capacity for representations of the same unperturbed images (clean exemplar manifolds VOneNets are far more stable than the standard ResNet50 network. At lower attack strengths where VOneNets' accuracies are minimally degraded, their normalized capacity remains stable, indicating that they represent the perturbed images with approximately the same capacity as clean images. In other words, the adversarial perturbations do not push the representation beyond that for clean images. In SM 3.7, we extend this analysis to variety of additional networks including two networks trained ad adversarially with different norms, Vet50 with no stochasticity during training or inference, and ResNet50 with stochastic activation layer mirroring that in VOneResNet50 (see Dapello, Marques et al. [14
Clean exemplar manifolds for non-stochastic models have the maximum capacity of 20
5

---

for network details.) Our trends hold up in each of these cases, with adversarially trained networks stabilizing the e-sized ball, and stochastic networks expanding it the point of the stochastic layer. but using similar capacity relative to clean images for small perturbations. In all experiments, we find similar geometric signatures for the Gaussian and Poisson VOneNets. For simplicity in the rest of the paper we focus on stochastic representations with Gaussian noise, but include an analysis of Poisson noise network trained on CIFAR-I0 SM 5.4.
Stochastic representations improve robustness in auditory networks
The results from the previous section highlight distinct geometric profiles for adversarially trained models and for models with stochastic responses. To demonstrate that the observed neural population geometry generalizes across modalities, compare biologically-insprred and adversarially trained auditory models trained to perform a speech recognition task. Our focus here is not on generating an auditory model that is fully defended against adversarial attacks, but rather to test whether the influence of stochastic representations and the geometric trends observed in the previous section generalizes across domains. We thus focus on T. attacks acknowledging that T.p audio attacks that successfully change network's s prediction are often audible to human listeners 40)
5.1 Models and dataset
Auditory models are trained to perform the word recognition task in the Word-Speaker--oe Noise dataset introduced in [9] Networks learn to distinguish the word present in the middle of two-second speech clip from 793 word classes. Word class manifolds are constructed from unique words with 50 unique speakers saying the word, drawn from the Wall Street Journal Corpus 42 (2500 unique speech clips). Exemplar manifolds are measured from a random selection of 100 example clips from the class manifold dataset, with 50 samples measured for each clip.
followed by ResNet50 architecture. The cochleagram consists of differentiable operations, allowing generation of adversarial examples the waveform. Unlike VOneNets, and to maintain consistency with previously published auditory models [ 9 the architecture maintains the wenu-relu-mxpooe before the first residual block, on the grounds that the cochleagram models the ear rather than primary auditory cortex. For stochastic models, Gaussian noise with standard deviation is added after the cochleagram representation (Figure [A), and we refer to this model as St0. A comparison network without noise (a ()) is similarly trained and evaluated (CochResNet50))
Auditory models contain a bhologically-insiiee 'cochleagram' representation 4
A.
D.
Effect stochasticity during training attack
Cochlesblock
Stool sostiss
Standard ANN layers
Accuracy at Linf 0.000
1.0
0.8
0.6
B.
C.
Robustness vs. Linf Attack for
Accuracy Vs. Noise Variance
0.4
Gaussian Noise Auditory Models
Measured at s=0.001
1.04
0
1.0
0.2
Accuracy
0.8
0=0.03125
0=0.6625
Accurael
0.8
0.0
0.6
0=0.125
0.6
No Stochasticity
Training
Stochasticity
Training + Attack Stochasticity
AT Linf s=0.001 .010
0.4
0=0.25
0.4
a=0.5
0.2
0.2
0.0
0.0
1
10
0.0
0.1
0.3 0.4 0.5 Gaussian Noise (o)
0.2
Attack Strength
Figure 4: Robustness of auditory networks trained with stochastic cochlear representations (A) Depiction of auditory network with stochastic cochleagram. (B) Performance of auditory networks trained with varying stochasticity levels evaluated on L adversarial attacks, averaged over 100 randomly chosen test examples of clean speech. Error bars are STD across 5 sets of test stimuli. (C) An intermediate level of Gaussian noise yields the best adversarial robustness. (D) Stochastic activations during training but not during testing yields improvements in adversarial robustness; maintaining stochasticity during adversarial generation and testing further increases robustness.

---

A robust model is achieved with adversarial training using Lee perturbations with = 10-3 and maximum of steps for the attack (ATCochResNett5), Training and adversarial robustness details are presented in SM 4.1 and SM 4.4 respectively.
5.2 Adversarial robustness in auditory models with stochastic cochleagrams
When setting the level of noise in the VOneBlock, neural responses from Macaque V1 to a specific image set were used to tune the relative amplitude of the stochastic representations. However, this type of neural data is not readily available for all sensory areas. We instead empirically found an optimal level of noise for our auditory models by varying the level of additive Gaussian noise in the stochastic cochleagram. The robustness of these models to Los perturbations is evaluated for different perturbation sizes. To ensure a reliable signal for adversarial attack generation, model gradients are sampled eight times for each PGD iteration. The resulting accuracy for each model is shown in Figure 4.. As noise is increased, the model's robustness to adversarial perturbations increases, but only up point, yielding a peak in the robustness curve (Figure (C). The accuracy for StochCochResNetS0 with noise during the training and attack is only slightly worse than ATCochResNet50 (Figure AD). Much like what was observed in VOneResNet50 [14], including noise during training but not during adversarial evaluation significantly increases adversarial robustness, suggesting that these improvements cannot be trivially explained by the stochastic component masking the gradients during the attack. Instead, the benefits appear to reflect downstream representational changes from training with stochasticity. The additional performance boost when stochasticity is included during inference suggests a secondary defense driven by the stochastic activations during evaluation. These results show that by including stochasticity in biologically-inss -inspiiee peripheral model we can improve performance on adversarial examples and by extension learn a more human-like representation.
5.3 Neural population geometry of auditory networks
FGSM Adversarial Exemplar Manifolds (t 0.001)
We investigated whether the neural population geometry used by StochCochRes is similar to that observed in the visual domain. As with the ImageNet experiments, clean
Manifold Capacity
StochCochResNet50
StochGochResNet500
(w/o inference noise)
CochReaNet50
Manifold Width
AeSochRosNet50
or perturbed audio is presented
2
to the model and intermediate representations are extracted,
waveform
cochleagram
cochleagram
(+ moise)
maxpool convt
blook
block 2
block 3
block 4
randomly projected to 5000
features (if the layer has more than 5000 features), and ana- lyzed with MFTMA.
g
Adv Manifold Capacity
Clean Manitald Capaelly
1.0.
waveform
cochleagram (+ noise)
avgpool
0.9.
An analysis of the e-sized ball exemplar manifolds reveals that the geometry of with
0.8.
0.7
StochCochResNet50 CochResNet50
0.6
StochCochResNet50 noise during training but not during evaluation is similar
0.5
ATCochResNet50
10
10 10 10
10
10
10
to that of ATCochResNet50, while the Seeee with noise during evaluation has much lower capacity for the adversarial exemplar manifolds late stages of the network, much like VOneRes- Net50 (Figure (A)). As in VOneResNet50, when the e-sized ball exemplar manifold capacity is normalized by the clean manifold capacity, the normalized capacity for
Figure 5: Adversarial exemplar manifold geometry of auditory models. (A) Capacity of adversarial exemplar manifolds for audi- tory models. For networks without stochasticity, the stochastic layer representation equal to the cochleagram representation. Eval- uating the capacity for the adversarial exemplar manifolds with stochasticity present during inference yields significantly lower capacity after the stochastic layer, while the model trained, but not tested, with stochasticity looks similar to the model achieved through adversarial training. Error bars are STD across 5 RP and MFTMA seeds. (B) Adversarial exemplar capacity normalized by the clean manifold capacity function of adversarial perturbation size. Unnormalized capacity detailed in SM 4.5. Error bars are STD across 5 RP and MFTMA seeds.
7

---

A.
Auditory Networks Class
B.
ImageNet Networks Class Manifold Geomatry avgpool
Manifold Geometry at avgpool
0.18
model
model
Center Correlation
0.16
StochCochResNetS0
0.13-
VOneResNet50
0.14
CochResNNe500
0.12
ResNet50
ATCochResNet50
ATResNat50
0.12
0.11
0.10
0.0
0.001
0.10
1/1020
0.08
0.002
1/255
0.06
0.09
2
8
3
Manifold Width
Manifold Width
Figure 6: Similarities between class manifold geometry in Auditory and ImageNet networks. Geometry of class manifolds for Auditory (A) and ImageNet (B) networks when computed from clean or adversarial stimuli.
StoshCochResNet50 stable with increasing epsilon from the stochastic cochleagram layer onwards TTigureBBB This suggests that etochCochResNee500 represents the adversarially perturbed sounds with the same manifold capacity and width as the clean sounds. By comparison, AeSochResNet50 does not show invariance to the adversarially perturbed stimuli until the avgpool layer.
To examine the key geometric factors associated with an increase in capacity in both the auditory and visual networks, we plot class manifold width vs. the manifold center correlation, as these two factors together determine the capacity estimate. Figure ( and [B show that in both visual and auditory networks an increase in size of the adversarial perturbation leads to an increase in manifold width and center correlation (two key variables leading to the capacity). While this general trend is present for all networks, the networks that more adversarially robust have less change across both metrics as the perturbation size increases. This provides further evidence that the manifold metrics are a useful way to interpret the internal geometries that lead to adversarial robustness, and points to the same underlying mechanism in visual and auditory modalities for the classification degradation in the presence of adversarial vulnerability specifically that the class manifolds become larger and more correlated when adversarial perturbations are present.
6 Manifold overlap and the opposing effects of noise on robust classification
In the previous sections we investigated how stochastic representations change the size and capacity of adversarial exemplar manifolds, but did not directly test whether adversarial examples fall within the manifolds elicited by stochastic activations. We hypothesize that for small perturbations, activations elicited from adversarial examples overlap with the exemplar manifold formed by the stochastic representation of a clean image, effectively eliminating vulnerability to small adversarial perturbations. Here, we directly investigate whether small adversarial perturbations indeed overlap with multiple stochastic representations evoked from the same stimulus, and investigate how this overlap trades off with task performance. To do so, we test whether an SVM can separate the adversarial exemplar manifolds from the clean exemplar manifolds generated for the same stimulus. As manifold inseparability as measured by SVM only one possible description of manifold overlap, we further detail another analysis using pairwise distance distributions in Section SM 8. Finally, we analyze the opposing effects of noise on clean and adversarial performance by investigating a CIFAR model trained with different noise levels.
6.1 CIFAR model and datasets
We investigate how the level of noise changes the manifold geometry in smaller model trained on the CIFAR- 10 dataset 45 with an architecture similar to ResNet18 (2). with the first conv-relu-mane -axpooo layers replaced by
Gabor filters and oollogiall-inspiee activation functions adapted from the Gaussian VeneNe In SM 5.4, we repeat the experiments with Poisson-like stochastic activations as well, where the trends are similar.
Mooe details about the model architecture and training be found in SM 5.1
8

---

6.2 Low perturbation strength adversarial examples overlap with stochastic representations
We measure the overlap between e-sized ball adversarial exemplar manifolds and clean exemplar manifolds at the stochastic representations of the networks (the output of the VOneBlock for the CIFAR and ImageNet models, and the stochastic cochleagram for the auditory model). For all models, the adversarial exemplar manifold is generated by running FGSM Too, varying the strength of the adversarial attack. The clean exemplar manifold generated by measuring multiple stochastic representations from the same natural stimulus. We train an SVM to separate the adversarial exemplar manifold from the clean exemplar manifold (Figure (7). For CIFÁR SVM experiments, the stimulus set includes 20 unique image exemplars, each with 1000 samples. For VOneResNei50 and StochCochResnett samples, and the features are downsampled with random projection to 5000 dimensions. In line with our hypothesis, the clean and adversarial manifolds become less overlapped as the adversarial attack strength increases and performance is degraded.
the stimulus set includes 10 unique stimulus exemplars, each with 5000
6.3 Adversarial robustness requires balancing exemplar and class information
Although increasing the variance of stochastic representations may
CIFAR-10 VOneNet
ImageNet VOneNet
StochCochResNet50
1.0
hide larger adversarial perturba-
0.8
tions, high noise levels may also
Mlaniola
0.6
0.4
reduce capacity for class mani-
0.2
folds and decrease model accu- racy. We hypothesize that the op- timal noise level for model ac- curacy and robustness requires balance between these two factors (Figure (A).. We empir- ically test this hypothesis, first by showing that the adversarial and clean exemplar manifolds be- come less linearly separable with increasing noise variance (Figure BC). We also evaluate the perfor- mance of the CIFAR model with varying noise levels across differ- ent adversarial attack strengths (Figure (D). As the variance of the stochastic responses increase, the clean performance decreases, while performance under ad- versarial attack depends non-linearly on the noise variance. Using MFTMA we examine the effect of varying levels of stochasticity on class manifold capacity and related geometric properties. We generated clean exemplar and class manifolds from stimulus set of 100 unique images, each with 50 noise samples. Figure R shows the dependence of manifold capacity on stochasticity level. As the noise level increases, both class and exemplar manifolds become more entangled and capacity drops. In addition, Figure FF shows that as the noise level increases, the manifold width increases. Thus, both the capacity, which characterizes manifold linear separability, and the geometry demonstrate that increasing noise levels makes manifolds larger, more entangled and less linearly separable. This type of optima is also observed when choosing the level of stochasticity for the auditory networks in Figure and additional auditory analysis of the noise level is found in section SM 4.6.
0.0
10
10
10 10
10
10
Perturbation Strength (c)
Adversarial exemplar manifolds measured at stochastic representation overlap with clean exemplar man- ifolds at smaller attack strengths. A binary SVM trained to separate adversarial and clean exemplar manifolds extracted from the stochastic representation of each network show that clean and adversarial exemplar manifolds measured from the same stimulus become less linearly separable as the attack strength decreases. Experiments are presented for CIFAR VOneNet (Error bar is STD across 20 images and 6 RP seeds), ImageNet VOneRes- Net50 (Error bar is STD across 10 images and 5 RP seeds) and StochCochResNet500 (Error bar STD across 10 images and RP seeds).
Figure 7:
7 Discussion
Using recently developed techniques to analyze the neural population geometry of variety of networks on clean and adversarially perturbed stimuli, our work provides new insight into mechanisms underlying adversarial robustness. First, we show key geometrical differences between adversarially trained networks and VOneNets operating with stochastic neural representations. Second, we demonstrate the generality of the usefulness of stochastic representations for defending against adversarial attacks by showing that the effects extend auditory models. It was not obvious a priori whether stochasticity would have the same benefit on auditory models in part because (unlike vision

---

A.
Adversarial and Clean Manifold Overlap
Class manifold capacity
C. Adversarial and Clean Exemplar Manifold Overlap
Adversariaa performance for
Gaussian CIFAR-10 VOneNet:
Natural Stimulus
Manifold Overlap
1.0
10
Clase
Accuracy
..8
Low noise
Neural noise
0.6
0.6
Adversary
Class
0.4
0.4
Attad
0.2
0.0
0.0
2
4
6
8
10
2
4
10
High noise
Gaussian Noise (a)
Gaussian Noise
E.
Manifold Capacity
F.
Manifold Geometry
Class boundary
Manifald Capacity
1.5
Marifold Witthh
B.
Class
20
Class
1.0
Exemplar
Exemplar
Adversarial and Clean
0.5
10
Manifold Overlap
Class Manifold Capacity
0.0
2.5
16.0
7.5 10.0
Gaussian Noise (o)
Gaussian Noise
Gaussian Noise
Figure 8: Stochastic representations induce opposing geometric effeets that determine model's performance (A) Illustration of the stochasticity level's effect on the representations. In the low-noise regime, clean and adversarial representations do not overlap significantly, but class manifolds are well separated. In the high-noise regime, clean and adversarial representations are more overlapped, but the class manifolds also becomes more entangled.
(B) These two properties trade off' as the level of Gaussian noise is varied. (C) Binary SVMs are trained to separate adversarial and clean exemplar manifolds representations from same image in CIFAR models, showing that clean and adversarial exemplar manifolds become less linearly separable as stochasticity increases. Error bar is STD from 20 images and random seeds. (D) Dependence between model performance and noise level at multiple attack strengths. Error bar is STD from 6 random seeds. (E) Capacity for both class and exemplar manifolds capacity decreases the stochasticity increases. Error bar STD from 6 random seeds. (F) Manifold width increases as the noise level increases. Error bar is STD from random seeds. (All representations are from the stochastic output of the VOneBlock)
models) they are typically trained with additive noise on the input 6 9 given that additive noise ubiquitous real-world auditory signals (because concurrent sounds sum together). Third, we show that stochastic representations of clean and adversarially perturbed stimuli overlap, helping to explain why the stochastic networks generalize better to perturbations below a certain threshold. Fourth, we isolate and quantify the competing effects of stochastic representations on network performance: while stochasticity increases the overlap between clean and adversarial activations, increasing stochasticity also creates more overlap between different class manifolds, making them less separable, and ultimately reducing peak clean network performance.
Here, we focused on linear methods to probe the mechanisms of adversarial robustness. Future work could benefit from more sophisticated methods to measure manifold overlap, and could also extend analysis to more naturally occurring corruptions [46]. Further, while we largely focus on the case of noise injected after fixed biologically inspired filters, more work is needed to explore what types of representations benefit from the addition of noise; for instance Dapello, Marques et al. found there was not as strong of a protective effect with stochasticity added to a standard convolutional filter. We leave the exact role of the neurally plausible filters, as used in both eet50 and StochCochhenet50, as a promising direction for future work. The presence of noise at all stages of the brain, suggests that if we can resolve when stochasticity is useful, this defense may be extensible to additional network layers for greater gains in robustness.
In theoretical neuroscience, much of the discussion on the role of stochasticity in neural coding has centered around the efficient representation of uncertainties associated with rask-relevant variables Our work adds to this line of research, by using geometry to demonstrate how stochasticity improves the neural population's robustness to adversarial perturbations unseen during training in deep networks. We hope that our work will motivate mechanistic explanations of biologically plausible robust computation in ANNs through the lens of geometry, and further identification of biological constraints that might inspire favorable changes in task-efficient neural representations.
47 48).
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by NSF Neuronex 1707398 (S.C.), the Gatsby Charitable Foundation GAT3708 (S.C.), BCS Computational Fellowship (S.C.), Intel Research Grant (S.C., H.L.),, Friends of the McGovern Institute Fellowship (J..FF,, a DOE CSGF Fellowship (J...F), the PhRMA Founda- tion Postdoctoral Fellowship in Informatics (T.M), the Semiconductor Research Corporation (SRC) (J.D., J.J.D... ). NIH grant ROIDC01 7970 (J.H.M.,, the Office of Naval Research grant MURI- 14407 (JJ..D.), the Simons Foundation grant SCGB-544 5429655 (J.J.D.), the MIT-IBM Watson AI Lab grant W1771646 (J.D, J.J.D.). All experiments were performed on the MIT BCS OpenMind Computing
Cluster.
References
[1] Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In Pereira, C C. Burges, I. Bottou, and Q Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097-1105. Curran Associates, Inc., 2012.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. December 2015.
[3] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:
2014.
[4] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[5] Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-365, 2016.
[6] Alexander JE Kell, Daniel I.K Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H McDermott. task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-644, 2018.
[7] Rishi Rajalingham, Flias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J Di- Carlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-ar deep artificial neural networks. Journal of Neuroscience, 38(33):7255-7269, 2018
[8] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint 1811. 12231, 2018.
[9] Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. Metamers of neural networks reveal divergence from human perceptual systems. In NeurIPS, pages 10078-10089, 2019.
[10] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. December 2013.
[11] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. August 2016.
[12] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks speech- to-text. In 2018 IEEE Security and Privacy Workshops (SPW), pages 1-7. IEEE, 2018
[13] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. June 2017.
[14] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D Cox, and James DiCarlo. Simulating primary visual cortex at the front of cnns improves robustness image perturbations. In NeurIPS, 2020.
[15] William Softky and Christof Koch. The highly irregular firing of cortical cells is inconsistent with temporal integration of random epsps. Journal of neuroscience, 13(1)3334-350, 1993.
[16] Aldo Faisal, Lue Selen, and Daniel M Wolpert. Noise in the nervous system. Nature reviews neuroscience, 94)3 2008.

---

[17]
Robbe LT Goris, J Anthony Movshon, and Eero P Simoncelli. Partitioning neuronal variability, Nature neuroscience, 17(6):858 865, 2014.
[18] Rodrigo Echeveste, Laurence Aitchison, Guillaume Hennequin, and Máté Lengyel. Cortical-like dynamics in recurrent circuits optimized for sampling-hase probabilistic inference. Nature neuroscience, 113-)1133-11499 2020.
[19] Zachary W Davis, Lyle Muller, Julio Martinez- Trujillo, Terrence Sejnowski, and John H Reynolds. Spontaneous travelling cortical waves gate perception in behaving primates. Nature, 587(7834):452433, 2020.
[20] Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
[21] Yuo Chung and LF Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks, arXiv preprint Seti:22044070555 2021.
[22] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. December 2017.
[23] Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. February 2019.
[24] Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature
Reviews Neuroscience, 22(11))703-718 2021.
[25] Nikolaus Kriegeskorte and Rogier A Kievit. Representational geometry: integrating cognition, computation, and the brain. Trends in cognitive sciences, 17(8):401-44 -11211--22222011 2013.
[26] Olivier J Hénaff and Eero P Simoncelli. Geodesics of learned representations. arXiv preprint arXiv:1511.0639) 2015.
[27] Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos. Nature neuroscience, page 1, 2019.
[28] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, pages 6109-6119, 2019.
[29] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 1610.01644, 2016.
[30] John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. arXiv preprint arXiv: 1909:09900000 190.0339, 2019.
[31] Anna A Ivanova, John Hewitt, and Noga Zaslavsky. Probing artificial neural networks: insights from neuroscience. arXiv preprint arXiv: 104.08977 2021.
[32] Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Linear readout of object manifolds. Physical Review F, (36)0000, 2016
[33] Uri Cohen, Sue Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
[34] Cory Stephenson, Jenelle Feather, Suchismita Padhy, Oguz Elibol, Hanlin Tang, Josh McDer- mott, and Sue Yeon Chung. Untangling in invariant speech recognition. In NeurlPS, 2019.
[35] James DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 13333333-34,, 2007.
[36] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security:
Circumventing defenses to adversarial examples, February 2018.
[37] J Deng, W Dong, R Socher, L Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages June 2009.
[38] William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into hilbert space, 1984.
[39] Candes and T Tao. Decoding by linear programming. IEEE Trans. Inf. Theory, 51(12):4203- 4215, December 2005.
12

---

[40] Lea Schönherr, Katharina Kohls, Steffen Zeiler, Thorsten Holz, and Dorothea Kolossa. Adver- sarial attacks against automatic speech recognition systems via psychoacoustic hiding. arXiv preprini arXiv: 1888005665, 2018.
[41] Yao Qin, Nicholas Carlini, Garrison Cottrell, Ian Goodfellow, and Colin Raffel. Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. In International conference on machine learning, pages 5231-5240. PMLR, 2019.
[42] Douglas B Paul and Janet Baker. The design for the wall street journal based esr corpus. In Proceedings of the workshop Speech and Natural Language, pages 357-362. Association for Computational Linguistics, 1992.
[43] Brian Glasberg and Brian CJ Moore. Derivation of auditory filter shapes from notched-noise data. Hearing research, 47(1-2):103-1111
1990.
[44] Josh H McDermott and Eero P Simoncelli. Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis. Neuron, 71(5):926 940, 2011.
[45] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[46] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. March 2019.
[47] Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. Bayesian inference with probabilistic population codes. Nature neuroscience, 911))1432-11438, 2006.
[48] Wei Ji Ma and Mehrdad Jazayeri. Neural coding of uncertainty and probability. Annual review of neuroxcience, 37:205-2222 2014.
13


Reviews:
[{"review_id": "gmyxZaQ1FHn", "reviewer": "Reviewer_D6LM", "summary": "The submission investigates the role of response stochasticity, a feature of biological neurons, in robustness to adversarial attacks in deep neural networks.  They use manifold analysis to compare the neural population geometry of standard-trained, adversarial-trained and stochastic networks and find geometrical signatures of the different conditions. They show that the robustness achieved through adversarial training is qualitatively different than that achieved by the stochastic network. They characterize how increasing noise affects the resulting manifolds. They observe a tradeoff between adversarial robustness and classification performance and can explain this tradeoff in terms of geometric features of the representations. These results (on images) are replicated in the auditory domain.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission employs recently developed manifold analysis techniques and sheds new light on a previously documented phenomenon. The demonstration of the value of stochasticity in auditory models is novel.\n\nMFTMA is an exciting analysis technique which was proposed a few years ago but has yet to be widely employed. I am pleased to see it used here where is has helped to shed light on an interesting phenomenon. The work is complete and the authors clearly identify avenues for future work which are not addressed in the present submission.\n\nIn general, the submission is clearly written. However, there may be room for improvement in the introduction + related work section (often two sections, but here combined into one). In particular, I found the logical jump from paragraph 2 to 3 unclear. The particular experimental design and methodology does not feel clearly motivated in the text (although I believe that it is motivated). I imagine the authors have tried to condense this first section to make room for the rest of the paper but it feels slightly too condensed.  \n\nminor points: \n- In the first paragraph the authors write \"establishing them as leading candidate models of human perception\". \"Human perception\" denotes a much broader set of capacities than those for which we have validated neural network models. Consider scaling back to \"visual object recognition\".\n- line 269: \"We include investigation how the level of noise...\". Are there some missing words here?\n\nThe main conclusions are novel. Some of the results replicate previous findings. The work constitutes progress towards understanding the role of stochasticity in neural computation, which has potential implications for both neuroscience and machine learning. It also constitutes a step towards biologically-inspired deep learning.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors clearly acknowledge the limitations of their analysis. They do not mention any potential negative societal impact but none comes to mind.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.25"}, "extra_text": {}}, {"review_id": "McpsZADnF9", "reviewer": "Reviewer_UJnB", "summary": "The authors note that stochasticity is an important component of recent attempts to improve network adversarial robustness by emulating known biological mechanisms. They also demonstrate their own similar result for a network trained on auditory signals. Motivated by this, they analyze the representations formed by a set of example networks to understand the relationship between representation geometry and robustness. Their analysis reveals differences among networks that underwent standard training, adversarial training, and those with stochasticity added. They claim that this analysis improves our understanding of mechanisms of robust perception as well as our understanding of the role stochasticity plays in biological computation.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Review summary:*\n\nThe MFTMA analysis method reveals clear differences among the networks tested. Specifically, I find it compelling how clearly the authors demonstrated that two strategies for achieving robustness can show opposing behavior in terms of their analysis. I note a bunch of nitpick concerns in my review below, but overall I found the paper to be well written and the results interesting. I am recommending the paper for acceptance, and will further increase my score if my concerns are addressed.\n\n*Strengths:*\n\n\nThe paper is well written and has helpful extended explanations in the appendix. Their analysis provides a very interesting perspective on how two different methods for achieving robustness (adversarial training vs noise training) can lead to alternative mechanisms. Their controls, both within the main text and in the appendix, are thorough and convincing of the overall result.\n\n*Weaknesses:*\n1) Claims:\n1.a) I think the paper falls short of the high-level contributions claimed in the last sentence of the abstract. As the authors note in the background section, there are a number of published works that demonstrate the tradeoffs between clean accuracy, training with noise perturbations, and adversarial robustness. Many of these, especially Dapello et al., note the relevance with respect to stochasticity in the brain. I do not see how their additional analysis sheds new light on the mechanisms of robust perception or provides a better understanding of the role stochasticity plays in biological computation. To be clear - I think the paper is certainly worthy of publication and makes notable contributions. Just not all of the ones claimed in that sentence.\n\n1.b) The authors note on lines 241-243 that “the two geometric properties show a similar dependence for the auditory (Figure 4A) and visual (Figure 4B) networks when varying the eps-sized perturbations used to construct the class manifolds.” I do not see this from the plots. I would agree that there is a shared general upward trend, but I do not agree that 4A and 4B show “similar dependence” between the variables measured. If nothing else, the authors should be more precise when describing the similarities.\n\n2) Clarifications:\n2.a) The authors say on lines 80-82 that the center correlation was not insightful for discriminating model defenses, but then use that metric in figure 4 A&B. I’m wondering why they found it useful here and not elsewhere? Or what they meant by the statement on lines 80-82.\n\n2.b) On lines 182-183 the authors note measuring manifold capacity for unperturbed images, i.e. clean exemplar manifolds. Earlier they state that the exemplar manifolds are constructed using either adversarial perturbations or from stochasticity of the network. So I’m wondering how one constructs images for a clean exemplar manifold for a non-stochastic network? Or put another way, how is the denominator of figure 2.c computed for the ResNet50 & ATResNet50 networks?\n\n2.c) The authors report mean capacity and width in figure 2. I think this is the mean across examples as well as across seeds. Is the STD also computed across examples and seeds? The figure caption says it is only computed across seeds. Is there a lot of variability across examples?\n\n2.d) I am unsure why there would be a gap between the orange and blue/green lines at the minimum strength perturbation for the avgpool subplot in figure 2.c. At the minimum strength perturbation, by definition, the vertical axis should have a value of 1, right? And indeed in earlier layers at this same perturbation strength the capacities are equal. So why does the ResNet50 lose so much capacity for the same perturbation size from conv1 to avgpool? It would also be helpful if the authors commented on the switch in ordering for ATResNet and the stochastic networks between the middle and right subplots.\n\n3) General curiosities (low priority):\n3.a) What sort of variability is there in the results with the chosen random projection matrix? I think one could construct pathological projection matrices that skews the MFTMA capacity and width scores. These are probably unlikely with random projections, but it would still be helpful to see resilience of the metric to the choice of random projection. I might have missed this in the appendix, though.\n\n3.b) There appears to be a pretty big difference in the overall trends of the networks when computing the class manifolds vs exemplar manifolds. Specifically, I think the claims made on lines 191-192 are much better supported by Figure 1 than Figure 2. I would be interested to hear what the authors think in general (i.e. at a high/discussion level) about how we should interpret the class vs exemplar manifold experiments.\n\n4) Nitpick, typos (lowest priority):\n4.a) The authors note on line 208 that “Unlike VOneNets, the architecture maintains the conv-relu-maxpool before the first residual block, on the grounds that the cochleagram models the ear rather than the primary auditory cortex.” I do not understand this justification. Any network transforming input signals (auditory or visual) would have to model an entire sensory pathway, from raw input signal to classification. I understand that VOneNets ignore all of the visual processing that occurs before V1. I do not see how this justifies adding the extra layer to the auditory network.\n\n4.b) It is not clear why the authors chose a line plot in figure 4c. Is the trend as one increases depth actually linear? From the plot it appears as though the capacity was only measured at the ‘waveform’ and ‘avgpool’ depths; were there intermediate points measured as well? It would be helpful if they clarified this, or used a scatter/bar plot if there were indeed only two points measured per network type.\n\n4.c) I am curious why there was a switch to reporting SEM instead of STD for figures 5 & 6.\n\n4.c) I found typos on lines 104, 169, and the fig 5 caption (“10 image and”).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "KbxKam7vQSL", "reviewer": "Reviewer_fXyv", "summary": "The paper compares networks that are adversarially trained with networks that have biologically inspired stochasticity in the first layer, including previous visual object-recognition networks and a new auditory network. The analysis is based on manifolds of classes and exemplars in representation space, and considers measures of manifold size and overlap. Robustness to adversarial attacks is also considered. The analysis highlights different responses of the stochastic and adversarially trained networks to adversarial attacks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThis is a new kind of analysis, and I don’t think it has been used to study adversarial robustness before. It makes sense to do so. \n\nQuality: \nThe work is technically sound and thorough. \n\nIn comparing visual networks, the paper varies two things at once. The paper focuses on the fact that VOneResNet50 has stochastic responses while ATResNet50 has adversarial training. But another difference is that VOneResNet50 has biologically inspired first-layer filters, while ATResNet50 has standard first-layer filters. It seems cleaner to either address the first difference in isolation, or to test all combinations. The authors leave this for future work (line 343) but it may be a limitation. \n\nClarity: \nThe writing is clear and engaging. \n\nSignificance: \nHere I am uncertain. The results with undefended and adversarially trained networks are unsurprising. It is interesting that stochasticity widens exemplar manifolds in early layers, contrary to adversarial training, but I am not sure what else I would have expected. I may need more guidance from the authors about what to take from the study.  \n\nThe last section of the paper shows that exemplar manifolds due to stochasticity and adversarial perturbations overlap, less so at high attack strengths, and more so with high stochasticity. Here again, I am not sure what else might have been expected. \n\nMinor: \n“basis of this stochasticity and its implications for information processing are open questions in neuroscience [19]” The reference is good but not recent. \n\nIn Figure 2B, I am not clear about the significance of the decreasing manifold width in ATResNet50, versus the fact that for other networks, width is higher at the output than the pixels. I don’t think it matters, but maybe the authors could discuss this in more detail, or add a plot of un-perturbed manifold widths for additional context. \n\nAnother helpful point of context would be the accuracy vs. attack strength added to Figure 1C,D. \n\nIn Figure 4C, the black line is easy to miss. Lines may not be the best representation in this plot because the ordinate isn’t continuous, and multiple levels are skipped. \n\nI missed which layers are analyzed in Figure 5. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work has a minor societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "The work does not raise ethical concerns.  ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "9BC3Fh1LYZ4", "reviewer": "Reviewer_8wmG", "summary": "It is well known that a deep learning model is fragile to adversarial perturbation, whereas a biological system is not. This study investigate the geometry of internal representations (especially, in the penultimate later) of a DL model, that trained using adversarial examples, and that with biologically-inspired component. The basic approach is based on what the authors call MFTMA.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Roughly speaking, the reviewer failed in finding a significance in the results and discussions.\n\n(1) The relationship with references [20,22] are unclear. The word \"MFTMA\" does not appear in the references. The supplementary material 2.1 suggests that it is a synonym fot the method in [8]. Please discuss the relationship with references.\n\n(2) The references [20,22] demonstrated the manifold size is related to the accuracy. Then, the contribution of this study is unclear. All experiments can be done using the accuracy as the measure instead of the manifold size. The only thing that the reviewer considers as the contribution is the suggestion that the manifold analysis is applicable to adversarial perturbation. If so, the introduction is inappropriate.\n\n(3) According to Figure 1, the top-1 accuracy is proportional to the manifold capacity. A vanilla ResNet loses its accuracy and manifold capacity with a strong attack, whereas a ResNet trained with adversarial perturbation does not. VOneResMet is more robust to attack than the vanilla. The similar results are found in auditory data recognition in Section 4.\n\nSeveral previous works such as Miyato et al. demonstrated that all the dropout, random perturbation, and adversarial training smooth the score function. While dropout and random perturbation are stochastic and inefficient compared to adversarial training, their contribution is similar to adversarial training. Hence, it is unsurprising that \"there exists an overlap between the stochastic representations of clean and adversarially perturbed stimuli\".\n\nEven through the very first paper by Goodfellow et al. suggested that the dropout is insufficient for robustness to the adversarial perturbation, many previous studies such as Dhillon et al. have attempted to improve the robustness by stochastic components. Hence, it is unsurprising that \"stochastic representations provide a defense against adversarial perturbations.\"\n\n- Miyato et al., \"Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning,\" IEEE TPAMI, 2019.\n- Goodfellow et al., \"Explaining and Harnessing Adversarial Examples,\" ICLR 2015.\n- Dhillon et al., \"Stochastic Activation Pruning for Robust Adversarial Defense,\" ICLR2018.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an excellent paper with good consensus and engagement among the reviewers for the acceptance. It derives novel insights for mechanisms of robust perception, and will likely inspire researchers in the fields of adversarial robustness, biologically-inspired neural networks, and computational neuroscience, all important and active sub-fields in NeurIPS conference. 
Decision: Accept (Poster)