Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception

1 Introduction . In recent years , artificial neural networks ( ANNs ) have come to dominate both visual object recognition and auditory recognition tasks [ 1 , 2 , 3 ] , establishing them as leading candidate models for several domains of human perception [ 4 , 5 , 6 , 7 ] . However , they still exhibit many non-human-like traits [ 8 , 9 ] . One such failure is in the existence of adversarial perturbations – small changes to stimuli explicitly crafted to fool a model that remain imperceptible to humans [ 10 , 11 , 12 ] – which demonstrate the fragility of some ANNs as models of biological perception . 1See https : //github.com/chung-neuroai-lab/adversarial-manifolds for accompanying code . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , Dapello , Marques et al . discovered that one such method , known as adversarial training [ 13 ] , not only reduces the network ’ s adversarial vulnerability but also yields network representations that are more similar to those in the primate primary visual cortex [ 14 ] . Motivated by this result , the authors developed VOneNets , a class of networks that simulate the primate primary visual cortex at the front of a convolutional neural network , and show improved robustness to adversarial attacks with no adversarial training . However , a number of questions remain unanswered – in particular , while both adversarially trained networks and VOneNets have improved adversarial robustness and also greater similarity to the primate primary visual cortex , it is unclear how VOneNets achieve robustness , and in particular if the mechanism of robustness is similar to that induced by adversarial training . A key component of robustness in VOneNets is the inclusion of stochastic representations during both training and inference , a feature inspired by biological sensory neurons which exhibit trial-to-trial variability across presentations of the same stimulus [ 15 ] . The implications of this stochasticity for information processing are open questions in neuroscience [ 16 , 17 , 18 , 19 ] . Pinpointing how this representational stochasticity contributes to robustness in VOneNets could drive further developments in the mechanisms of robust perception . Here we use recently developed manifold analysis techniques from computational neuroscience [ 20 ] to look beyond accuracy and investigate the internal neural population geometry [ 21 ] of standard , adversarially trained , and biologically-inspired stochastic networks in response to clean and adversarially perturbed examples in both visual and auditory domains . We present several key findings : • Using manifold analysis , we demonstrate that standard , adversarially trained , and stochastic networks each have distinct geometric signatures in response to clean and adversarially perturbed stimuli , shedding light on varied robustness mechanisms . • We demonstrate the generality of our findings by translating the results to a novel biologically-inspired auditory ANN , StochCochResNet50 , that includes stochastic responses . Stochasticity makes auditory networks more robust to adversarial perturbations , and the underlying neural population geometry is largely consistent with that in vision networks . • Analysis of stochastic networks reveals a protective overlap between the representations of adversarial examples and clean stimuli . We quantitatively survey the stochasticity conditions leading to the overlap , and map a competing geometric effect mediating a trade-off between clean and adversarial performance . 2 Related work . Previous work in machine learning has reported that additive noise can improve the adversarial robustness of a model : Liu et al . [ 22 ] used random noise applied to pixels and intermediate layers to improve adversarial robustness , and Cohen et al . [ 23 ] demonstrated a method to transform a model with Gaussian noise in the pixel space to one with certified robustness to attacks of a given strength . Unlike VOneNets , both [ 22 ] and [ 23 ] rely on using multiple inference passes with different noise samples to improve robustness . Further , while both of these works suggest new defenses , none analyze the properties of stochastic representations that lead to robustness . The need to understand internal mechanisms of biological and artificial neural networks gave rise to the field of neural population geometry [ 24 , 21 ] , an line of work exploring geometric properties of high-dimensional neural representations . To capture the complexity of neural representations in ANNs , various geometric analyses have been proposed , including representation similarity analysis [ 25 ] , geodesics [ 26 ] , curvature [ 27 ] , and intrinsic dimensionality [ 28 ] . Another popular approach is through supervised linear probes [ 29 ] , i.e. , linear classifiers trained on top of these representations . However , recent work has discussed how such analyses with linear classifiers are limited [ 30 ] , and that more structural studies are needed for investigating the internal layers of networks [ 31 ] . A recent theoretical development based on replica theory in statistical physics [ 32 , 20 , 33 ] provides a solution to this , by formally connecting the geometry and linearly decodable information embedded in neural populations . By using this approach , our study situates the analysis of robustness mechanisms into the developing field of neural population geometry [ 24 , 21 ] . 3 Methods and experimental setup . 3.1 Replica-based manifold analysis . In this paper , we use replica mean-field theoretic manifold analysis ( MFTMA , hereafter ) [ 20 , 34 , 33 ] , which formally connects the linear decodability of object manifolds , defined as a set of stimulusevoked representations in neural state space [ 35 ] , to their geometrical properties . Here we provide a brief description of the key quantities ( See SM 1 and [ 20 ] for a complete treatment of MFTMA ) . Using this framework , we analyze the size , shape , and distribution of object manifolds as they are transformed throughout the network , gaining insight into the neural population geometry underlying performance of the classification task . Specifically , given P object manifolds ( measured as feature point clouds ) in N dimensions , MFTMA returns capacity ( a measure of linear separability ) and the manifold dimensions , radii , and center correlations associated with capacity estimation . Manifold capacity ( α = P/N ) refers to the maximum number of object manifolds ( Pmax ) that can be linearly separated given N features , and characterizes the linearly-decodable object information per feature dimension . MFTMA estimates α through measures of the manifold dimension ( DM ) , manifold radius ( RM ) , and manifold center correlation , which refer to dimensionality , size , and distribution of object manifolds relevant for the linear classification . In our analysis , we combine the manifold dimension and radii together into a single manifold width measure , defined as RM · √ DM , which captures the width of the convex hull of a manifold . Manifold width formally links the linear separability of object manifolds with their underlying geometric structure . Specifically , small values of manifold width yield more linearly separable manifold geometry . The correlation between locations of the object manifolds also plays a role in determining manifold capacity , as more correlated manifolds are less separable . For our analysis , center correlation is calculated as the average of the absolute value of the cosine similarity between pairs of object manifold centroids . In this work , we characterize two types of manifolds : 1 ) Class manifolds : each manifold is defined by the activations evoked by multiple examples drawn from a specific class of visual or auditory stimuli ( i.e. , object or word identity ) . The variability within the manifold can come from different exemplars within the class , but may also be influenced by adversarial perturbations , and/or a layer with stochastic activations . 2 ) Exemplar manifolds : each manifold is defined by the activations evoked by multiple instances of a single exemplar ( i.e. , image or utterance ) with the manifold variability reflecting either norm-bounded adversarial perturbations2 and/or the influence of stochasticity . Although the class manifold analysis is most tightly coupled to the accuracy of the network on a classification task , analyzing the exemplar manifolds gives insight into how the class manifolds are constructed . Specifically , multiple types of exemplar manifold geometries can lead to the same class manifold capacity and width ( Figure 1 ) . 3.2 Quantifying object manifold overlap . The MFTMA framework provides a geometric description of exemplar manifolds evoked by adversarial and clean stimuli in the case of multiple distributed object manifolds . However , we also characterize the overlap between adversarial and clean exemplar manifolds generated from the same stimuli ( on the grounds that overlap should produce robustness to the adversarial perturbations in question ) . To do so , we use a notion of object manifold overlap defined as the generalization error of a linear Support Vector Machine ( SVM ) fit to separate two sets of representations , and report the chance-normalized error rate on the held-out data . A normalized error rate of 1 means that two 2We replicate many experiments for non-adversarial but random perturbations within the -sized ball to directly compare model geometry measured on the same stimuli . Results are similar and provided in SM 6. representations are completely overlapped , and 0 means that they are completely separable and thus not overlapped . We use the scikit-learn SVM implementation with a train/test split of 80/20 . 3.3 Adversarial attacks . For performing adversarial attacks , we use either the single-step fast gradient sign method ( FGSM ) with a random starting location or multi-step projected gradient descent ( PGD ) . Unless otherwise specified , we use L∞ norm constrained attacks . All attacks are untargeted and performed on a model-by-model basis . When the goal is to evaluate adversarial accuracy in newly created networks with stochastic internal representations , we use ensemble-PGD , where each step is in the mean direction of k samples of the noisy gradient to ensure a useful signal [ 36 ] . For more details , see SM 2 . 4 Manifold analysis of robustness in ImageNet-trained networks . We first investigate the properties of ImageNet-trained models , aiming to compare the geometric signatures of standard , adversarially trained , and stochastic networks . 4.1 Models and Dataset . We use images sampled from the ImageNet [ 37 ] test set . For class manifold analysis , the clean stimulus set consists of 50 classes , with each class containing 50 unique exemplar images for a total of 2500 unique images . For exemplar manifold analysis , 100 unique images are sampled from the ImageNet test set and each is perturbed with FGSM from a random starting location 50 times for 5000 unique images . We analyzed three publicly available ImageNet models , including ResNet50 with standard training , ResNet50 adversarially trained with an L∞ = 4/255 penalty ( ATResNet50 ) , and VOneResNet50 , a ResNet50-based model with the first conv-relu-maxpool layers replaced by a linear-nonlinear-Poisson model front-end ( called the VOneBlock ) with Gabor filters and noise fitted to primate neuronal data . In addition , we analyze a novel variant of VOneResNet50 ( GVOneResNet50 ) with additive Gaussian noise scaled to match the mean over all units in the output of the VOneBlock in response to a set of reference stimuli . GVOneResNet50 performs similarly to the original VOneResNet50 in clean and adversarial conditions ( see SM 3.3 for more details ) . While the first two models are deterministic , VOneResNet50 and GVOneResNet50 have stochastic representations at the output of the VOneBlock . For additional details see SM 3 . 4.2 MFTMA reveals unique robustness strategies for VOneNet and adversarial training . Our analysis begins by observing the effects of norm-bounded , gradient-based adversarial attacks on the class manifold geometry of representations in ResNet50 , ATResNet50 , VOneResNet50 , and GVOneResNet50 . For all experiments , images perturbed with PGD L∞ constraints of ∈ [ 0 , 1/1020 , 1/255 ] are shown to a model and intermediate representations are extracted , randomly projected to 5000 features [ 38 , 39 ] , and analyzed with MFTMA ( Figure 2A ) . Much like previously reported with MFTMA [ 33 ] , models develop separable representations in later layers , with capacity peaking in the final layer before classification ( see SM 3.5 ) . First we focus on this penultimate layer , where class manifold representations have become linearly separable and where capacity thus peaks . For the full layer-wise results , see SM 3.5 . We empirically confirm that capacity of the penultimate layer is predictive of the top-1 accuracy across models and attack strengths ( Figure 2B ) , indicating that MFTMA is sensitive to the attacks we are investigating . MFTMA exposes how capacity ( Figure 2C ) and class manifold width ( Figure 2D ) vary in the four models as a function of perturbation strength . Adversarial attacks cause the width of class manifolds to grow most rapidly in ResNet50 , to a lesser degree in VOneNets , and least of all in ATResNet50 . Next , to investigate how the models represent adversarial perturbations around individual images , we introduce another MFTMA-based approach . Instead of class manifolds , we consider exemplar manifolds consisting of points sampled using FGSM with L∞ constraints of ∈ [ 0 , 1/1020 , 1/255 , 4/255 , 8/255 ] from a random starting location in the epsilon-sized ball around 100 exemplar images , with the goal of tracing how these -sized ball exemplar manifolds develop as they travel through subsequent layers of our networks of interest ( Figure 3A ) . Restricting our analysis to the layer-wise trajectory of = 8/255 sized ball exemplar manifolds ( Figure 3B ) , the most salient feature is how distinct the trajectories are for the ATResNet50 and VOneNets . Our results indicate the defense mechanism induced through adversarial training generally stabilizes the width of the -sized ball around an exemplar as it propagates through the network , effectively mapping small perturbations around the clean image to small regions in later layers of the network . By contrast , the width of the -sized ball increases in VOneNets and the standard ResNet50 . In fact , at the VOneBlock output , the exemplar manifolds become highly entangled , suggesting different mechanisms of robustness for adversarially trained and stochastic models . How then do stochastic responses improve robustness ? Figure 3C demonstrates that when the capacity of the adversarially induced exemplar manifolds is normalized by the capacity for representations of the same unperturbed images ( clean exemplar manifolds ) 3 , VOneNets are far more stable than the standard ResNet50 network . At lower attack strengths where VOneNets ’ accuracies are minimally degraded , their normalized capacity remains stable , indicating that they represent the perturbed images with approximately the same capacity as clean images . In other words , the adversarial perturbations do not push the representation beyond that for clean images . In SM 3.7 , we extend this analysis to a variety of additional networks including two networks trained adversarially with different norms , VOneResNet50 with no stochasticity during training or inference , and ResNet50 with a stochastic activation layer mirroring that in VOneResNet50 ( see Dapello , Marques et al . [ 14 ] 3Clean exemplar manifolds for non-stochastic models have the maximum capacity of 2 [ 20 ] . for network details . ) Our trends hold up in each of these cases , with adversarially trained networks stabilizing the -sized ball , and stochastic networks expanding it at the point of the stochastic layer , but using similar capacity relative to clean images for small perturbations . In all experiments , we find similar geometric signatures for the Gaussian and Poisson VOneNets . For simplicity in the rest of the paper we focus on stochastic representations with Gaussian noise , but include an analysis of a Poisson noise network trained on CIFAR-10 in SM 5.4 .

Summary: The submission investigates the role of response stochasticity, a feature of biological neurons, in robustness to adversarial attacks in deep neural networks.  They use manifold analysis to compare the neural population geometry of standard-trained, adversarial-trained and stochastic networks and find geometrical signatures of the different conditions. They show that the robustness achieved through adversarial training is qualitatively different than that achieved by the stochastic network. They characterize how increasing noise affects the resulting manifolds. They observe a tradeoff between adversarial robustness and classification performance and can explain this tradeoff in terms of geometric features of the representations. These results (on images) are replicated in the auditory domain.