Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception

1 Introduction . In recent years , artificial neural networks ( ANNs ) have come to dominate both visual object recognition and auditory recognition tasks [ 1 , 2 , 3 ] , establishing them as leading candidate models for several domains of human perception [ 4 , 5 , 6 , 7 ] . However , they still exhibit many non-human-like traits [ 8 , 9 ] . One such failure is in the existence of adversarial perturbations – small changes to stimuli explicitly crafted to fool a model that remain imperceptible to humans [ 10 , 11 , 12 ] – which demonstrate the fragility of some ANNs as models of biological perception . 1See https : //github.com/chung-neuroai-lab/adversarial-manifolds for accompanying code . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , Dapello , Marques et al . discovered that one such method , known as adversarial training [ 13 ] , not only reduces the network ’ s adversarial vulnerability but also yields network representations that are more similar to those in the primate primary visual cortex [ 14 ] . Motivated by this result , the authors developed VOneNets , a class of networks that simulate the primate primary visual cortex at the front of a convolutional neural network , and show improved robustness to adversarial attacks with no adversarial training . However , a number of questions remain unanswered – in particular , while both adversarially trained networks and VOneNets have improved adversarial robustness and also greater similarity to the primate primary visual cortex , it is unclear how VOneNets achieve robustness , and in particular if the mechanism of robustness is similar to that induced by adversarial training . A key component of robustness in VOneNets is the inclusion of stochastic representations during both training and inference , a feature inspired by biological sensory neurons which exhibit trial-to-trial variability across presentations of the same stimulus [ 15 ] . The implications of this stochasticity for information processing are open questions in neuroscience [ 16 , 17 , 18 , 19 ] . Pinpointing how this representational stochasticity contributes to robustness in VOneNets could drive further developments in the mechanisms of robust perception . Here we use recently developed manifold analysis techniques from computational neuroscience [ 20 ] to look beyond accuracy and investigate the internal neural population geometry [ 21 ] of standard , adversarially trained , and biologically-inspired stochastic networks in response to clean and adversarially perturbed examples in both visual and auditory domains . We present several key findings : • Using manifold analysis , we demonstrate that standard , adversarially trained , and stochastic networks each have distinct geometric signatures in response to clean and adversarially perturbed stimuli , shedding light on varied robustness mechanisms . • We demonstrate the generality of our findings by translating the results to a novel biologically-inspired auditory ANN , StochCochResNet50 , that includes stochastic responses . Stochasticity makes auditory networks more robust to adversarial perturbations , and the underlying neural population geometry is largely consistent with that in vision networks . • Analysis of stochastic networks reveals a protective overlap between the representations of adversarial examples and clean stimuli . We quantitatively survey the stochasticity conditions leading to the overlap , and map a competing geometric effect mediating a trade-off between clean and adversarial performance . 2 Related work . Previous work in machine learning has reported that additive noise can improve the adversarial robustness of a model : Liu et al . [ 22 ] used random noise applied to pixels and intermediate layers to improve adversarial robustness , and Cohen et al . [ 23 ] demonstrated a method to transform a model with Gaussian noise in the pixel space to one with certified robustness to attacks of a given strength . Unlike VOneNets , both [ 22 ] and [ 23 ] rely on using multiple inference passes with different noise samples to improve robustness . Further , while both of these works suggest new defenses , none analyze the properties of stochastic representations that lead to robustness . The need to understand internal mechanisms of biological and artificial neural networks gave rise to the field of neural population geometry [ 24 , 21 ] , an line of work exploring geometric properties of high-dimensional neural representations . To capture the complexity of neural representations in ANNs , various geometric analyses have been proposed , including representation similarity analysis [ 25 ] , geodesics [ 26 ] , curvature [ 27 ] , and intrinsic dimensionality [ 28 ] . Another popular approach is through supervised linear probes [ 29 ] , i.e. , linear classifiers trained on top of these representations . However , recent work has discussed how such analyses with linear classifiers are limited [ 30 ] , and that more structural studies are needed for investigating the internal layers of networks [ 31 ] . A recent theoretical development based on replica theory in statistical physics [ 32 , 20 , 33 ] provides a solution to this , by formally connecting the geometry and linearly decodable information embedded in neural populations . By using this approach , our study situates the analysis of robustness mechanisms into the developing field of neural population geometry [ 24 , 21 ] . 3 Methods and experimental setup . 3.1 Replica-based manifold analysis . In this paper , we use replica mean-field theoretic manifold analysis ( MFTMA , hereafter ) [ 20 , 34 , 33 ] , which formally connects the linear decodability of object manifolds , defined as a set of stimulusevoked representations in neural state space [ 35 ] , to their geometrical properties . Here we provide a brief description of the key quantities ( See SM 1 and [ 20 ] for a complete treatment of MFTMA ) . Using this framework , we analyze the size , shape , and distribution of object manifolds as they are transformed throughout the network , gaining insight into the neural population geometry underlying performance of the classification task . Specifically , given P object manifolds ( measured as feature point clouds ) in N dimensions , MFTMA returns capacity ( a measure of linear separability ) and the manifold dimensions , radii , and center correlations associated with capacity estimation . Manifold capacity ( α = P/N ) refers to the maximum number of object manifolds ( Pmax ) that can be linearly separated given N features , and characterizes the linearly-decodable object information per feature dimension . MFTMA estimates α through measures of the manifold dimension ( DM ) , manifold radius ( RM ) , and manifold center correlation , which refer to dimensionality , size , and distribution of object manifolds relevant for the linear classification . In our analysis , we combine the manifold dimension and radii together into a single manifold width measure , defined as RM · √ DM , which captures the width of the convex hull of a manifold . Manifold width formally links the linear separability of object manifolds with their underlying geometric structure . Specifically , small values of manifold width yield more linearly separable manifold geometry . The correlation between locations of the object manifolds also plays a role in determining manifold capacity , as more correlated manifolds are less separable . For our analysis , center correlation is calculated as the average of the absolute value of the cosine similarity between pairs of object manifold centroids . In this work , we characterize two types of manifolds : 1 ) Class manifolds : each manifold is defined by the activations evoked by multiple examples drawn from a specific class of visual or auditory stimuli ( i.e. , object or word identity ) . The variability within the manifold can come from different exemplars within the class , but may also be influenced by adversarial perturbations , and/or a layer with stochastic activations . 2 ) Exemplar manifolds : each manifold is defined by the activations evoked by multiple instances of a single exemplar ( i.e. , image or utterance ) with the manifold variability reflecting either norm-bounded adversarial perturbations2 and/or the influence of stochasticity . Although the class manifold analysis is most tightly coupled to the accuracy of the network on a classification task , analyzing the exemplar manifolds gives insight into how the class manifolds are constructed . Specifically , multiple types of exemplar manifold geometries can lead to the same class manifold capacity and width ( Figure 1 ) . 3.2 Quantifying object manifold overlap . The MFTMA framework provides a geometric description of exemplar manifolds evoked by adversarial and clean stimuli in the case of multiple distributed object manifolds . However , we also characterize the overlap between adversarial and clean exemplar manifolds generated from the same stimuli ( on the grounds that overlap should produce robustness to the adversarial perturbations in question ) . To do so , we use a notion of object manifold overlap defined as the generalization error of a linear Support Vector Machine ( SVM ) fit to separate two sets of representations , and report the chance-normalized error rate on the held-out data . A normalized error rate of 1 means that two 2We replicate many experiments for non-adversarial but random perturbations within the -sized ball to directly compare model geometry measured on the same stimuli . Results are similar and provided in SM 6. representations are completely overlapped , and 0 means that they are completely separable and thus not overlapped . We use the scikit-learn SVM implementation with a train/test split of 80/20 . 3.3 Adversarial attacks . For performing adversarial attacks , we use either the single-step fast gradient sign method ( FGSM ) with a random starting location or multi-step projected gradient descent ( PGD ) . Unless otherwise specified , we use L∞ norm constrained attacks . All attacks are untargeted and performed on a model-by-model basis . When the goal is to evaluate adversarial accuracy in newly created networks with stochastic internal representations , we use ensemble-PGD , where each step is in the mean direction of k samples of the noisy gradient to ensure a useful signal [ 36 ] . For more details , see SM 2 . 4 Manifold analysis of robustness in ImageNet-trained networks . We first investigate the properties of ImageNet-trained models , aiming to compare the geometric signatures of standard , adversarially trained , and stochastic networks . 4.1 Models and Dataset . We use images sampled from the ImageNet [ 37 ] test set . For class manifold analysis , the clean stimulus set consists of 50 classes , with each class containing 50 unique exemplar images for a total of 2500 unique images . For exemplar manifold analysis , 100 unique images are sampled from the ImageNet test set and each is perturbed with FGSM from a random starting location 50 times for 5000 unique images . We analyzed three publicly available ImageNet models , including ResNet50 with standard training , ResNet50 adversarially trained with an L∞ = 4/255 penalty ( ATResNet50 ) , and VOneResNet50 , a ResNet50-based model with the first conv-relu-maxpool layers replaced by a linear-nonlinear-Poisson model front-end ( called the VOneBlock ) with Gabor filters and noise fitted to primate neuronal data . In addition , we analyze a novel variant of VOneResNet50 ( GVOneResNet50 ) with additive Gaussian noise scaled to match the mean over all units in the output of the VOneBlock in response to a set of reference stimuli . GVOneResNet50 performs similarly to the original VOneResNet50 in clean and adversarial conditions ( see SM 3.3 for more details ) . While the first two models are deterministic , VOneResNet50 and GVOneResNet50 have stochastic representations at the output of the VOneBlock . For additional details see SM 3 . 4.2 MFTMA reveals unique robustness strategies for VOneNet and adversarial training . Our analysis begins by observing the effects of norm-bounded , gradient-based adversarial attacks on the class manifold geometry of representations in ResNet50 , ATResNet50 , VOneResNet50 , and GVOneResNet50 . For all experiments , images perturbed with PGD L∞ constraints of ∈ [ 0 , 1/1020 , 1/255 ] are shown to a model and intermediate representations are extracted , randomly projected to 5000 features [ 38 , 39 ] , and analyzed with MFTMA ( Figure 2A ) . Much like previously reported with MFTMA [ 33 ] , models develop separable representations in later layers , with capacity peaking in the final layer before classification ( see SM 3.5 ) . First we focus on this penultimate layer , where class manifold representations have become linearly separable and where capacity thus peaks . For the full layer-wise results , see SM 3.5 . We empirically confirm that capacity of the penultimate layer is predictive of the top-1 accuracy across models and attack strengths ( Figure 2B ) , indicating that MFTMA is sensitive to the attacks we are investigating . MFTMA exposes how capacity ( Figure 2C ) and class manifold width ( Figure 2D ) vary in the four models as a function of perturbation strength . Adversarial attacks cause the width of class manifolds to grow most rapidly in ResNet50 , to a lesser degree in VOneNets , and least of all in ATResNet50 . Next , to investigate how the models represent adversarial perturbations around individual images , we introduce another MFTMA-based approach . Instead of class manifolds , we consider exemplar manifolds consisting of points sampled using FGSM with L∞ constraints of ∈ [ 0 , 1/1020 , 1/255 , 4/255 , 8/255 ] from a random starting location in the epsilon-sized ball around 100 exemplar images , with the goal of tracing how these -sized ball exemplar manifolds develop as they travel through subsequent layers of our networks of interest ( Figure 3A ) . Restricting our analysis to the layer-wise trajectory of = 8/255 sized ball exemplar manifolds ( Figure 3B ) , the most salient feature is how distinct the trajectories are for the ATResNet50 and VOneNets . Our results indicate the defense mechanism induced through adversarial training generally stabilizes the width of the -sized ball around an exemplar as it propagates through the network , effectively mapping small perturbations around the clean image to small regions in later layers of the network . By contrast , the width of the -sized ball increases in VOneNets and the standard ResNet50 . In fact , at the VOneBlock output , the exemplar manifolds become highly entangled , suggesting different mechanisms of robustness for adversarially trained and stochastic models . How then do stochastic responses improve robustness ? Figure 3C demonstrates that when the capacity of the adversarially induced exemplar manifolds is normalized by the capacity for representations of the same unperturbed images ( clean exemplar manifolds ) 3 , VOneNets are far more stable than the standard ResNet50 network . At lower attack strengths where VOneNets ’ accuracies are minimally degraded , their normalized capacity remains stable , indicating that they represent the perturbed images with approximately the same capacity as clean images . In other words , the adversarial perturbations do not push the representation beyond that for clean images . In SM 3.7 , we extend this analysis to a variety of additional networks including two networks trained adversarially with different norms , VOneResNet50 with no stochasticity during training or inference , and ResNet50 with a stochastic activation layer mirroring that in VOneResNet50 ( see Dapello , Marques et al . [ 14 ] 3Clean exemplar manifolds for non-stochastic models have the maximum capacity of 2 [ 20 ] . for network details . ) Our trends hold up in each of these cases , with adversarially trained networks stabilizing the -sized ball , and stochastic networks expanding it at the point of the stochastic layer , but using similar capacity relative to clean images for small perturbations . In all experiments , we find similar geometric signatures for the Gaussian and Poisson VOneNets . For simplicity in the rest of the paper we focus on stochastic representations with Gaussian noise , but include an analysis of a Poisson noise network trained on CIFAR-10 in SM 5.4 .

Summary: It is well known that a deep learning model is fragile to adversarial perturbation, whereas a biological system is not. This study investigate the geometry of internal representations (especially, in the penultimate later) of a DL model, that trained using adversarial examples, and that with biologically-inspired component. The basic approach is based on what the authors call MFTMA.