Manipulating dropout reveals an optimal balance of efficiency and robustness in biological and machine visual systems
Authors: Jacob S. Prince
Gabriel Fajardo
George A. Alvarez
Talia Konkle
Venue: ICLR 2024 poster (2024)
Keywords: Efficient coding
object representation
dropout
robustness
human fMRI
occipitotemporal cortex
cognitive neuroscience
distributed coding
Primary area: applications to neuroscience & cognitive science

Abstract: According to the efficient coding hypothesis, neural populations encode information optimally when representations are high-dimensional and uncorrelated. However, such codes may carry a cost in terms of generalization and robustness. Past empirical studies of early visual cortex (V1) in rodents have suggested that this tradeoff indeed constrains sensory representations. However, it remains unclear whether these insights generalize across the hierarchy of the human visual system, and particularly to object representations in high-level occipitotemporal cortex (OTC). To gain new empirical clarity, here we develop a family of object recognition models with parametrically varying dropout proportion $p$, which induces systematically varying dimensionality of internal responses (while controlling all other inductive biases). We find that increasing dropout produces an increasingly smooth, low-dimensional representational space. Optimal robustness to lesioning is observed at around 70% dropout, after which both accuracy and robustness decline. Representational comparison to large-scale 7T fMRI data from occipitotemporal cortex in the Natural Scenes Dataset reveals that this optimal degree of dropout is also associated with maximal emergent neural predictivity. Finally, using new techniques for achieving denoised estimates of the eigenspectrum of human fMRI responses, we compare the rate of eigenspectrum decay between model and brain feature spaces. We observe that the match between model and brain representations is associated with a common balance between efficiency and robustness in the representational space. These results suggest that varying dropout may reveal an optimal point of balance between the efficiency of high-dimensional codes and the robustness of low dimensional codes in hierarchical vision systems.

Published as a conference paper at ICLR 2024
MANIPULATING DROPOUT REVEALS AN OPTIMAL BALANCE OF EFFICIENCY AND ROBUSTNESS IN BIO- LOGICAL AND MACHINE VISUAL SYSTEMS
Jacob Prince'; Gabriel Fajardo George A. Alvarez 1 & Talia Konkle Harvard University, Department of Psychology Boston College, Department of Psychology and Neuroscience
ABSTRACT
According to the efficient coding hypothesis, neural populations encode informa- tion optimally when representations are high-dimensional and uncorrelated. How- ever, such codes may carry cost in terms of generalization and robustness. Past empirical studies of early visual cortex (V1) in rodents have suggested that this tradeoff indeed constrains sensory representations. However, it remains unclear whether these insights generalize across the hierarchy of the human visual system, and particularly to object representations in high-level occipitotemporal cortex (OTC). To gain new empirical clarity, here we develop a family of object recog- nition models with parametrically varying dropout proportion (p), which induces systematically varying dimensionality of internal responses (while controlling all other inductive biases). We find that increasing dropout produces an increasingly smooth, low- oww-imensional representational space. Optimal robustness to lesion- ing is observed at around 70% dropout, after which both accuracy and robustness decline. Representational comparison to large-scale 7T fMRI data from occipi- totemporal cortex in the Natural Scenes Dataset reveals that this optimal degree of dropout is also associated with maximal emergent neural predictivity. Finally, using new techniques for achieving denoised estimates of the eigenspectrum of hu- man fMRI responses, we compare the rate of eigenspectrum decay between model and brain feature spaces. We observe that the match between model and brain representations associated with common balance between efficiency and ro- bustness in the representational space. These results suggest that varying dropout may reveal an optimal point of balance between the efficiency of high-ddmensionall codes and the robustness of low dimensional codes in hierarchical vision systems.
INTRODUCTION
Biological and artificial visual systems must convert rich high- high-dimensional input data into a more compact, meaningful representational format to support downstream tasks. A core tension in this process relates to the nature of the sensory coding scheme. On one hand, the efficient coding hypoth- esis posits that visual systems should employ sparse, high-dimensional representations capture the statistical regularities of the natural world with minimal redundancy (Barlow 1959; Olshausen & Field 1996]. Simoncelli & Olshausen 2001 Olshausen & Field 2004; Ganguli & Sompolinsky/ 2012. Alternativly,, distributed coding schemes implement a broader distribution of information
the ex- pense of efficiency (Shadlen & Newsome) 1998 Reich et al. [2001] Haxby et al.) [200]] Cunningham & Yu 2014). It remains unclear how the human visual system navigates this trade-off to function optimally in naturalistic contexts. To address this gap, here we directly instantiate this theoretical tension in a family of object recognition neural network models, which enables us to study where the human visual system falls within the sparse-to tonisss-t-distbbttee coding continuum.
across neurons or computational units, thereby enhancing robustness and generalizability
Sparse coding schemes aim to maximize representational capacity by employing efficient informa- tion encoding with minimal redundancy (Barlow 1959). Prior work suggests the benefit of sparse
Corresponding author: jprince harvardd.eddu

---

Published as a conference paper at ICLR 2024
Vinje & Gallant 2000) and for tasks like few-shot learning (Sorscher et al. 2022). ). While useful for representing the complexity of sensory inputs with fine-grained detail, such codes may be vulnera- ble to noise and perturbations, potentially yielding unstable population responses given small input changes (Stringer al.] 2019). Furthermore, extracting meaningful information from sparse rep- resentations may impose a significant readout challenge for downstream neurons or computational units, Precisely because the information is localized, readout depends on knowing "where to look" within the large, lige-simensional space to extract the relevant signals.
codes as basis feature set in early stages of visual processing Olshausen & Field 1997
In contrast. distributed coding schemes involve a more expansive set of units in representing a given stimulus, thereby introducing redundancy into the population code (Reich et al. 2001). Distributed codes reside in lower-dimensional spaces due to the inherent redundancy among the neurons' tuning, which leads to correlated activity that can be captured with fewer unique dimensions (Cunningham & Yu, 2014). Empirical studies suggest that the visual hierarchy of primate inferotemporal cor- tex (IT) may pressure toward low-dimensional representations of object information (Lehky et al.) 2014 DiCarlo & Cox 2007). Others have theorized that lower-dimensional codes offer benefits of adversarial robustness (Amsaleg et al., 2020 Kong al. 2022) and improved separability of ob- ject class manifolds (Cohen et al., 2020 Sorscher et al. 2022). While such coding strategies may afford these advantages, increased redundancy may eventually begin to impose a representational cost, constraining the system's 's capacity to encode a diverse range of important features.
To study these influences on sensory representation, related work has attempted to quantify the dimensionality of visual population responses within both biological and artificial systems. For instance, investigation of large-scale data from mouse primary visual cortex (VI) led to the proposal that particular dimensionalittess quantified as rates of eigenspectral decay, may reflect an optimal balance between high- and oow-dimensional coding schemes (Stringer et al., 2019). This finding challenges the longstanding view that V1 implements maximally sparse codes. Instead, it suggests a more complex framework for understanding visual coding strategies, one that incorporates a balance between high- and low-dimensional codes. Despite these advances, substantial gaps remain in our understanding of coding principles across the human visual hierarchy.
Deep neural networks (DNNs) now offer a powerful framework to operationalize and scrutinize this sparse-to-distributed coding continuum (Figure IA), Here, we do so by developing a class of models where single inductive bias dropout regularization (Hinton et al. 2012) is varied while keeping other inductive biases constant, isolating the impact of dropout on representation learning. Specifically, we train models with stochastic removal of neurons during each training iteration, varying the proportion of dropout across models (Figure IB). This regularization method is known to limit overfitting and enhance model generalization (Baldi & Sadowski 2013 Srivastaval
et al. 2014 Liu et al. 20023..
The logic here is that, with increasing proportions of dropout, models will be pressured against assigning high weights to individual connections, since those units might not be consistently avail- able. Thus, in object recognition models, information should consequently becomes more broadly distributed across the layer. As result, increasing dropout should result in smoother, lower- dimensional geometry of the population code. Our study first verifies these representational signa- tures empirically. Then, we leverage this form of controlled model variation to examine the effect of dropout on representational capacity, geometry, lesioning robustness, and emergent brain predic- tivity. We report three novel findings:
1. We confirm that "controlled model rearing" with increasing dropout p yields models with increasingly distributed codes and lower-dimensional eigenspectra.
2. Varying dropout preserves task performance while systematically altering representational geometry, revealing an optimal point of robustness to simulated model "lesions".
3. Model-to-brain comparison with high-resolution fMRI data reveals that the most robust model also shows the highest degree of emergent brain alignment.
2

---

Published as a conference paper at ICLR 2024
2 RESULTS
2.1 INCREASING DROPOUT REDUCES DIMENSIONALITY OF MODEL REPRESENTATIONS
To investigate the relationship between dropout regularization and the dimensionality of learned representations, we trained a family of models with controlled dropout regularization. The base encoder model was an AlexNet architecture (Krizhevsky et al. 2012) with dropout implemented at fully connected layers fc6 and fc7. We trained a family of 10 models, with dropout probabilities (p) varying from 0 to 0.9, incremented steps of 0.1 (Figure IB).
All models were trained on the ImageNet dataset for 1000-way object classification (Deng et al., 2009), using the standard cross-entropy loss function. Models were trained using stochastic gradient descent on three Nvidia A100 GPUs with 40 GB RAM. Efficient data-loading
achieved via the Fast-Forward Computer Vision (FFCV) library (Leclere et al.] 2023). We used a one-cycle learning rate schedule, linearly ramping from an initial learning rate set to 1 x 10-4 times the peak learning rate (0.05), which was reached at epoch 15, before linearly descending to 2 x 10-5 over the remainder of training. Image resolution started at 160, then increased to 192 on epoch 71 (implemented as a linear ramp between epochs 65 and 76, rounded to the nearest multiple of 32). Given that dropout was already implemented as a regularization method, no further weight decay was applied. All training runs used a batch size of 512 and momentum value of 0.9.
A
B
Sparse Coding
Distributed Coding
Controlad Model Rearing (N=10)
Tuning profiies
Bigomapootuum
Turnng profiles
Egponppccrum
Aotic
Resporte
6.1
High- demo1abnna
0.D
Jow- ommonaional
0.00
2.001
Stirulus K0
10 1,000
Stimuius value
100 1,000 PC
dropout variation p [0. 0.1, 0.2, 0.3,0.4, 0.4, 0.5, 0.6, 0.7, 0.8
dimension
Stringer et al 2019 Experimental Sctupp
D
Model Eigenspectra
Mouse VI
feacodingg ale
Dlgenspedrum
Egerspectrun
Spectral docay
m.28 0,800 mm
0.885 mm
10a
1.4
10-
C2 - 1,04
10
10-
107
3.0
10 10
10
10F
10
20"1
dimension
Log PC dimendion
Dropout
Figure 1: Controlled model rearing with varied dropout p yields variation between sparse and distributed coding schemes. (A) Schematic depiction of the tension between sparse, high- dimensional codes and distributed, low-dimensional codes. Images adapted from Stringer et al.] 2019 (B) Overview of controlled model rearing procedure. 10 models were trained with varying dropout p from 0 to 0.9, in increments of 0.1. (C) Experimental approach from Stringer et al., 2019. Calcium imaging was used to record the simultaneous population activity of thousands of neurons in mouse primary visual cortex (V1) in response to natural images. From these brain responses, the neural eigenspectrum was estimated for early visual cortex. The alpha (rate of spectral decay) was estimated with a linear fit over the log eigenspectrum. (D) For each of the dropout models, the log eigenspectrum of layer fc6 activations (measured at inference time, with dropout disabled) is shown (left plot), with the first 250 dimensions plotted. The corresponding alpha values are shown for each model (right plot).
First we assessed the dimensionality of learned representations over the set of 10 models. To do so, we computed the activation matrices over a sample of 1000 images from the ImageNet validation set focusing AlexNet layer fc6, which has been previously shown to have a high degree of brain alignment (Prince et al.) 2023, Bao et al., 2020). From these activations, we followed a similar approach to that of [Stringer et al. 2019 which quantifies the decay rate of the log- trenrssfffored eigenspectrum (Figure IC), as follows. Activation data were standardized to zero-mean and unit- variance. The PCA-derived eigenspectrum was then sorted in decreasing order, normalized by its
3

---

Published as a conference paper at ICLR 2024
sum, and truncated to the first 250 components for subsequent analysis. A power-law fit to this eigenspectrum was computed as:
log(A,) = log(t) 3
(I)
Optimal parameters were estimated via weighted least squares regression, which aims to emphasize the importance of the leading principal components by inversely weighting them according to their rank, thereby ensuring a more robust estimation of the decay rate a in the eigenspectrum:
b= -xxXxxx- - X Wy
Here, W is a diagonal weight matrix with w; = 1. y (oag(ragg)) and X contains - log(1) and a constant term. The parameter a, which is the first element of the resulting b, serves our metric for the rate of eigenspectrum decay, indicating the effective dimensionality of the feature space.
The resulting eigenspectra for the parameterized of dropout models shown in Figure 1D. These show systematic variation in the resulting shape of the eigenspecta, with corresponding vari- ation in the estimated alpha parameters. We observed systematic reduction in dimensionality of object representations with increasing dropout p. These results confirm that dropout proportion serves an effective knob for controlling the dimensionality of the learned representations.
2.2 VARYING DROPOUT ALTERS REPRESENTATIONAL. GEOMETRY WHILE PRESERVING TASK PERFORMANCE
Next, we examined the impact of dropout variation on both task performance and on the nature of the internal representations learned across the layers of these models. Model performance the ImageNet dataset, using the top-5 accuracy metric, is shown in Figure 2A. Despite the extensive range of dropout rates, all models exhibited surprisingly competent performance. Peak accuracy was observed for models trained with dropout of 0.5. (We note that the original AlexNet model included dropout p=0.5, likely based on hyperparameter sweep; our results converge with this setting). However, we also note that all models have performance within a 10% accuracy range. Thus, all models are capable of effectively learning the task hand, despite wide range of dropout probability.
2.2.1 REPRESENTATIONAL TRAJECTORY ANALYSIS
We next sought to understand the impact of dropout variation on the internal representations learned across the layers of these models. To do so, we used Representational Trajectory Analysis (Kall-) mayer et al. 2020) see also Mehrer et al. 2020). This method creates a visualization of the "repre- sentational trajectory" that different models take from the pixel space to their final representational format (Figure 1B). In this visualization, model layers with similar representational geometries are nearby in the plot, and model layers with more different geometries are further away.
To apply this analysis, for each layer L of every model, activations were extracted to image set I, consisting of 250 sample images from the validation set. This yielded an activation matrix A where each row corresponds to the activation pattern elicited by single image. Next, the Pearson dissimilarity for this layer was computed to form a representational dissimilarity matrix RDM,, a square matrix that captures the dissimilarity between activation patterns for all image pairs:
RDM =1 cov(Ali,,), An.))) (i))(
(3)
Next, for each RDM,, the vectorized lower triangular portions was extracted, and concatenated to construct the matrix M. Finally, the Pearson dissimilarity between all pairs of rows in M was computed, yielding a Deta-Dissimilarityy Matrix (MDM):
MDM(m, n) = cov(M :), (nn:)
(M(m,;))oo (M (n,:))
This MDM matrix reflects how dissimilar every model layer's 's representation is from every other model layer, across all models. To visualize the structure in this matrix intuitively, the MDM matrix
4

---

Published as a conference paper at ICLR 2024
A
ImagaNet Accuracy
Model Leetone
130
Floono
bocasiiiin accuuracy
TOp accurssyy
Imagathle Sart
Aesohat
20
Isoo
lson
Proportion
B
Roprosantational Trajectorios
D
Accuracy: mild lesion
Ascurnyy moderate Insion
Accurnny: severe losion
56 basainee acouraey
surput
MADIS dimanaion 2
2022
224 3 22222535
Dropoup
Dropout0
Dropoutp
dimension
30%
00% unio Resoned
Figure 2: Varying dropout preserves task performance while altering representational geome- try, revealing optimal point of robustness simulated lesioning. (A) Top-5 ImageNet accu- racy after 100 training epochs, across the dropout models. (B) Representational Trajectory Analysis (Kallmayer et al., 2020): showing the layer-to-layer representational dissimilarity structure of all 10 models projected into 2D space using multidimensional scaling (MDS), with traces connecting the input and output stages of each model.
(C) Model lesioning procedure. Percent of baseline top-5 ImageNet accuracy is plotted as function of the proportion of units randomly lesioned in layer fc6. Error bars reflect the standard deviation accuracy over the 10 iterations.
(D) Bar graphs summarize the observed outcomes for mild (20% layer units), moderate (50% layer units), and severe (80% layer units) lesions.
was input to 2D multidimensional scaling (MDS), yielding x-y coordinates for each model layer, which are shown 2D scatter plot. Layers from the same model are connected with line, visualizing the 'representational trajectory" from the input (pixel space) to output (fc8). In this space, the farther apart two model layers are, the more different their representational geometries.
Inspection of these trajectories in Figure 1B reveals systematic variations in representational ge- ometries learned across the layers of these models. The largest variation occurs predominantly in the fully connected layers, where dropout was explicitly applied. Interestingly, representational differences across these models also extended the earlier convolutional layers, underscoring the cascading impact of localized dropout across the hierarchical structure of the network.
2.3
OPTIMAL ROBUSTNESS TO LESIONS IS OBSERVED AT HIGH (BUT NOT EXTREME) LEVELS OF DROPOUT
We next sought understand the functional consequences of varying dropout. To do so, we applied simulated "lesions" to these models, drawing upon neuropsychological paradigms that assess the impact of damage object-responsive cortex (Barton et al., 2002 Moro et al. 2008; Kanwisher & Barton, 201T). Note that dropout regularization occurs during training but not during inference. In this analysis, we are effectively applying dropout during inference time, as a way of lesioning or damaging the model. Presumably, the models trained with the most dropout will be the most robust to lesions during inference.
For this analysis, lesions were applied to layer fc6 in each model during inference (see Appendix for a similar analysis of fc7). Randomized groups units were lesioned by setting their outputs to zero, where we varied the scope of the lesion between O to 95% of layer units, in steps of 10%. Top-5 ImageNet accuracy was subsequently re-evaluated. This lesioning protocol was performed iteratively 10 times, each with different random subset of units targeted. This approach aligns with prior work examining the robustness of neural networks to selective perturbations (Cheney et al. 2017 Casper al. 2021).
5

---

Published as a conference paper at ICLR 2024
When lesioning the model trained with dropout, monotonic decline in performance ob- served with increasing lesion sizes, as expected (Figure 2C-D). As the dropout rate increased, so did robustness, enabling substantial proportions of units in layer fc6 to be lesioned with minimal decrement in performance. This trend, however, reached an inflection point around p = 0.7, beyond which the models exhibited brittle behavior under large lesions. These findings validate that dropout can effectively mitigate the damaging effects of lesions, especially at mid-to-high levels of p. How- ever, the results show that it not the case that optimal robustness observed at the most extreme levels of dropout. And, it is not the case that this lesion-robust model (p = 0.7) is the same as the most accurate model where no dropout applied at inference (p 0.5).
Intriguingly, this analysis suggests the existence of an ideal dropout level that effectively balances the merits of both low- and high-dimensional codes within the DNN model (i.e., robustness and representational capacity). These in silico findings further hint that the human visual system may navigate a similar trade-off between sparsity and robustness. To address this possibility, we next examined how these models relate to representations measured in condition-rich fMRI study of human object-responsive cortex.
2.4
OPTIMAL ROBUSTNESS IS ASSOCIATED WITH MAXIMAL HUMAN BRAIN ALIGNMENT
The aim of our next analysis was to compare the emergent alignment between late-stage represen- tational geometries of our dropout models and the representational geometry measured in human high-level visual cortex. To do so, we leveraged fMRI recordings from the Natural Scenes Dataset (NSD, Figure 3A, (Alen et al. 2022). NSD contains measurements of over 70,000 unique stimuli from the Microsoft Common Objects in Context (COCO) dataset (Lin et al.) 2014) at high resolution (7T field strength, 1.6-s TR, 1.8mm3 voxel size). Here, we focused on the brain responses to the set of 515 COCO stimuli that all subjects viewed 3 times across scan sessions. All responses were estimated using a new, publicly available GLM toolbox (GLMsingle: Prince et al. 20222. which im- plements optimized denoising and regularization procedures to accurately measure changes in brain activity evoked by experimental stimuli.
We targeted our analysis to object-responsive cortex ("occipitotemporal cortex" in humans -T which is homologous to "Inferotemporal" IT cortex in non-human primates). The sector was defined following the procedures in Conwell et al. 2022] Specifically, within a broad visual system mask ("nsdgeneral" ROI), the selected voxels included those in the mid-to-high ventral and lateral ROIs ("streams" ROIs), supplemented by voxels from 1l category- ROIs with a t-contrast statistic I. The 8 NSD subjects yielded an average data dimensionality of 34,195 in their OTC sectors. Importantly, this sector encompasses high-level object-responsive cortex only, and excludes data from earlier visual regions (e.g. V1-V4). Thus, the scope of our analysis goes beyond replicating the work Stringee et al. 2019 which was limited to area V1 in mice.
To perform model-brain comparison, we used classical Representational Similarity Analysis (RSA; Kriegeskorte et al., 2008). schematized in Figure 3A-B. First, the brain activations to 515 images were transformed into a representational dissimilarity matrix (RDM), as in equation (3). Note that the representational geometry of this brain sector was highly reliable (Conwell et al. 2022),, making this brain region a strong mapping target for our dropout models. The same 515 images were shown to each model, and activations from the fe6 layer (dimensionality 4,096 units) were recorded, and converted into representational dissimilarity matrices (RDMs) following the same procedure. Finally, the lower triangular values of the model and brain RDMs were correlated, with Pearson r values plotted in Figure 3C. Note that this classical RSA procedure differs from more flexible, and more typical, model-ttobrrin linking procedures, which allow for feature re-weighting to best try to warp the model representational geometry to fit the brain geometry. Here, no re-weighting is allowed, revealing the degree of emergent alignment between model and brain geometries.
The results in Figure 3C show that across the models, all showed some degree of representational alignment with the brain. However, surprisingly, and in parallel with our earlier findings, layer fcó of the model trained with p = 0.7 dropout was not only most robust to lesioning, but also showed the highest degree of alignment with human brain representations. This finding adds further plausibility to the hypothesis that dropout manipulation instantiates a brain-relevant tradeoff between efficiency and robustness. Moreover, the apparent synergy between model robustness and emergent

---

Published as a conference paper at ICLR 2024
A
Naturai Scenes Dalasst; Alien et 2022
Model-brain PISA conveaations
Pearson
B
Layer 1oB units
RDM
Shmul
Dropout 0
D
coverience
coveriance
Signal
E
Human OIO elgenepectra
Rate of dacay pa
10
mainn abphe
p= aloha en 1.17
Log vulance
10
1133
10c
empincally measured
estimatied using taal repertions
decompose: "denaised PCA
0.4
Log dimension
Humans
Figure 3: Model-brain mapping reveals a link between lesion robustness and brain predictiv- ity, associated with similar rates of spectral decay. (A) Schematic overview of OTC data from the Natural Scenes Dataset (Allen et al. 2022). Highly reliable 7T fMRI responses to 515 Microsoft mapping via classical (unweighted) representational sim- ilarity analysis (cRSA). (B) Model layer fc6 features are extracted for the 515 test images and com- pared to brain RDMs using Pearson correlation.
COCO images are subject to brain-mode
(C) Summary of model-brain cRSA prediction levels for the 8 NSD) subjects, by dropout level. (D) Overview of the Generative Modeling of Signal and Noise approach (cvnlab) 2022). Signal covariance structure estimated by subtracting empir- ically measured noise covariance (estimated using trial repetitions) from the full data covariance structure. Brain region eigenspectra are then computed directly from the signal covariance matrix. (E) Human OTC eigenspectra for the NSD subjects (left), with alpha values summarized (gray bars, center) and compared to the dropout models (colored bars, right).
brain alignment raises an intriguing possibility: is this correspondence between neural systems also reflected similar rates of cigenspectrum decay?
2.5 EIGEnSPECTRUM OF UBJECT-RESPONEE CORTEX
To further probe the observed model-brain alignment, decay of the brain representations, in order to compare to the model eigenspectra. Critically, while the models are noiseless systems, the brain measures eigenspectrum directly would also reflect the noise structure, which poses thorny methodological challenge. For instance, if the noise present in the data were to exhibit high-dimensional struc- ture relative to the signal, then it could systematically bias naive eigenspectrum estimates (e.g. computed by applying PCA directly to the data) the downward direction. address this issue, employed novel technique known Generative Modeling of Signal and Noise (GSN, cvn- lab (022). This approach leverages available stimulus repetitions to distinguish between signal and noise covariance structure, thereby yielding more reliable, "denoised PCA" from which the brain's 's cigenspectra can be estimated.
computed the rate of eigenspectrum
not. Thus, simply calculating the brain's
2.5.1 GENERATIVE MODELING OF SIGNAL AND NOISE
For given ROI, the GSN method estimates multivariate Gaussian distributions characterizing the signal and the noise under the assumption that observed responses can be characterized as sums of samples from the signal and noise distributions. This approach is similar in spirit to the cross- validated principal component analysis (cvPCA) method developed by SSringer et al.) 2019 in that both techniques aim to separate signal-relater variance from trial-to-trial noise. However, cvPCA achieves this by computing the covariance between training and test presentations of identical

---

Published as a conference paper at ICLR 2024
stimulus ensemble, to isolate stimulus-related variance. In contrast, GSN utilizes stimulus repeti- tions to model both the signal and noise explicitly, which aims to provide a more comprehensive description of the empirical data. As such, GSN provides principled, unbiased method for extract- ing eigenspectra from noisy data.
To briefly summarize the approach: GSN assumes that for each condition, trial-to-trial variability (noise) adheres to a zero-mean multivariate Gaussian distribution, a property shared across condi- tions. Signal variability across conditions is also modeled as a multivariate Gaussian whose param- eters are expected to vary from the noise distribution. The model can be summarized as:
D X1 X2.
X1 ~ (Mtiinaal Erignal )
(5)
~ ~VO, Ennise)
Here, D denotes the observed multivariate response for a given trial, X1 and X2 represent the multivariate responses due to signal and noise, respectively. Parameters Msignal and Esignal specify the mean and covariance of the signal Gaussian, whereas Emoise is the covariance of the noise Gaussian. This modeling framework is generative, as it directly characterizes the data-generating processes.
The crux of the GSN approach lies in parameter estimation for both signal and noise distributions, to be fully described and validated in a forthcoming manuscript. Here we provide a brief, non- exhaustive overview of the GSN algorithm, which capitalizes on the property that the sum of samples from two Gaussian distributions is also Gaussian-distriuuted First, to estimate the noise covariance, we start with a data matrix X (n fMRI voxels xe conditions xt trials). To estimate the noise structure, we calculate the covariance of voxel responses over stimulus repetitions separately for each condition, and then average the covariance across conditions:
D Enoise
cov(X;)
Next, we estimate the data covariance of voxel responses after averaging over stimulus repetitions:
datajs] = cov(X)
Finally, to infer the signal covariance, we subtract the estimated noise covariance from the estimated data covariance (Figure 3D). Due to averaging across trials, the variance of the noise is reduced by a factor of t in the trial-averaged data:
Esigual Edat((] (n]
noise
The GSN procedure next implements check for whether the estimated covariance matrices valid and if not, computes the nearest (Frobenius matrix norm) symmetric positive semi-definite matrix to given square matrix (Higham 1988] see eeellablab] 2022).
Once the signal covariance structure, Esignal, was estimated for OTC data from each NSD subject, directly computed its eigenspectrum using matrix decomposition. Figure 3E shows the eigen- spectrum of object-responsive cortex seperately for each of the 8 participants. As was previously observed in mouse V1 (Stringer et al. (2019), we see a characteristic linear fall-off on the log-log plot of the eigenvalues. The a decay was computed for each participant following the same procedure applied the dropout models' eigenspectra, with mean a 1.13 across participants (standard deviation a = 0.05).
For comparison to the dropout models, we related these estimated values to those arising from model layer fc6 across the model family (Figure 3E, see also Appendix sec. [A.)). This analysis revealed that the model trained with p 0.7 dropout, which was most robust to lesions and brain predictive, also demonstrated rate of eigenspectrum decay (a value) that most closely matched that of human OTC. This observation adds a further layer of plausibility to the hypothesis that the advantages of dropout in model robustness and efficiency are not just computationally useful but also neurobiologically relevant. One implication of these results is that the brain't 's representation may not be overly dependent single units specific read-out connections. Further, the close match in a values suggests that the alignment between model and brain representations may be driven by similar constraints efficiency and robustness within their respective feature spaces.
8

---

Published as a conference paper at ICLR 2024
3 LIMITATIONS
Here we focused on dropout regularization as a way to operationalize a set of representations that vary along the sparse-to-distribuled axis. However, it is possible that the percentage of dropout may reflect a pressure to be more-or-less distributed (rather than pushing towards sparsity directly). Alter- native regularization techniques such as L1 and L2 penalties on the weights could also be explored. It is an open empirical question whether these regularization techniques have distinct consequences on the geometry of the representations, and their emergent robustness and brain correspondence.
Additionally, here we focused on the dimensionality and structure of the representation of block 6 of an AlexNet architecture, and object-responsive cortex within the human visual system. Given the correspondences in the emergent robustness, alignment, and eigenspectra, it will be informative to understand whether these spectral signatures of the representational space predict emergent brain alignment across other model architectures, and across other stages of the visual hierarchy (Canatar et al. 2023 Conwell et al. 2022 Elmoznino & Bonner 2022 Sorscher et al. 2022 Appendix|A.4
4 CONCLUSION
In this work, we have instantiated the theoretical tension between sparse and distributed coding schemes with controlled regularization over set of deep neural network (DNN) models trained on object recognition. According to the efficient coding hypothesis, sparse, high-dimensional represen- tations offer optimal information encoding the expense of robustness (Olshausen & Field 2004).
Conversely, more distributed (ower-dimensional) coding schemes provide enhanced robustness by introducing representational redundancy, at some cost information capacity (Stringer al. 2019 Kong et al. 2022). Our approach involved "controlled rearing" of a set of models with varying dropout proportion (p) during training. Three novel findings emerged: (1) dropout variation system- atically altered the dimensionality of model representations, operationalizing a continuum of more sparse to more distributed codes; (2) varying dropout influenced the learned representational ge- ometries of these models, while generally preserving task performance, revealing an optimal point of robustness during simulated lesions at inference; and, (3) the model with optimal robustness to lesions was also associated with maximal emergent brain alignment, and comparable rates of spec- tral decay. Overall, these results suggest that varying dropout reveals an optimal point of balance between the efficiency of nigh-dimensional codes and the robustness of low dimensional codes in biological and artificial vision systems.
It is open question what the parallel is between dropout regularization during model training and underlying neurobiological mechanisms. On one hand, dropout has been likened to synaptic fai failures, and variable neurotransmitter release probabilities in neural systems (Shadlen & Newsome 1998 McKee et al. (002I). This is a rather direct mechanistic parallel (though, if so, this factor should be operating both training and inference). However, another possibility is that the brain arrives at this representational format and dimensionality though completely different mechanisms, e.g. by interfacing with associative memory systems, which could pressure both for redundancy to aid gen- eralization, and, for sufficient sparsity to avoid memory "collisions" during readout (Olshausen & Field. 2004, Palm 2013). On this account, dropout mechanisms in models may have no relationship to the underlying biological operations, and still help promote brain-like formats by proxy, through their regularization of the learning process. Understanding how and why neurobiological systems may implement such forms of regularization is an important area for further investigation.
Broadly, the current work expands a growing body of research on the spectral properties of repre- sentations in artificial and biological visual systems. For instance, Canatar al. 2023 have recently proposed a method for analytically relating the brain predictivity of a DNN model to its spectral characteristics, and the orientation of its eigenvectors. In related work, Khosla et al. 2023 have introduced a framework for identifying "privileged axes" in brain and DNN representations, sug- gesting that such axes function to promote increased sparsity and efficiency of the neural code. And, Sorscher et al. 22022havv identified striking mismatches between the geometry and dimensionality of manifolds in the primate visual pathway and of trained DNNs, in the context of few-shot learn- ing paradigms. Collectively, these studies reflect necessary progress beyond the binary debate of sparse versus distributed codes, and toward more nuanced theories of information coding sensory systems, which can be tested empirically through controlled "rearing" of neural network models.

---

Published as a conference paper at ICLR 2024
REFERENCES
Emily J Allen, Ghislain vee,, Yihan Wu, Jesse I. Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 16-126, 2022.
Laurent Amsaleg, James Bailey, Amélie Barbe, Sarah M Erfani, Teddy Furon, Michael E Houle, Miloš Radovanović, and Xuan Vinh Nguyen. High intrinsic dimensionality facilitates adversarial attack: Theoretical evidence. IEEE Transactions on Information Forensics and Security, 16:854- 865, 2020.
Pierre Baldi and Peter J Sadowski. Understanding dropout. Advances in neural information pro- cessing systems, 26, 2013.
Pinglei Bao, Liang She, Mason McGill, and Doris Y Tsao. A map of object space in primate inferotemporal cortex. Nature, 583(7814):103- 108, 2020.
Horace Barlow. Sensory mechanisms, the reduction redundancy, and intelligence. The mecha- nisation of thought processes, 10:535-539 1959.
Jason JS Barton, Daniel Z Press, Julian P Keenan, and Margaret O' O'Connor. Lesions of the fusiform face area impair perception of facial configuration in prosopagnosia. Neurology, 58(1):71-78, 2002.
Abdulkadir Canatar, Jenelle Feather, Albert Wakhloo, and Yeon Chung. A spectral theory of neural prediction and alignment. arXiv preprint urXiv:2309. 2ww:2239../22.1 2023.
Stephen Casper, Xavier Boix, Vanessa D' Amario, Ling Guo, Martin Schrimpf, Kasper Vinken, and Gabriel Kreiman. Frivolous units: Wider networks are not really that wide. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6921-6929, 2021.
Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the robustness of convolutional neural networks to internal architecture and weight perturbations. arXiv preprint arXiv: 17X3::1703008255: 2017.
Uri Cohen, Sue Chung, Daniel Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation brains and machines? BioRxiv, pp. 2022-03, 03, 2022.
John P Cunningham and Byron M Yu. Dimensionality reduction for large-scale neural recordings. Nature neuroscience, (7011))500-1159 2014.
cvnlab. GSN: Generative modeling of signal and noise. 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 11(8)33333411 2007.
Eric Elmoznino and Michael F Bonner. High-peefforing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv, pp. 2022-07, 2022.
Surya Ganguli and Haim Sompolinsky. Compressed sensing, sparsity, and dimensionality in neu- ronal information processing and data analysis. Annual review of neuroscience, 35:485-508, 2012.
James V Haxby, M Ida Gobbini, Maura L Furey, Alumit Ishai, Jennifer L Schouten, and Pietro Pietrini. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293 2430,, 2001.
10

---

Published as a conference paper at ICLR 2024
Nicholas J Higham. Computing a nearest symmetric positive semidefinite matrix. Linear algebra and its applications, 103: 10:-1881 1988.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 2012.
Aylin Kallmayer, Jacob Prince, and Talia Konkle. Comparing representations that support object, scene, and face recognition using representational trajectory analysis. Journal of Vision, 20(1 I): 861-861, 2020.
Nancy Kanwisher and Jason JS Barton. The functional architecture of the face system: Integrating evidence from fmri and patient studies. The Oxford handbook of face perception, pp. 111-129, 2011.
Meenakshi Khosla, Josh McDermott, and Nancy Kanwisher. Privileged representational axes in biological and artificial neural networks. In 2023 Conference on Cognitive Computational Neu- roscience. Cognitive Computational Neuroscience, 2023, doi: 10.32470/ccn.20222 262..166-00 URL
https::/doi.... org/10. 32470/ ccn 002......66000
Nathan CL. Kong, Eshed Margalit, Justin I. Gardner, and Anthony M Norcia. Increasing neural network robustness improves match macaque eigenspectrum, spatial frequency preference and predictivity. PLOS Computational Biology, 18(1 18(1):e1009733, 2022.
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, pp. 4, 2008.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV: Accelerating training by removing data bottlenecks. In Computer Vision and Puttern Recognition (CVPR), 2023. https: tttpb//ggttt...tt//ege/e.eb... thub. com/ wibffo/fffee
Sidney R Lehky, Roozbeh Kiani, Hossein Esteky, and Keiji Tanaka. Dimensionality of object rep- resentations in monkey inferotemporal cortex. Neural computation, 26(10):2135-2162, 2014.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755, Springer, 2014.
Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell. Dropout reduces underfit- ting, arXiv preprint 203:0:2303.015000 2023.
Kevin I. McKee, Ian C Crandell, Rishidev Chaudhuri, and Randall C O'Reilly. Locally learned synaptic dropout for complete bayesian inference. arXiv preprint arXiv:2111.09780 2021.
Johannes Mehrer, Courtney J Spoerer, Nikolaus Kriegeskorte, and Tim C Kietzmann. Individual differences among deep neural network models. Nature communications, 11(1):5725, 2020.
Valentina Moro, Cosimo Urgesi, Simone Pernigo, Paola Lanteri, Mariella Pazzaglia, and Salva- tore Maria Aglioti. The neural basis of body form and body action agnosia. Neuron, 60(2): 236-246, 2008
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (8166)3))607-669 1996.
Bruno A Olshausen and David J Field. Sparse coding with overcomplete basis set: A strategy employed by vI? Vision Research, 37(23):331133325 1997.
Bruno Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neuro- biology, 14(4):4814487 2004.
11

---

Published as a conference paper at ICLR 2024
Günther Palm. Neural associative memories and sparse coding. Neural Networks, 37:165-171 2013.
Jacob Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using gimsingle. Elife, 11:77999, 2022.
Jacob S Prince, George A Alvarez, and Talia Konkle. A contrastive coding account of category selectivity in the ventral visual stream. bioRxiv, pp. 2023-08 2023.
Daniel S Reich, Ferenc Mechler, and Jonathan D Victor. Independent and redundant information in nearby cortical neurons. Science, 294(5551):2566 2568, 2001.
Michael N Shadlen and William T Newsome. The variable discharge of cortical neurons: impli- cations for connectivity, computation, and information coding. Journal of neuroscience, 18(10): 3870-3896, 1998.
Eero Simoncelli and Bruno Olshausen. Natural image statisties and neural representation. Annual review of neuroscience, 24()):1193-1216, 2001.
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry under- lies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43): e2200800119, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-115) 19589 2014.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth Harris. High-dimensional geometry of population responses in visual cortex. Nature, 571(7765):361- 365, 2019.
William Vinje and Jack Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, 287(5456):123-12776 2000.
ACKNOWLEDGMENTS
We thank all members of Harvard Vision Lab for helpful conversations throughout this project and for feedback on the manuscript. This work was funded by: Natural Science Foundation CAREER Grant # 19424438 (TK); National Defense Science and Engineering Graduate Fellowship (JSP).
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 QUANTIFYING ACTIVATION SPARSITY
We examine both the lifetime and population sparsity of layer fe6 activations across the dropout models. Lifetime sparsity refers to the sparsity observed in single neuron's 's activity over multiple inputs. The lifetime sparsity for a unit u over a dataset of images is calculated as follows:
Lifetime Sparsity,
Σ KE1
((=..=0)
[T]
where [4] represents the number of images, Lu,i denotes the activation of unit u for image and ((z... =0) is the indicator function that is if the unit is inactive (the activation is less than or equal 0), and otherwise.
Population sparsity describes the proportion of inactive neurons across the layer's 's response pattern for single input. The population sparsity for image over the of units is given by:
Population Sparsity,
X(
(10)
U
NEU
where [U] is the total number of units, and Fa,i is defined before.
These metrics are shown in Figure 4, revealing that both lifetime and population sparsity decrease as dropout p increases from 0 to 0.9, These results are consistent with the observation that the effective dimensionality of layer activations tends to decreases as dropout increases.
Lifetime and population sparsity by dropout proportion (fc6)
1.0
Lifetime Sparsity Population Sparsity
0.8
Sparsity (Proprtion inadtive units)
0.4
0.2
0.0
0.0
0.1
0.2
0.3
0.6
0.7
0.8
Dropout proportion
Figure 4: Analysis of sparsity. Activation sparsity levels in fc6 are reported for both lifetime and population sparsity metrics, plotted for models with increasing dropout along the x-axis. Each dot an images, from the set 1000 probe images (1 from each category).
13

---

Published as a conference paper at ICLR 2024
A.2 EIGENSPECTRA COMPARISON WITH DIFFERENT PROBE IMAGE SETS
In the main text, human OTC eigenspectra were assessed with 515 COCO images from the Natural Scenes Dataset, while the model eigenspectra assessed with sample of the validation set of ImageNet (in-distribution). Here we also assessed whether the model eigenspectra would be depen- dent on the choice of probe We directly compare fc6 eigenspectra computed over the ImageNet validation stimuli with those computed over the 515 COCO stimuli from the fMRI analysis. The results are shown in Figure and show very similar outcomes across these two image datasets (average difference between across layers = 0.02).
ImageNet probe set (n=1000 imgs)
COCO probe set (n=515 imgs)
10a
Log variance
10a
10a
100
022 4aro 6.9
00 40%
10
2-022-0ha 0239 1.02
4bme
0-055 upts 1.55
10
p- -8m 1.00
003a -100
10-1
p=00. upta 1:31
p 6.0 abpu
3.7
10
10a
102
100
10
102
Log PC dimension
Log PC dimension
Figure 5: Comparing ImageNet and COCO probe stimuli. Layer fc6 eigenspectra and corre- sponding alpha estimates are plotted for model responses to probe set of 1000 ImageNet valida- tion stimuli (left) and a probe set of the 515 Microsoft COCO stimuli that were used for the fMRI analyses (right).
14

---

Published as a conference paper at ICLR 2024
A.3 ANALYSES OF LAYER FC7
A parallel set of post-hoc were analyses conducted on layer fc7, are shown in Figure 6. We find the model with p 0.7 dropout again had the highest emergent brain alignment.
However, we note that this layer was generally less strongly correlated with the brain sector than layer fc6. Further, layer fc7 also showed less convergence across other signatures. For example, the cigenspectra of the fc7 layer of this model was generally steeper, meaning that the fc7 layer of model with less drop-out best matched the slope of OTC eigenspectra (p 0.3). Further, lesions to layer fc7 at inferences did not show u-shape curve, and instead showed more robustness with increasing dropout proportion.
A
Eigenspectra (fc7)
B
Robustness lesioning (fc7)
% basoline accuracy
Log veriance
107
80-
mild
ledion
10
moderate leslon
40-
10
11
23-
severe Iol01
100
10
10a
0.0
0.1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Log dimenelon
Proportion units lesioned
C
RSA correlationss (fc7)
RSA correlations (fc8- referenca))
0.5
0.4
Pearson
0.3
Pearson
8.2
0.1
0.3
0.0
011
03
Dropout
Dropout p
Figure 6: Analysis of Layer fc7. (A) fc7 eigenspectra are plotted with corresponding alpha esti- mates over the same probe set of 1000 ImageNet validation stimuli used for previous analyses. (B) Lesioning outcomes. Percent of each model's baseline top-5 ImageNet accuracy is expressed as function proportion fe7 units randomly lesioned at inference.
(C) RSA correlations for layer fc7 shown, with dots reflecting the 8 NSD subjects. (D) As point of reference, RSA correlations for layer fe6 are shown.
There are several implications of these results. First, the features in fc7 are less well aligned to the OTC brain sector data than fe6. Second, these results also demonstrate that simple measure of effective dimensionality (the slope of the eigenspectrum) is not itself sufficient metric to predict emergent brain alignment. Further, seems that lesions closer to the output stage show a more direct relationship between dropout proportion during training and lesioning during inference. Together these results highlight that eigenspectra slope, lesion robustness, and emergent brain alignment do not necessarily all hang together across all layers, providing room for future work to gain deeper theoretical understanding of the relationship among these constructs (Canatar et al. (003)).
15

---

Published as a conference paper at ICLR 2024
A.4 REPLICATING ANALYSES USING HUMAN VI DATA
We perform initial analyses to extend our paradigm from occipitotemporal cortex (OTC) to human V1. Specifically, analyze data from the ventral aspect of V1 across the 8 NSD subjects (mean dimensionality = 2,650). Data processing and model-brain comparison occur in the same manner as described for comparisons with the OTC sector.
First, layer-wise analyses reveal that the 'maxpooll' stage of the dropout models is consistently the most predictive of human VI activity (Figure 7A). Notably, the overall alignment of these models with V1 is lower compared to the observed fe6-OTC prediction levels. We also observe that varying amounts of dropout applied at the fc6 and fc7 layers does not significantly alter the representational characteristics in 'maxpooll', shown in both representational trajectory analysis (Figure 2B) and in the effective dimensionality and emergent brain alignment assessments (Figures 7B and 7C, respectively).
These findings suggest that manipulating dropout at earlier stages in the network, possibly within the first block, might be required in order to extend the current paradigm to study human VI repre- sentations. Our initial findings indicate that V1 representations may be of lower dimensionality than those in the OTC (Figure 7D), providing starting point for further investigation into the eigenspec- tra across different areas of the human visual system.
A
B
RSA corrolations with human V1
Layer maxpool1 summary
0.5
0275
0.53
0.4
Subject mean
Pearson
0283
0.15
0.2
0.1
0323
0.03
0991
reut.
H82 rerpod
0.0
0.1
0.2
0.3
0.5
Layer
Dropout p
D
Layer maxpool1 eigenspectra
Humen V1 eigenspectra
Log varianca
Log variance
10a
54501 VI 31000 1.45
V2 ub-
10a
54205 V2 aphs 1.587
wa2/23 V2 agd
540077 V1 31h0 3.445
a22008 V2.
agpha 2.42
10
10:
10a
12P
10'
107
Log PC dimension
Log PC dimension
Figure 7: Analysis of human V1 data. (A) RSA correlations are summarized over the 8 NSD subjects for layers convl through conv3. Lines reflect the mean over subjects, with shaded regions reflecting the SEM. The same 515 test stimuli are used as in the primary analyses. (B) Complete RSA correlations for layer maxpooll are shown, with dots reflecting the 8 subjects. (C) Model layer maxpooll cigenspectra are plotted in log-log space, with alpha values reflecting the slope of the linear fit to the log eigenspectrum. Eigenspectra computed over probe set of 1000 ImageNet validation stimuli. (D) Human V1 eigenspectra estimated using the GSN technique. All analysis procedures identical to those previously reported for OTC.
16


Reviews:
[{"review_id": "tV8ItZlZYF", "reviewer": "Reviewer_wjr3", "summary": "The paper use dropout as an approach to control the sparsity of neural representation in deep neural network (AlexNet), and evaluated the robustness of the resulting networks to lesions to one of the layers. The dropout rate of 0.7 was found to be most robust to lesion at inference time. Coincidently, at this level of dropout, the geometry of neural representation in the same layer also shows the highest correlation to the neural representational geometry to a set of COCO dataset images, in occipitaltemporal cortex of human. Further investigating the rate of decay in the eigenspectrum of the brain region and those of DNNs with different dropout rates, the decay rate of the model with 0.7 dropout rate also matches the decay rate of the human brain. Together, the result appears to suggest that, if dropout rate can be considered as a valid way of controlling for the sparsity of neural encoding, then perhaps the brain chooses its level of sparsity at a level most robust to lesioning of \"neurons\".", "questions": "Although the GSM is not a focus in this paper, I think in principle it is not guaranteed that the subtraction of the empirical noise covariance matrix from data covariance matrix results in a positive definite matrix. In other words, the eigenvalues can be negative. So I am worried that this approach is not generalizable if any readers want to test the hypothesis of this paper on other neural networks.\n\nAlthough AlexNet only applied dropout to the two layers investigated here, in principle it can be applied to any layer. One natural question is how dropout at earlier layers influence the coding scheme at the layer of investigation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The result shows a high consistency among three results: the dropout rate yielding the highest robustness of the DNN against high-level lesion, the dropout that yields neural representation best matching that of the object-recognition related brain region, and the the dropout rate yielding the same decay rate in the eigenspectrum of the covariance structure of fMRI spatial patterns across images.\n\nIt used a novel approach to separate the spatial covariance structure due to potentially noise and that due to signal in fMRI activity", "weaknesses": "Although the decay rate of eigenspectrum is an intuitive way of showing the effective dimensions being used to encode stimuli, I still feel there is a lack of direct demonstration of the level of sparseness resembling the orientation tuning example in Figure 1A. If indeed the neural network learns a sparse coding, I suppose you will find that the number of positive responses in the relu6 (or a layer that the authors consider appropriate) at inference stage varies according to the dropout rate at training time, or at least the number of units showing response above a threshold (not necessarily zero) should show such a dependency on dropout rate.\n\n\nThe robustness drop due to lesion at inference time only appears apparent at very high level of lesion. What is the implication of this for the brain: do we expect that such level of unexpected inactivation of neurons is common for our object recognition regions, so that it is necessary to adjust the region's selection in the continuum between sparse and distributed coding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gz0oqnDVuO", "reviewer": "Reviewer_yavH", "summary": "The paper introduces a neurobiological link between dropout, a common ML regularization method and human brain processing. \n\nKey findings include:\n* There was a reduction in object representation dimensionality with increased dropout levels for NNs\n* NN performance was relatively consistent across different levels of dropout, first monotonically increasing and then monotonically decreasing\n* Representational trajectory analysis revealed most differences in the fully connected layers where dropout was applied \n* Dropout mitigates effect of [neuron] lesioning, but to a limited extent\n* A model that was most robust to [neuron] lesioning (dropout p=0.7) mostly closely matched human brain representations obtained from an FMRI study and evaluated using Representational Similarity Analysis (RSA).", "questions": "I did not understand why human responses were compared to network responses but using different datasets (as discussed in section 2.4, human fMRI was obtained using Natural Scenes Dataset and compared to results on ImageNet using networks)? Authors should either explain the motivation of not running networks on the Natural Scenes dataset to match human fMRI, or report results using a single dataset.\n\nIt would be useful to understand the effect of dropout on the trajectories from representational trajectory analysis reported in Figure 1B. Specifically, is it dropout that induces the highest variation in the fc layers, or the underlying network architecture?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a systematic analysis of dropout and [neuron] lesioning with a neuro-biological perspective in mind. Experiments highlight interesting and novel findings with respect to model variations due to dropout and newly discovered connections to human brain processing. Overall, the paper makes advances in understanding spectral properties learnt by humans and neural networks.  \n\nAuthors explain the limitations of their work in terms of considering only one type of regularizations.", "weaknesses": "The paper would benefit from more clarity and better organization. It was somewhat difficult to understand key components of the paper. For instance, the definition of “lesioning”, i.e., pruning neurons in an NN layer, which is central to the paper message, is found in a figure caption (Figure 2), but should appear in the main paper. \n\nSpecific points:\n* It is not clear how the layer for lesioning (relu6) was selected or whether results will still hold if a different layer would be selected. \n* Results are reported using a single architecture (AlexNet), and may not generalize to other NNs. Also, the statement in the beginning of conclusion “schemes within a family of deep neural network (DNN)” needs to be adapted to better reflect the type of networks considered. Specifically, here family seem to refer not to architecture variations, but changes in the level of dropout within an architecture. \n* Authors should discuss the significance of the reported findings for the ML community. I thought that many of the reported analyses were cleverly executed, but I struggled to understand the main message behind the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSD4AAp4Mp", "reviewer": "Reviewer_h1kR", "summary": "To understand the trade-off between high-dimensional and uncorrelated efficient codes and distributed codes known better for generalization and robustness, the authors trained a family of object recognition models with parametrically varying dropout proportions. They found that a higher proportion of dropouts results in more smooth and low-dimensional representational space, with 70% of dropouts offering optimal robustness (against simulated brain lesions). Interestingly, this is also associated with the highest degree of emergent brain alignment for fMRI data in humans. Furthermore, the match between the model and brain representations is associated with a common balance between the efficiency of a high-dimensional code and the robustness of a low-dimensional code.", "questions": "While it is still fantastic to prove empirically a simple and intuitive idea is true elegantly,  is it possible to show this analytically?  What would be the analytical approach?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea and approach are simple, and the results are compelling and conceptually reasonable. The study illuminates the relationship between sparse codes and distributed codes. Using simulated lesions to evaluate robustness is an interesting innovation. The paper is well written. The presentation is clear and well-organized. The best alignment between human fMRI representation and of the representation of the model with the greatest robustness is very interesting, suggesting robustness, in addition to efficiency, is indeed a very important criterion for learning brain representation. So, the study does provide insights into the brain. I rank this an acceptable paper, with \"grade\" somewhere between 6 and 7, so I round it up to 6, because there is no \"7\".", "weaknesses": "The idea seems obvious and intuitive, and the approach and the work are perhaps too simple.  \nThe work is most empirical and does not have much theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QV4oBdSnQ2", "reviewer": "Reviewer_YU1u", "summary": "In this study, the researchers manipulated the degree of dropout in AlexNet, and then computed the eigenspectrum of the population responses in fc6, following Stringer, 2019. The results showed that increasing the degree of dropout reduced the representational dimension in fc6 and that dropout=0.7 achieved the best test results during the inference process. The authors also computed the eigenspectra of voxel responses in human visual cortex using NSD data and found that it was similar to AlexNet at dropout=0.7, while the correspondence between fc6 and voxel responses was strongest at dropout=0.7.\n\nI am generally positive about this paper. It replicates the analysis of mouse V1 data in human fMRI data.", "questions": "1. In Figure 1, I am not sure why dropout was only implemented in fc6 and fc7 in model training? In a typical training, dropout was applied to all layers\n2. I am wondering about the results of fc7 using the similar analysis in Figs. 2&3\n3. In Figure 2, dropout=0.5 is the best in terms of top-5 ImageNet accuracy (Fig. 2A) but dropout=0.7 is the best in terms of accuracy with unit lession (Fig. 2D). It is debatable what exactly the criteria we emphasize when talking about coding efficiency.\n4. In Figure 3, how many voxels and how many fc6 units are used? I suspect that the numbers are very different.\n5. The original study by Stringer, 2019 only recorded neurons in mouse V1. However, the ROI used here included several low- and high-level cortex. Do you expect the eigenspectrum to be different across visual areas in humans. Actually, I would say that this is the novel point to make compared to simply replicating the analyses in Stringer, 2019.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Using a large-scale fMRI dataset and computational analyses to address the coding principles of human visual cortex\n2. A novel approach is developed to estimate the signal correlation matrix and and the noise correlation matrix in population responses", "weaknesses": "see my questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)