Sparse components distinguish visual pathways & their alignment to neural networks
Authors: Ammar I Marvi
Nancy Kanwisher
Meenakshi Khosla
Venue: ICLR 2025 Spotlight (2025)
Keywords: visual representations
alignment
sparse decomposition
neural pathways
brain and machine vision
Primary area: applications to neuroscience & cognitive science

Abstract: The ventral, dorsal, and lateral streams in high-level human visual cortex are implicated in distinct functional processes. Yet, deep neural networks (DNNs) trained on a single task model the entire visual system surprisingly well, hinting at common computational principles across these pathways. To explore this inconsistency, we applied a novel sparse decomposition approach to identify the dominant components of visual representations within each stream. Consistent with traditional neuroscience research, we find a clear difference in component response profiles across the three visual streams—identifying components selective for faces, places, bodies, text, and food in the ventral stream; social interactions, implied motion, and hand actions in the lateral stream; and some less interpretable components in the dorsal stream. Building on this, we introduce Sparse Component Alignment (SCA), a new method for measuring representational alignment between brains and machines that better captures the latent neural tuning of these two visual systems. We find that standard visual DNNs are more aligned with ventral than either dorsal or lateral representations. SCA reveals these distinctions with greater resolution than conventional population-level geometry, offering a measure of representational alignment that is sensitive to a system’s underlying axes of neural tuning.

Published as a conference paper at ICLR 2025
SPARSE COMPONENTS DISTINGUISH VISUAL PATH- WAYS & THEIR ALIGNMENT TO NEURAL NETWORKS
Ammar I Marvi
Nancy G Kanwisher
Meenakshi Khosla UCSD
MIT
MIT
ama rvi (mim edu
ngk@mi @mi t. edu
mkhos 1 a duccdd. edu
ABSTRACT
The ventral, dorsal, and lateral streams high-level human visual cortex are implicated in distinct functional processes. Yet, deep neural networks (DNNs) trained on a single task model the entire visual system surprisingly well, hint- ing at common computational principles across these pathways. To explore this inconsistency, we applied a novel sparse decomposition approach to identify the dominant components of visual representations within each stream. Consistent with traditional neuroscience research, we find a clear difference in component re- sponse profiles across the three visual streams crrrams-ideenffyinng components selective for faces, places, bodies, text, and food in the ventral stream; social interactions, implied motion, and hand actions the lateral stream; and some less interpretable components in the dorsal stream. Building on this, we introduce Sparse Compo- nent Aligmment (SCA), new method for measuring representational alignment between brains and machines that better captures the latent neural tuning of these two visual systems. Using SCA, we find that standard visual DNNs are more aligned with ventral than either dorsal or lateral representations. SCA reveals these distinctions with greater resolution than conventional population-level geometry, offering measure of representational alignment that is sensitive to system's s underlying axes of neural tuning.
INTRODUCTION
Understanding the differences in how the ventral, dorsal, and lateral visual streams process informa- tion is crucial for building good models of human vision. Extensive evidence indicates that visual information is processed along three functionally distinct cortical pathways: the ventral stream, im- plicated in object recognition, the dorsal stream in visually guided action, and the lateral stream in motion and social information processing (Mishkin & Ungerleider] 1982 Pitcher & Ungerlei- der 2021). In order carry out such different functions, the ventral, dorsal, and lateral streams surely represent visual information in fundamentally distinct ways: what are the precise compu- fations underlying these functions, and how do visual representations differ across the three pathways?
Over the last decade, deep neural networks (DNNs) trained for object classification have been shown to exhibit similar internal activations (Yamins et al., 2014), functional selectivities (Blauch et al.
2022] Dobs et al.] 2022), and behavioral capabilities (Dobs et al., 2023) Rajalingham et al., 2018) to the human ventral visual pathway. The hierarchy of cascading layers with shared convolutional filters was inspired by known neural architecture (LeCun et al. 1989), and indeed standard measures of similarity provide strikingly high alignment between visual representations in DNNs and the ventral stream (Yamins & DiCarlo 2016). However, recent studies have suggested that these same and similar DNNs also capture responses in the dorsal and lateral stream similarly well (Finzi al., 2024 Conwell et al. 2024), see also (Margalit et al. 2024). How can the same computational model capture the diversity of function across these pathways? Here we address two specific questions: withinnddssiinusseeee the representations in the ventral, dorsal, and lateral streams? And second, why do current measures of representational alignment between brains and DNNs often fail to reflect these differences?
first, what-if
"htps//aimarvi.iihuun iithuu...

---

Published as a conference paper at ICLR 2025
To answer the first question applied data-driven approach identify the dominant components of the fMRI response to natural images the three visual pathways of human high-level cortex, and in intermediate layers of DNNs. We observed qualitatively distinct response profiles in the three visual streams, with often interpretable selectivities: scenes, faces, bodies, food, and text in the ven- tral stream; group interactions, implied motion, hand actions, scenes, and reach-spaces in the lateral stream; and scenes and implied motion in the dorsal stream. We also collected behavioral ratings provide rigorous, quantitative evaluation of these interpretations. These findings recapitulate known category selectivity (Kanwisher 2010) and dissociations between the three visual pathways.
a)
stinuli
component
component
stirult
comporent
Sparse Component Alignment (SCA)
b)
st.imll
X50
components
coneeree
rark under
comperent
C, fe
IE sare. othcIvise
average
Sc3muls
Figure 1: Schematic overview of the data-driven component modeling approach. (a) We used Bayesian non-negative matrix factorization (NMF) to decompose a given voxel x stimuli matrix into two lower rank matrices representing component responses R and the corresponding weights of anatomical voxels W. (b) For each iteration, connectivity matrices C are created using rank- ordered component responses, where each cell of the connectivity matrix C.j represents whether pair of stimuli J maximally load onto the same component. Binary matrices are averaged across all iterations to produce single image connectivity matrix.
To answer the second question, measured brain-mode alignment using two standard meth- ods: linear encoding models (Yamins et al. 2014) and representational similarity analysis (RSA) (Kriegeskorte| 2008 Consistent with earlier work, both linear encoding and RSA suggest high degree of alignment between the visual representations DNNs and each of the visual pathways. However, these methods-due inherent rotational invariances are insensitive the tuning prop- erties of individual neurons, and so we developed a novel, complementary method for measuring representational alignment that retains such sensitivity. Sparse Component Alignment (SCA) uses dominant, data-dria ddia-n components to capture the activity of sparse sub uub-populations of neurons, thus providing an alternative measure of similarity that is specific a system's ss native axes of neural tuning. SCA reveals clear alignment of DNNs to the ventral visual pathway but much weaker align- ment to the lateral and dorsal pathways. These findings suggest that DNNs trained on natural images better capture the computations of the ventral visual pathway, and they further raise the question of what models might better capture the lateral and dorsal streams.
2 METHOD
2.1 BAYESIAN NON-NEGATIVE MATRIX FACTORIZATION
We applied data-driven Bayesian non-negative matrix factorization (NMF) (Schmidt et al.) 2009) to identify the dominant components of visual representations in each stream in four subjects of
2

---

Published as a conference paper at ICLR 2025
the Natural Scenes Dataset (NSD), massive naturalistic fMRI dataset (Allen al. 2022). This method decomposes a neural response matrix into its dominant components, represented as the product of two lower rank matrices consisting of anatomical voxel weights for each component and the corresponding component responses. Importantly, NMF is free of any a priori hypotheses regarding the dataset the spatial layout of voxels, relying solely on the data's inherent statistical structure. Our method follows a similar approach as NNrman-Haigneree et al. (2015) and Khosla al.] (2022) and makes minimal assumptions about the inferred components, requiring only that both matrices in the decomposition contain non-negative entries.
The neural response matrix D R.V.V with S images and V voxels modeled as:
(1) where R E R.C. is the component response matrix with C' components, W E R.C.V is the voxel weight matrix, and E E R.V.V is a residuals matrix. Bayesian NMF introduces priors for the pa- rameters of all these matrices (R. W, E) to enable probabilistic inference. Specifically, following Schmidt et al. (2009), we assume a normal likelihood for the data, where the residuals E are mod- eled as zero-mean Gaussian noise with variance o²:
RW + E D,
PD W, E) ~ II,= I1,= v-N (Ds,.i (WR) u. o²)
Independent exponential priors are placed on R and V, enforcing non-negativity:
(R) II==I...:==1,,.... exp(-Ps,c RA, E Rz
and P(W) ~ -I=,,..C==1,... Ve. exp( -c, W wew), We, 12
where Pa.c and Te.u are scale parameters. The posterior distributions for R and W are rectified Gaussian, while the variance o² has an inverse-gamee -aamaa posterior. We infer these parameters via Markov Chain Monte Carlo (MCMC) sampling.
The use of hypothesis-free methods to decompose nigh-dimensional data into more interpretable components has become increasingly popular with the introduction of large-scale datasets, taking advantage of statistical regularities within big data to find underlying, dominant response patterns. Common methods like principal component analysis (PCA) aim to linearly decompose data along an orthogonal set of dimensions, while others like t-SNE and autoencoders assume non-linearities to explain complex relationships within a dataset under fewer constraints.
Our motivation in using Bayesian NMF to decompose brain data four-fold. First, unlike PCA or ICA, NMF-derived components are not constrained by orthogonality or independence, which better aligns with empirical evidence suggesting that neural responses of distinct components are often inter-dependent (Pnevmatikakiis et al. 2016). Second, the non-negative constraint on matrices W and R aids biological interpretation of fMRI data. Negative response magnitudes in are in- consistent with neural responses in the ventral visual pathway, which usually increase after stimulus presentation, and the presence of negative values in W violates our modeling assumption that this matrix represents the relative anatomical weights of each component in every voxel. Third, while PCA is invariant to rotations in neural space, NMF is not and recovers different components before and after arationn-dvaatageenn axes or tuning functions over others. Finally, in comparison to standard NMF algorithms, Bayesian NMF more readily discovers natural sparsity in the underlying data. Empirical studies of neu- ronal spiking suggest that neural responses are inherently sparse, and biological wiring costs point toward sparse connections between interacting brain regions (Olshausen & Field 2004; [Brllo] 2012 Chklovskii et al., 2002). As result, downstream brain regions rarely have access to complete upstream neural activity. Bayesian NMF is well- for discovering such sparsity and-um unlike standard NMF or PCA-infer sparse components that map well onto the original latent data. For these reasons-whiel we validate in simulated data (see Figure 2 we opted for Bayesian NMF over other decomposition methods.
when modeling neural data that often consistently favor certain
and-unlike
The Bayesian NMF algorithm is stochastic, so to reliably model dominant components we apply N = 50 iterations of the NMF algorithm for each subject. A consensus set of weight and response matrices is collected across these iterations, which is then aggregated to produce the final compo- nent weight and response matrices A.1 for further details). The number of components in each iteration is a free parameter, which we fix at C' = 20 for consistency across subjects, streams, mod- els, and layers. This was principally motivated by previous study that used Bayesian information
3

---

Published as a conference paper at ICLR 2025
criteria estimate the optimal number of components in modeling the ventral visual stream (Khosla) iet al. 2022). However, we note that similar results also arise when deriving between 10 to 30 com- ponents. We then identified the most consistent components across subjects using a shared set of 1, 1,000 images viewed by each subject.
We similarly applied the NMF algorithm to DNN feature activations extracted in response to the same set of NSD stimuli. DNNs were pre-trained on ImageNet-Ik (Deng et al. (2009) and varied in architecture and objective. Model backbones included AlexNet (Krizhevsky al. 2012). ResNet- 50 (He et al. 2016), and ViT (Dosovitskiy el al.] 2020), and models were trained with category- or self-supervision (ResNet-50 w/ MoCo-v2 (Chen et al. 2020b) & SimCLR (Chen al., 2020a), ViT w/ DINO (Caron et al.] 2021)). Specifically, for each model we obtained feature activations separately for each of the four subjects' 10, 000 viewed images. The resulting unit x image matrix was treated identically to the subject voxel image matrix for further analysis.
2.2 REPRESENTATIONAL ALIGNMENT
Measures of representational alignment fall into one of two broad categories: (A) measures that establish an explicit mapping between single-neuron dimensions, and (8) measures that compare stimulus x stimulus dissimilarities of a population (Sucholutsky et al. 2023 Harvey et al. 2023
Linear encoding falls into the set of methods belonging to category A and involves linearly combin- ing responses from model units to predict voxel responses by This approach is well-established in the neuroscience literature, as it aims to optimally align model and brain response spaces through linear transformations while minimizing the introduction of complex non-linearities. These trans- formations are often preferred due to the assumption that downstream readout mechanisms apply approximately- -inear functions their inputs (Cao & Yamins (022).. We extracted feature activa- tions from the ultimate (AlexNet) or penultimate (ResNet-50) pooling layer of convolutional models, or from the best performing attention head in vision transformers (ViT), and used the activations to predict neural responses to shared set of 1, 000 images viewed by each subject (via ridge re- gression with an 80/20 train/test split). Similarity between models and brains was calculated as the coefficient of determination between predicted and actual neural responses in individual subjects.
RSA belongs to category and characterizes the geometry of stimulus representations in a high- dimensional neural space. We calculated the population-level dissimilarity of neural responses for each pair of the shared 1, 000 images, which we summarized in a representational dissimilarity matrix (RDM) for each subject and model. To assess the similarity between patterns of brain activity across all stimuli, we utilized correlation as our measure. We measured the second-order similarity between model representations and brain responses by extracting the upper triangular entries of each RDM and calculating the Spearman's 's rank correlation (p) between models and brains.
Importantly, both methods deseribed above treat rotations nuisance transformations and are thus explicitly insensitive to the specific tuning of individual neurons (see Figure While this invariance may be useful in evaluating distributed, population-level responses, it is perhaps less desirable when representations consistently favor certain axes over others, as is the case in functional regions of interest selective for visual categories in the brain.
2.3 SPARSE COMPONENT ALIGNMENT
To systematically assess the consequences of rotational invariance, we propose new technique Sparse Component Alignment (SCA) that measures representational alignment while retaining sensitivity to neural tuning. Of category B and similar to RSA, SCA assesses stimulus-level rep- resentational dissimilarities using Bayesian NMF-derived sparse components. However, instead of relying on population geometry, SCA computes pairwise distances between stimuli based on the likelihood that they are processed by the same dominant component.
The SCA algorithm is described in Figure | and Algorithm | For each iteration of Bayesian NMF, we decompose the neural response matrix into two lower-rank matrices representing voxel weights W E C.V and component responses R E S.C.C Using an overlapping set of images, we first rank-order R and then construct stimulus stimulus matrix C C.C. where each entry ca takes a value of one or zero based on whether that pair of stimuli produce the highest response in the same component. Mathematically, the connectivity matrices seek to partition images based on their
4

---

Published as a conference paper at ICLR 2025
most contributing basis component. We define each entry of the connectivity matrix C' as, if samples i, maximally load the component,
{
Cj o
otherwise
These matrices are averaged across all iterations of Bayesian NMF to produce a single Image Con- nectivity Matrix (ICM), where the off-diagonal describes how frequently two stimuli are processed by the same dominant component.
Finally, we extract the upper triangular entries of these connectivity matrices and measure the simi- larity between systems using Pearson's 's correlation coefficient (r).
Algorithm Sparse Component Alignment (SCA)
1: procedure CONNECTIVITY MATRIX(D)
Neural response matrix D
2: for <- N do
> iterations N
3:
C = 03,5
# of stimuli S
4:
Wn :e D
5:
for j t : C' where j do
of components C
6:
f rank-sort( )
7:
r rank-sort ).:)
8:
C = [0]000100)
end for 10: end for
9:
11: C E C"
> Connectivity matrix C
12:
end procedure
13:
14:
corr(CA,, CB)
Leveraging the sparse decomposition of neural representations, we compare SCA with another simi- larity measure within category A: the Component Matching Score (CMS). This score optimizes over permutations to align components between two representations, X and Y, as follows:
S(X, Y) = max
1))
1=1
where
rj corr (X.,, )
In this context, represents C. x C permutation matrix, allowing us to find the optimal alignment between the extracted components, accounting for possible permutations. We also measure the isolated effects of NMF pre-processing by using the components in traditional aligmment metrics (see A.2 and Figure 7 for further details).
2.3.1
BIOLOGICAL RELEVANCE OF SCA
Many measures of representational geometry implicitly assume that downstream brain regions have access to the entire neural population response. However, biological systems face wiring constraints that limit the number of connections between regions. In reality, neurons in downstream areas "read out" information from only a small subset of neurons in their input. By applying sparse decompositions neural representations, ologically interpretable components. These sparse components may offer a more realistic reflection of the information that downstream structures can access. Consequently, the image connectivity matrices as defined above can be interpreted as capturing the likelihood that two stimuli are to be routed the same downstream neural structures, thus retaining sensitivity to neural tuning axes.
aim distill these population-level responses into bi-
RESULTS
3.1 SIMULATIONS
Through simulations, we first demonstrate the sensitivity of our proposed SCA framework to subtle changes in the axes of representations, showing that small perturbations significantly reduce
5

---

Published as a conference paper at ICLR 2025
oporss Tareer crnooeere
UNIT GOOPEL
X
SIREE (
b)
orle
plans ronate
Figure 2: Simulations of latent component recovery and rotation sensitivity Different methods used to recover the latent components of simulated data X. (a) A sparse decomposition finds the op- timal mapping of original-to-inferees components (top, red-outlined matrix entries). Unlike sparse NMF (snmf), Bayesian NMF (bnmf) jointly infers sparsity in W and R (bottom, gray bars). (b) NMF but not PCA components are dissimilar after correcting for rotation, measured via Pearson's 's (note: overlapping PCA components) (c) Sparse component alignment (SCA) demonstrates clear sensitivity to minor perturbations in the native axes of the representation; specifically, increasing the extent of axis rotations (X X, -hhther through larger angles or greater number of 2D planes in more substantial decreases alignment.
alignment. To begin, we generate simulated data as mixture of sparse latent components and mixing coefficients. The data matrix X E " is constructed as the product of two sub-matrices: [Rm,k,, which simulates the response of & components to m stimulus conditions, and A R containing the mixing coefficients. The resulting equation, with added noise c, is defined as:
X LA+E
We then attempted recover the latent components using four decomposition methods: PCA, stan- dard NMF, sparse NMF, and Bayesian NMF (for further details, see A.3). Sparse NMF is similar Bayesian NMF but infers sparsity in only a single sub-matrix. Figure 2 shows that sparse decomposition indeed finds the optimal alignment between latent and inferred components.
Next, we introduce small adjustments the native axes of representations in X through rotation matrix R parameterized by 0., (for details, see A.4). We then apply these rotations:
(5) and derive components using both PCA and NMF, As expected, find that PCA but NMF components remain unchanged after correcting for rotation (Figure for details see A.5
X, XR,
Finally, study how the angle and number 2D rotations affect the alignment between X and X, as measured using SCA and RSA. This sensitivity analysis serves as proof concept, revealing that larger rotation angles (0t,) and increased number of 2D rotations (c) correspond pronounced reduction in alignment with SCA (Figure 2). In contrast, axis-insensitive measures like RSA deem all these rotated representations X, as highly similar X. These findings further validate SCA's strong sensitivity to rotations in the native axes of tuning in a representation.
more
3.2 HYPOTHESIS FREE DISTINCTION OF VISUAL STREAMS
We first applied the NMF algorithm to decompose neural responses into their dominant components using IMRI data from four subjects of the NSD. The resulting W and R matrices explained much of the variance in the neural response and performs almost as well as PCA, which sets an upper bound on the variance explained by any linear decomposition. We also empirically measured the sparsity of the resulting decomposition via the dispersion of W and R following Hoyer (2004) (Figure also and 8).

---

Published as a conference paper at ICLR 2025
nmf
bnmf
pca
motrix
a)
2.6
2.5
5.4
8:3
explained variance
5.2
sparsity
tateral vestral
matrrix
2.0
3.b
2.4
3.2
1aterst vestral.
untroleed
Figure 3: Examination of data decomposition. (a) Explained variance of an example neural re- sponse matrix D in brain and models. (b) Bayesian priors produce sparse components in non- negative matrix factorization (NMF). Measured sparsity of example weight W (top) response R (bottom) matrices of components derivered from NMF, bayesian NMF, and PCA in the brain and models. Note: bars for standard NMF are present but close to zero.
We extracted the most consistent components (median inter-subject consistency > 0.5) separately in the dorsal, ventral, and lateral stream and qualitatively examined their response profiles (ie. the images producing the highest and lowest responses in each component). A majority of the response profiles offered highly interpretable category selectivities, which we further quantitatively tested with behavioral saliency rating collected online (for details, see A.7). Component selectivity in the ventral stream replicates previous studies (Khosla et al. 20022). while the components and interpre- tations derived for the dorsal and lateral stream that present here wholly novel.
The response profiles of each component is plotted in Figure The same 1, 1,000 images are repre- sented in each subplot via individual sticks and colored by the average saliency rating for their respective category. Images are rank-ordered by component response magnitude (y-axis, arbitrary units (a.u.)). Alongside each sub-plot, display three of the images producing the highest re- sponse, as well the correlation between normalized saliency ratings and component responses. We find components selective for scenes (r 0.632), faces (r 0.799), bodies (r = 0.604), and text (r = 0.439) in the ventral stream; group interactions (r = 0.454), implied motion (r 0.660), hand actions (r 0.448), scenes (r 0.299), and reachspaces (r 0.310) in the lateral stream; and scenes (r = 0.393) and implied motion (r = 0.428) in the dorsal stream.
0.695). food
The component response profiles indicate distinet visual representations and functional roles for each of the dorsal, ventral, and lateral streams. In particular, this method refines the role of the lateral stream social information processing, namely that separate components are selective for group interactions, hand actions, and reachspaces. We emphasize that this three-way way dissociation is free of any a priori hypotheses regarding spatial layout and/or functional segregation, resulting only from the statistics and biases within the data and stimuli.
We also examined the response profiles of components derived from the hidden layers of various DNNs (see Figure TOp. Untrained models seem selective for low-level features, while pre-trained models showed selectivities with dominant motifs for handful of categories including faces, scenes, words, and various animals. Qualitatively, it's clear that the response profiles in DNN components are dominated by visual similarity. We note that variations in training objective and architecture did not produce notably different response profiles the models we tested here.
3.3 MODEL-BRAIN ALIGNMENT
We quantified representational alignment between DNNs and each of the dorsal, ventral, and lateral streams using linear encoding, RSA, and SCA and CMS. For clarity. we focus on the alignment of pre-trained and untrained AlexNet models with the three visual streams, though the full set of models

---

Published as a conference paper at ICLR 2025
b}
<)
ventral
latersl
donset
P12
Tou
groups!
faces: 2.00
metion:
66
0.45
Tood:
C6
hands:
0:45
d
teadine: 0.70
TeRctscerw: D.M
Figure 4: Component response profiles and preferred stimuli. Plots depicting the response pro- files of the most consistent components across the four subjects in the (a) ventral (blue), (b) lateral (red), and (c) dorsal (green) streams. Each subplot shows the same 1, 000 images (depicted as sticks) rank-ordered by their evoked component response (y-axis, a.u) colored by their average saliency rating to component-specific prompt. The correlation between saliency ratings and component responses are provided in each subplot, along with three of the component's 's preferred stimuli. (d) Visualization of the anatomical masks used to demarcate each visual stream.
exhibited similar patterns, as summarized in Figure 5 A linear readout suggests that pre-trained
models predict neural activity similarly well in the dorsal (r = 0.232), lateral (r 0.179), and ventral (r = 0.180) stream well above bascline alignment to an untrained model (dorsal: r = 0.120, lateral: = 0.096, ventral: r 0.091). When using RSA, we observed similar though notably higher levels of alignment to the ventral (r = 0.347) than dorsal (r = 0.199) or lateral (r = 0.222) streams. Finally, SCA suggests markedly higher alignment between standard vision models and the ventral (r 0.187) stream, and drops significantly in both the lateral (r = 0.047) and dorsal (r = 0.058) streams to levels approaching that of an untrained baseline. This measure is non-trivially related to the construction of connectivity matrices, as the same dominant components-ffllovine a strict 1-1 mapping as used in CMS show only a modest similarity across all streams (see also A.2 and Figure for further tests of NMF pre-processing).
Past studies have shown that representational alignment extends across the processing hierarchy of the ventral visual stream and DNNs. We sought to test if SCA also captured this hierarchical fit by extracting intermediate layer activations from the pre-trained Alexnet model. As expected, later layers in the model better captured neural responses in higher-level ventral visual stream, and this pattern persisted in the dorsal and lateral streams though to far lesser extent.
Altogether, these results point toward two important conclusions. First-in contrast to standard ikaatin-innvaiian measures. which suggest similar functional representations across the dorsal, ven- tral, and lateral streams- -eraams-CAAAA reveals much greater alignment to just the ventral pathway. Second, this alignment is specific to models optimized for object recognition and drops for untrained models, hinting shared design to capture visual similarity by both DNNs and the ventral visual pathway.
3.4 BEHAVIORAL SIMILARITY
While alignment with neural data provides valuable insight into a system's internal mechanisms, human behavior often offers a more explicit reflection of how we represent high-level visual infor- mation. Tools like RSA particularly well-suited to capture the similarity between neural repre- sentations and behavior, which would otherwise be difficult to quantify. We leveraged this capability to analyze the Meadows dataset-a behavioral dataset from the NSD-iin which four participants
8

---

Published as a conference paper at ICLR 2025
a)
unt relmed
danno
vit
resnet5e
FOCO
SERCET
dlexnet
0.33
Linear enceding
sca
1-1 mapping (cas)
0.22
slestarity Er
0-2
0.13
0.30
0.1
0.8
8.96
6:0
b)
dorsal
lateral
ventral
0.30
0.25
sieturity (r)
0.20
0.30
0.05
Figure 5: Alignment of deep neural networks (DNNs) to the brain. The measured alignment between visual representations in the brain-in dorsal, lateral, and ventral streams-and the same set 7 visual DNNs. The untrained model is in white, and pre-trained models are colored in various shades of grey. (a) From left to right, similarity is measured by linear encoding, representational similarity analysis (RSA), sparse component aligmment (SCA), and the 1-1 component matching score (CMS). (b) Alignment between each pathway and intermediate layers of pre-trained AlexNet model, using SCA.
arranged subset of stimuli along two dimensions based their perceived similarity. The pairwise distances between stimuli were then used to construct a behavioral RDM.
We used either RDMs or ICMs derived from neural representations to better understand how differ- ent stimulus-leved representations align with high-level behavior, quantified by computing Pearson's 's correlation between the corresponding similarity matrices. With RSA, behavior is most aligned with visual representations in the brain's 's ventral stream and in models optimized for object recognition. Alignment begins to drop for representations the lateral stream, falls further the dorsal stream, and is lowest in models with untrained weights. Do the connectivity matrices used in SCA capture similar patterns of information as the dissimilarity matrices used in RSA? As shown in Figure 6 analysis using connectivity matrices results in similar overall pattern, with the ventral stream and task-optimized models showing the highest alignment to behavior. However, we don't 't observe the same intermediate levels of alignment with the lateral and dorsal stream. Importantly, Bayesian NMF-derived connectivity matrices seem to capture similar information as representations used in RSA while relying on much sparser coding structure. All of this suggests that the connectivity matrices derived using Bayesian NMF effectively capture behaviorally information.
DISCUSSION
Here we sought to resolve the apparent contradiction between prior findings demonstrating (i) dis- tinct functions of the ventral, lateral, and dorsal visual pathways in the brain, versus (ii) the similar fit of all three pathways artificial networks optimized for object recognition. First, we used a sparse decomposition approach to identify the dominant components of visual representations in four subjects from the NSD, an IMRI dataset containing neural responses to thousands of natural- istic images. Separate analyses of the dorsal, lateral, and ventral visual pathways, as well as in suite of DNNs trained for object recognition, revealed distinct components for each pathway. We then introduced Sparse Component Alignment (SCA) to measure the alignment of DNNs visual representations in the brain, and we found markedly higher alignment to the ventral than either the lateral or dorsal streams. This finding is invisible standard alignment metrics due to their rota-

---

Published as a conference paper at ICLR 2025
a)
image connectivity retrix
representational dissimilarity netrix
b)
5.35
2.23
6.3
rerenntationl disteclerity retri
0-2
2:32
1.03
0.02
Literal
restral
alcene
Figure 6: Alignment to behavior. (a) Using either representational dissimilarity matrices (light) or image connectivity matrices (dark), we measured the alignment between brains (left) and models (right) to high-level visual representations derived from the Meadows dataset. Connectivity matrices capture similar patterns of alignment while maintaining higher degree of sparsity. (b) Both con- nectivity (top) and dissimilarity (bottom) matrices capture behaviorally-releaan categories. Face (red) and scene (blue) images depicted below each matrix colored sticks.
tional invariance. We thus conclude that DNNs share similar axes of neural tuning as neurons in the ventral visual stream.
4.1 FUNCTIONAL DIFFERENCES BETWEEN VENTRAL, LATERAL, AND DORSAL PATHWAYS
Using non-negative matrix factorization (NMF), characterized neural response profiles free of a priori functional and/or spatial hypotheses. Consistent with prior results, we find components selective for faces, scenes, bodies, food, and words in the ventral stream. In addition, interpretation of novel components selective for group interactions, scenes, hand-related actions, motion, and reachspaces in the lateral stream, and for group interactions and scenes in the dorsal stream. These results reinforce a litany of existing neuroimaging, behavioral, electrophysiologi- cal, and computational findings implicating the ventral stream in object recognition, the lateral in dynamic social perception, and the dorsal in visually guided action.
offer
The fine-grained functional organization in the lateral and dorsal streams has remained less clear than in the ventral pathway. Our findings show that a sparse decomposition of even static snapshots well-suited for understanding these less-explored brain regions. At the same time, the methods we used here do not fully capture the representations and computations of the dorsal and lateral streams, which would require collecting neural responses to wider variety of stimuli and tasks.
4.2 ALIGNMENT OF EACH PATHWAY TO DNNS
Our finding distinct functional roles in each pathway heightens the mystery of how all three path- ways could be similarly aligned the same maaee-rrinned DNN. We argue that standard metrics of alignment-due to rotational invariance and insensitivity to specific tuning axes-are insensitive to the functional differences SCA, a novel method that captures the tuning of neurons and network units. Using SCA, we find a pattern of results implying that units in DNN share similar tuning properties as category-selective neurons in the ventral stream. That image-traine trained DNNs converge onto similar representations as the ventral pathway suggests that computations in both of these systems are driven by statistical regular- ities in static visual input, providing an intuitive explanation for our results. The relatively weaker alignment to dorsal and lateral representations highlights the limitations of current task objectives, architectures, and datasets, and may call for fundamentally different approaches to modeling visual pathways. Perhaps video-trained networks would better fit neural responses that are sensitive to motion, Bayesian models of social cognition and physical scene understanding to better fit the lateral and dorsal streams, respectively. Object recognition just one piece of the broader puzzle of human vision.
find in the neural responses
pathways. We therefore introduce
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was made possible by funding from the NIH grant EY033843 to N.G.K. and the NSF Science and Technology Center - N.G.K
Miserr--ente Center for Brains, Minds, and Machines Grant CCF-1231216 to
CODE AVAILABILITY
Code for the analyses performed in this study will made available at ut/ttps://githhbb com/ aimarvi/NSD Setarvi/NSDstream..
REFERENCES
Emily J. Allen, Ghislain St- Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dow- dle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25:116-126 2022. ISSN 15461726. doi: 00.1088/441593-021-00000
H. B. Barlow. Possible Principles Underlying the Transformations of Sensory Messages, pp. 216- 234. The MIT Press, 9 2012. doi: 10.755 /t/755//mmmpes//77322255111.0..0
Nicholas M. Blauch, Marlene Behrmann, and David C. Plaut. A connectivity constiiityy-cntttaineed computa- tional account of topographic organization in primate high-level visual cortex. Proceedings of the National Academy of Sciences, 119, 2022. ISSN 0027-8424. doi: 10. 10.1073/pnas.21 11116116611 19.
Rosa Cao and Daniel Yamins. Explanatory models in neuroscience, part 1: Taking mechanistic abstraction seriously. Cognitive Systems Research, pp. 101244, 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https:
//arxiv. aaxxiv.org/abs//100 14229
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020a. URL
2002. 05709
Xinlei Chen, Haoqi Fan, Ross B Girshick, and Kaiming He. Improved baselines with mo- mentum contrastive learning. ArXiv, 04292 2020b. URL httpss:/appi.
Gemanticscholar. wwtp://aaaa..oooooooooooaaoaa:aa
Dmitri B. Chklovskii, Thomas Schikorski, and Charles F. Stevens. Wiring optimization cortical
circuits. Neuron, 34:341-344 4 2002. ISSN 08966273, doi: 10. 10.1016/50886666233022000686
William R. Clements, Peter C. Humphreys, Benjamin Metcalf, W. Steven Kolthammer, and Ian A. Walmsley. Optimal design for universal multiport interferometers. Optica, 1605 Dec 2016. doi: 10. 10.1364/OPTICA.3.000 460. abst www.:efnnnotiiiiom
12 1460
Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15(1)33383, Oct 2024. ISSN 2041-1723. doi: 1031038/ 441467-024-53147.1 URL / orgeii.org//00 1038 0111338/44467--0---333
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 255. IEEE, 6 2009. ISBN 978-1-422 1188--44244-9922 8. doi: 1L.100/CCVRR2220052066..
Katharina Dobs, Julio Martinez, Alexander J. E. Kell, and Nancy Kanwisher. Brain-like functional specialization emerges spontaneously in deep neural networks. Science Advances, 8, 3 2022. ISSN 2375-2548. doi: 10.1 121//sciadv..a118113...

---

Published as a conference paper at ICLR 2025
Katharina Dobs, Joanne Yuan, Julio Martinez, and Nancy Kanwisher. Behavioral signatures of face perception emerge in deep neural networks optimized for face recognition. Proceedings of the National Academy of Sciences, 120, 2023. ISSN 0027-8424. doi:
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. 10 2020.
Dawn Finzi, Eshed Margalit, Kendrick Kay, Daniel Yamins, and Kalanit Grill-Spector. single computational objective drives specialization of streams in visual cortex. bioRtiv, 2024. doi: 10. 1201/2023.12.19.5777 12. 9.572460. URL httt:t:/t/////omm
1ttp//0..111/222222 12 19........5.
Sarah E. Harvey, Brett W. Larsen, and Alex H. Williams. Duality of bures and shape distances with implications for comparing neural representations. 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEFE Conference on Computer Vision and Puttern Recognition (CVPR), pp- 770-778. IEFE, 6 2016. ISBN 978-1-4673-88511 doi: 10.1 R..1199CCVR.20016
Patrik O. Hoyer.
Non-negative matrix factorization with sparseness constraints. Mach. Learn. Res., 5:1457-1469, December 2004. ISSN 1532-4435.
Nancy Kanwisher. Functional specificity in the human brain: A window into the functional archi- tecture of the mind. Proceedings of the National Academy of Sciences, 116-25): 2010. doi: www.3//aaa.1000062100 1073/pnas. 1 005062107
URL https: /www wttppp.......///////////...
Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32:4159 4171.e9, 2022. ISSN 18790445. doi: 10. 10160006....... cub. www.............0. 2022.08.009.
Dylan Kotliar, Adrian Veres, M Aurel Nagy, Shervin Tabrizi, Eran Hodis, Douglas Melton, and Pardis C Sabeti. Identifying gene expression programs of cell-type identity and cellular activity with single-cell raa-sqq. eLife, 8, 2019. ISSN 2050-084X. doi: (0.7554/eLifee438833
Nikolaus Kriegeskorte. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2008 ISSN 16625137. doi: 10.3389/neuro.66. 004.2008.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F Pereira, C J Burges, L. Bottou, and K Q Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012. URL https: ptttps://proceeednggs wwwpogs..euurigee neurips. cc cc/paper_.... fol_ffleee files//aper//012/ pdf
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back- propagation network. In Neural Information Processing Systems, 1989. URL httpps://pp..
Eshed Margalit, Hyodong Lee, Dawn Finzi, James J. DiCarlo, Kalanit Grill-Spector, and Daniel L.K. Yamins. A unifying framework for functional organization in early and higher ventral visual cortex. Neuron, 2024. ISSN 08966273. doi:
Mortimer Mishkin and Leslie G. Ungerleider. Contribution of striate inputs to the visuospatial functions of parieco-praoccipital cortex in monkeys. Behavioural Brain Research, 6:57-77, 1982. ISSN 01664328. doi: 10. (0016/0666-4422888))00881848
Sam Norman Haignere, Nancy G. Kanwisher, and Josh H. McDermott. Distinct cortical pathways for music and speech revealed by hypothesis voxel decomposition. Neuron, 88:1281-1222 2015. ISSN 10974199. doi: 10. 10066/..eurron.2111 11.03..
12

---

Published as a conference paper at ICLR 2025
B Olshausen and D Field. Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14: 481-487, 2004. ISSN 09594388. doi: 10.1016f..comb2204400000000
Didier Pinchon and Pierre Siohan. Angular parametrization of rectangular paraunitary matri- ces. In Mathematics, 2016. URL
125926810
David Pitcher and Leslie G. Ungerleider. Evidence for third visual pathway specialized for social perception. Trends in Cognitive Sciences, 25:100- 110, 2 2021. ISSN 13646613. doi: 10. 10.1016/j. Ncs.2020.11.000
Eftychios A. Pnevmatikakis, Daniel Soudry, Yuanjun Gao, Timothy A. Machado, Josh Merel, David Pfau, Thomas Reardon, Yu Mu, Clay Lacefield, Weijian Yang, Misha Ahrens, Randy Bruno, Thomas M. Jessell, Darcy S. Peterka, Rafael Yuste, and Liam Paninski. Simultaneous denoising, deconvolution, and demixing of calcium imaging data. Neuron, 89:285-299, 2016. ISSN 08966273. doi: w.100//..eeuron.222 1..07.
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled represen- tations and group structure of dynamical environments. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 19727-19737. Curran Associates, Inc., 2020. URL https::/prroceediggs...nnet wttpipssccccccceeo cc/paper Flles//apee//220/Eiie
Rishi Rajalingham, Blias Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J. Di- Carlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-arr deep artificial neural networks. The Journal of Neuro- science, 18:7255-7266 8 2018. ISSN 0270-6474. doi: I0.123//EURRSSC.0388 18.20888811188....81888111111111
Mikkel N Schmidt, Ole Winther, and Lars Kai Hansen. Bayesian non-negative matrix factorization. In International Conference on Agents, 2009. URL httpss///pp....eantiicccci www.eeatttatiitccooeeo.o
Org/CorpusTD. ID: 12871123
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qi- uyi Zhang, Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas P. O' Connell, Thomas Unterthiner, Andrew K. Lampinen, Klaus-Robert Müller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment. 10 2023.
Daniel L K Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19:356-365, 2016. ISSN 1097-6256. doi: 10. 1038/nn 4244.
Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. hierarchical models predict neural responses in higher visual
DiCarlo. Performance-optimize cortex. Proceedings of the National Academy of Sciences, 11 111:8619-866 9624, 6 2014. ISSN 0027- 8424. doi: 10.1073/pnas. 320T3//pna..1401122 1403112111.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 CONSENSUS PROCEDURE FOR AGGREGATING RESULTS ACROSS BAYESIAN NMF ITERATIONS
Following the approach of Kotliar et al. 2019) and Khosla al. (2022), aggregate results across N = 50 iterations Bayesian NMF using a consensus algorithm. This algorithm processes the estimated component response matrices from all runs by horizontally stacking them into a large matrix, where each row represents a stimulus and each column corresponds to a component from of the iterations. The total number of columns equals the number of iterations (50) multiplied by the number of components per iteration (20).
To ensure the stability of the components, the algorithm first identifies and removes unreliable com- ponents through an outlier detection procedure. Components are considered outliers if their Eu- clidean distance from the nearest neighbors exceeds threshold of 0.8, indicating they cannot be replicated across runs.
Once outliers are removed, the remaining components from all iterations are grouped into C clusters. The median of each cluster is then selected as the consensus response profile for the corresponding component.
To obtain the final voxel (or unit) weight matrix for each subject (or network), we identify in each individual Bayesian NMF run run-the the component indices that show the highest correlation with the C consensus component response profiles. The voxel/unit weights for these indices are normalized (to sum to 1) and averaged across runs, yielding consensus voxel/unit weights for each component.
untrained
dino vit
resnet50
moco
sinclr
alexnet
a
component linear encoding
component rsa
0.3
simi Siiitarity (r)
8.4
8.3
8.2-
0.1
8.8
dorsal
lateral
ventral
Lateral
Figure 7: Standard aligmment measures using Bayesian NMF components. Representational alignment between DNNs and the dorsal, lateral, and ventral streams, as measured by (a) linear encoding and (b) representational similarity alignment (RSA). In contrast to Figure these methods use component responses in place of unit/voxel responses.
A.2 EVALUATION BAYESIAN NMF PRE-PROCESSING
To further assess the effect of chosen matrix decomposition method, measured representa- tional alignment using standard measures but with component-rafee than unit/voxel unit/voxel-responee rer-posee (Figure 7 Bayesian NMF pre-processing has some effect the general pattern of alignment but does not completely account for the results given by Image Connectivity Matrices and SCA.
A.3 SIMULATIONS ON THE RECOVERY OF SPARSE LATENT COMPONENTS
We evaluated the effectiveness of various matrix factorization techniques in recovering the true latent factors from data generated mixture of sparse latent components and sparse mixing coefficients. We construct the data matrix X E mm,, as product of two sparse matrices: L E mg. which simulates the response of & components m stimulus conditions and A k which contains the mixing coefficients. An additive noise component is included, resulting the following equation:
X LAT +
14

---

Published as a conference paper at ICLR 2025
Here, each entry of the matrices {L, A} is drawn independently from random uniform distribution, while the noise term é sampled from a normal distribution with o = 0.01.
Subsequently, we applied three matrix factorization non-negative matrix factorization (NMF), and Bayesian NMF-ou the simulated data matrix X, setting the number of components to & for each method. To assess the similarity of the inferred component response matrix L' from each method and the ground truth latent factor matrix L, we computed the following similarity score:
cetttodss-pinncippe component analysis (PCA),
S L') max
Ti)
where ry = corr(L (L. n.
In this context, represents k permutation matrix, allowing to find the optimal alignment between the recovered and true component matrices under permutations.
A.4 ROTATION MATRICES
We construct the rotations in n. dimensions as the product of m(n-1) plane rotations
(Pinchon &
2
Siohan 2016
Clements al. 2016 Quessard et 2020
f(0 01.............................. 1,0
II (o,()
where (O.,) denotes the rotation in the i.j plane embedded within the n-dimensional repre- sentation, characterized by the angle 0z.j- Each 2D rotation affects only two coordinates time, leaving the others unchanged. For example, to rotate 3-dimensional representation, combine individual rotations across each 2D plane within the 3-dimensional space. By parameterizing these rotations with angles (01,2, 01,3, 02,3), we can express (0,3(01.)) as follows:
10,3(0,,3) (
cos 01.3 sin 01.3
sin 01,3
cos 01,3/
The overall rotation matrix can then be obtained by multiplying these matrices:
R =
By sampling rotation matrices in this manner, we control fi.j to set the magnitude of each 2D rotation. We set 0z, be constant across all i,j, choosing from (лr /20, π 1/40, (0, π n/60, x/80]. For each of these angles, construct the final rotation matrices by composing varying number (c) of 2D rotations, where is drawn from 10 linearly spaced values between 0 and "(n-l), Here,
x(n-1) corresponds to the rotation across all possible planes in n-dimensional representational 2
space.
A.5 SIMILARITY OF ROTATED COMPONENTS
We derived components from the same set of simulated data before and after applying a set of random rotations (see A.4 We used both PCA and Bayesian NMF derive two components from sparse, three- diree--imniional data, and applied the appropriate inverse transform to correct for rotation. Finally, find the optimal mapping of original to rotated components, and measured their similarity using Pearson's 's We also measured the relationship between components using cosine similarity and found similar results.
15

---

Published as a conference paper at ICLR 2025
0.4
02
0.4
20.0
2orel
urtrate
7.5
1.2
0.5
0.5
6.0
Taterel
stowet
Figure 8: Empirical measurements of sparsity in NMF decomposition. We measured the sparsity of NMF-derived components by characterizing the distributions of W and by their (a) dispersion, (b) kurtosis, and (c) skewness
A.6 EMPIRICAL MEASUREMENT SPARSITY
We characterized the distribution of W and R for each decomposition using empirical measures of sparsity and statistical measures of kurtosis and skew.
We follow the method defined by Hoyer 2004) to measure the dispersion of given vector R":
sparsity(x)) Vn E(I,))//VV
Vn
which measures relationships between the L1 and L2 norm and evaluates to 1 if contains only single non-zero element.
We also used the th sample moment m, to quantitatively measure the statistical properties of æ. Specifically we used m2, M3, and my to measure excess kurtosis x and skewness
m,
T)", (x)
n
m4
m
K
and
? =
m
m
A.7 BEHAVTORAL SALIENCY RATINGS
To further test our interpretations of component response profiles, we collected subjective salience ratings to the shared set of 1, 000 images viewed by all four subjects. Ratings collected from
16

---

Published as a conference paper at ICLR 2025
b)
dorsal
1004
Figure 9: Extended response profiles of neural components. Ten of the images that produced the highest response for given component in each of the four subjects. Response profiles are shown for the components with highest inter-subject consistency in the (a) ventral, (b) lateral, and (c) dorsal streams.
total of 125 subjects via Prolific, online crowd-sourcing platform. In the experiment, participants were asked to rate a random sample of 250 images according to single prompt on Likert scale of 1 (not all) (very much). Prior to the onset of the first trial, prompt was randomly sampled from the following list:
1. To what extent is motion occurring in this image?
2. How prominent are hands and/or hand-directed actions in this image?
3. To what extent could you reach the contents of this image moving only your arms
4. To what extent does the image depict place (either indoor or outdoor)?
5. How prominent in this image are groups of people interacting with each other and/or groups or people engaged in joint activity?
The initial prompt was kept the same throughout the experiment. Images remained on-screen until the participant provided response. Next, displayed feedback showing the given score, followed by a 500 millisecond inter-stimulus interval.
In addition the behavioral data we collected here, we also used set of saliency ratings obtained in earlier project that included prompts on scenes, faces, bodies, text, and food. Images were similarly rated by online participants, or by two independent experts scene-selective cortex who were asked to predict how strongly the image would drive the scene-selective cortex. Further details on the experimental method can be found in Khosla et al. (2022).
17

---

Published as a conference paper at ICLR 2025
Untrained
b)
Lexnet
c)
noco
d)
zeanet
vit
dine
sinelz
Figure 10: Response profiles model components. Ten the images that produced the highest response for a given component in each model. Separate activations were extracted for each of the four subject's s 10, 000 images, leading to four sets of components for each model. Response profiles shown for the components with highest inter-subject consistency in (a) untrained AlexNet, and pre-trained (b) AlexNet, (c) ResNet-50 (w/ MOCOv2), (d) ResNet-50 (supervised), (e) Vision transformer (ViT, supervised), ViT (w/ DINO), and ResNet-50 (w/ SimCLR) models.
18


Reviews:
[{"review_id": "02l3EoTSGm", "reviewer": "Reviewer_23vv", "summary": "In this work, the authors apply a novel method (Sparse Component Alignment) to the problem of differentiating neural response profiles across three distinct streams of the human ventral visual pathway -- all of which are strongly theorized (after decades of intensive neuropsychological and neuroimaging work) to be functionally and representationally distinct, but which many statistical methods do not so saliently differentiate. The authors also assess the advantage of this method in the domain of brain-to-model mappings -- another area where what seems a priori to be salient functional and representational differences often fail to manifest with the statistical methods most popular for assessing them.", "questions": "- There are a number of software packages that allow for the comparison of many different neural network models. Why not try this method on more of them? A graph that shows the differentiability of models more canonical methods versus SCA would (in my mind) be a powerful amplifier to this work.)\n- The authors mention that the still-remaining non-differentiability of the lateral and dorsal streams is something that we likely need further / better / different brain data assays to meaningfully address. I am entirely on board with this argument, but one natural question that arises here is whether SCA (or any method) can be used not only for differentiation on current data, but for proposing different kinds stimuli that could maximize differentiation in future data? (Think controversial stimuli / selection). \n- A bit of a follow-up on the last point (perhaps something worth addressing depending on the reaction of other reviewers). If at end of day, certain kinds of differentiation can only be achieved with the right kinds of data... doesn't this somewhat undermine one of the central motivations for using these \"hypothesis-free\" methods in the first place?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper constitutes an important contribution and meaningful step towards resolving a deep and lingering concern in computational neuroscience: How do we reconcile data-driven methods with data-derived theory? Methods developed by machine learning researchers and proponents of \"hypothesis-free\" (a somewhat unfortunate term) structural decomposition / discovery can sometimes clash or produce results counterintuitive to or directly at odds with observations made by experimental neuroscientists and clinicians in well-documented and amply-replicated experimental scenarios. And the issue of \"resolution\" tends to be one of the defining attributes of this debate. The finding by the authors that there are as-of-yet under-explored statistical methods that advance the mission to use \"less biased\" / less error-prone \"hypothesis-free\" data-driven techniques (with scare quotes galore) in a way that nevertheless accords with theoretical priors seems (in my mind) to open the door to new discovery, and far less \"two steps forward, one step back\"-kind-of debate along the way.", "weaknesses": "While I do not necessarily fault the authors for this (especially given the importance of the first-order business that involves showing this method to be intuitive on the task of differentiating response profiles across the well-separated \"streams\" of visual cortex), the use of only a few neural network models does not give particularly strong confidence that this method is relevant yet to the differentiation of otherwise very similar-looking brain models (another problem canonical decomposition have heretofore struggled with.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4uTU6JKCk", "reviewer": "Reviewer_EmTJ", "summary": "The paper introduces a new method, Sparse Component Alignment, for comparing models to brain measurements. The method is used to compare fMRI measurements from the Natural Scenes Dataset with 3 Deep neural network architectures trained on ImageNet-1k. Compared with linear (Euclidean) metrics, or RSA, the proposed method stronger alignment with the ventral stream compared with the dorsal or lateral streams.  The authors also compare to behavioral data (the Meadows dataset - sec 3.4), and find similar aligment with the ventral stream.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Development of methods for brain-model comparison is an important area. The authors have developed a promising new method, and have used it to demonstrate  that DNN recognition networks are more similar to ventral stream than dorsal or lateral. Experiments comparing the three brain regions, for 3 different DNN architectures, and 3 different similarity measures, are extensive.", "weaknesses": "The contributions of the paper are both methododological and scientific.  I think both have potential, but found both to be somewhat limited as presented.  \n\n- Method: The intuitive motivation for the SCA method (that it is sensitive  to changes in the axes of the representation) makes sense, but after reading it 3 times, I still don't understand how the mathematical construction of SCA leverages that, nor the extent to which the results should really be attributed to the BNMF pre-processing.  If the method is to be relied upon, I think practioners will want either a derivation tying it to some objective, or some sort of bounds on senstivity to axis changes, or an analysis of why it exhibits this sensitivity to axis changes.  Alternatively, one could provide a more thorough set of empirical tests showing the conditions under which the method works or fails.  For example, how sensitive are results to the choice of the BNMF dimensionality? The simulations provided (fig 2) are quite limited.\n\n- Scientific result: The main result, that recognition models (across three different architectures) are more aligned with ventral stream than dorsal or lateral streams is nice, albeit expected.  RSA already provides this result, in a slightly weaker form (text on p. 8, fig 5).  It was not clear to me how to fairly compare the r-values achived by the two methods.  Nor (as mentioned above) how much the result arises from the BNMF pre-processing.  What if you compute RSA on the BNMF components?  \n\nWriting/presentation could be improved.  \n- please indicate up front (abstract/intro) that you're analyzing fMRI data (it was not apparent until sec 2.1).\n- I think you could reduce space spent describing Bayesian NMF, which is a known method \n- Provide a more thorough description of the SCA calculation - I read it several times and am still not fully understanding the construction.\n- Provide a more  thorough description of the results shown in Fig 4", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z7pnB0ySeh", "reviewer": "Reviewer_d41b", "summary": "This paper is an attempt to solve the apparent contradiction between the observation that the ventral, dorsal, and lateral streams in human vision have different functions; and the fact that neural activity in each of these tends to be similarly correlated with deep neural-net (DNN) activity trained on a single task. The authors use Bayesian non-negative matrix factorization (NMF) to identify the different selectivities for the three visual streams from fMRI data, confirming previous results. They next introduce a different way of assessing the similarity between neural-net activations and brain recordings that finds that DNNs correlate much better with the ventral stream than with the dorsal and lateral ones.", "questions": "* Why is NMF worse than BNMF with simulated data (Fig. 2)?\n  * ...and why is the similarity between inferred and actual components not perfect? If I understand correctly from eq. (4), the data should exactly conform to the model here.\n* I don't understand the $\\theta$ colorbar in Fig. 2b: what is the range actually? I see it going from $\\pi/2\\theta$ to $\\theta$, which I can't really parse.\n* Why is the NMF bar missing from most conditions in Fig. 3b?\n* In section 3.2 (line 319), is it standard NMF or Bayesian that is employed? If standard, why not Bayesian?\n* The color palette for the lateral stream (Fig. 4b) should go from light pink to red instead of yellow to red, to avoid a confusing change in hue (and also to be consistent with the other two panels).\n* The tick and axis labels in most of the figures (Figs. 2, 3, 4, 5, 6) should be significantly bigger.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Originality**\n* the authors observe that the rotational symmetry that is built into existing measures of representational alignment disregards the fact that biological constraints (such as the cost of neural wiring) would in fact be very sensitive to rotations that could, e.g., convert a very sparse representation (that is easy to read out with few neurons) to a very broad one.\n\n**Quality**\n* the work seems carefully done, with controls to check that the proposed methods recover known results and that they work on simulated data.\n\n**Clarity**\n* the paper is very well written with overall clear figures and detailed explanations.\n\n**Significance**\n* understanding the relation between artificial neural nets and their biological counterparts is important both for using the artificial networks to aid in understand the brain, and conversely, as a potential route to gleam inspiration from biological neural circuits to apply to artificial nets.", "weaknesses": "* Fig. 2 is really confusing\n  * Both panels seem to mix together the question of inferring the original components correctly with the question of sensitivity to rotations.\n  * I would suggest separating the two parts:\n    * First, take out the \"rotate\" part from panel a, focus only on reconstruction, and show bar plot of similarity for different methods.\n    * Second, add explanation of rotation and show its effect on SCA.\n      * Might also be useful to add lines to show that NMF and PCA don't change at all when you rotate.\n  * It's very confusing to have the bars colored in Fig. 2b, as the shades of gray employed there do not match the ones from Fig. 3.\n    * If the shade only indicates the similarity level, then it is redundant with the height of the bars, and I would suggest getting rid of it.\n*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yXTPHmnRy0", "reviewer": "Reviewer_PbjK", "summary": "This paper introduces Sparse Component Alignment (SCA) for quantifying alignment between two learning systems through a data-driven approach (non-negative matrix factorization). This is done by identifying dominant components that capture sparse neuronal activity. SCA is specific to a system’s native axes of neural tuning. The authors find components selective for faces, scenes, bodies, food, and words in the ventral stream. Furthermore, representations generated by different deep neural networks are found to have a greater alignment with the ventral over dorsal or lateral representations.", "questions": "How big is the ViT evaluated—is it base-16?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Quantifying model to brain similarity is an important research direction that has seen substantive progress in the last few years, and I appreciate the author’s contributions towards the same.\n- The manuscript is written in a very straightforward way, and it is easy to follow.\n- SCA is biologically and mathematically grounded without a priori hypotheses on neural response profiles.\n- The authors show that DANNs, at least the ones tested, show a greater similarity to the ventral stream under SCA over other methods.", "weaknesses": "- Lines 46-9: “How can the same computational model capture the diversity of function across these pathways?” Could the authors ground this question more concretely given that the Topographic DANN (Margalit et al., 2024) was NOT trained to perform only object recognition but instead also imposed a spatial constraint on model units to find separation of visual processing into distinct streams? It was precisely topography and self-supervision that led to that outcome.\n- I would have preferred seeing the TDANN’s similarity to the different streams computed using SCA, given that it was cited in the introduction. More generally, I would have preferred seeing more breadth in the computational models evaluated—what do the authors find when they use DANNs that are trained not for object recognition but instead for object detection (such as on the Pascal VOC or MS-COCO datasets) or action recognition (on the Kinetics dataset). Do these models, under SCA, show a higher (or at least somewhat better, given limitations of available neural data) match to the dorsal and lateral streams?\n\n**Things (manuscript writing) that can be improved but did not impact my score:**\n- I was having a really hard time looking at the figures since the particular font used makes visibility really low unless the figure is zoomed in a lot. It would be nice if the authors could use a darker and bolder font like Arial or Calibri for the figures.\n- Line 97 (and similar phrases throughout the manuscript): “SCA reveals clear alignment of DNNs to the VVP …”. The authors should make it clearer that only those DNNs that have been (a) evaluated in this work, and (b) trained to perform object categorization. DNNs in general is a very broad term given only 4 trained networks were evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)