Title: Brain Dissection: fMRI-trained Networks Reveal Spatial Selectivity in the Processing of Natural Images
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computational Neuroscience

Deep Neural Networks

Visual Neuroscience

Visual Streams

Scene Perception

Brain Imaging
Abstract: The alignment between deep neural network (DNN) features and cortical responses currently provides the most accurate quantitative explanation for higher visual areas. At the same time, these model features have been critiqued as uninterpretable explanations, trading one black box (the human brain) for another (a neural network). In this paper, we train networks to directly predict, from scratch, brain responses to images from a large-scale dataset of natural scenes (Allen et. al., 2021). We then use "network dissection" (Bau et. al., 2017), an explainable AI technique used for enhancing neural network interpretability by identifying and localizing the most significant features in images for individual units of a trained network, and which has been used to study category selectivity in the human brain (Khosla & Wehbe, 2022). We adapt this approach to create a hypothesis-neutral model that is then used to explore the tuning properties of specific visual regions beyond category selectivity, which we call "brain dissection". We use brain dissection to examine a range of ecologically important, intermediate properties, including depth, surface normals, curvature, and object relations across sub-regions of the parietal, lateral, and ventral visual streams, and scene-selective regions. Our findings reveal distinct preferences in brain regions for interpreting visual scenes, with ventro-lateral areas favoring closer and curvier features, medial and parietal areas opting for more varied and flatter 3D elements, and the parietal region uniquely preferring spatial relations. Scene-selective regions exhibit varied preferences, as the retrosplenial complex prefers distant and outdoor features, while the occipital and parahippocampal place areas favor proximity, verticality, and in the case of the OPA, indoor elements. Such findings show the potential of using explainable AI to uncover spatial feature selectivity across the visual cortex, contributing to a deeper, more fine-grained understanding of the functional characteristics of human visual cortex when viewing natural scenes.

Brain Dissection: fMRI-trained Networks Reveal Spatial Selectivity in the Processing of Natural Images

Abstract

The alignment between deep neural network (DNN) features and cortical responses currently provides the most accurate quantitative explanation for higher visual areas [1, 2, 3, 4]. At the same time, these model features have been critiqued as uninterpretable explanations, trading one black box (the human brain) for another (a neural network). In this paper, we train networks to directly predict, from scratch, brain responses to images from a large-scale dataset of natural scenes [5]. We then use 'network dissection' [6], an explainable AI technique used for enhancing neural network interpretability by identifying and localizing the most significant features in images for individual units of a trained network, and which has been used to study category selectivity in the human brain [7]. We adapt this approach to create a hypothesis-neutral model that is then used to explore the tuning properties of specific visual regions beyond category selectivity, which we call 'brain dissection'. We use brain dissection to examine a range of ecologically important, intermediate properties, including depth, surface normals, curvature, and object relations across sub-regions of the parietal, lateral, and ventral visual streams, and scene-selective regions. Our findings reveal distinct preferences in brain regions for interpreting visual scenes, with ventro-lateral areas favoring closer and curvier features, medial and parietal areas opting for more varied and flatter 3D elements, and the parietal region uniquely preferring spatial relations. Scene-selective regions exhibit varied preferences, as the retrosplenial complex prefers distant and outdoor features, while the occipital and parahippocampal place areas favor proximity, verticality, and in the case of the OPA, indoor elements. Such findings show the potential of using explainable AI to uncover spatial feature selectivity across the visual cortex, contributing to a deeper, more fine-grained understanding of the functional characteristics of human visual cortex when viewing natural scenes.

1 Introduction

Humans possess an extraordinary ability to process and interpret visual information; however, our understanding of how such information is represented and processed within the brain remains incomplete. While advances in modeling visual cortex have been made through the use of abstract stimuli and the study of evoked neuronal responses, these efforts have primarily focused on low-level visual features [8] or high-level categorical selectivity [9]. In reality, the brain utilizes numerous ecologically-relevant intermediate features to perceive the world, such as 3D information, physical relationships, and object properties [10, 11].

Characterizing high-level areas in the visual cortex presents challenges due to the circular problem of finding optimal stimuli. Although deductive [12, 13, 14, 15, 16] and inductive [17, 18, 19, 20] approaches have identified a wide range of functionally-defined brain regions, these methods are not scalable for exploring a wider range of brain areas and diverse features. To address these limitations, we leverage 'network dissection' [6], a method that enables exploring the feature selectivity of neural network units at scale. Combined with brain response-optimized networks trained on fMRI data for subjects viewing natural images, this method offers a hypothesis-neutral approach to uncovering feature tuning properties across the visual system [7, 21]. While Khosla and Wehbe [7] demonstrate the effectiveness of network dissection for validating known category selectivities, here we report that this combined approach, brain dissection , can be extended to examine more granular features, such as depth, curvature, and object relations.

We apply brain dissection across the visual cortex to uncover functional differences between regions. In particular, the human visual system is believed to be organized into distinctive processing streams: the parietal, lateral, and ventral visual streams. While previous experimental and modeling work has made progress in characterizing each of these streams [22, 23, 24, 1, 25, 26], at a fine-grained level, their distinct roles in visual processing remain under-specified. Functional networks within these streams likewise remain functionally under-specified. For example, precise differences among the three components of the 'scene network', the occipital place area (OPA), parahippocampal place area (PPA), and the retrosplenial complex (RSC) [27] remain unclear for naturalistic, large-scale viewing of scenes with prior work providing primarily broad functional characterizations in terms of the representation of scene layout [28, 29, 30], 3D features [20, 31, 32, 33], scene objects [15, 7], and navigational affordances [34, 35].

Utilizing a hypothesis-neutral approach with brain response optimization, we demonstrate feature selectivity differences across multiple visual streams and scene regions, encompassing a wide variety of highly specific feature properties. Importantly, our findings are based on subjects viewing natural images and do not require a separate experiment for each category or visual feature.

A

Training

x#voxels

voxel-specific

spatial weights

voxel-specific

featuremap

-β—β

voxel-specific 1x1 conv

ROl-specific

featureweights

convolutional

neuralnetwork

B

Dissection

Surface

Depth

Normals

voxel-specific

featuremap

Category

voxel-specific

Relations

binarymask

Attributes

overlapping3Dcurvature

Shading

Threshold

region

convolutionalfeature weights

ROl-specific

voxel-specific

1x1conv

neuralnetwork

2 Brain Dissection

Brain dissection provides a hypothesis-neutral approach for identifying the most significant image features for predicting the response of a specific voxel. The method trains a convolutional neural network tailored to predict voxel responses within a defined sub-region. By training a backbone network for this sub-region and incorporating a linear readout for each voxel, the network learns to extract image features crucial for predicting each voxel (Fig. 1A). After training, we 'dissect' the network by inputting held-out images and extracting regions that the network considers most relevant for each voxel (Fig. 1B). Subsequently, we analyze properties of the voxel-selective regions within the images. Our training and dissection procedure closely follows Khosla and Wehbe [7].

2.1 fMRI Response-Optimized Network Training

We trained a set of neural network encoding models to predict fMRI voxel responses to NSD images (Fig. 1A). Visual cortex was divided into sub-regions based on the Human Connectome Project (HCP) atlas [36], with a separate convolutional neural network (CNN) backbone shared across all voxels within each sub-region. For each voxel in a sub-region, a linear readout was applied to the backbone output features to predict individual voxel responses to the images. The linear readout was factorized into voxel-specific spatial and feature dimensions, as described by Klindt et al. [37]. Spatial weights correspond to the most responsive locations of a voxel in an image, while feature weights represent the visual information to which a voxel is most responsive.

Given the activations of the last layer of the ROI-specific backbone network ( A ∈ R C × H × W ), the spatial weights for a voxel in the ROI ( w s ∈ R H × W ) were multiplied channel-wise to A . The feature weights ( w f ∈ R C ) were applied to each spatial position of A . The output is a voxel-specific feature map, which was summed and added to a learned bias to obtain the predicted voxel response. The network was trained using mean-squared error with the true voxel response. We trained each network using NSD data from 8 subjects, including 68,400 training images, and 3,600 validation images. We show predictivity performance of our networks in Appendix Section S3.2.

While we use a CNN for all the analysis in the main paper, we also report results for the scene regions using a transformer architecture, and attention heads for the voxel-specific feature maps with consistent results in the Appendix Section S3.1.

2.2 Dissection Procedure

During dissection (see Fig. 1B), the spatial weights were discarded and only the feature weights were used to focus the analysis on visual selectivity for each voxel. For each voxel k , we obtained its corresponding ROI-specific backbone network f roi (with k ∈ roi ) and voxel-specific feature weights w k f ∈ R C . For every input image x in the evaluation dataset, we passed the image through f roi to obtain activations A . The feature weights ( w k f ∈ R C ) were then applied to each spatial position of A using a 1x1 convolution. The output is a voxel-specific feature map A k ∈ R H × W . The top quantile level T k is determined such that P ( A k > T k ) = 0.01 over every spatial location of the activation map.

To compare a low-resolution voxel activation map to the input-resolution annotation L c for some concept or measure c (e.g., L c could be a category mask, depth map, surface normal map, etc.), the activation map is scaled up to the mask resolution S k from A k using bilinear interpolation. S k is then thresholded into a binary segmentation: M k = S k ≥ T k , selecting all regions for which the activation exceeds the threshold T k .

As in Bau et al. [6], to score each voxel k for a discrete category concept c , we compute the data-set-wide intersection over union (IOU) score:

To score the selectivity of each voxel k against a continuous concept c (e.g., depth), we compute the median value within the region defined by M k , and take the average median across all images as the selective value for that voxel:

where L c ( x )[ M k ( x )] represents the values in L c ( x ) defined by the mask M k ( x ) .

We exclude voxels that lack selectivity across all evaluation set images by comparing the distribution of L_c masked by M_k against a uniformly distributed (full image) mask. This exclusion used the Kolmogorov-Smirnov test, with a significance threshold of p>0.01.

While we present results using the Network Dissection interpretability method [6] throughout the main paper, we demonstrate in Appendix Section S3.1 the feasibility of using other interpretability techniques. These include gradCAM [38] and transformer raw attention [39], with which we obtain consistent results for the scene regions.

3 Experiments

3.1 Brain Data

fMRI dataset. We used the Natural Scenes Dataset (NSD) [5], which consists of high-resolution fMRI responses to naturalistic images from Microsoft COCO [40]. NSD contains 7T fMRI responses (1.8 mm, 1.6 s) from 8 subjects who each viewed 9,000-10,000 distinct color natural scenes (22,000-30,000 trials). Subjects fixated centrally and performed a long-term continuous image recognition task. The noise ceiling (NC) was estimated in each voxel as described in Allen et al. [5]. We only include voxels with NC ≥ 10% variance. Notably, the dataset contains complex and sometimes crowded images of various everyday objects in their natural contexts at varied viewpoints. The stimulus set is thus more typical of real-world vision and allows us to characterize neural representations and computations under more ecological conditions.

Regions of Interest (ROIs). We used the 'streams' anatomical atlas from NSD to define three ROIs that cover the parietal, lateral, and ventral visual streams (mid- and high-level ROIs were grouped). We then split each of the ROIs into sub-regions using the HCP atlas, including any ROI in the HCP atlas with at least 50% voxel overlap with any of the streams. We added a fourth ROI, which we call 'Medial', and took all HCP regions overlapping with RSC. We further split the 'Ventral' ROI into 'Ventrolateral' and 'Ventromedial', where 'Ventromedial' included HCP regions overlapping with PPA. A separate network was trained for each of the HCP sub-regions. We also examined three scene ROIs - RSC, OPA, and PPA - as defined by the category functional localizer.

3.2 Measures

Unless otherwise specified, we evaluate on the Places365 dataset [41], and use the top 1000 images per sub-region per subject as determined by the predicted responses to the images by the sub-region network. Thus, the evaluation takes into account: 1) the top-most-activating images for a region; 2) the areas of the image that most contribute to the response as determined by network dissection. We perform brain dissection on the following measures:

Depth. We analyze the selectivity of depth, which refers to the perceived distance between the viewer and different regions of the scene. To obtain depth maps for each evaluation image in Places365, we use the ZoeDepth metric depth estimation network [42], which has been shown to exhibit exceptional zero-shot generalization performance on indoor and outdoor scenes. We evaluate two measures of depth: 1) metric depth , which represents the absolute depth in meters for each pixel in the image, and 2) relative depth , which denotes depth maps normalized to a range of 0-1 for each image. Consequently, absolute depth captures the overall nearness or farness across the images, while relative depth focuses on the nearness or farness within an individual image.

Surface Normals. Surface normals, vectors perpendicular to surfaces, are essential for spatial cognition, as they indicate surface orientation and reflect the spatial coordinate system. Using the XTC network [43], we estimate normals for each Places365 image. The x, y, and z components are segmented into 512 bins (8 per component; 8 3 = 512 combinations), and voxel selectivity is reported as the bin number.

3D Curvature. Curvature features provide information about 3D structure and are invariant under rigid transformations. We use the XTC network [43] for estimating principal curvatures.

Shading. One way to infer scene geometry is 'shape from shading' [44, 45] using the intrinsic image decomposition I = AS , where S is a shading function parameterized by lighting and depth. We use the XTC network [43] for estimating shading.

Segment Anything [46] was used to obtain segmentation masks from the bounding boxes for each of the annotations.

4 Results

MetricDepth

Surface Normals

Guassian Curvature

Shading

2

6

8

10 meters

40

60

80

100

120

0.050.10 0.15 0.20

0.25

0.30.40.5

0.60.70.8

Medial

Ventrolateral

Parietal

Ventromedial

Lateral

RSC

OPA

PPA

We employed brain dissection to investigate the scene features represented throughout visual cortex. Our investigation focused on the Ventromedial, Ventrolateral, Lateral, Parietal, and Medial regions, as well as the scene network - PPA, OPA, and RSC. Our initial examination centered on identifying and quantifying the differences between these regions based on four spatial measures, namely, depth, surface normals, 3D curvature, and shading. These measures were chosen to cover a range of ecologically-relevant spatial features. Following our spatial analysis, we explored how these identified differences are reflected in the processing of object categories, attributes, and spatial relations. This aspect of our study sheds light on the relationship between the observed variations in spatial features and the resultant impact on higher-order visual processing - specifically how we perceive and interpret objects and their relationships within a given scene. Finally, we quantified the spatial clustering of voxels for each measure to better understand the spatial organization for different visual features.

4.1 3D Spatial Dissection

4.1.1 Brain Flatmaps

To gain a better understanding of how 3D structure is represented in the brain, we used brain dissection to quantify the selectivity for each voxel across the high-level visual cortex for the four

spatial measures. This process allowed us to create a detailed selectivity map for each voxel that reveals how 2D inputs are transformed into 3D representations that enable reasoning about the physical world. Figure 2 displays these selectivity maps on a cortical flatmap of the brain. We observe that Lateral areas demonstrate selectivity for close distances, aligning with the overlap of Lateral ROIs with regions selective for bodies, frequently featured in the foreground of images. Notably, we observe that the depth selectivity for the Lateral and Ventrolateral ROIs are similar, and stand in contrast with the rest of the ROIs. This pattern is also true for the other spatial metrics (Surface Normals, Gaussian Curvature and Shading) we study, where the Lateral and Ventrolateral appear as a distinct cluster from the Ventromedial, Parietal and Medial ROIs. In contrast, we observed high selectivity for far distances in the Medial and Ventromedial areas, suggesting a specialized role for behaviors that require processing and understanding far-reaching aspects of a visual scene (e.g., navigation). Additionally, we observed a strong gradient in the selectivity for depth in Ventral areas. The gradient transitioned from near-preferring depths in the Ventrolateral ROI to far-preferring depths in the Ventromedial ROI. Such a gradient is likely to be important for building up scene representations that capture a wide variety of behaviorally relevant affordances. Finally, we observed distinct patterns related to the processing of surface normals. Large areas of the Lateral and Ventrolateral regions showed a preference for surface normals in the negative X direction, pointing back at the camera (again, consistent with the preference of these areas for bodies and objects). In contrast, the Parietal and Medial areas predominantly preferred surface normals oriented in the positive Z direction (pointing upwards) and along the Y direction (pointing right and left).

MetricDepth

RelativeDepth

SurfaceNormals

Shading

GuassianCurvature

4

6

**

*

*

***

0.3

0.2

***

**

**

***

***

100

50

75

***

***

***

**

**

0.8

0.6

0.4

**

***

***

***

**

0.15

0.10

**

**

**

**

***

2

**

0.1

25

0.2

0.05

0

medial

lateral

Lateral

Parietal

Medial

0.0

medial

lateral

Lateral

Parietal

Medial

0

medial

lateral

Lateral

Parietal

Medial

0.0

medial

lateral

Lateral

Parietal

Medial

0.00

medial

lateral

Lateral

Parietal

Medial

4.1.2 Spatial Dissection of High-level Visual Areas

We quantify the differences between the high-level visual ROIs in terms of mean ± standard deviation for four 3D spatial measures: depth (metric & relative), surface normals, shading, and Gaussian curvature (Fig. 3). We observe differences in depth preferences between the Medial and Ventrolateral ROIs. Medial and Ventromedial areas exhibit significantly larger absolute and relative depth selectivity compared to the Ventrolateral and Lateral regions ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). Conversely, Ventrolateral and Lateral areas have significantly lower average absolute depths than the Medial and Ventromedial areas, as depicted in Figure 3 ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). This disparity highlights the specialized roles these regions play in processing natural images across varying depth distributions, catering to both nearby and distant objects. Concerning the processing of surface normals, we observe that the Ventromedial region significantly prefers normals indicative of surfaces pointing in the 'up' direction ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). Parietal and Medial areas show a stronger preference for surface normals oriented to the left and right compared to the other ROIs ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). In contrast, the Ventrolateral and Lateral regions have a preference for surface normals pointing back at the camera ( p < 0 . 01 ; post hoc Tukey test; N = 8 ).

Examining the shading preferences across the ROIs, we again notice a clear distinction. Ventrolateral areas prefer the lightest shading, while Medial and Ventromedial areas prefer the darkest shading ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). This observation reaffirms the notion of functional segregation in the visual cortex, with different regions specializing in processing specific visual attributes, in the case of shading, perhaps reflecting additional cues to scene depth.

Lastly, we consider Gaussian curvature, a measure that captures the degree of 3D curvature of a surface. Higher Gaussian curvature implies a more spherical geometry. Ventrolateral and Lateral areas significantly prefer higher Gaussian curvature (Fig. 3), indicating a preference for more spherical

surfaces ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). In contrast, Parietal and Ventromedial areas prefer lower Gaussian curvature, suggesting a preference for flatter surfaces ( p < 0 . 01 ; post hoc Tukey test; N = 8 ). Ventrolateral and Lateral areas may be encoding spatial features relevant to object appearance for both recognition and grasping, while Parietal and Ventromedial areas may be encoding scene layout in terms of large-scale surfaces and scene elements. Such findings add another layer to our understanding of how different areas of the visual cortex process and interpret 3D scene structure.

4.1.3 Spatial Dissection of Scene Areas

In this section, we delve into the differences between the scene ROIs (PPA, OPA, RSC) with respect to the four 3D spatial measures, providing a quantified analysis of their specialized roles in scene perception. We illustrate the voxel-selectivity on brain flatmaps (Fig. 4A) and plot the mean ± standard deviation ( N = 8 ) for each of the each ROIs and measures (Fig. 4B). These distributions and mappings elucidate the distinct patterns of selectivity across the scene ROIs.

There is a considerable increase in both metric and relative depth for RSC compared to both PPA and OPA, as evidenced in Figure 4B ( p < 0 . 001 ; post hoc Tukey test; N = 8 ). This pattern indicates that RSC in processing far-reaching information in scenes, consistent with the claim that RSC is preferentially involved in navigation [47]. Additionally, we observe a distinct shift in surface normal preference, with RSC showing a preference for more right/left surface normal bins compared to OPA and PPA, which favor upright surface normals ( p < 0 . 001 ; post hoc Tukey test; N = 8 ). This distinction underscores the functional segregation between these scene ROIs in interpreting surface orientation and suggest, again, that RSC may represent information supporting navigation, while OPA and PPA may represent information more relevant to interacting within a local region of space.

To illustrate these observations, we identified voxels that have the closest selectivity to the ROI mean for depth or surface normals for PPA, OPA, and RSC, and plotted the voxel's top five images by predicted response (Fig. 5). We include corresponding dissection masks below these on images and either depth or surface normal images. RSC exhibits a preference for far depths and more 'right/left' surface normals, indicative of vertical structures relevant to navigation. In contrast, OPA and PPA exhibit a preference for 'up' surface normals which are relevant to representing the local scene layout. Once more, these results highlight the distinct functional roles these areas play in scene perception.

A

MetricDepth

SurfaceNormals

GuassianCurvature

Shading

RS

PPA

PPA

2

4

6

8

10meters

40

60

80

100120

0.05 0.100.15 0.200.25

0.30.40.5.0.60.70.8

B

MetricDepth

RelativeDepth

SurfaceNormals

Shading

GuassianCurvature

***

***

***

***

150

***

***

*

***

*

n.s.

0.5

125

0.5

***

0.125

10.0

0.4

100

0.4

0.100

7.5

0.3

75

0.3

0.075

5.0

0.2

50

0.2

0.050

2.5

0.1

25

0.1

0.025

0.0-

PPA

RSC

0.0

RSC

0

RSC

0.0

PPA

OPA

RSC

0.000

RSC

PPA

OPA

RSC

PPA(MetricDepth)

OPA(MetricDepth)

RSC(MetricDepth)

Unit 1

Unit 2

Unit 1

Unit 2

Unit 1

Unit 2

Unit 3

Unit 4

Unit 3

Unit 4

Unit 3

Unit4

PPA

OPA

RSC

PPA (Surface Normals)

OPA(Surface Normals)

RSC(Surface Normals)

Unit 1

Unit 2

Unit 1

Unit 2

Unit 1

Unit 2

Unit 3

Unit 4

Jnit3

Unit 4

Unit 3

Unit 4

OPA

RSC

PPA

MedianIOu

0.012

0.016

0.014

0.016

0.014

0.012

0.010

0000

0.008

0.010

0.010

0.008

0.006

0.006

0.006

0.004

0.004

0.004

0.002

0.002

0.002

0.000

toilet

uo

white

to the left of

to theright of

floor

counter

table

poom

in front of

below

black

above

Ke6

0.000

to theleft of

behind

roof

window

UO

to theright of

le

white

large

blue

brick

trees

leM

0.000

to the right of

UO

white

floor

to theleft of

near

u

toilet

counter

poom

black

bathroom

below

window

Ventromedial

Ventrolateral

Lateral

Parietal

Medial

MedianIOU

429

4413

4.68

4004

4.62

8.099

84875

4.4910

8.9625

1.0t5

1.906

uo

p ya a0 0

MO

6.005

1.409

fuueam

upou

[aoey

山

vewom

护

Addey

4.05

1.080

uew

uewoM

Aoq

ue

1.004

1.090

l.mi

44906

4.2 Object Relations, Attributes and Categories

We utilize the GQA dataset [48] in concert with brain dissection to gain insights into the selectivity differences for object relationship, attributes, and categories within an image. The median IOU (mean ± standard deviation across 8 subjects) for the top 20 concepts for each ROI is reported in Figure 6. With respect to high-level visual ROIs, we observe distinct preferences: Parietal areas prefer inanimate and scene objects, such as walls and tables, that are more relevant to both navigation and physically interacting with objects. In contrast, Lateral areas exhibit a preference for person and animate objects, while Ventrolateral areas favor persons and objects - representations that support social interactions and scene understanding. When examining object relations, Parietal areas showed a strong preference for spatial relations such as 'in front of,' 'near,' 'in,' and 'above'. Lateral areas favor relations involving people, such as 'wearing,' 'holding,' and 'with'. In terms of attributes, Parietal areas show a preference for colors and materials, whereas Lateral areas favor animate attributes involving people, such as 'playing,' 'standing,' and 'happy'. These preferences are largely aligned with prior results showing Parietal preferentially processes spatial visual information [22], Lateral preferentially processes social visual information [24], and Ventral preferentially processes objects and faces [23].

As illustrated in Figure 6, within scene ROIs, OPA shows greater selectivity for indoor scene elements, such as chairs, tables, and floors. PPA exhibits a mix of indoor and outdoor preferences, whereas RSC shows a clear preference for outdoor scene elements like trees, buildings, and the sky, once again reinforcing a dichotomy between local space interactions and navigation. Furthermore, when

considering attributes, RSC again demonstrates more selectivity for 'outdoor' attributes compared to PPA and OPA, which show higher selectivity for indoor elements. These findings underscore the distinct functional implications of these ROI selectivity profiles for processing visual scenes.

4.3 Topographic Organization of the Spatial Preferences

We investigated the topographical organization of the spatial measures within each ROI. High-level visual areas are topographically organized, comprising multiple hierarchically organized areas that are selective for specific domains such as faces and scenes [49, 50, 12, 13, 14, 15, 16, 51]. However, whether and how this topographical organization applies to the spatial measures used here remains unclear. Given a specific type of spatial information, we explored: (a) if the voxels in an ROI are differently tuned for this type of information (e.g., sub-groups of voxels are tuned for different surface normal orientations); and if so, (b) does the tuning follow a smooth topographical organization with relatively large clusters processing similar information. To quantify the degree of clustering by selectivity similarity, for each measure we computed the average cluster proportion (cluster size normalized by total voxels in the ROI) (considering clusters > 5 voxels) that each voxel belongs to, and report the results compared to a random permutation of the voxels (Fig. 7). We employed connected components [52] on the 3D voxel data to determine whether two neighboring clusters belong to the same cluster. For continuous measures, such as depth, we consider two voxels the same if they fall within δ of each other. We used 1/100 th of the data range across all ROIs as the δ for each of the continuous measures. For discrete concepts, such as object category, we consider two neighbors to be in the same cluster if they belong to the same discrete category.

A cluster analysis was run for each ROI. We report the average cluster size for a given measure compared to the average cluster size for the measure when the voxels are randomly permuted ((Fig. 7; p < 0 . 05 paired t -test; N = 8 )). We visualize the spatial clusters on flatmaps: All of the high level visual areas showed clustering significantly different from random for all spatial measure, indicating that their voxels are tuned for different values of each measure in a way that is topographically organized. At the level of the scene areas however, RSC was not distinguishable from chance for metric depth, relative depth, shading, gaussian curvature and category. This suggests that except for surface normals, the voxels in RSC are tuned similarly. Similarly, the results suggest that the voxels in PPA are tuned differently only for metric depth, and the voxels in OPA are tuned differently for metric depth, relative depth and surface normals. These findings highlight the robust topographical organization within the ROIs, extending beyond traditional domain-specific areas. Critically, this extended organization highlights the fact that, as a whole, visual cortex is representing a broad range of ecologically relevant properties that support complex behavior [53].

A

Parietal Clusters (Metric Depth)

Parietal Clusters (Surface Normals)

B

MetricDepth

Relative Depth

0.4

size mean

size mean permute

0.3

'su

Parietal ClusterswithPermutations

(MetricDepth)

Parietal ClusterswithPermutations

(SurfaceNormals)

0.15

***

0.2

0.1

***

**

8**

*8*

***

0.0

0.00

lateral

Lateral

Parietal

Medial

PPA

o

lateral

Lateral

Parietal

Medial

SurfaceNormals

Shading

GaussianCurvature

Category

0.25

n.5.

0.5

n.s.

n.s.

n.S

0.25

0.4

n.5

0.4

0.20

0.3

0.3

0.15

0.10

口

***

口

**

***

**

***

0.2

0.1

***

0.2

0.1

***

***

0.10

0.05

nS.

nS

汇

***

n.5

0.00-

e

lateral

Lateral

Parietal

eipae

OPA

0.0

medial

lateral

Lateral

Parietal

0.0

medial

lateral

Parietal

RSC

0.00

lateral

Lateral

Parietal

5 Conclusion - Discussion

A clearer understanding of representational progression in natural vision requires developing hypothesis-neutral models and interpretable methods. We leverage advances in quantifying feature selectivity of artificial neural networks [6], merged with response-optimized models trained on fMRI data [7], to better characterize spatial properties of a broad range of visual brain regions. This method, "brain dissection," enables granular examination of feature preferences within multiple visual processing streams and functional sub-regions like the scene network. Contrary to earlier studies [7], our work focuses on characterizing visual sub-region preferences in terms of spatial properties and high-level categories (e.g., faces, places). Dissecting spatial properties provides complementary perspectives on visual processing: relationships, orientations, positions of objects, and surfaces within a scene in terms of spatial features (see also [10]).

We observed specific preferences in regions like the RSC, OPA, and PPA. The RSC demonstrated a pronounced preference for greater depths, outdoor object categories, attributes, relations, and predominantly "right/left" surface normals. In contrast, the OPA exhibited preference for proximate depths, intricate 3D geometries, indoor scene object categories, relations, attributes, and a higher inclination towards 'upward' surface normals. These findings align with literature suggesting OPA is primed for local navigational affordances [20, 34], while RSC is geared towards facilitating landmarkbased spatial-memory retrieval [54, 55]. The PPA preferences spanned a middle ground between OPA and RSC, slightly leaning towards OPA, supporting its role in encoding scene structure at a coarser scale than OPA [34]. A salient finding is RSC's pronounced selectivity for vertical surface normals, contrasting with PPA and OPA's preference for horizontal supporting structures, like tabletops and floors, highlighting PPA's sensitivity to scene layout and surface arrangement [56].

Further distinguishing between high-level visual ROIs, our study revealed gradients from ventrolateral to medial areas. Ventro-lateral regions exhibited a preference for closer depths, predominantly horizontal surface normals, and darker shading, while medial areas showed opposite preferences, aligning with distinct processing of foreground objects and distant background elements in these pathways [54]. Added granularity was evident in variability in depth and surface normal selectivity across voxels in medial and parietal regions, indicative of specialized regions for different 3D profiles and global shape processing [57, 58]. This pattern may also reflect differences in coordinate systems for spatial information representation. Ventral areas' preference for camera-oriented surface normals indicates an egocentric frame of reference, while medial and parietal areas' preference for ground plane-oriented surface normals indicates an allocentric frame of reference. Parietal areas favored inanimate objects and spatial relations, Lateral areas preferred persons and animate objects, and Ventral areas favored both, aligning with previous literature. Furthermore, the parietal region's pronounced selectivity for spatial relations (like 'on', 'near', etc.) underscores the significance of spatial relation encoding for this area, corroborating recent research [58].

Limitations. Our approach relies on network dissection and therefore inherits some of its limitations. Network dissection considers each output unit in the network (in our case corresponding to an fMRI voxel) in isolation during interpretation, thus it is not capable of identifying the selectivity of a group of output units. We thus identify the selectivity of individual voxels, which is in line with the voxel-wise modeling school of thought, but not with multivariate pattern analysis. Furthermore, in natural images, there are inherent correlations between specific categories and their spatial attributes (e.g., scenes tend to be rectilinear, pointing up, with far away features, while bodies tend to be close up, facing the camera, etc. [10, 11]). Lastly, our approach relies on pretrained networks to derive spatial measures from images, and this can introduce some estimation errors. We mitigate this limitation by focusing on ROIs that are widely held to encode spatial properties.

A comprehensive understanding of natural vision should cover early visual processing, ecologicallybased intermediate features, and high-level, semantically-grounded representations. Current knowledge largely focuses on the first and third areas. Brain dissection offers potential for uncovering data-driven insights into the full visual processing hierarchy.

References

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences , 111(23):8619-8624, 2014.

Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv , page 407007, 2018.

Aria Y. Wang, Kendrick Kay, Thomas Naselaris, Michael J. Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. bioRxiv , 2022.

Colin Conwell, Jacob Prince, George Alvarez, Talia Konkle, and Kendrick Kay. Opportunistic experiments on a large-scale survey of diverse artificial vision models in prediction of 7t human fMRI data. In Conference on Cognitive Computational Neuroscience , 2022.

Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience , 25(1):116-126, 2022. doi: 10.1038/s41593-021-00962-x.

David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6541-6549, 2017.

Meenakshi Khosla and Leila Wehbe. High-level visual areas act like domain-general filters with strong selectivity and functional specialization. bioRxiv , pages 2022-03, 2022.

Matteo Carandini, Jonathan B Demb, Valerio Mante, David J Tolhurst, Yang Dan, Bruno A Olshausen, Jack L Gallant, and Nicole C Rust. Do We Know What the Early Visual System Does? Journal of Neuroscience , 25(46):10577-10597, 2005.

Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annual Review of Neuroscience , 27:649-677, 2004.

Margaret M. Henderson, Michael J. Tarr, and Leila Wehbe. A texture statistics encoding model reveals hierarchical feature selectivity across human visual cortex. Journal of Neuroscience , 2023.

Margaret M. Henderson, Michael J. Tarr, and Leila Wehbe. Low-level tuning biases in higher visual cortex reflect the semantic informativeness of visual features. Journal of Vision , 23(4): 8-8, 2023.

Kalanit Grill-Spector, Tammar Kushnir, Talma Hendler, and Rafael Malach. The dynamics of object-selective activation correlate with recognition performance in humans. Nature neuroscience , 3(8):837-843, 2000.

J Sergent, S Ohta, and B MacDonald. Functional neuroanatomy of face and object processing: A positron emission tomography study. Brain , 115:15-36, 1992.

Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience , 17(11): 4302-4311, 1997.

Geoffrey K Aguirre, Eric Zarahn, and Mark D'Esposito. An area within human ventral cortex sensitive to 'building' stimuli: evidence and implications. Neuron , 21(2):373-383, 1998.

Bruce D McCandliss, Laurent Cohen, and Stanislas Dehaene. The visual word form area: expertise for reading in the fusiform gyrus. Trends in cognitive sciences , 7(7):293-299, 2003.

Michael C-K Wu, Stephen V David, and Jack L Gallant. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci. , 29:477-505, 2006.

Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage , 56(2):400-410, 2011.

Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron , 76(6):1210-1224, 2012.

Mark D Lescroart and Jack L Gallant. Human scene-selective areas represent 3d configurations of surfaces. Neuron , 101(1):178-192, 2019.

Meenakshi Khosla, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Characterizing the ventral visual stream with response-optimized neural encoding models. Advances in Neural Information Processing Systems , 35:9389-9402, 2022.

Leslie G Ungerleider. Two cortical visual systems. Analysis of visual behavior , pages 549-586, 1982.

Bhavin R Sheth and Ryan Young. Two visual pathways in primates based on sampling of space: exploitation and exploration of visual information. Frontiers in integrative neuroscience , 10:37, 2016.

Emalie McMahon, Michael F Bonner, and Leyla Isik. Hierarchical organization of social action features along the lateral visual pathway. 2023.

Vladislav Ayzenberg and Marlene Behrmann. The where, what, and how of object recognition. Trends in Cognitive Sciences , 2023.

Dawn Finzi, Daniel LK Yamins, Kendrick Kay, and Kalanit Grill-Spector. Do deep convolutional neural networks accurately model representations beyond the ventral stream? In 2022 Conference on Cognitive Computational Neuroscience , 2022.

Daniel D Dilks, Joshua B Julian, Alexander M Paunov, and Nancy Kanwisher. The occipital place area is causally and selectively involved in scene perception. Journal of Neuroscience , 33 (4):1331-1336, 2013.

Merim Bilali´ c, Tobias Lindig, and Luca Turella. Parsing rooms: the role of the ppa and rsc in perceiving object relations and spatial layout. Brain Structure and Function , 224:2505-2524, 2019.

Dwight J Kravitz, Cynthia S Peng, and Chris I Baker. Real-world scene representations in high-level visual cortex: it's the spaces more than the places. Journal of Neuroscience , 31(20): 7322-7333, 2011.

Linda Henriksson, Marieke Mur, and Nikolaus Kriegeskorte. Rapid invariant encoding of scene layout in human opa. Neuron , 103(1):161-171, 2019.

Moira R Dillon, Andrew S Persichetti, Elizabeth S Spelke, and Daniel D Dilks. Places in the brain: Bridging layout and object geometry in scene-selective cortex. Cerebral Cortex , 28(7): 2365-2374, 2018.

Samoni Nag, Daniel Berman, and Julie D Golomb. Category-selective areas in human visual cortex exhibit preferences for stimulus depth. Neuroimage , 196:289-301, 2019.

Soojin Park and Marvin M Chun. Different roles of the parahippocampal place area (ppa) and retrosplenial cortex (rsc) in panoramic scene perception. Neuroimage , 47(4):1747-1756, 2009.

Michael F Bonner and Russell A Epstein. Coding of navigational affordances in the human visual system. Proceedings of the National Academy of Sciences , 114(18):4793-4798, 2017.

Jeongho Park and Soojin Park. Coding of navigational distance and functional constraint of boundaries in the human scene-selective cortex. Journal of Neuroscience , 40(18):3621-3630, 2020.

Chu-Chung Huang, Edmund T Rolls, Jianfeng Feng, and Ching-Po Lin. An extended human connectome project multimodal parcellation atlas of the human cortex and subcortical areas. Brain Structure and Function , 227(3):763-778, 2022.

David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating 'what' and 'where'. Advances in Neural Information Processing Systems , 30, 2017.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR , abs/1405.0312, 2014. URL http://arxiv.org/abs/ 1405.0312 .

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1452-1464, 2017.

Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 , 2023.

Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11197-11206, 2020.

Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence , 37(8):1670-1687, 2014.

Edward H Adelson and Alex P Pentland. The perception of shading and reflectance. Perception as Bayesian inference , 409:423, 1996.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Merim Bilali´ c, Tobias Lindig, and Luca Turella. Parsing rooms: the role of the PPA and RSC in perceiving object relations and spatial layout. Brain Struct. Funct. , 224(7):2505-2524, 2019.

Cheng Zhang, Wei-Lun Chao, and Dong Xuan. An empirical study on leveraging scene graphs for visual question answering. arXiv preprint arXiv:1907.12133 , 2019.

Nicholas M Blauch, Marlene Behrmann, and David C Plaut. A connectivity-constrained computational account of topographic organization in primate high-level visual cortex. Proceedings of the National Academy of Sciences , 119(3):e2112566119, 2022.

Dawn Finzi, Eshed Margalit, Kendrick Kay, Daniel LK Yamins, and Kalanit Grill-Spector. Topographic DCNNs trained on a single self-supervised task capture the functional organization of cortex into visual processing streams. In SVRHM 2022 Workshop @ NeurIPS , 2022. URL https://openreview.net/forum?id=E1iY-d13smd .

Nidhi Jain, Aria Wang, Margaret M Henderson, Ruogu Lin, Jacob S Prince, Michael J Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Commun. Biol. , 6(1):175, 2023.

Azriel Rosenfeld and John L Pfaltz. Sequential operations in digital picture processing. Journal of the ACM (JACM) , 13(4):471-494, 1966.

Stefania Bracci and Hans P. Op de Beeck. Understanding human object vision: A picture is worth a thousand representations. Annual Review of Psychology , 74(1):113-135, 2023.

Russell A Epstein. Parahippocampal and retrosplenial contributions to human spatial navigation. Trends in cognitive sciences , 12(10):388-396, 2008.

Stephen D Auger, Sinéad L Mullally, and Eleanor A Maguire. Retrosplenial cortex codes for permanent landmarks. 2012.

Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature , 392(6676):598-601, 1998.

Andrew E Welchman. The human brain in depth: how we see in 3d. Annual review of vision science , 2:345-376, 2016.

Vladislav Ayzenberg and Marlene Behrmann. The dorsal visual pathway represents objectcentered spatial relations for object recognition. Journal of Neuroscience , 42(23):4693-4710, 2022.

Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology , 32:1-13, 2022.

James S Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant. Pycortex: an interactive surface visualizer for fmri. Frontiers in neuroinformatics , page 23, 2015.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'12, page 1097-1105, Red Hook, NY, USA, 2012. Curran Associates Inc.

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV) , pages 418-434, 2018.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633-641, 2017.

S1 Overview

Section S2 contains more details of the methods described in the main paper. Section S3 provides additional results from the experiments.

We include an interactive project page, which includes exploratory brain flatmaps for each of our dissection measures, and code for training and evaluating the networks. Please visit the following link to access the project page: https://brain-dissection.github.io/

S2 Implementation details

S2.1 Network Architecture and Training

Our architecture and training largely follow Khosla et al. [59], except we use regular convolutions instead of E(2)-steerable convolutions. We found that using regular convolutions instead of E(2)-steerable convolutions does not significantly affect dissection performance while providing significantly improved training speed and memory efficiency. We train a separate convolutional neural network (CNN) for each ROI sub-region shared across all 8 subjects. The CNN consists of four 'blocks', each with two convolutional layers followed by average pooling. We use batch normalization and early stopping. For each voxel in the ROI sub-region (across for all subjects), we employ a linear readout model on top of the feature space to predict the responses of individual voxels in a specific brain region. The linear readout is factorized into spatial and feature dimensions following [37]. This allows us to separate image spatial tuning (the 'where') from feature tuning (the 'what'). The spatial features have been shown to correlate with the population receptive fields (pRFs) of the voxels [21]. We train with a mean squared error loss between the predicted voxel response and the true voxel response. During the dissection procedure, we only use the feature tuning. We train all models on a single Nvidia GeForce RTX 3090 GPU. The number of parameters in the CNN is 784697 and the number of parameters in the transformer network (see next section) is 7053009.

S2.2 Training & Validation Curves

We provide encoding model training and validation curves for PPA, OPA and RSC functionallydefined ROIs in Figure S1 and Figure S2. We utilize early stopping to terminate training and obtain an evaluation checkpoint by checking if the validation performance (Pearson correlation) is lower than the highest validation performance for 10 epochs.

S2.3 Additional Evaluation Details

For network dissection, we use the code repository from Bau et al. [6]. We use pycortex to visualize voxel data [60]. Data are projected onto 2D brain flatmaps from 3D voxel data using trilinear interpolation for continuous data (e.g., depth) and nearest interpolation for discrete data (e.g., category).

S3 Additional Results

S3.1 Performance of Response-Optimized and Baseline Models

We have provided the Pearson correlation coefficient plotted on a flatmap for all ROIs on a held-out test set of NSD data in Figure S3 for example Subject S1. Additionally, we compare our model's mean Pearson correlation to the features from Lescroart and Gallant [20] and ImageNet task-optimized features [61] on held-out NSD brain data using Pearson correlation. We the features fit to brain data via Ridge regression. We computed the features in Lescroart and Gallant [20] for NSD images using the estimation networks. As shown in Figure S3, our brain dissection model demonstrates a significant improvement, nearly 2x across scene ROIs, in its alignment with the brain responses compared to the baseline features from [20]. Our model also aligns in performance with AlexNet ImageNet features, even though the AlexNet network having a significantly larger parameter size and trained on significantly more images.

train/mse_loss

train_streams_subjs12345678_PPA

- train_streams_subjs12345678_0PA

- train_streams_subjs12345678_RSC

0.6

0.4

0.2

Step

0

50k

100k

150k

200k

250k

validation/total_corr

- train_streams_subjs12345678_PPA

- train_streams_subjs12345678_0PA

train_streams_subjs12345678_RSC

0.4

0.3

0.2

0.1

Step

0

50k

100k

150k

200k

250k

S3.2 Results using Different Architectures and Interpretability Methods

We integrated both gradCAM [38] and raw attention techniques [39]. The former utilizes input features combined with network gradients, while the latter leverages attention scores from transformer architectures. As shown in Figure S4, our results affirm consistency across diverse interpretability methods and network architectures.

S3.3 GQAData Visualization

We show example images and annotations for the GQA dataset in Figure S5. We used Segment Anything [46] to obtain the segmentation masks from the GQA box annotations.

S3.4 Histograms of the Spatial Measures

We show the distribution of voxel preferences for each high-level visual ROI in Figure S6 and for the scene ROIs in Figure S7. We observe that subjects are largely consistent in their distributions for each ROI. We also observe that some ROIs and measures are multi-modal, such as in several measures in Medial and Gaussian curvature in Ventro-lateral and Lateral. This may indicate a further specialization of voxel clusters in these areas.

C

0.6

CNNresponse-optimized

transformerresponse-optimized

0.5

task-optimized(lmageNet)

Pearson correlation

0.4

0.20.40.60.8

0.3

0.2

0.1

0.0

PPA

OPA

RSC

0.20.40.60.8

S3.5 Places365 Evaluation

In addition to the GQA dataset, we evaluate for category selectivity in the Places365 dataset, where segmentation masks are obtained using the Unified Perceptual Parsing image segmentation network [62] previously trained on 20,000 scene-centric images from the ADE20k dataset [63]. We report median IOU (mean ± standard deviation across 8 subjects) in Figure S8.

S3.6 Unit Visualizations

We include additional single-voxel visualizations for RSC, OPA, and PPA in Figure S9, Figure S10, and Figure S11, respectively.

S3.7 Brain Flatmaps

In this section, we include brain flatmaps for all eight subjects for metric depth (Figure S12), relative depth (Figure S13), surface normals (Figure S14), Guassian curvature (Figure S15), and shading (Figure S16).

S3.8 WordCloud Visualizations

We offer a WordCloud representation to visualize the top 20 categories that an ROI selects for. In this visualization, the category size represents the magnitude of the median IOU. We show WordClouds for all high-level visual ROIs and scene ROIs for the GQA dataset (Figure S17) and the Places365 dataset (Figure S18).

Network Dissection (CNN)

Absolute depth

Surface normals

***

***

Surface Normal Bin

150

***

***

Depth (meters)

12.5

125

10.0

7.5

15

5.0

505

2.5

0.0

PPA

OPA

RSC

PPA

OPA

Raw Attention (Transformer)

12

***

***

Bin

***

Depth (meters)

Normal

120

100

80

60

Surface

40

0

PPA

OPA

RSC

PPA

OPA

RSC

gradCAM (CNN)

***

***

Surface Normal Bin

**

n.s.

Depth (meters)

10

125

550250

6

2

0

PPA

OPA

RSC

OPA

RSC

HO DC

NOGLASS

?

RelativeDepth

Surface Normals

Shading

Guassian Curvature

subject 1

subject 2

subject 3

subject 4

subject 5

subject 6

medial

subject 7

subject 8

lateral

Lateral

Parietal

Medial

5

10

0.2

0.4

50

100

0.50

0.75

0.1

0.2

MetricDepth

Relative Depth

Surface Normals

Shading

Guassian Curvature

subject 1

subject 2

subject 3

subject 4

PPA

subject 5

subject6

subject 7

subject 8

OPA

RSC

5

10

0.2

0.4

50

100

0.50

0.75

0.1

0.2

Ventromedial

Ventrolateral

Lateral

0.014

0.05

0.012

0.04

0.04

0.010

0.03

0.008

0.03

0.006

0.02

0.02

0.004

0.01

0.002

0.01

0.000

floor

wall-b

pooM

!

llem

J-11m

white

yellow

painted

fabric

-bupn

wall-I

green

0.00

head

skin

person-t

pa

person

person-r

person-1

orange

purple

person-b

neck

hair

fabric

0.00

head

skin

person-t

person

person-1

pueu

pa

e

person-b

purple

fabric

orange

leg

neck

black

torso

Parietal

Medial

0.016

0.030

0.014

0.025

0.012

0.010

0.020

0.008

0.015

0.006

0.004

0.010

0.002

0.005

0.000

ud

pa

fabric

floor

person

qpeoqnis

floor-r

purple

ana

white

green

0.000

head

skin

q-llem

person

poom

person-1

leM

wall-r

building-t

1-eM

black

S1

S2

S3

S5

S6

S7

S8

2

4個

6

10 meters

S1

S2

S3

S5

S6

S7

S8

0.1 0.15 0.2 0.25 0.3 0.35 0.4

S1

S2

S3

S5

S6

S7

S8

40

60

80

100

120

S1

S2

S3

S5

S6

S7

S8

0.05 0.10 0.15 0.20 0.25

S1

S2

S3

S5

S6

S7

S8

0.3

0.40.5.0.60.7 0.8

Ventrolateral

gir

I

sitting on

boy

hairholding

".face

hat

wearing

in front of

man

to the left of

woman

standing

playing head

to the right of happy

Lateral

whi te

to the left of leg

●hand player

wea

arm swinging shirt

man

hitting

woman

on

fboy

person

to the right of

in frontof

Medial

building

signr

to the left of

buildings

near

in front of

on

wall

k

red white

train

bus

brick

window

behind

car

to the right of

tall

OPA

to

the left(

of

in front of

to

white

bathtub

et

floor

1

wood

plate

gray

table.

desk

on

near

sink counter

bed

d to the right of

Ventromedial

to the left of

tall

near

.5

bathroom·

black

•wall

behind

bus

white.

on

mirror

above

window

train

wood

o the right oflarge

to

Parietal

round

orange

sky.

tot

the left of

near

red

table

holding

on

above

to the right of

in

wood

wearin

g

blue

of

plate bowl

man

white

RSC

in front of to theright of

building

train

sky:

roof

trees

snq

tall

buildingsbehind

white

blue

to

the

"left

of

brick

near

window

above

PPA

white

counter above

tot

the left of

on

in floor..traintable

in front of

wall

toilet

to.the

rig

ght

of

desknear

black

wood

bathroom

Ventrolateral

metal pink

painting

wood

head

hairhand

paper

-grey·painted

person-1

S

skin

person-t

red

wall-r wall-t

Lateral

foot

person-t

head

person-r

skin

red

purple

fabric

arm

per son

wood Person-b

pinkhand

person-1

orange

torso

leg

Medial

building-r

building

glass

wall-1

1

pink

wall

wa

shelf

building.

building-b

wall-t

window

yellow

wall-r

-t

poOM

metal

building-l orange

OPA

floor

polished stone

floor-b

t

bed-1

grey

一

fabric

brownmetalfloor

carpet

ceraml

black

woodfloor-ltablel

tile

bed

bed-t

Ventromedial

metal

building

wall-r.

green

wall-t

ceiling-b building-1

painted

window-b

window

wall

wall-b

white

yell

.ow

building-r

glass wall-1

wood

Parietal

personperson-1

purple

skin

sky

red

leg

person-b

fabric

orange

person-t

pink

ceiling

arm

wood

sky-b person-r

metal

seat

RSC

glass building-r

building

building-t

ceiling

ceiling-r

wa 1l

painted sky-l sky

white

grey

blue

sky-b,

sky-r

building-l

window ceiling-b

ceiling-l wall-1

PPA

fabricceiling-l carpet

ceiling

wood

floor-r

table

ceiling

floor-l building-t

ceiling-r

white

tile

blue

brown floor-b floor

floor-t

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors train CNNs to predict the responses of individual voxels across several ROIs to the natural scenes dataset (NSD). They then develop and employ a strategy termed “brain dissection” to uncover the properties/features of images for which specific visual regions are tuned. They highlight gradients within subregions of the visual stream that correspond to tuning variation along mid-level image characteristics including depth, curvature, and object relations.\n\nSTRENGTHS:\nOriginality: The methodology, as the authors note, is conceptually similar to Khosla & Wehbe (2022) but is applied towards evaluating sub-region tuning of mid-level visual features rather than reaffirming category selectivity. This is an effective use and reference of an existing computational strategy towards novel scientific investigation. My only comment would be to include some more comprehensive methods and related work description, instead of relying on the reader going back to this original work for the conceptual motivation and validation. A few sentences will suffice.\n\nQuality: This submission is technically sound, uses appropriate methods, and the claims are well supported for the most part. The authors should, however, be a bit more cautious about claims tying the trends they observe in RSC and OPA/PPA to specific functional preferences, e.g., in 4.1.3 and 4.3, as these are somewhat speculative hypotheses that were not explicitly tested, e.g., tying “up” normal to preferences for an “allocentric frame of reference.”\n\nClarity: The submission is clearly written and well organized.\n\nSignificance: The results are important, tie in with existing literature and discussions, and will inform other researchers interested in visual stream organization.\n\nWEAKNESSES:\nWrapped in with above and discussed again below.\n\nQUESTIONS:\nAs noted above, I would recommend expanding the discussion of related work and methodology to make for a clearer and more well-packaged reading. It would also be beneficial to tie in more related work supporting or refuting the hypotheses introduced about the subdivision of the “scene network”, such that these data can be appropriately placed into context with other findings and approaches. This limited connection to existing literature, context, and related findings is the work’s greatest weakness.\n\nLIMITATIONS:\nMethodological limitations are mostly well discussed. An additional sentence about limitations with this type of ROI modeling strategy in general, e.g., potential lack of transfer or generalization to real experiments, should be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUnderstanding the organization of higher level information representation in the brain is a challenging task in neuroscience. Modern deep learning methods together with big data of brain recording have opened up new opportunities for constructing large-scale models in a data-driven way and for gaining valuable insights about information processing in the brain. The present paper explores this idea by training a deep neural network model for predicting human brain fMRI from natural scene images. By analyzing the feature map generated by the model, the paper reveals spatial and functional organization of the brain for a wide range of high-level visual features.\n\nSTRENGTHS:\nOverall, I find that the paper is well written and that it addresses important questions at the intersection of machine learning and neuroscience.\n\nWEAKNESSES:\nI have a number of concerns about the results and particularly, technical details presented in the paper, which prevented from accepting it directly.\n\nQUESTIONS:\nMajor comments\n1.\tThe paper mentions that the model is trained on the NSD dataset including 68400 training images and 3600 validation images. However, the performance of the trained network on estimating fMRI data on the training/validation dataset is missing. Since the main point of the paper is to use a DNN model to gain insights about feature representation in the brain, it is crucial to establish an understanding of how well the model can predict brain activity in the first place. I notice that in the supplementary material, the performance of the model is reported for 3 ROIs (PPA, OPA, RSC) in terms of the Pearson correlation coefficient. However, unless the performance of the model is clearly reported for all ROIs, it is difficult to meaningfully interpret the analysis based on the proposed brain dissection method. For instance, the correlation between the DNN prediction and the ground-truth fMRI data in each ROI could be presented as a flatmap.\n2.\tOne concern regarding the calculation of selectivity (Eq. 1-2) is that a voxel is assigned to a feature value disregarding whether the voxel is truly selective at all. For instance, if L_c masked by M_k across all images is uniformly distributed, then no selectivity can be said about this voxel. To remedy this, the author should rule out these non-selective voxels from the analysis, otherwise it could lead to misleading interpretations of the result.\n\n\nMinor comments\n1.\tThe dimensionality of the neural network model should be clearly stated for reproducibility.\n2.\tSince the paper is trained on human fMRI data, a discussion about the biological plausibility of the model would be appropriate. For instance, how much can we infer from the feature representation in the model about that in the real brain? Can we make any experimentally testable predictions?\n\nLIMITATIONS:\nThe paper technically sounds. But the technical details are not clear and guarantee reproductivity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study uses the network dissect method to investigate the feature selectivity of RSC, OFA, and PPA in the human brain. This method is called \"brian dissection\". In particular, this study focuses on some ecologically important intermediate features, such as depth, surface normals, curvatures, and object relations. Results showed that the three regions show distinct feature preferences.\n\nSTRENGTHS:\n1. To my best knowledge, this is the first study to apply the network dissect method to examine the voxel preferences.\n2. The overall method is clear and the presentation is good.\n\nWEAKNESSES:\n1. It is unclear to me why this study only focuses on some intermediate features, such as depth, surface normals, curvature. In theory, this method can be also used to study both low-level and high-level visual features.\n2. The theoretical contribution is unclear. I agree this study may be the first application of network dissection on voxel preferences. But the idea here is generally incremental. It can certainly obtain some new findings because the method per se is new. But I don't see clear progress being made as compared to previous methods. These types of results may be good for a neuroscience journal. The key point which is missing here is that how these representations are formed.\n\nQUESTIONS:\nMy major concern is why this method is better than previous methods. For example, a simple encoding model was developed in ref.[1] and capture several important properties of scene-selective regions. I hope the authors provide evidence why brain dissection is better than this simple encoding-regression model. I understand that brain dissection certainly gives different results and feature maps because they are different methods. But this is not the reason why brain dissection is superior. \n\nOr, different dissection methods could be used and show that the results are highly consistent??\n\n\n[1] Lescroart MD, Gallant JL. Human Scene-Selective Areas Represent 3D Configurations of Surfaces. Neuron. 2019 Jan 2;101(1):178-192.e7. doi: 10.1016/j.neuron.2018.11.004.\n\nLIMITATIONS:\n1. I trust the results. But the results are completely data-driven. I didn't see how the results add much to our understanding of the function of high-level visual regions.\n2. My feeling is that this type of result is suitable for neuroscience journals like Neuroimage. I didn't see any new algorithm-level novelty. It is certainly not interesting to the machine learning community. The results here may be of interests to neuroscientists. I mean the results are valid. But this study only displays some results but does not contribute to how the brain form such representation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper utilized network dissection model to understand how human brain is functionally mapped to perception of natural scenes. The proposed method is used to examine a range of ecologically important, intermediate properties, including depth, surface normals, curvature, and object relations and find consistent feature selectivity differences.

STRENGTHS:
- The paper is well written and easy to understand. 
- The paper introduces interesting discussions by employing an AI model to a neuroimaging study.

WEAKNESSES:
- While the paper performs a very interesting experiment from neuroscience perspective, there is not much of a technical contribution. The paper employs an existing model, i.e., network dissection, and does not propose a new or novel approach. 
- There is no baseline experiment and there is no way to validate the gain or improvement from the proposed method. 
- I think this paper has novelty, but perhaps not in a way that the NeurIPS community expects. At least I would like to see a proposal of a novel approach that lead to novel and improved findings.

QUESTIONS:
- As far as I know, there are various methods that provide interpretability, e.g., Chefer et al., CVPR 2021. Will the result remain the same if other models are adopted other than the network dissection (which is a bit outdated). 
- What would be the feature maps like if some naive models were to be deployed, e.g., MLP or simple CNNs with Class Activation Map?
- Is NSD dataset the only dataset that connects fMRI to natural scene? If so, this would justify experimenting only on a single benchmark regardless of generalization issue. 
- What are OPA, RSC, PPA in the abstract? They should be fully spelled out.

LIMITATIONS:
The authors address limitation of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06