SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To illustrate these observations, we identified voxels that have the closest selectivity to the ROI mean for depth or surface normals for PPA, OPA, and RSC, and plotted the voxel's top five images by predicted response (Fig. 5). We include corresponding dissection masks below these on images and either depth or surface normal images. RSC exhibits a preference for far depths and more "right/left" surface normals, indicative of vertical structures relevant to navigation. In contrast, OPA and PPA exhibit a preference for "up" surface normals which are relevant to representing the local scene layout. Once more, these results highlight the distinct functional roles these areas play in scene perception.
Figure 4: A. Flatmaps showing voxel-prefrences in OPA, PPA, and RSC for the four spatial measures. B. Mean  $\pm$  standard deviation ( $N = 8$  subjects) for each of the high-level visual ROIs.  $^{***} = p < 0.001$ ,  $^{**} = p < 0.01$ ,  $^{*} = p < 0.05$ , n.s. = not significant  $p > 0.05$  (post hoc Tukey test;  $N = 8$  subjects).
Figure 5: Voxels that have the closest selectivity (by euclidean distance) to the ROI mean of depth (top) or surface normals (bottom) for PPA, OPA, and RSC. An example image is shown with a representative voxel (left) and the top five images by predicted response for the top four voxels (right) for each ROI. Dissection masks are visualized on the images and either the depth or surface normal images. RSC prefers far-reaching depth and vertical structures, while OPA and PPA prefers more local depths and a mix of vertical and horizontal structures.
Figure 6: Median IOU (mean  $\pm$  standard deviation;  $N = 8$  subjects) for the GQA dataset [48] for each of the scene ROIs (top) and high-level visual ROIs (bottom).
4.2 Object Relations, Attributes and Categories
We utilize the GQA dataset [48] in concert with brain dissection to gain insights into the selectivity differences for object relationship, attributes, and categories within an image. The median IOU (mean  $\pm$  standard deviation across 8 subjects) for the top 20 concepts for each ROI is reported in Figure 6. With respect to high-level visual ROIs, we observe distinct preferences: Parietal areas prefer inanimate and scene objects, such as walls and tables, that are more relevant to both navigation and physically interacting with objects. In contrast, Lateral areas exhibit a preference for person and animate objects, while Ventrolateral areas favor persons and objects - representations that support social interactions and scene understanding. When examining object relations, Parietal areas showed a strong preference for spatial relations such as 'in front of,' 'near,' 'in,' and 'above'. Lateral areas favor relations involving people, such as 'wearing,' 'holding,' and 'with'. In terms of attributes, Parietal areas show a preference for colors and materials, whereas Lateral areas favor animate attributes involving people, such as 'playing,' 'standing,' and 'happy'. These preferences are largely aligned with prior results showing Parietal preferentially processes spatial visual information [22], Lateral preferentially processes social visual information [24], and Ventral preferentially processes objects and faces [23].
As illustrated in Figure 6, within scene ROIs, OPA shows greater selectivity for indoor scene elements, such as chairs, tables, and floors. PPA exhibits a mix of indoor and outdoor preferences, whereas RSC shows a clear preference for outdoor scene elements like trees, buildings, and the sky, once again reinforcing a dichotomy between local space interactions and navigation. Furthermore, when
considering attributes, RSC again demonstrates more selectivity for "outdoor" attributes compared to PPA and OPA, which show higher selectivity for indoor elements. These findings underscore the distinct functional implications of these ROI selectivity profiles for processing visual scenes.
4.3 Topographic Organization of the Spatial Preferences
We investigated the topographical organization of the spatial measures within each ROI. High-level visual areas are topographically organized, comprising multiple hierarchically organized areas that are selective for specific domains such as faces and scenes [49, 50, 12, 13, 14, 15, 16, 51]. However, whether and how this topographical organization applies to the spatial measures used here remains unclear. Given a specific type of spatial information, we explored: (a) if the voxels in an ROI are differently tuned for this type of information (e.g., sub-groups of voxels are tuned for different surface normal orientations); and if so, (b) does the tuning follow a smooth topographical organization with relatively large clusters processing similar information. To quantify the degree of clustering by selectivity similarity, for each measure we computed the average cluster proportion (cluster size normalized by total voxels in the ROI) (considering clusters  $>5$  voxels) that each voxel belongs to, and report the results compared to a random permutation of the voxels (Fig. 7). We employed connected components [52] on the 3D voxel data to determine whether two neighboring clusters belong to the same cluster. For continuous measures, such as depth, we consider two voxels the same if they fall within  $\delta$  of each other. We used  $1/100^{th}$  of the data range across all ROIs as the  $\delta$  for each of the continuous measures. For discrete concepts, such as object category, we consider two neighbors to be in the same cluster if they belong to the same discrete category.
A cluster analysis was run for each ROI. We report the average cluster size for a given measure compared to the average cluster size for the measure when the voxels are randomly permuted ((Fig. 7;  $p < 0.05$  paired  $t$ -test;  $N = 8$ ). We visualize the spatial clusters on flatmaps: All of the high level visual areas showed clustering significantly different from random for all spatial measure, indicating that their voxels are tuned for different values of each measure in a way that is topographically organized. At the level of the scene areas however, RSC was not distinguishable from chance for metric depth, relative depth, shading, gaussian curvature and category. This suggests that except for surface normals, the voxels in RSC are tuned similarly. Similarly, the results suggest that the voxels in PPA are tuned differently only for metric depth, and the voxels in OPA are tuned differently for metric depth, relative depth and surface normals. These findings highlight the robust topographical organization within the ROIs, extending beyond traditional domain-specific areas. Critically, this extended organization highlights the fact that, as a whole, visual cortex is representing a broad range of ecologically relevant properties that support complex behavior [53].
Figure 7: A. Metric depth (left) and surface normals (right) spatial clusters on flatmaps for example subject S1, where each color is a different cluster with (bottom) and without (top) random spatial permutations. B. Bar plots showing mean  $\pm$  standard deviation of average cluster size per voxel for each ROI.  $^{***} = p < 0.001$ ,  $^{**} = p < 0.01$ ,  $^* = p < 0.05$ , n.s. = not significant  $p > 0.05$  (paired t-test;  $N = 8$ ).
5 Conclusion - Discussion
A clearer understanding of representational progression in natural vision requires developing hypothesis-neutral models and interpretable methods. We leverage advances in quantifying feature selectivity of artificial neural networks [6], merged with response-optimized models trained on fMRI data [7], to better characterize spatial properties of a broad range of visual brain regions. This method, "brain dissection," enables granular examination of feature preferences within multiple visual processing streams and functional sub-regions like the scene network. Contrary to earlier studies [7], our work focuses on characterizing visual sub-region preferences in terms of spatial properties and high-level categories (e.g., faces, places). Dissecting spatial properties provides complementary perspectives on visual processing: relationships, orientations, positions of objects, and surfaces within a scene in terms of spatial features (see also [10]).
We observed specific preferences in regions like the RSC, OPA, and PPA. The RSC demonstrated a pronounced preference for greater depths, outdoor object categories, attributes, relations, and predominantly "right/left" surface normals. In contrast, the OPA exhibited preference for proximate depths, intricate 3D geometries, indoor scene object categories, relations, attributes, and a higher inclination towards "upward" surface normals. These findings align with literature suggesting OPA is primed for local navigational affordances [20, 34], while RSC is geared towards facilitating landmark-based spatial-memory retrieval [54, 55]. The PPA preferences spanned a middle ground between OPA and RSC, slightly leaning towards OPA, supporting its role in encoding scene structure at a coarser scale than OPA [34]. A salient finding is RSC's pronounced selectivity for vertical surface normals, contrasting with PPA and OPA's preference for horizontal supporting structures, like tabletops and floors, highlighting PPA's sensitivity to scene layout and surface arrangement [56].
Further distinguishing between high-level visual ROIs, our study revealed gradients from ventrolateral to medial areas. Ventro-lateral regions exhibited a preference for closer depths, predominantly horizontal surface normals, and darker shading, while medial areas showed opposite preferences, aligning with distinct processing of foreground objects and distant background elements in these pathways [54]. Added granularity was evident in variability in depth and surface normal selectivity across voxels in medial and parietal regions, indicative of specialized regions for different 3D profiles and global shape processing [57, 58]. This pattern may also reflect differences in coordinate systems for spatial information representation. Ventral areas' preference for camera-oriented surface normals indicates an egocentric frame of reference, while medial and parietal areas' preference for ground plane-oriented surface normals indicates an allocentric frame of reference. Parietal areas favored inanimate objects and spatial relations, Lateral areas preferred persons and animate objects, and Ventral areas favored both, aligning with previous literature. Furthermore, the parietal region's pronounced selectivity for spatial relations (like 'on', 'near', etc.) underscores the significance of spatial relation encoding for this area, corroborating recent research [58].
Limitations. Our approach relies on network dissection and therefore inherits some of its limitations. Network dissection considers each output unit in the network (in our case corresponding to an fMRI voxel) in isolation during interpretation, thus it is not capable of identifying the selectivity of a group of output units. We thus identify the selectivity of individual voxels, which is in line with the voxel-wise modeling school of thought, but not with multivariate pattern analysis. Furthermore, in natural images, there are inherent correlations between specific categories and their spatial attributes (e.g., scenes tend to be rectilinear, pointing up, with far away features, while bodies tend to be close up, facing the camera, etc. [10, 11]). Lastly, our approach relies on pretrained networks to derive spatial measures from images, and this can introduce some estimation errors. We mitigate this limitation by focusing on ROIs that are widely held to encode spatial properties.
A comprehensive understanding of natural vision should cover early visual processing, ecologically-based intermediate features, and high-level, semantically-grounded representations. Current knowledge largely focuses on the first and third areas. Brain dissection offers potential for uncovering data-driven insights into the full visual processing hierarchy.
[52] Azriel Rosenfeld and John L Pfaltz. Sequential operations in digital picture processing. Journal of the ACM (JACM), 13(4):471-494, 1966.
[53] Stefania Bracci and Hans P. Op de Beeck. Understanding human object vision: A picture is worth a thousand representations. Annual Review of Psychology, 74(1):113-135, 2023.
[54] Russell A Epstein. Parahippocampal and retrosplenial contributions to human spatial navigation. Trends in cognitive sciences, 12(10):388-396, 2008.
[55] Stephen D Auger, Sinead L Mullally, and Eleanor A Maguire. Retrosplenial cortex codes for permanent landmarks. 2012.
[56] Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, 392(6676):598-601, 1998.
[57] Andrew E Welchman. The human brain in depth: how we see in 3d. Annual review of vision science, 2:345-376, 2016.
[58] Vladislav Ayzenberg and Marlene Behrmann. The dorsal visual pathway represents object-centered spatial relations for object recognition. Journal of Neuroscience, 42(23):4693-4710, 2022.
[59] Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. *Current Biology*, 32:1-13, 2022.
[60] James S Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant. Pycortex: an interactive surface visualizer for fmri. Frontiers in neuroinformatics, page 23, 2015.
[61] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS'12, page 1097-1105, Red Hook, NY, USA, 2012. Curran Associates Inc.
[62] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.
[63] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
S1 Overview
Section S2 contains more details of the methods described in the main paper. Section S3 provides additional results from the experiments.
We include an interactive project page, which includes exploratory brain flatmaps for each of our dissection measures, and code for training and evaluating the networks. Please visit the following link to access the project page: https://brain-dissection.github.io/
S2 Implementation details
S2.1 Network Architecture and Training
Our architecture and training largely follow Khosla et al. [59], except we use regular convolutions instead of E(2)-steerable convolutions. We found that using regular convolutions instead of E(2)-steerable convolutions does not significantly affect dissection performance while providing significantly improved training speed and memory efficiency. We train a separate convolutional neural network (CNN) for each ROI sub-region shared across all 8 subjects. The CNN consists of four "blocks", each with two convolutional layers followed by average pooling. We use batch normalization and early stopping. For each voxel in the ROI sub-region (across for all subjects), we employ a linear readout model on top of the feature space to predict the responses of individual voxels in a specific brain region. The linear readout is factorized into spatial and feature dimensions following [37]. This allows us to separate image spatial tuning (the "where") from feature tuning (the "what"). The spatial features have been shown to correlate with the population receptive fields (pRFs) of the voxels [21]. We train with a mean squared error loss between the predicted voxel response and the true voxel response. During the dissection procedure, we only use the feature tuning. We train all models on a single Nvidia GeForce RTX 3090 GPU. The number of parameters in the CNN is 784697 and the number of parameters in the transformer network (see next section) is 7053009.
S2.2 Training & Validation Curves
We provide encoding model training and validation curves for PPA, OPA and RSC functionally-defined ROIs in Figure S1 and Figure S2. We utilize early stopping to terminate training and obtain an evaluation checkpoint by checking if the validation performance (Pearson correlation) is lower than the highest validation performance for 10 epochs.
S2.3 Additional Evaluation Details
For network dissection, we use the code repository from Bau et al. [6]. We use pycortex to visualize voxel data [60]. Data are projected onto 2D brain flatmaps from 3D voxel data using trilinear interpolation for continuous data (e.g., depth) and nearest interpolation for discrete data (e.g., category).
S3 Additional Results
S3.1 Performance of Response-Optimized and Baseline Models
We have provided the Pearson correlation coefficient plotted on a flatmap for all ROIs on a held-out test set of NSD data in Figure S3 for example Subject S1. Additionally, we compare our model's mean Pearson correlation to the features from Lescroart and Gallant [20] and ImageNet task-optimized features [61] on held-out NSD brain data using Pearson correlation. We the features fit to brain data via Ridge regression. We computed the features in Lescroart and Gallant [20] for NSD images using the estimation networks. As shown in Figure S3, our brain dissection model demonstrates a significant improvement, nearly  $2\mathrm{x}$  across scene ROIs, in its alignment with the brain responses compared to the baseline features from [20]. Our model also aligns in performance with AlexNet ImageNet features, even though the AlexNet network having a significantly larger parameter size and trained on significantly more images.
Figure S1: Mean squared error training curves for encoding models for PPA (gray), OPA (pink), and RSC (green) ROIs. X-axis is number of training steps.
Figure S2: Pearson Correlation validation curves for encoding models for PPA (gray), OPA (pink), and RSC (green) ROIs. X-axis is number of training steps. Training was terminated using early stopping based on the validation curves.
S3.2 Results using Different Architectures and Interpretability Methods
We integrated both gradCAM [38] and raw attention techniques [39]. The former utilizes input features combined with network gradients, while the latter leverages attention scores from transformer architectures. As shown in Figure S4, our results affirm consistency across diverse interpretability methods and network architectures.
S3.3 GQA Data Visualization
We show example images and annotations for the GQA dataset in Figure S5. We used Segment Anything [46] to obtain the segmentation masks from the GQA box annotations.
S3.4 Histograms of the Spatial Measures
We show the distribution of voxel preferences for each high-level visual ROI in Figure S6 and for the scene ROIs in Figure S7. We observe that subjects are largely consistent in their distributions for each ROI. We also observe that some ROIs and measures are multi-modal, such as in several measures in Medial and Gaussian curvature in Ventro-lateral and Lateral. This may indicate a further specialization of voxel clusters in these areas.
Figure S3: A. Pearson correlation coefficient plotted on a flatmap for all ROIs on a held-out test set of NSD data for example Subject S1 for our response-optimized networks. B. Same as A but for the features of Lescroart and Gallant [20]. C. Mean Pearson correlation for our model, the features from Lescroart and Gallant [20], and the ImageNet task-optimized features [61] on held-out NSD brain data using Pearson correlation for the scene-selective ROIs.
S3.5 Places365 Evaluation
In addition to the GQA dataset, we evaluate for category selectivity in the Places365 dataset, where segmentation masks are obtained using the Unified Perceptual Parsing image segmentation network [62] previously trained on 20,000 scene-centric images from the ADE20k dataset [63]. We report median IOU (mean  $\pm$  standard deviation across 8 subjects) in Figure S8.
S3.6 Unit Visualizations
We include additional single-voxel visualizations for RSC, OPA, and PPA in Figure S9, Figure S10, and Figure S11, respectively.
S3.7 Brain Flatmaps
In this section, we include brain flatmaps for all eight subjects for metric depth (Figure S12), relative depth (Figure S13), surface normals (Figure S14), Gaussian curvature (Figure S15), and shading (Figure S16).
S3.8 WordCloud Visualizations
We offer a WordCloud representation to visualize the top 20 categories that an ROI selects for. In this visualization, the category size represents the magnitude of the median IOU. We show WordClouds for all high-level visual ROIs and scene ROIs for the GQA dataset (Figure S17) and the Places365 dataset (Figure S18).
Network Dissection (CNN)
Raw Attention (Transformer)
gradCAM (CNN)
Figure S4: Depth and surface normal results for the Network Dissection [6], gradCAM [38] and raw attention techniques [39] for the scene-selective ROIs.
Figure S5: GQA image with bounding box annotations (left). Segment Anything [46] output given the box annotations as input (right).
Figure S6: Histogram representation of each measure for each high-level visual ROI. Each color represents a different subject. Gray vertical dotted lines indicate the grand mean, and red vertical dotted lines indicate the  $95\%$  confidence intervals.
Figure S7: Histogram representation of each measure for each scene ROI. Each color represents a different subject. Gray vertical dotted lines indicate the grand mean, and red vertical dotted lines indicate the  $95\%$  confidence intervals.
Figure S8: The median IOU (mean  $\pm$  standard deviation across 8 subjects) for the top 20 concepts for each high-level visual ROI for the places365 dataset.
Figure S9: Unit visualization plots for four additional RSC units aligned with the mean depth selectivity of the ROI (note images are down-sampled to save memory). Each unit displays its top 50 images by predicted response for that unit. For the 50 images, we show 7 sets of images (by row - top to bottom): 1. original RGB; 2. Dissection mask overlaid on the RGB image; 3. Dissection mask overlaid on the surface normal map; 4. Dissection mask overlaid on the depth map; 5. Dissection mask overlaid on the Gaussian curvature map; 6. Dissection mask overlaid on the shading map; 7. heatmap visualization of the voxel feature map for the image.
Figure S10: Unit visualization plots for four additional OPA units aligned with the mean depth selectivity of the ROI (note that images are down-sampled to save memory). Same format as Figure S9.
Figure S11: Unit visualization plots for four additional PPA units aligned with the mean depth selectivity of the ROI (note that images are down-sampled to save memory). Same format as Figure S9.
Figure S12: Metric depth on flatmaps colored by selectivity per voxel for subjects S1-S8.
Figure S13: Relative depth on flatmaps colored by selectivity per voxel for S1-S8.
Figure S14: Surface normals on flatmaps colored by selectivity per voxel for S1-S8.
Figure S15: Gaussian curvature on flatmaps colored by selectivity per voxel for S1-S8.
Figure S16: Shading on flatmaps colored by selectivity per voxel for S1-S8.
Ventrolateral
Ventromedial
Parietal
Medial
Figure S17: WordCloud for top 20 categories that an ROI selects for in the GQA dataset. In this visualization, the category size represents the magnitude of the median IOU
Figure S18: WordCloud for top 20 categories that an ROI selects for in the Places365 dataset. In this visualization, the category size represents the magnitude of the median IOU