A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision

Domain: neuroscience
Published: 2026-02-09
Editorial outcome: major_revision

Abstract
Large-scale visual neural datasets such as the Natural Scenes Dataset (NSD) are enabling models of the brain with performances beyond what was possible just a decade ago. However, because the stimuli of these datasets typically live within a common naturalistic visual distribution, they make it challenging to implement out-of-distribution (OOD) generalization tests crucial for the development of robust brain models. Here, we address this by releasing NSD-synthetic, a dataset of 7T fMRI responses from the same eight NSD participants for 284 synthetic images. We show that NSD-synthetic’s fMRI responses reliably encode stimulus-related information and are OOD with respect to NSD; that OOD generalization tests on NSD-synthetic reveal differences between brain models that are not detected in-distribution; and that the degree of OOD (quantified as the test data distance from the training data) is predictive of the magnitude of model failures. Together, NSD-synthetic enables OOD generalization tests that facilitate the development of more robust models of visual processing.

Introduction
Recently, there has been an increase in both collection and use of large-scale visual neural datasets that extensively sample brain responses to visual stimuli, suggesting that the field of vision science is entering a new era of big open data1–7. One example of this recent trend in the context of human fMRI is the Natural Scenes Dataset (NSD) which provides high-quality 7T fMRI responses to over 70,000 images of naturalistic scenes across 8 participants1. Its unprecedented size and quality have made NSD one of the most popular go-to datasets for research in computational neuroscience, where data-hungry machine- and deep-learning models are used to understand the functioning of the brain8–13. Within three years since its release, NSD has been used in hundreds of research projects, leading to the development of state-of-the-art predictive models of neural responses to visual stimulation14–16 and to new theory formation17–21. Thus, large-scale visual neural datasets such as NSD are enabling new progress in computational models of the brain, leading to new theoretical insights.

However, it remains difficult to assess the level of out-of-distribution (OOD) generalization of brain models built using recent large-scale visual neural datasets, that is, whether model predictions generalize outside of the stimulus distribution on which they are trained22. This is because, despite the unprecedented size of these datasets, their stimuli typically live within the same visual distribution, which comprises only a fraction of the vast visual space that our brains process during our lifetime. For example, through clustering and reconstruction analyses, Shirakawa and colleagues showed that NSD’s stimulus images have limited visuo-semantic diversity23. As a result, brain models are typically tested in-distribution (ID), that is, within the visual distribution on which they are trained. However, OOD tests are critical for visual neuroscience research in three significant ways. First, successful models should predict brain responses under a broad range of situations. In that sense, OOD generalization serves as a stricter but essential assessment of model robustness and validity than ID tests. Poor OOD generalization indicates that theoretical inferences of brain function might not apply beyond stimuli from the train distribution, suggesting that further development is needed to capture stimulus-to-brain-response relationships. Second, breaking down generalization scores across different types of OOD conditions can reveal stimulus properties that neural models fail to account for24, hence providing explicit targets for model improvement. Third, OOD generalization scores serve to distinguish among models that may otherwise have similar ID test scores25,26. Finding that one of several similarly-performing models generalizes better OOD can reveal important properties that make models successful (e.g., architecture, training diet, learning objective), informing the engineering of more robust models and inspiring new hypotheses about brain function.

To address the lack of OOD components in large-scale visual neural datasets and enable crucial OOD generalization tests of brain models, here we release a companion dataset to NSD called NSD-synthetic (for the remainder of the paper, we refer to NSD as NSD-core, to distinguish it from NSD-synthetic). NSD-synthetic consists of fMRI responses for an additional scan session from the same eight participants of NSD-core. During this session, fMRI responses were measured to 284 synthetic (non-naturalistic) stimuli while the participant performed either a fixation task or a one-back task. Through computational analyses and modeling we show that NSD-synthetic’s fMRI responses reliably encode stimulus-related information (Fig. 2) and that the responses are OOD with respect to NSD-core (Fig. 3). Furthermore, we provide proof of principle that encoding models trained on NSD-core generalize OOD on NSD-synthetic with lower prediction accuracies compared to ID generalization on NSD-core (Fig. 4); that OOD tests on NSD-synthetic reveal differences between encoding models not detected by ID tests on NSD-core (Fig. 5); that the degree of OOD is a useful indicator of the magnitude of model failures (Fig. 6); and that the concept of OOD can be usefully applied even within the domain of naturalistic stimuli (Fig. 7). Together, as the OOD companion of NSD-core, NSD-synthetic enables strict OOD generalization tests critical for development of more robust models of visual processing and the formulation of more accurate theories of human vision.

Results
NSD-synthetic stimuli and experimental design
To enable strict OOD generalization tests of computational models of visual processing, we created a set of 284 synthetic images that are far removed from the natural scene images in NSD-core (Fig. 1a). These synthetic images are divided into 8 classes, and every class contains multiple subclasses of 4 images each. The image classes include: various types of noise (4 subclasses, 16 images); natural scenes (2 subclasses, 8 images); manipulated version of natural scenes (3 subclasses, 12 images); contrast modulation (4 subclasses, 16 images); phase-coherence modulation (4 subclasses, 16 images); single words varying in position (10 subclasses, 40 images); spiral gratings varying in orientation and spatial frequency (28 subclasses, 112 images); and chromatic noise varying in hue (16 subclasses, 64 images) (Fig. 1b).

We presented these images in a rapid event-related design consisting of 4-s trials (2-s ON, 2-s OFF; instead of NSD-core’s 3-s ON, 1-s OFF trials) while measuring fMRI responses (7T, 1.8-mm resolution, TR 1.6 s) from each of the eight NSD participants (Fig. 1c). To assess potential task dependence of neural responses, participants performed a fixation task and a one-back task in alternating runs (instead of NSD-core’s long-term continuous recognition task) (Fig. 1d). In the fixation task, participants reported brightness increments and decrements of the fixation dot. In the one-back task, participants reported whether the current image was identical to the previous image. The participants exhibited good behavioral compliance: overall percent correct in the fixation task was 98%, 97%, 80%, 94%, 97%, 92%, 76%, and 83%, and overall d’ in the one-back task was 3.3, 2.6, 1.6, 2.7, 2.6, 3.2, 2.2, and 1.9. Since, for the purposes of this paper, our goal is to showcase NSD-synthetic’s out-of-distribution image properties, we pooled fMRI responses across tasks and analyzed fMRI responses with respect to the stimulus presented.

NSD-synthetic’s fMRI responses reliably encode stimulus-related information
NSD-synthetic’s main purpose is to facilitate out-of-distribution (OOD) generalization tests aimed at improving the robustness of computational models of the brain. A prerequisite of these tests is that NSD-synthetic’s fMRI responses must reliably encode the different stimulus images. To assess this, we calculated NSD-synthetic’s noise ceiling signal-to-noise ratio (NCSNR), a measure of stimulus-related signal in the fMRI responses, using methods introduced in previous work1. For all participants, we found that NCSNR scores are high over visual cortex, ranging approximately from 0.75 to 2 (Fig. 2a). This indicates that 36–80% variance in single-trial responses reflects signals driven by the stimulus. As a comparison, NSD-core’s NCSNR scores range approximately from 0.5 to 1.25, indicating that 20–60% variance in single-trial responses reflects signals driven by the stimulus (Supplementary Fig. 1). Hence, stimulus-related information is indeed reliably encoded in NSD-synthetic’s fMRI responses. Within visual cortex, NSD-synthetic’s NCSNR is highest in early areas compared to intermediate, ventral, dorsal, and lateral areas. We believe this reflects the limited visual and semantic complexity of NSD-synthetic’s stimuli, which leads to reduced signals in higher visual areas that preferentially respond to more complex visual features27–32. Supporting this intuition, the visually and semantically more complex naturalistic images from NSD-core lead to NCSNR scores that are more uniform across visual cortex (Supplementary Fig. 1).

To further assess the quality of the encoded visual information, we next analyzed NSD-synthetic’s univariate and multivariate fMRI responses from both early visual cortex (EVC) and high-level visual cortex (HVC) regions of interest (ROIs). We chose ROIs based on their known responsiveness for the visual features defining NSD-synthetic’s stimulus images, so as to best characterize the visual information encoded in NSD-synthetic’s fMRI responses. The EVC ROIs consisted of V1, V2, V3, and hV4, due to their responsiveness to low- and mid-level visual features such as spatial frequency33, contrast energy34, edges35, and color36. The HVC ROIs consisted of the parahippocampal place area (PPA) due to its responsiveness to visual scenes29 and of the visual word form area (VWFA) due to its responsiveness to visual words37.

We begin by showing that NSD-synthetic’s univariate fMRI responses—defined as the average activity over all vertices within each ROI—for each image subclass reproduce known tunings of EVC and HVC ROIs. In Fig. 2b, we see that images of scenes activate PPA more than images consisting of noise, images with low contrast, and images with low phase-coherence, in line with PPA’s tuning to visual environments29. Additionally, univariate responses increase substantially with increasing contrast for EVC ROIs38. Finally, images consisting of noise and images with low phase coherence activate EVC ROIs more than HVC ROIs, in line with EVC’s selectivity for low-level visual features such as contrast energy34. In Fig. 2c, we find that chromatic noise activates EVC more than HVC, in line with EVC’s role in the processing of color36 and the fact that the chromatic noise patterns lack high-level structure. In Fig. 2d, we see that single words activate VWFA the most, especially when presented foveally (i.e., Word4 Pos3, Word6 Pos3), in line with VWFA’s tuning to visual words37. In Fig. 2e, we see that spiral gratings activate EVC more than HVC, in line with EVC’s general responsiveness to contrast energy34. Among all EVC ROIs, V1 is driven by spiral gratings the most, and shows preferential tuning for certain spatial frequencies within each spiral grating type (e.g., Spiral A SF4, Spiral B SF4, Spiral C SF4)33,39,40.

Next, we analyze the visual information in NSD-synthetic’s multivariate fMRI responses—i.e., population response patterns over all vertices within each ROI. Through representational similarity analysis (RSA)41, we assessed the similarity of multivariate fMRI responses between each participant and ROI. For each ROI, we found that NSD-synthetic’s multivariate fMRI responses are correlated between participants (RSA scores within the cyan boxes in Fig. 2f), indicating that participants have shared visual representations. The cross-participant RSA scores are higher for EVC than for HVC, in line with the NCSNR and univariate response results indicating EVC ROIs are more responsive to NSD-synthetic’s stimulus images than HVC ROIs (Fig. 2a–e). We also found that the multivariate fMRI responses are correlated between ROIs (RSA scores outside the cyan boxes in Fig. 2f), again more so for EVC ROIs. In particular, RSA scores from ROIs in the same participant are elevated (k-th subdiagonals in Fig. 2f), which may be indicative of spatial noise correlations shared across areas. However, the multivariate responses from different ROIs are also highly similar across participants (off-subdiagonal entries in Fig. 2f). Since noise is uncorrelated between participants, this bypasses the effect of spatial noise correlations and indicates similarity of visual representations across areas.

In summary, NSD-synthetic’s fMRI responses reliably encode stimulus-related information with the strongest stimulus-driven responses occurring in EVC ROIs. The encoded information conforms to known tuning properties of EVC and HVC ROIs, and is consistent across participants.

NSD-synthetic’s fMRI responses are out-of-distribution with respect to NSD-core
To be well suited for OOD generalization tests, NSD-synthetic’s fMRI responses should not only reliably encode stimulus-related visual information, but should also exhibit a different distribution compared to NSD-core’s responses. To ascertain this, we reduced NSD-synthetic and NSD-core’s fMRI responses to two dimensions using multidimensional scaling (MDS)42,43, and compared the resulting two-dimensional MDS embeddings. To evaluate whether differences in distribution are due to unwanted session-specific effects, we applied MDS jointly on NSD-synthetic’s single-trial fMRI responses and on the single-trial fMRI responses from two NSD-core scan sessions (sessions 10 and 20), aggregated across participants. We found that the single-trial fMRI responses for NSD-synthetic form a distinct cluster compared to the single-trial fMRI responses from NSD-core (Fig. 3a). The fact that the single-trial fMRI responses from the two NSD-core scan sessions overlap in the MDS embedding space suggests that the difference between NSD-synthetic and NSD-core’s data distributions is not driven by session-specific effects, but rather by stimulus-related signals.

These two-dimensional MDS embeddings additionally revealed two clear patterns visible by eye. First, NSD-synthetic’s responses for scene images are closer in the MDS embedding space to NSD-core (Fig. 3a), which is reasonable given that NSD-core consists of fMRI responses to scenes. Second, NSD-synthetic’s fMRI responses cluster based on the image class to which they belong (Fig. 3a). To visualize this more effectively, we applied MDS on NSD-synthetic’s trial-averaged fMRI responses for the 284 stimulus images only, and plotted these images positioned at the two-dimensional MDS embedding coordinates (Fig. 3b). Again, we see a clear clustering based on image classes. This clustering is consistent with properties of NSD-synthetic’s stimulus images that are visible by eye, and is geometrically similar to the class clustering from applying MDS jointly on NSD-synthetic and NSD-core’s responses (Fig. 3a).

In summary, NSD-synthetic’s fMRI responses are differently distributed (i.e., OOD) with respect to NSD-core. These distributional differences are driven by the distinctness of NSD-synthetic’s visual stimuli, and the similarity structure of NSD-synthetic’s fMRI responses appears to mirror what is visible in the stimulus images.

NSD-synthetic enables out-of-distribution generalization tests for models of the brain
Having established that NSD-synthetic is OOD with respect to NSD-core, we turn to assessing whether NSD-synthetic is useful for OOD generalization tests of computational models of the brain. As computational models we used neural encoding models, that is, predictive algorithms that generate brain responses to arbitrary stimulus images13,44–47. We expected that encoding models trained on NSD-core would generalize OOD on NSD-synthetic with lower prediction accuracies than they would for ID generalization.

The encoding models consisted of linear regressions that map image features–downsampled to 250 principal components (PCs) using principal component analysis (PCA)48– onto the fMRI responses of each vertex. For image features we used the activations of AlexNet49, a convolutional deep neural network pretrained on image classification that is a well-established standard in computational visual neuroscience research. We trained the linear regression weights using image features and fMRI responses from NSD-core, and tested their generalization performance both ID and OOD. For ID tests, we used a separate data partition of NSD-core not used for model training; for OOD tests, we used NSD-synthetic. We quantified generalization performance by correlating the predicted fMRI responses with their recorded (i.e., experimentally collected) counterparts, obtaining ID and OOD explained variance scores for each participant and vertex (Fig. 4a). Critically, explained variance scores are determined not only by the encoding model’s generalization performance, but also by the level of noise in the recorded fMRI responses (which might differ between the ID and OOD fMRI responses). Thus, to make the ID and OOD explained variance scores more meaningful and directly comparable, we normalized them with the noise ceiling of the corresponding recorded fMRI responses (Fig. 4b). This resulted in ID and OOD noise-ceiling-normalized explained variance scores that indicate the amount of explainable variance accounted for by the models for each vertex and participant (Fig. 4c).

As expected, we found that the encoding models trained on NSD-core predict visual fMRI responses better ID than OOD (Fig. 4c,d) (P=10−7, paired sample t-test, one-sided, N=8 participants). The fact that OOD generalization is lower is crucial, as it indicates that further improvement is needed to capture stimulus-to-brain-response relationships, thus validating NSD-synthetic’s usefulness for model and theory development. We further observed that decreases in OOD performance compared to ID performance are consistent across both lower- and higher-level visual areas (see red regions in Fig. 4d). Importantly, these decreases in performance are not due to lack of explainable signal in NSD-synthetic’s fMRI responses, but rather to model failures. When plotting the vertex-wise explained variance scores against their noise ceilings, we found that compared to ID tests on NSD-core, the encoding models’ explained variance scores were consistently lower relative to the noise ceiling for OOD tests on NSD-synthetic, across both lower- and higher-level visual areas (Fig. 4e).

To ensure that the lower OOD performances are meaningful and robust, we performed three controls. First, to ensure that the lower performance on NSD-synthetic isn’t simply due to the fact that it was collected in a separate scan session, we trained and tested the encoding models using fMRI responses from separate NSD-core scan sessions (Supplementary Fig. 2). Second, to ensure that the specific regression method that one chooses to use does not change our conclusions, we trained and tested the encoding models using individual AlexNet layers, by either mapping each layer’s first 250 PCs onto fMRI responses with linear regression, or by mapping the layer’s full feature space using ridge regression (Supplementary Figs. 3-4). Third, to ensure that our results are not contingent on a specific choice of deep neural network, we trained and tested the encoding models on image features from deep neural networks other than AlexNet (Supplementary Figs. 5–7). In all three control analyses, we observed lower OOD performances for NSD-synthetic than for NSD-core. Importantly, controlling for session-specific effects led to a considerable decrease in ID (Δ mean noise-ceiling-normalized explained variance score of 6.5%), but not OOD (Δ mean noise-ceiling-normalized explained variance score of 0.18%) generalization. Therefore, for a clearer comparison between ID and OOD generalization performances using NSD-synthetic, we recommend researchers to train and test their models on data from independent scan sessions. Furthermore, neither of the two regression methods led to best OOD generalization performances for all AlexNet layers, suggesting that the optimal regression strategy might be contingent on the specific data scenario.

In summary, we showed that encoding models trained on NSD-core generalize ID and, to a lower extent, OOD on NSD-synthetic. The lower OOD generalization performance opens the door for using NSD-synthetic in combination with NSD-core to perform OOD generalization tests for model and theory improvement.

Out-of-distribution generalization tests reveal differences between brain models not detected by in-distribution tests
After demonstrating that NSD-synthetic enables OOD generalization tests, we ask whether these tests reveal differences between brain models that are not detected by ID tests. If so, NSD-synthetic would provide unique insight essential for developing more robust models of the brain.

To assess this, we compared the ID and OOD noise-ceiling-normalized explained variance scores of AlexNet-based encoding models (Fig. 5a) with the scores of encoding models based on a more modern computer vision architecture, a vision transformer pretrained on image classification called vit_b_3250 (Fig. 5b). When tested ID, we found that AlexNet outperformed vit_b_32 in lower-level visual areas (blue areas in Fig. 5c, left panel) and that vit_b_32 outperformed AlexNet in higher-level visual areas (red areas in Fig. 5c, left panel). This ID difference between the two models was modest, peaking at 10% of absolute explained variance difference (Fig. 5c, left panel). Surprisingly, when testing the models OOD we found that vit_b_32 instead outperformed AlexNet across both lower- and higher-level visual areas (red areas in Fig. 5c, right panel). This OOD difference between the two models was much stronger than their ID difference, peaking at 25% of absolute explained variance difference (Fig. 5c, right panel). Thus, OOD tests revealed differences between brain models not detected by ID tests. These differences suggest that vit_b_32 is a more robust model of visual cortex compared to AlexNet. Furthermore, the magnitudes of absolute explained variance difference scores indicate that OOD tests allow to better disentangle brain models compared to ID tests.

In a last comparison, we assessed whether OOD tests reveal unique differences also between brain encoding models based on highly similar deep neural networks. Specifically, we compared encoding models based on ResNet5051, a convolutional neural network with skip connections pretrained on task-supervised image classification, with encoding models based on MoCo52, consisting of the same ResNet50 architecture but trained with self-supervised contrastive learning53. When tested ID, we found that ResNet50 outperformed MoCo in higher-level visual areas (blue areas in Fig. 5d, left panel), and that MoCo outperformed ResNet50 in lower-level visual areas (red areas in Fig. 5d, left panel). When tested OOD, MoCo instead outperformed ResNet50 across both lower- and higher-level visual areas (red areas in Fig. 5d, right panel). The ID difference between the two models was smaller (peaking at 5% of absolute explained variance difference, Fig. 5d, left panel) compared to their OOD difference (peaking at 10% of absolute explained variance difference, Fig. 5d, right panel), again indicating that OOD tests allow to better disentangle brain models compared to ID tests. Thus, OOD tests reveal differences not detected by ID tests also for encoding models based on highly similar deep neural networks.

To ensure that the OOD differences not detected by ID tests are meaningful and robust, we performed two controls. First, to ensure that our results are not simply due to the fact that NSD-synthetic was collected in a separate scan session, we trained and tested the encoding models using fMRI responses from separate NSD-core scan sessions (Supplementary Fig. 8). Second, to ensure that the specific regression method that one chooses to use does not change our conclusions, we trained and tested the encoding models on image features from individual ResNet50 or MoCo layers, by either mapping each layer’s first 250 PCs onto fMRI responses with linear regression, or by mapping the layer’s full feature space using ridge regression (Supplementary Figs. 9–12). In both control analyses, OOD tests revealed differences between models not detected by ID tests.

In summary, these results offer a proof of principle that OOD tests reveal differences between models that are not detected ID. This indicates that the outcomes of OOD and ID generalization tests do not imply one another, and therefore that NSD-synthetic provides unique insight for model development.

The degree of out-of-distribution is informative of brain model failures
The neural responses on which a brain model is tested can exhibit various degrees of OOD–that is, various degrees of distributional distance–with respect to the responses on which the model is trained. Therefore, we ask whether there is a relationship between the extent to which the fMRI responses used to test an encoding model are differently distributed from the train responses and the encoding model’s generalization performance to these test responses. If so, the degree of OOD could serve as an indicator of the stimulus properties that encoding models might fail to account for, which may help guide the engineering of more robust models24,54. We operationally define the degree of OOD as the Euclidean distance between the fMRI response to a given stimulus of interest and the NSD-core train responses in two-dimensional MDS space.

Using MDS, we reduced the trial-averaged fMRI responses for NSD-synthetic (used to test the encoding models) and for NSD-core (used to train the models), to two dimensions. Within this two-dimensional embedding space, we then computed the Euclidean distance between the responses of each NSD-synthetic image and the responses of each NSD-core image, and averaged these distance scores across all NSD-core images, as well as across NSD-synthetic images belonging to the same image class. This resulted in distributional distance scores indicating, for each NSD-synthetic image class, the degree of OOD of its fMRI responses from NSD-core’s responses (Fig. 6a).

Next, we compared these distributional distance scores with the noise-ceiling-normalized encoding accuracy scores of vision-transformer-based encoding models trained on NSD-core and tested on each NSD-synthetic image class (Fig. 6b). We found that the distributional distance scores are anticorrelated with the encoding accuracy scores (Spearman’s ρ=-0.36, P=0.0004, one-sample t-test, one-sided, N=8 participants) (Fig. 6c), indicating that encoding models generalize better to fMRI responses that have lower distributional distance to the responses on which the models are trained.

As a complementary way of assessing the encoding models’ generalization performance for the different NSD-synthetic image classes, we used the models’ predicted fMRI responses to identify, in a zero-shot fashion, the stimulus images of NSD-synthetic’s recorded fMRI responses2,44,55,56 (choosing among 8 randomly selected images per NSD-synthetic image class, for a total of 64 images; the final identification scores reflect the average across 1000 analysis iterations, each with a different random selection of 8 images per image class) (Fig. 6d). We found that the ranks of the correct stimulus image (i.e., the position at which the correct image appears among the models’ choices) correlate with the distributional distance scores (Spearman’s ρ=0.65, P=10−6, one-sample t-test, one-sided, N=8 participants) (Fig. 6e), indicating that encoding models more reliably identify stimulus images when the corresponding fMRI responses have lower distributional distance to the model’s train responses.

We obtained quantitatively similar results when using encoding models based on other deep neural network architectures (Supplementary Figs. 13–15), indicating that the correlation between distributional distance scores and encoding model performance is not contingent on a specific choice of deep neural network.

In summary, these results show that the degree to which an encoding model’s test fMRI responses are OOD with respect to the model’s train responses is informative of the model’s generalization performance to these test responses. Thus, the degree of OOD is a useful indicator of the stimulus properties that encoding models might fail to account for.

NSD enables out-of-distribution tests of brain models across both naturalistic and synthetic stimulus images
Up to here, we implemented OOD generalization tests using NSD-synthetic. To highlight that OOD tests need not be restricted to neural responses for synthetic stimuli, in this last analysis, we assess the OOD generalization performance of vision-transformer-based encoding models to fMRI responses for both naturalistic and synthetic images.

We first used a combination of MDS and clustering algorithms to divide NSD-core’s fMRI responses into train, ID test, and naturalistic OOD test splits23. For each participant, we reduced the trial-averaged fMRI responses for NSD-core’s images to two dimensions through MDS, and partitioned this two-dimensional embedding space into 15 clusters using k-means clustering. For the naturalistic OOD test split we considered the fMRI responses from the cluster with highest Euclidean distance from the centroids of the remaining 14 clusters and, within it, selected fMRI responses for the 284 images with the highest Euclidean distance from the remaining clusters. For the ID test split we selected fMRI responses for 284 images randomly selected from the remaining 14 clusters. As the train split, we selected the images from the remaining 14 clusters not belonging to the ID test split. These procedures ensured that the naturalistic OOD test responses are maximally distant in MDS embedding space from the distribution of the train responses, and that the train and ID test responses come from the same distribution (Fig. 7a). We then trained encoding models using the train split and evaluated them ID (on NSD-core), as well as on naturalistic (on NSD-core) and synthetic (on NSD-synthetic) OOD test fMRI responses.

We obtained higher noise-ceiling-normalized encoding accuracy scores for ID testing on NSD-core, followed by naturalistic OOD testing on NSD-core, and finally by synthetic OOD testing on NSD-synthetic (Fig. 7b). We obtained the same ranking when using the encoding models’ predicted fMRI responses to identify, in a zero-shot fashion, the stimulus images of the ID, naturalistic OOD, and synthetic OOD test fMRI responses (choosing among all 852 images from the three test splits) (Fig. 7c). This ranking was expected, as it reflects the fact that the ID, naturalistic OOD, and synthetic OOD test fMRI responses are increasingly distant from the train responses in MDS space (Fig. 7a).

To ensure that this ranking between test splits is meaningful and robust, we performed two controls. First, to ensure that the ranking is not contingent on a specific choice of deep neural network, we trained and tested the encoding models on image features from deep neural networks other than the vision transformer (Supplementary Figs. 16–18). Second, to ensure that the way in which the test splits are defined does not change our conclusions, we selected the images of the ID and naturalistic OOD test splits based, not on empirically observed fMRI responses, but rather on the features of a deep neural network (Supplementary Fig. 19). In both control analyses, we observed decreasing model generalisation performances for ID, naturalistic OOD, and synthetic OOD test splits.

In conclusion, these results showcase the strength of combining NSD-core and NSD-synthetic to test the OOD robustness of brain models on neural responses. The results also demonstrate that the concept of OOD is not restricted to artificial stimuli, but that through carefully-designed train-test splits it can be usefully applied even within the domain of naturalistic stimuli.

Methods
Procedures for data acquisition, data pre-processing, and GLM analysis were the same as in NSD-core. For full details of these procedures, please refer to NSD-core’s paper1. Below, we provide a detailed summary of the procedures.

Participants
The NSD-synthetic experiment was conducted with the same set of eight participants who took part in NSD-core. These participants (subj01–subj08) were right-handed with normal or corrected-to-normal visual acuity and no color blindness, and consisted of two males and six females, with age range 19–32 years. Informed written consent was obtained from all participants, and the experimental protocol was approved by the University of Minnesota Institutional Review Board. Participants were compensated at a rate of $30 per hour, plus performance bonuses. Additional participant information, including height, weight, handedness and visual acuity, was logged and is available online.

MRI data acquisition
MRI data were collected at the Center for Magnetic Resonance Research at the University of Minnesota. Functional MRI data were collected using a 7T Siemens Magnetom passively shielded scanner and a single-channel-transmit, 32-channel-receive RF head coil (Nova Medical). Data for NSD-synthetic were collected in one scan session that took place after completion of the NSD-core experiment. The fMRI data were collected using gradient-echo EPI at 1.8 mm isotropic resolution with whole-brain coverage (84 axial slices, slice thickness 1.8 mm, phase encode direction anterior-to-posterior, TR 1600 ms, TE 22.0 ms, partial Fourier 7/8, iPAT 2, multi-band slice acceleration factor 3). In addition to the EPI scans, the scan session also included dual-echo field maps for post hoc correction of EPI spatial distortion (same overall slice slab as the EPI data, 2.2 mm × 2.2 mm × 3.6 mm resolution, TE1 8.16 ms, TE2 9.18 ms). The acquisition structure for the NSD-synthetic scan session was [FXOXOFXOXOF], where F indicates a field map, X indicates a fixation run (268 TRs), and O indicates a one-back run (268 TRs).

Stimulus display and scanner peripherals
Stimuli were presented using a Cambridge Research Systems BOLDscreen 32 LCD monitor positioned at the head of the 7T scanner bed. The monitor operated at a resolution of 1920 pixels × 1080 pixels at 120 Hz. Participants viewed the monitor via a mirror mounted on the RF coil. The size of the monitor image was 69.84 cm (width) × 39.29 cm (height), and the viewing distance was 176.5 cm. Measurements of display spectral power density were obtained using a PR-655 spectroradiometer (Photo Research). A Mac Pro computer controlled stimulus presentation using code based on Psychophysics Toolbox 3.0.14. Behavioral responses were recorded using a button box (Current Designs). Ear plugs were used to reduce acoustic noise, and head motion was mitigated using head cases.

Eye-tracking was performed using an EyeLink 1000 system (SR Research). We caution that the eye-tracking data are variable in quality due to difficulties achieving sufficient pupil contrast in some participants. In addition, eye-tracking data for the first NSD-synthetic run for subj05 were corrupted and are therefore not available. For full details of eye-tracking acquisition and pre-processing, see NSD-core’s paper1. Overall, we found that participants were generally able to maintain good central fixation during our various fMRI experiments, with the possible exception of subj05 (see NSD-core’s Extended Data Fig. 41).

The NSD-synthetic experiment
Stimuli. Stimuli consisted of 284 distinct images. All images were presented under two different tasks, a fixation task and a one-back task. A high-level description of the images is as follows (the number of images is indicated in parenthes): white noise (4), white noise with a large block size (4), pink noise (4), natural scenes (4), upside-down versions of these scenes (4), Mooney versions of these scenes (4), line-drawing versions of these scenes (4), contrast-modulated natural scenes (4 scenes × 5 contrast levels (100%, 50%, 10%, 6%, 4%) = 20), phase-coherence-modulated natural scenes (4 scenes × 4 coherence levels (75%, 50%, 25%, 0%) = 16), single words (2 word lengths × 5 positions × 4 words = 40), spiral gratings varying in orientation and spatial frequency (112), and chromatic pink noise varying in hue (68). Images were prepared at a resolution of 1360 pixels (width) × 714 pixels (height), corresponding to a visual field extent of 16° × 8.4°. Most images occupy the central 8.4° × 8.4°, which is the stimulus size used in NSD-core; a few word stimuli extend beyond this central region (details below). Of the 284 images, the first 220 images are achromatic. Pixel luminance values for these images were prepared in the range 0–1 and then transformed using a color lookup table that increases linearly in luminance (L+M) with the chromaticity of D65. In this lookup table, the RGB values corresponding to the minimum and maximum luminance are (0, 0, 0) and (252, 220, 216), respectively. The remaining 64 images are chromatic, and their design is described in further detail below.

The images are hierarchically organized: there are 284 distinct images belonging to 71 image subclasses, and these 71 image subclasses belong to 8 image classes (Noise, Natural scenes, Manipulated scenes, Contrast modulation, Phase-coherence modulation, Single words, Spiral gratings, and Chromatic noise). Below, we describe each of the image subclasses. The parentheses at the end of each description indicate the associated image class.

White noise: Image subclass 1 (images 1–4). Each image consists of pixels whose luminance values were randomly drawn from a uniform distribution between 0 and 1. Four distinct images were generated (Noise).

White noise (large block): Image subclass 2 (images 5–8). Pixels are grouped into block sizes of 42 pixels × 42 pixels (0.49° × 0.49°). Luminance values for each block were randomly drawn from a uniform distribution between 0 and 1. Four distinct images were generated (Noise).

Pink noise: Image subclass 3 (images 9–12). Images with 1/f amplitude spectra were generated. This was achieved by multiplying white Gaussian noise by the desired amplitude spectrum. For each image, pixel values from 3.5 standard deviations below and above the mean were linearly mapped to the range 0–1. Four distinct images were generated (Noise).

Natural scenes: Image subclass 4 (images 13–16). Four natural scenes were chosen from the stimulus set used in a previous study69; the motivation for choosing from this stimulus set is that the natural scenes have associated line drawings (which are used in image subclass 7). The four chosen scenes depict mountains, a highway, a street, and a desk. Each natural scene was converted to grayscale, square cropped, resized to 714 pixels × 714 pixels, converted to luminance values assuming a display gamma of 2.0, and then contrast-normalized such that the 0.1 and 99.9 percentiles were linearly mapped to the range 0–1 (Natural scenes).

Upside-down scenes: Image subclass 5 (images 17–20). The prepared images from image subclass 4 were rotated by 180° (Manipulated scenes).

Mooney scenes: Image subclass 6 (images 21–24). We binarized the prepared images from image subclass 4 by thresholding each image at the median pixel value. The resulting images have pixel values of either 0 (black) or 1 (white), and are akin to Mooney images68. Participants were not explicitly pre-exposed to the intact versions of the images, but were presumably familiarized with the intact images over the course of the scan session (Manipulated scenes).

Line-drawing scenes: Image subclass 7 (images 25–28). Line drawings were obtained from a previous study69. In brief, trained artists at the Lotus Hill Research Institute created line drawings by tracing the contours of color photographs using a custom interface. The sequence and coordinates of each line stroke were digitally recorded. The line drawings associated with the natural scenes used in image subclass 4 were square cropped, resized to 714 pixels × 714 pixels, and then prepared as black lines (0) on a gray background (0.5) (Manipulated scenes).

Natural scenes (second set): Image subclass 8 (images 29–32). A second set of four natural scenes were chosen from the stimulus set used in the previous study69. The four chosen scenes depict a forest, a superhighway, a boulevard, and cubicles. The same image preparation procedures were applied as in image subclass 4 (Natural scenes).

Contrast 50%, Contrast 10%, Contrast 6%, Contrast 4%: Image subclasses 9–12 (images 33–48). The prepared images from image subclass 8 were manipulated to have contrast levels of 50%, 10%, 6%, and 4%. This was achieved by designating the prepared images as having a contrast level of 100% and by scaling pixel luminance values towards 0.5 by the appropriate amount (Contrast modulation).

Phase 75%, Phase 50%, Phase 25%, Phase 0%: Image subclasses 13–16 (images 49–64). We took the prepared images from image subclass 9 (the ‘Contrast 50%‘ subclass) and manipulated the phase coherence level. Specifically, the phase spectrum of each image was blended with a random phase spectrum (e.g., ‘Phase 75%‘ indicates a blend of 75% original and 25% random). A different random phase spectrum was used for each image and each coherence level. Pixel luminance values were then truncated to fit the range 0–1 (only a very small amount of clipping occurred) (Phase-coherence modulation).

Word4 Pos1–5, Word6 Pos1–5: Image subclasses 17–26 (images 65–104). Each image contains a single word (black on a gray background) written in a sans-serif, monospaced font (Liberation Mono). Half of the words are 4 letters long and presented in a relatively large font size (x-height 0.4°, center-to-center spacing 0.43°, word bounding box height 0.69°), and half of the words are 6 letters long in a smaller font size (x-height 0.27°, center-to-center spacing 0.28°, word bounding box height 0.46°). The 4-letter words roughly match the 6-letter words in horizontal extent (average word width 1.64°). The 4- and 6-letter words (20 in each set) are closely matched in lexical frequency. The words span a wide range of syntactic categories (parts of speech). Each word was placed at one of five possible positions along the horizontal meridian (–6°, –3°, 0°, 3°, 6°). For each combination of word length and position, four images were generated, yielding a total of 2 word lengths × 5 positions × 4 words = 40 images. Note that the words positioned at –6° and 6° are the only images (out of the 284 NSD-synthetic images) with content that lies outside of the central 8.4° × 8.4° region. We caution that models trained on NSD-core may generalize poorly to these images due to their relatively peripheral image content (Single words).

Spiral A–D SF1–6, Spiral E–H SF4: Image subclasses 27–54 (images 105–216). These images are sinusoidal gratings in log-polar coordinates (we refer to these as ‘spiral gratings’ but they can also be called ‘log-polar gratings’). The primary advantage of spiral gratings over conventional Cartesian sinusoidal gratings is that for spiral gratings, spatial frequency varies inversely with eccentricity, thereby enabling more efficient sampling of neural tuning preferences (neurons at higher eccentricities are tuned for lower spatial frequencies)71. We created spiral gratings at full contrast. We prepared four types of spirals, Spiral A–D (pinwheels, forward spirals, annuli, reverse spirals), at six spatial frequency levels (L=6.0, 11.0, 20.0, 37.0, 69.0, 128.0), and we prepared four additional types of spirals, Spiral E–H (various intermediate spirals), at one spatial frequency level (L=37.0). For a given point in the visual field with eccentricity E (in degrees), the local spatial frequency (in cycles per degree) is given by L = (2πE); hence, for a fixed spatial frequency level L, different spirals have identical local spatial frequency content (but differ in local orientation). To avoid aliasing, a central circle was cut out of each image. The diameter of this circle was 4, 8, 14, 26, 48, and 86 pixels (0.05°, 0.09°, 0.16°, 0.31°, 0.56°, and 1.01°) for the six spatial frequency levels, respectively. We generated a total of 4 types of spirals (Spiral A–D) × 6 spatial frequency levels + 4 types of spirals (Spiral E–H) × 1 spatial frequency level = 28 image subclasses. For each image subclass, four images were generated, corresponding to four equally spaced grating phases. Hence, the total number of images was 28 image subclasses × 4 phases = 112 images. For general information about the design of spiral gratings, please see this previous study71 (Spiral gratings).

Hue 00: Image subclass 55 (images 217–220). These images are the same as the prepared images from image subclass 3 (pink noise with 1/f amplitude spectra) but lowered in contrast. The images are metameric in chromaticity with standard illuminant D65. Four images were generated (Noise).

Hue 01–16: Image subclasses 56–71 (images 221–284). These 64 images are the only chromatic images in NSD-synthetic. To create these images, we selected 16 different hues by choosing evenly spaced angles in a version of the MacLeod-Boynton chromaticity diagram72 that is based on the Stockman, MacLeod, and Johnson cone fundamentals73. We then determined the maximum saturation by fitting a circle with the maximum available radius within the gamut of the BOLDscreen display. Finally, for each hue, we created isoluminant images by taking the 4 pink noise patterns from image subclass 55 and using these patterns to modulate between zero saturation (the D65 gray point) and maximum saturation (within the determined circle). To ensure isoluminance for individual observers, we created participant-specific versions of the 64 images based on heterochromatic flicker photometry results obtained from each participant (Chromatic noise).

For most image subclasses (comprising a total of 268 images), images naturally come in groups of four for which we do not expect much variation in brain activity. For example, for the ‘White noise’ subclass, the four generated images are quite similar to each other (they are just different samples of noise). As another example, for the ‘Spiral A SF1’ subclass, the four generated images differ only with respect to the specific phase of the underlying spiral grating. Hence, for these image subclasses, we present images only once per task (with the four presentations serving as quasi-repetitions). In contrast, for the ‘Natural scenes’, ‘Upside-down scenes’, ‘Mooney scenes’, and ‘Line-drawing scenes’ subclasses (comprising a total of 16 images), each distinct scene might be expected to produce a substantially different brain activity pattern. Hence, for these image subclasses, we present images four times per task. Overall, for a complete set of trials for one task, we require 268 images × 1 presentation + 16 images × 4 presentations = 332 stimulus trials.

Trial and run design. Each trial lasted 4 s and consisted of the presentation of an image for 2 s, followed by a 2 s gap. (The motivation for the shorter 2 s duration compared to the 3 s duration in NSD-core was to reduce the strong adaptation effects caused by gratings.) In total, 107 trials were conducted in a run; thus, each run lasted 428 s. The first three trials (12 s) and the last four trials (16 s) were blank trials. The remaining 100 trials were divided into 93 stimulus trials and seven blank trials. The blank trials were randomly positioned in each run such that the minimum and maximum number of continuous stimulus trials was nine trials (36 s) and 14 trials (56 s), respectively.

We allocated four runs to complete the data for the fixation task. To determine trial ordering for these four runs, we first created a random ordering of the complete set of 332 stimulus trials, splitting these trials into 4 runs (83 stimulus trials each). This random ordering was subject to the constraint that no image was repeated back-to-back within a run. Then, for each run, we randomly selected 10 of the stimulus trials to be repeated as one-back trials. This yielded, for each run, a total of 83 regular stimulus trials + 10 inserted one-back trials = 93 stimulus trials.

A total of eight runs were collected in the NSD-synthetic scan session, with participants alternating between the fixation task and the one-back task. To ensure that differences in brain activity across tasks are not due to stimulus-related differences, we took the run design for the fixation task and used it identically for the one-back task (albeit changing the run order). Specifically, the tasks performed in the eight runs were XOXOXOXO, where X indicates the fixation task and O indicates the one-back task, and the run designs for the eight runs were ACBDCADB, where A, B, C, and D correspond to the four runs designed for the fixation task (as described above). All stimulus and experiment characteristics (e.g., trial ordering, one-back events, dot color changes) were kept identical across participants.

Stimulus presentation and task. The BOLDscreen monitor was configured to behave as a linear display device, and all stimuli were delivered using a linear color lookup table. (For natural scene stimuli, we prepared the luminance of these stimuli to simulate standard display gamma; see details above.) Stimuli occupied 16.0° (width) × 8.4° (height). Stimulus presentation was locked to the refresh rate of the BOLDscreen monitor, which was nearly exactly 120 Hz. Empirically, we confirmed that the durations of runs were highly reliable, ranging from 427.94 s to 427.96 s. Throughout each run (including blank trials), a small semi-transparent fixation dot with a black border (0.2° × 0.2°, 50% opacity) was present at the center of the stimuli. Every 1.4 s, the luminance of the dot was randomly set to one of five different linearly spaced levels (0.17, 0.37, 0.58, 0.79, 1), with consecutive repetitions allowed. Stimuli were shown against a gray background (luminance level 0.5).

The fixation task (odd runs) was intended to be a challenging task that draws cognitive resources away from the stimuli and towards the small fixation dot74. In the fixation task, participants were instructed to fixate the central dot and to press button 1 using the index finger of their right hand when the dot darkens and button 2 using the middle finger of their right hand when the dot lightens. Participants were informed that the dot brightness sometimes stays constant. Participants were encouraged to ignore the stimulus images and focus on the color of the dot.

The one-back task (even runs) was intended to encourage perception of the stimuli. In the one-back task, participants were instructed to fixate the central dot (ignoring the dot color changes) and to press button 1 if the presented image is different from the previous image and button 2 if the presented image was identical to the previous image. In a sense, this task is similar to the continuous recognition task performed in NSD-core where participants were asked to judge whether each presented image is new (not yet seen) or old (presented before). Participants were warned that some images may look degraded, noisy, or abstract, that some images may be positioned off to the side, and that some images may be quite similar but not identical to others. Participants were additionally instructed to continue to fixate and wait for the next image in the event of blank trials.

Summary of image repetitions and trial counts. Across the entire NSD-synthetic experiment, there are a total of 93 stimulus trials × 8 runs = 744 stimulus trials. Of these stimulus trials, 80 trials are one-back trials (10.8%). Considering all trials except the one-back trials, 268 images are presented 1 time per task and 16 images are presented 4 times per task. Considering all trials, including the one-back trials, 236 images are presented 1 time per task, 32 images are presented 2 times per task, 8 images are presented 4 times per task, and 8 images are presented 5 times per task.

Pre-processing of MRI data
Details on MRI pre-processing are provided in the Supplementary Information of NSD-core’s paper1. In brief, T1-weighted anatomical data were processed using FreeSurfer to create cortical surface representations. Functional data were pre-processed using one temporal resampling to correct for slice time differences and one spatial resampling to correct for head motion within and across scan sessions, EPI distortion, and gradient nonlinearities. Two versions of the functional data were prepared: a 1.8 mm standard-resolution preparation (temporal resolution 1.333 s) and an upsampled 1.0-mm high-resolution preparation (temporal resolution 1.000 s). Population receptive field and functional localizer experiments included in NSD were used to define retinotopic and category-selective regions of interest (ROIs), respectively. The defined ROIs include retinotopic visual areas (V1, V2, V3, hV4) as well as category-selective regions such as extrastriate body area (EBA), fusiform face area (FFA), parahippocampal place area (PPA), and visual word form area (VWFA).

GLM analysis of fMRI data
We performed a GLM analysis of NSD-synthetic’s pre-processed time-series data. This GLM analysis was the same as performed for NSD-core, and provides single-trial BOLD response amplitude estimates (‘betas’) in units of percent signal change. Our GLM approach combines three analysis components: a library of hemodynamic response functions (HRFs), adaptation of the GLMdenoise technique75,76 to a single-trial GLM framework, and application of ridge regression77 as a method for dampening the noise inflation caused by correlated single-trial GLM regressors. The GLM approach has been implemented in a software tool called GLMsingle78.

In implementing the GLM analysis for NSD-synthetic, we made the following design choices. First, the fixation and one-back tasks were treated separately. Hence, each image was associated with two distinct conditions: one condition reflects the presentation of the image during the fixation task and the other condition reflects the presentation of the image during the one-back task. Second, because the GLMdenoise and ridge regression techniques require condition repeats (so that cross-validated optimization of hyperparameters can be performed), we designated certain groups of images as repeats of a single condition. This choice reflects the expectation that the images within each group are likely to give very similar evoked brain responses. We specifically designated each of image subclasses 1, 2, 3, and 27–71 as consisting of four images that are repeats of a single condition. Third, to minimize assumptions and avoid complications stemming from any repetition suppression (adaptation) effects, we separately coded the one-back trials such that each one-back trial was assigned its own single-trial regressor. As such, the responses evoked by the one-back trials have minimal influence on the optimization of the GLMdenoise and ridge regression analysis components.

Analysis of behavioral data
For the fixation task, we extracted the first button response obtained within a time window extending 0–1400 ms after each dot color change. We recorded whether the button response was correct (i.e., button 1 for brightness decrements, button 2 for brightness increments). We counted any case in which no button was pressed as an incorrect response. Percent correct was used to summarize performance.

For the one-back task, for each stimulus trial (excluding the first), we extracted the first button response obtained within a time window extending 250–4250 ms after the onset of the stimulus image. We then interpreted the responses as reflecting a simple signal detection experiment. Trials in which the image was identical to the previously presented image (one-back trials) were treated as ‘signal present’ trials. For these trials, button 2 (“old”) corresponds to a hit and button 1 (“new”) corresponds to a miss. All other trials were treated as ‘signal absent’ trials. For these trials, button 2 (“old”) corresponds to a false alarm and button 1 (“new”) corresponds to a correct rejection. We counted any trial in which no button was pressed as an incorrect response (either a miss or a false alarm). Across trials, we computed the hit rate and the false alarm rate. We then converted these rates into the d’ sensitivity index based on the inverse cumulative distribution function of the standard normal distribution. In this calculation, to prevent numerical explosion, we set the minimum and maximum possible values for the hit rate and false alarm rate to be 0.01 and 0.99, respectively.

Analysis of fMRI data
Below, we describe the specific analyses of NSD demonstrated in this paper. These analyses go beyond the data preparation as described above. For all analyses we used the pre-processed fMRI betas version 3 (‘nsdsyntheticbetas_fithrf_GLMdenoise_RR’ for NSD-synthetic and ‘betas_fithrf_GLMdenoise_RR’ for NSD-core), prepared in FreeSurfer’s fsaverage space. (Beta version 3 reflects all GLM analysis components as described above.) For the analyses shown in this paper, we interpret responses based on the presented stimuli, aggregating across the tasks performed by the participant.

To reduce the effect of noise on results, for each participant, we utilized vertices with noise ceiling signal-to-noise ratio (NCSNR) scores above a threshold (except for the NCSNR visualization and for the encoding modeling, where we analyzed all available vertices). We quantified the NCSNR following the method proposed in NSD-core’s data release paper1, and selected 0.6 as the NCSNR threshold. We estimated the NCSNR independently for NSD-synthetic and NSD-core. Note that in the context of NSD-synthetic, we computed a single NCSNR estimate using fMRI responses from both tasks. Hence, our NCSNR estimate regards genuine task-related variability as noise and may therefore underestimate signals in the data.

Univariate fMRI response analysis. To compute NSD-synthetic’s univariate fMRI responses, we sequentially averaged the fMRI responses across four dimensions: first across repeated trials (resulting in one trial-averaged response for each of the 284 NSD-synthetic stimulus images); second across vertices within each ROI (V1, V2, V3, hV4, PPA, VWFA); third across all images from the same subclass; and fourth across all participants. This resulted in one univariate response for each image subclass and ROI, indicating the extent to which a given ROI is activated by each image subclass.

Representational similarity analysis. Using representational similarity analysis (RSA)41, we assessed the similarity of NSD-synthetic’s multivariate fMRI responses between the ROIs of all participants (V1, V2, V3, hV4, PPA, VWFA). We began by averaging the multivariate fMRI responses across repeated trials, resulting in one trial-averaged response for each of the 284 NSD-synthetic stimulus images. Next, for each ROI and participant, we built representational similarity matrices (RSMs), symmetric matrices indicating the similarity (Pearson’s r) of multivariate responses for each pair of stimulus images. Finally, for each pair of RSMs, we correlated (Pearson’s r) the lower triangle of the two RSMs. This resulted in one RSA score for each combination of participant and ROI, indicating the similarity of visual information encoded in their multivariate fMRI responses.

Multidimensional scaling. To compare NSD-synthetic’s and NSD-core’s data distributions, we applied multidimensional scaling (MDS)42,43 to single-trial fMRI responses aggregated across the two datasets (Fig. 3a). This MDS analysis determines a low-dimensional space that best approximates the similarity of responses between all stimulus image pairs. To improve stability of results, we appended fMRI vertex responses across participants before applying MDS. Furthermore, to assess potential differences in distributions due to session-specific effects (e.g., incidental differences in overall fMRI response magnitude across sessions), we aggregated the 744 single-trial fMRI responses from NSD-synthetic to the first 744 (out of 750) single-trial fMRI responses from two NSD-core scan sessions (sessions 10 and 20). The MDS analysis ultimately produced a two-dimensional array of shape (2,232 trials × 2 embedding dimensions).

We also applied MDS on only NSD-synthetic’s fMRI responses (without NSD-core) (Fig. 3b). To improve stability of results, for each of the 284 images, we averaged the fMRI responses across repeated trials and appended vertex responses across participants. This MDS analysis resulted in a two-dimensional array of shape (284 images × 2 embedding dimensions).

Encoding models. We trained and tested encoding models that linearly map stimulus image features onto fMRI responses. The image features consisted of the layer-wise activations of four deep neural networks, all trained on the ILSVRC-2012 image set79. The first deep neural network was AlexNet, a convolutional neural network pretrained on image classification49, for which we used the activations from the output of the last sublayer of each of the eight model layers. The second deep neural network was vit_b_32, a vision transformer pretrained on image classification50, for which we used the activations from the output of the last sublayer of each of the twelve model layers. The third deep neural network was ResNet5051, a convolutional neural network with skip connections pretrained on task-supervised image classification, for which we used the activations from the output of the last sublayer of each of the four convolutional model layers. The fourth deep neural network was MoCo52, consisting of the same ResNet50 architecture but trained with self-supervised contrastive learning53, for which we used the activations from the output of the last sublayer of each of the four model layers.

We trained an independent encoding model using the image features of each deep neural network. The train data split consisted of NSD-core’s 9000 participant-unique images and corresponding fMRI responses (i.e., the images uniquely seen by individual participants). During training, we first pre-processed each image by center-cropping it to square size using as size the image’s smallest dimension; resized it to 224 × 224 pixels; applied a square-root transformation to the RGB values if necessary (we used NSD-core images as-is and applied a square root transformation to NSD-synthetic images to match the NSD-core images); scaled its values to the range [0, 1]; and normalized the scaled RGB values using mean = [0.485, 0.456, 0.406] and standard deviation = [0.229, 0.224, 0.225]. We then fed each of the pre-processed train images to one of the four deep neural networks described above, extracted the corresponding image features, and concatenated these features across layers. Next, to reduce computational costs while retaining the most important axes of variation, we reduced the dimensionality of the concatenated features to 250 principal components using principal components analysis (PCA)48. Finally, we trained independent linear regressions that mapped these PCA-reduced features onto the fMRI responses of each vertex in each participant.

We additionally trained encoding models independently for each deep neural network layer, using either linear or cross-validated ridge regression. In the case of linear regression, we reduced the dimensionality of the image features from each deep neural network layer to 250 principal components. In the case of ridge regression, we used the full image features of each deep neural network layer without dimensionality reduction through PCA.

We tested these encoding models both in-distribution (ID) and out-of-distribution (OOD). For ID, we tested models on 284 of the 515 NSD-core shared images that all participants saw three times. For OOD, we tested models on NSD-synthetic’s 284 stimulus images. For both the ID and OOD testing images, we used the same procedure described above for obtaining PCA-reduced features, and then mapped these features onto fMRI responses using the trained encoding models weights. This generated predicted responses for ID and OOD testing images for all vertices and participants.

To assess the encoding models’ ID and OOD generalization performance, we compared the predicted fMRI responses with their recorded (i.e., experimentally collected) counterparts, using Pearson’s correlation. We correlated predicted and recorded fMRI responses independently for each vertex and participant, across the 284 ID or 284 OOD images, resulting in both ID and OOD correlation scores for each vertex and participant. These correlation scores reflect encoding model generalization performance, but are also affected by the level of noise in the recorded fMRI responses (which might not be equal between ID and OOD fMRI responses). Thus, to make the ID and OOD explained variance scores directly comparable, we normalized the ID and OOD generalization performance by the noise ceiling of the corresponding recorded fMRI responses. First, we squared all correlation scores after setting negative scores to zero, thus obtaining explained variance (r2) for each vertex and participant. Second, following the method proposed in NSD-core’s data release paper1, we computed each vertex’s noise ceiling signal-to-noise ratio (NCSNR), independently for NSD-synthetic and NSD-core’s test fMRI responses. We converted these NCSNR scores into noise ceiling scores, indicating the amount of variance contributed by the signal expressed as a percentage of the total amount of variance in the data. These noise ceilings indicate the upper bound of model performance, given the noise in the data. Third, we divided the r2 ID and OOD scores with the corresponding noise ceilings. This resulted in noise-ceiling-normalized explained variance scores that indicate the amount of explainable variance that is accounted for by the models, for each vertex and participant.

We applied the same encoding model procedure when controlling for session-specific effects (Supplementary Figs. 2 and 8), when testing the models on individual NSD-synthetic image classes (Fig. 6), and when testing the models on fMRI responses for both naturalistic and synthetic stimulus images (Fig. 7), with the only difference being the train and test data used.

Zero-shot image identification. As a complementary way of assessing the encoding models’ generalization performance, we used the models’ predicted fMRI responses to identify, in a zero-shot fashion, the stimulus images of the trial-averaged recorded test fMRI responses2,44,55,56. We started by correlating (Pearson’s r) the vertex activity pattern of the recorded fMRI response for each test image, with the encoding-model-predicted vertex activity patterns for the candidate images (including for the target image of interest). For the recorded vertex activity pattern for each test image we then ranked, from highest to lowest, its correlation scores with the predicted vertex activity patterns for all candidate images. Finally, we stored the rank of the correct stimulus image (i.e., the position at which the vertex activity pattern of the recorded fMRI response is correlated with the vertex activity pattern of the predicted fMRI response for the same image). This resulted in one rank score for each test image, where lower ranks indicate a more robust identification of the recorded fMRI response stimulus images, using predicted fMRI responses from encoding models.

Software
Data analyses were carried out in Python 3, using the following libraries (version in parenthesis): GLMsingle (v1.2); h5py (v3.1.0); Matplotlib (v3.9.4); NiBabel (v4.0.2); Nilearn (v0.9.2); Numpy (v1.26.4); nsdcode (v1.0.0); Pandas (v1.5.1); Pillow (v9.2.0); pycortex (v1.2.10); PyTorch (v1.13.0); scikit-learn (v1.1.1); SciPy (v1.12.0); Torchvision (v0.14.0); tqdm (v4.64.1).

fMRI data collection and preprocessing were carried out in MATLAB 2020a, using Psychtoolbox-3 FreeSurfer, and SPM12.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This manuscript introduces a valuable fMRI dataset collected from subjects who also participated in the NSD(-core) project, with neural responses recorded for a carefully curated set of synthetic stimuli spanning eight distinct families (e.g., noise images, Mooney images, spiral gratings). The authors perform univariate and multivariate analyses and provide evidence of reliable neural responses—especially in early visual areas. They further position these stimuli as out-of-distribution relative to NSD-core and report a significant drop in encoding performance, suggesting the potential utility of this dataset for developing more robust vision models.\nThe release of this dataset and accompanying analysis code is a noteworthy contribution, particularly because it extends the widely used NSD-core dataset with a diverse set of synthetic stimuli collected from the same participants. The public availability of the data, along with a well-documented and user-friendly codebase, strongly supports reproducibility.\nHowever, significant concerns exist regarding the analysis and interpretation of the collected data. Key aspects, such as the conceptualization of out-of-distribution (OOD) conditions, the control for potential experimental confounds, and the rigor of the analytical validation, need further attention to firmly establish the manuscript's conclusions. As a result, while the dataset is a valuable contribution, the current analyses may not yet fully substantiate claims of major advances significant to the field and the journal's focus.\n## Major comments\nThe implicit assumption would be that NSD-core sufficiently spans the natural image distribution. However, this assumption has been challenged in recent work. For example, Shirakawa et al. (2025, _Neural Networks_) pointed out that NSD-core exhibits limited diversity in natural images to just around 40 semantic categories, though it contains 30,000 brain samples per subject. Moreover, the same study demonstrated that failure of decoding OOD stimulus feature by redesigning training-test split of NSD-core. These results highlight that generalization to OOD is already quite challenging even within the domain of natural images, and that the notion of “OOD” should not be limited to synthetic stimuli alone.\nIn the present study, however, no effort has been made to evaluate model generalization to natural images beyond the training distribution. As a result, the observed drop in encoding performance for synthetic stimuli is difficult to interpret: it remains unclear whether the observed degradation is due to the stimuli being OOD, or additionally due to their synthetic images' nature.\nTo properly assess generalization to OOD stimuli, the authors should consider incorporating natural images that are semantically OOD with respect to the training data, not just synthetic images. Without a more rigorous treatment of the concept of OOD, the study does not yet provide the kind of clear and generalizable insight and falls short of establishing the broader significance of its findings to the field.\nAlthough the manuscript briefly acknowledges this point in the Discussion (around L374), a more thorough examination of these confounds is essential to support the claim that NSD-synthetic is a useful benchmark for model and theory improvement. Additional analyses are required to quantify and mitigate the potential effects of session-specific variability. For example, the authors could perform an additional encoding analysis on NSD-core in which the model is trained using only runs that do not overlap with the test data—i.e., by explicitly excluding any training data from the same run as each test sample. This would allow a cleaner comparison focused solely on the effect of stimulus distribution, minimizing potential confounds due to run- or session-level dependencies. While implementing such an analysis may require additional effort, it would significantly reduce the influence of these confounds, and provide a clearer assessment of how stimulus distribution impacts performance.\nWithout such additional controls analysis, the observed performance gap between NSD-core and NSD-synthetic cannot be conclusively attributed to distributional differences, and thus the validity of the core claim remains in question. In its current form, the manuscript does not meet the rigor expected within this field.\nTo evaluate this possibility, the authors can analyze the explained variance ratio or PCA reconstruction error for both NSD-core test images and NSD-synthetic images. This would help determine whether the PCA disproportionately suppresses information from OOD stimuli. Additionally, running encoding analyses using features from either lower or higher DNN layers independently, rather than combining them, would be better to isolate the contributions of low- versus high-level features clearly.\nWithout such analysis, it remains uncertain whether the observed encoding performance—and comparisons across models—truly reflect differences in model generalization, or are instead confounded by limitations in the feature representation itself. In its current form, the work does not sufficiently demonstrate that the observed effects are attributable to the models rather than artifacts of feature construction. As it stands, the work lacks the evidential clarity required to support the key claims.\nRecent work has increasingly raised concerns about the validity and interpretability of encoding analyses. For instance, Nonaka et al. reported that in feature-selection-based encoding models, only around 40% of model units contributed to prediction, suggesting that much of the model’s representational richness may be ignored in the encoding analysis. Moreover, there is growing skepticism about the overreliance on encoding performance metrics alone, particularly in the context of benchmarks such as brain-score. Malhotra and Bowers (2024), for example, showed that encoding accuracy can sometimes depend more on image background than on object content, raising questions about what these models are actually learning to predict. In the current study, it also remains unclear what aspects of the image features are being used for prediction—whether they reflect ideal image content, semantic content, trivial properties such as average luminance, session-specific noise, or other unexamined factors. While the recent work by Schaeffer et al. (2024) has several problems, their claim about the fundamental concerns of frameworks that focus primarily on maximizing encoding scores should not be neglected.\nTaking these recent discussions into account, this study that relies exclusively on standard encoding performance—without addressing potential confounding factors or validating the interpretability of the predicted responses—falls short of demonstrating meaningful progress in the field. The authors should consider conducting more multifaceted evaluations—such as testing whether the predicted responses can identify the presented stimuli—to better assess whether the predictions genuinely reflect stimulus related information.\n## Minor comments\nAlthough the manuscript introduces a useful dataset, it does not currently meet the journal’s standards for presenting conceptually novel and methodologically rigorous work that offers significant advances for specialists in the field. For these reasons, I do not recommend this manuscript for publication in this journal.\n**Reference:**\nMalhotra, G. & Bowers, J. Predicting brain activation does not license conclusions regarding DNN-brain alignment: The case of Brain-Score. in _Cognitive Computational Neuroscience_ (Boston, USA, 2024).\n[https://2024.ccneuro.org/pdf/309_Paper_authored_Neural_Predictivity_CCN2024.pdf]\n(https://2024.ccneuro.org/pdf/309_Paper_authored_Neural_Predictivity_CCN2024.pdf).\nNonaka, S., Majima, K., Aoki, S. C. & Kamitani, Y. Brain hierarchy score: Which deep neural networks are hierarchically brain-like? _iScience_ **24**, 103013 (2021).\nSchaeffer, R. _et al._ Position: Maximizing Neural Regression Scores May Not Identify Good Models of the Brain. in UniReps workshop (Vancouver, Canada, 2024). [https://openreview.net/pdf?id=vbtj05J68r](https://openreview.net/pdf?id=vbtj05J68r)\nShirakawa _et al._ Spurious reconstruction from brain activity. _Neural Networks (in press)_ (2025).\n[https://doi.org/10.48550/arXiv.2405.10078](https://doi.org/10.48550/arXiv.2405.10078)", "reviewer_2": "I co-reviewed this manuscript with one of the reviewers who provided the listed reports. This is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition for Early Career Researchers who co-review manuscripts.", "reviewer_3": "A very valuable resource for the visual neuroscience community. The original Natural Scenes Dataset of 7T fMRI responses to over 9,000 scenes (Nature Neuroscience, 2021) is one of the largest and most used open datasets in visual neuroscience (557 citations). The current paper announces and describes a supplement to this dataset, which seems to have been collected around the same time, but only released in March of this year. The supplement consists of 7T fMRI responses from the same individual subjects, to an additional 284 synthetic images (either manipulated, e.g. via contrast reduction or phase scrambling; or artificially generated e.g. colour fields or spatial-frequency-controlled spiral patterns). Although the new data supplement is much smaller than the original core dataset, it will be useful both for testing encoding models outside the distribution of natural images (the main purpose advocated by the authors) and for probing the cortical representation of colour, orientation, and spatial frequency. The stimuli in the dataset are probably not as exhaustively parameterised for this second purpose as a researcher would desire (e.g. colour hue space is sampled, but not the saturation and lightness axes), but the dataset will nevertheless be helpful for some carefully-chosen research questions (as well as serving as the intended out-of-distribution test set).\nThe paper is very clearly written and figures and methods are excellently presented. The analyses of the paper go beyond the \"minimum required\" to describe and quality-check the data, and also evaluate a number of deep neural network based encoding models.\nMINOR COMMENTS\n- line 46 \"intelligence\" seems like an odd way to characterise the research that would be done with this dataset. Why not \"visual neuroscience\" or similar?\n- Figure 1B \"Manipulates scenes\" should read \"Manipulated scenes\"\n- lines 96-101: It would be good to already acknowledge in the main text two important differences between the NSD-core and NSD-synthetic datasets - the temporal design (2s vs 3s on), and the task (mixture of fixation and same/different tasks, vs continuous old/new task). These differences do limit the comparability of the main and supplemental dataset and should be acknowledged early.\n- line 128: It would be helpful to mention how the noise ceilings measured in this dataset compare to those in the NSD-core dataset.\n- line 300: ResNet is a feedforward architecture with skip connections, in my understanding, not a recurrent network (i.e. it has no temporal dynamics or memory over stimuli).", "reviewer_4": "The authors present NSD-synthetic, a companion dataset to the Natural Scenes Dataset (NSD), consisting of 7T fMRI responses from 8 subjects viewing 284 synthetic images. This dataset is designed to enable out-of-distribution (OOD) generalization tests for computational models of visual processing. The authors demonstrate that neural responses to these synthetic stimuli are reliably encoded and are indeed OOD relative to NSD-core. They also present encoding model comparisons showing differences between self-supervised and task-supervised models in OOD settings, limited to early/mid visual cortex where the synthetic stimuli drive robust responses.\nStrengths\nThis is an exceptionally valuable resource for the computational neuroscience community. The dataset addresses a critical gap in the field - namely, the ability to test whether models of visual processing generalize beyond the natural image distributions they are typically trained on. The experimental design is rigorous, using the same 8 subjects from NSD-core and maintaining high data quality standards with 7T fMRI at 1.8mm resolution. The open release of this data will enable researchers to develop more robust models and better understand the limitations of current approaches.\nThe technical quality is, as always from Ale, excellent. Comprehensive validation shows that the synthetic stimuli reliably drive neural responses, particularly in early visual areas. The use of multidimensional scaling to confirm the OOD nature of the responses is convincing, and the variety of synthetic stimuli (noise patterns, manipulated scenes, gratings, words) provides good coverage of different visual features.\nOf particular interest, but also a potential confound, is the choice of artificial stimuli. as deep neural networks are (often) surprisingly bad using these type of stimuli. Using stimuli like spiral gratings, phase-scrambled images, etc. etc., the dataset probes a known weakness of current computational models. This makes NSD-synthetic an excellent diagnostic tool for understanding where and why DNNs fail to capture human visual processing.\nMajor/Medium point\n* The authors show that self-supervised models (MoCo) outperform task-supervised models (ResNet-50) in OOD generalization, and that vision transformers outperform CNNs. For early visual cortex (V1-V4), where the synthetic stimuli drive robust responses, these findings appear valid and represent an important discovery about differential model architectures. The high noise ceiling values in these areas (Figure 2A) support meaningful model comparisons. We do not really know what happens in other parts of cortex, or if the models would generalise better towards OOD stimuli for areas in higher Cortex (e.g. OOD stimuli that the cortex would respond to). Not sure what to do about this, you could look at OOD effects vs SNR effects. It could also really be that they are restricted to early cortex. But in any case I think it is important to reframe the conclusion a little bit to reflect that the OOD generalization findings are strongest for early visual cortex, with important caveats about interpretability in higher areas.\n* The paper lacks discussion of why this might occur. and why this occurs for early cortex. Is the dictionary of these networks just better? Or because contrastive learning explicitly trains models to be invariant to various image transformations, and that this generalises towards these artificial stimuli?\nIn this cortex it might be an idea to look at layers in MOCO in relation to effects?\nSome mechanistic insight would really help."}, "author_response": "Dear Reviewers,\nPlease find the detailed point-by-point response below. Reviewer comments are highlighted\nin gray, and the corresponding responses indented. Within the responses, extracts from the\nmanuscript are “quoted in italics”, and extracts reflecting changes to the manuscript are\n“quoted in italics and red font”. Manuscript line/page/figure numbers are indicated in bold font.\nBlue underscored text denotes hyperlinks.\nReviewer #1\nRemarks to the Author\nThis manuscript introduces a valuable fMRI dataset collected from subjects who also\nparticipated in the NSD(-core) project, with neural responses recorded for a carefully curated\nset of synthetic stimuli spanning eight distinct families (e.g., noise images, Mooney images,\nspiral gratings). The authors perform univariate and multivariate analyses and provide\nevidence of reliable neural responses—especially in early visual areas. They further position\nthese stimuli as out-of-distribution relative to NSD-core and report a significant drop in\nencoding performance, suggesting the potential utility of this dataset for developing more\nrobust vision models.\nThe release of this dataset and accompanying analysis code is a noteworthy contribution,\nparticularly because it extends the widely used NSD-core dataset with a diverse set of\nsynthetic stimuli collected from the same participants. The public availability of the data,\nalong with a well-documented and user-friendly codebase, strongly supports reproducibility.\nHowever, significant concerns exist regarding the analysis and interpretation of the collected\ndata. Key aspects, such as the conceptualization of out-of-distribution (OOD) conditions,\nthe control for potential experimental confounds, and the rigor of the analytical validation,\nneed further attention to firmly establish the manuscript's conclusions. As a result, while the\ndataset is a valuable contribution, the current analyses may not yet fully substantiate claims\nof major advances significant to the field and the journal's focus.\nWe thank Reviewer #1 for the positive evaluation and helpful comments, which\nresulted in valuable additional analyses and discussions. In brief, we:\n2\n(1) Added a new Results section and a Discussion paragraph on OOD generalization\ntests using fMRI responses for both naturalistic and synthetic images.\n(2) Added a new Results section and a Discussion paragraph on the OOD\ngeneralization performance for individual NSD-synthetic image classes.\n(3) Added new analyses and a Discussion paragraph related to the interpretability of\nOOD generalisation tests using the encoding modeling approach.\n(4) Added several control analyses to ensure that our encoding results are not a\nspurious product of session-specific noise, nor of the type of image features used.\n(5) Added clarifications as requested throughout the\nmanuscript. Major Comments:\n1. The most significant concern pertains to the treatment of out-of-distribution (OOD) stimuli\nin this study. In the Introduction (around line 45), the authors define in-distribution (ID) stimuli\nas those derived from the training distribution and OOD stimuli as those from a different\ndistribution. I agree with this definition, but the study exclusively treats OOD as synthetic\nimages and generalizes results from these synthetic stimuli as representative of OOD\nperformance more broadly. This operationalization is problematic.\nThe implicit assumption would be that NSD-core sufficiently spans the natural image\ndistribution. However, this assumption has been challenged in recent work. For example,\nShirakawa et al. (2025, _Neural Networks_) pointed out that NSD-core exhibits limited\ndiversity in natural images to just around 40 semantic categories, though it contains 30,000\nbrain samples per subject. Moreover, the same study demonstrated that failure of decoding\nOOD stimulus feature by redesigning training-test split of NSD-core. These results highlight\nthat generalization to OOD is already quite challenging even within the domain of natural\nimages, and that the notion of “OOD” should not be limited to synthetic stimuli alone.\nIn the present study, however, no effort has been made to evaluate model generalization to\nnatural images beyond the training distribution. As a result, the observed drop in encoding\nperformance for synthetic stimuli is difficult to interpret: it remains unclear whether the\nobserved degradation is due to the stimuli being OOD, or additionally due to their synthetic\nimages' nature.\nTo properly assess generalization to OOD stimuli, the authors should consider incorporating\nnatural images that are semantically OOD with respect to the training data, not just synthetic\nimages. Without a more rigorous treatment of the concept of OOD, the study does not yet\nprovide the kind of clear and generalizable insight and falls short of establishing the broader\nsignificance of its findings to the field.\nWe thank the Reviewer for the suggestion, which led to a new Results section where we\ntested the OOD generalization of encoding models for both naturalistic (from NSD-core)\nand synthetic (from NSD-synthetic) images. We obtained highest encoding accuracy\nscores for ID testing on NSD-core, followed by lower accuracy scores for\n3\nnaturalistic OOD testing on NSD-core, and finally by synthetic OOD testing on NSD-\nsynthetic. Together, this new analysis showcases the potential of combining NSD-core\nand NSD-synthetic to test the OOD robustness of brain models on neural responses for\nboth naturalistic and synthetic stimuli, and that the concept of OOD is not restricted to\nartificial stimuli but can be usefully applied even within the domain of naturalistic stimuli\n(Results (Fig. 7), page 20, lines 423-477):\n“NSD enables out-of-distribution tests of brain models across both naturalistic\nand synthetic stimulus images\nFig. 7 | NSD enables out-of-distribution tests of brain models across both\nnaturalistic and synthetic stimulus images. We trained vision-transformer-based\nencoding models on NSD-core, and tested their generalization across three\nconditions: ID on NSD-core, naturalistic OOD on NSD-core, and synthetic OOD on\nNSD-synthetic. a, MDS embedding of trial-averaged fMRI responses (aggregated\nacross participants) for the train images (gray dots), for the ID test images (orange\ndots), for the naturalistic OOD test images (yellow dots), and for the synthetic OOD\ntest images (turquoise dots). b, Encoding models’ explained variance for the three\ntest conditions, normalized by the noise ceiling. This figure shows results averaged\nacross participants on flattened cortical surfaces. To reduce false positives, for each\nvertex we averaged results from only those participants with a noise ceiling greater\nthan 0.3 for both NSD-core and NSD-synthetic (vertices for which no participant has a\nnoise ceiling above 0.3 are not shown). c, Encoding models’ zero-shot identification\nscores of the recorded fMRI response stimulus images, for each of the three test\nconditions, on violin plots. The y-axis represents the correct image rank, that is, the\nposition at which the correct image appears among the models’ choices (choosing\namong all 284 images of the corresponding test split), with lower ranks indicating a\nmore robust image identification. The colored violin shapes indicate the distribution\nof observations, with wider sections corresponding to higher density of values. Small\ntransparent dots indicate the correct image rank for single images and participants.\nLarge black dots indicate the average correct image rank across all participants and\nimages within each test condition, with the corresponding score reported above the\ndot. Horizontal dashed lines indicate chance identification scores.\nTo highlight\nthat OOD tests need not be restricted to neural responses for synthetic stimuli, in this\nlast analysis we assess the OOD generalization performance of vision-transformer-\nbased encoding models to fMRI responses for both naturalistic and synthetic images.\nWe first used a combination of MDS and clustering algorithms to divide NSD-core's fMRI\nresponses into train, ID test, and naturalistic OOD test splits. For each participant, we\nreduced the trial-averaged fMRI responses for NSD-core’s images to two dimensions\nthrough MDS, and partitioned this two-dimensional embedding space into 15 clusters\nusing k-means clustering. For the naturalistic OOD test split we considered the fMRI\nresponses from the cluster with highest Euclidean distance from the centroids of the\nremaining 14 clusters and, within it, selected fMRI responses for the 284 images with the\nhighest Euclidean distance from the remaining clusters. For the ID test split we selected\nfMRI responses for 284 images randomly selected from the remaining 14 clusters. As\nthe train split we selected the images from the remaining 9 clusters not belonging to the\nID test split. These procedures ensured that the naturalistic OOD test responses are\nmaximally distant in MDS embedding space from the distribution of the train responses,\nand that the train and ID test responses come from the same distribution (Fig. 7a). We\nthen trained encoding models using the train split and evaluated them ID (on NSD-core),\nas well as on naturalistic (on NSD-core) and synthetic (on NSD-synthetic) OOD test fMRI\nresponses.\nWe obtained higher noise-ceiling-normalized encoding accuracy scores for ID testing on\nNSD-core, followed by naturalistic OOD testing on NSD-core, and finally by synthetic\nOOD testing on NSD-synthetic (Fig. 7b). We obtained the same ranking when using the\nencoding models’ predicted fMRI responses to identify, in a zero-shot fashion, the\nstimulus images of the ID, naturalistic OOD, and synthetic OOD test fMRI responses\n(choosing among all images of the corresponding test split) (Fig. 7c). This ranking was\nexpected, as it reflects the fact that the ID, naturalistic OOD, and synthetic OOD test\nfMRI responses are increasingly distant from the train responses in MDS space (Fig.\n7a). We obtained quantitatively similar results when using encoding models based on\nother deep neural network architectures (Supplementary Figs. 16-18), indicating that\nthese findings are not contingent on a specific choice of deep neural network.\nIn conclusion, these results showcase the strength of combining NSD-core and NSD-\nsynthetic to test the OOD robustness of brain models on neural responses. The results\nalso demonstrate that the concept of OOD is not restricted to artificial stimuli but can be\nusefully applied even within the domain of naturalistic stimuli.”\nFurthermore, to reflect that OOD generalization tests can involve both naturalistic and\nsynthetic stimulus images, we also added the following paragraph to the Discussion\n(Discussion, page 23, lines 537-549):\n“To obtain valid inferences, the details, quantifications, and interpretation of OOD tests\nrequire careful consideration. Here, we comment on four specific aspects that\nresearchers should consider when performing OOD generalization tests.\nto the train data, which in turn affects the generalization performance of computational\nmodels of the brain. We showed this for test fMRI responses for different NSD-\nsynthetic image classes (Fig. 6), as well as for test fMRI responses for naturalistic and\nsynthetic images (Fig. 7). Our empirical results reinforce the fact that OOD\ngeneralization performance of brain models cannot be fully described by a simple\nsummary number, but lives in a continuum. Combining NSD-core and NSD-synthetic\nprovides a way to test several degrees of OOD generalization across this continuum,\nusing fMRI responses for both naturalistic and synthetic stimuli. We envision that this\nwill endow researchers with greater flexibility in model testing, resulting in a more\nnuanced account of the OOD generalization performance of brain models.”\nFinally, we present the new Method section relative to the zero-shot identification\nanalysis.\n(Methods, page 35, lines 1026-1037): “Zero-shot image identification. As a\ncomplementary way of assessing the encoding models’ generalization performance, we\nused the models’ predicted fMRI responses to identify, in a zero-shot fashion, the\nstimulus images of the trial-averaged recorded test fMRI responses (Kay et al., 2008;\nHorikawa & Kamitani, 2017; Seeliger et al., 2018). We started by correlating (Pearson’s\nr) the vertex activity pattern of the recorded fMRI response for each test image, with the\nencoding-model-predicted vertex activity patterns for all test images. For the recorded\nvertex activity pattern for each test image we then ranked, from highest to lowest, its\ncorrelation scores with the predicted vertex activity patterns for all test images. Finally,\nwe stored the rank of the correct stimulus image (i.e., the position at which the vertex\nactivity pattern of the recorded fMRI response is correlated with the vertex activity pattern\nof the predicted fMRI response for the same image). This resulted in one rank score for\neach test image, where lower ranks indicate a more robust identification of the recorded\nfMRI response stimulus images, using predicted fMRI responses from encoding\nmodels.”\n2. The encoding analysis in this paper aims to assess encoding performance differences\ncaused by stimulus distributions (NSD-core vs NSD-synthetic). However, the data split\ndesigns between the two differ in an important way: NSD-core training and test data come\nfrom the same fMRI runs, while NSD-synthetic test data come from entirely separate\nsessions. Hence, the performance drop for NSD-synthetic may not solely reflect OOD\ndifficulty, but also such confounding factors.\nAlthough the manuscript briefly acknowledges this point in the Discussion (around L374), a\nmore thorough examination of these confounds is essential to support the claim that\nNSD-synthetic is a useful benchmark for model and theory improvement. Additional\nanalyses are required to quantify and mitigate the potential effects of session-specific\nvariability. For example, the authors could perform an additional encoding analysis on\nNSD-core in which the model is trained using only runs that do not overlap with the test\ndata—i.e., by explicitly excluding any training data from the same run as each test sample.\nThis would allow a cleaner comparison focused solely on the effect of stimulus distribution,\nminimizing potential confounds due to run- or session-level dependencies. While\nimplementing such an analysis may require additional effort, it would significantly reduce the\ninfluence of these confounds, and provide a clearer assessment of how stimulus distribution\nimpacts performance.\nWithout such additional controls analysis, the observed performance gap between NSD-core\nand NSD-synthetic cannot be conclusively attributed to distributional differences, and thus\nthe validity of the core claim remains in question. In its current form, the manuscript does not\nmeet the rigor expected within this field.\nWe fully agree that session-specific effects might artificially cause reduced performance\nfor models generalizing to the scan session with synthetic stimuli responses. To control\nfor session-specific effects in our encoding analyses, as suggested by the Reviewer,\nwe trained and tested encoding models using fMRI responses from non-overlapping\nNSD-core scan sessions. This did not change our findings, namely that out-of-\ndistribution (OOD) tests on NSD-synthetic result in lower generalization performances\nthan in-distribution (ID) tests on NSD-core (Fig. 4), and that NSD-synthetic reveals\ndifferences between models that are not detected from ID tests (Fig. 5). Therefore, this\nexcludes session-specific effects as a confounding factor in our analyses.\n(Results (Fig. 4), page 13, lines 276-285): “To ensure that the lower OOD\nperformances are meaningful and robust, we performed three controls. First, to ensure\nthat the lower performance on NSD-synthetic is not simply due to the fact that it was\ncollected in a separate scan session, we trained and tested the encoding models using\nfMRI responses from separate NSD-core scan sessions (Supplementary Fig. 2).\nSecond, to ensure that the specific regression method that one chooses to use does not\nchange our conclusions, we trained and tested the encoding models on image features\nfrom individual AlexNet layers using linear or ridge regression (Supplementary Figs. 3-\n4). Third, to ensure that our results are not contingent on a specific choice of deep neural\nnetwork, we trained and tested the encoding models on image features from deep neural\nnetworks other than AlexNet (Supplementary Figs. 5-7). In all three cases, we\nobserved lower OOD performances for NSD-synthetic than for NSD-core.”\n7\n“Supplementary Fig. 2 | In-distribution and out-of-distribution generalization\ncomparison of AlexNet-based encoding models, controlling for session-\nspecific effects. To control for session-specific effects, we trained and tested\nAlexNet-based encoding models using fMRI responses from separate NSD-core scan\nsessions. For each participant, we trained the encoding models using image features\nand fMRI responses from all but the last 5 NSD-core scan sessions. We then tested\nthese models’ in distribution (ID) generalization on 284 image conditions that were\nonly presented during NSD-core’s last 5 scan sessions, and their out-of-distribution\n(OOD) generalization on the 284 image conditions from NSD-synthetic. This figure\nshows results averaged across participants on flattened cortical surfaces. a, Model\nexplained variance normalized by the noise ceiling. To reduce false positives, for each\nvertex we averaged results from only those participants with a noise ceiling greater\nthan 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown).\nb, Difference between ID and OOD noise-ceiling-normalized explained variance. For\neach vertex, we averaged results from only those participants with noise ceiling\ngreater than 0.3 for both NSD-core and NSD-synthetic. The asterisk indicates a\nsignificant difference between the ID and OOD noise-ceiling-normalized explained\nvariance scores (P = 10-7, paired sample t-test, one-sided, N = 8 participants). c,\nScatterplots of vertex-wise explained variance scores against their noise ceilings for\nID (NSD-core) and OOD (NSD-synthetic) generalization tests, color coded by the\nvisual stream that each vertex belongs to. For each vertex, we plotted results from\nonly those participants with noise ceiling greater than 0.3 for both NSD-core and\nNSD-synthetic (same as in panel b). Black crosses indicate median scores across all\nvertices.”\n(Results (Fig. 5), page 15, lines 335-342): “To ensure that the OOD differences not\ndetected by ID tests are meaningful and robust, we performed two controls. First, to ensure\nthat our results are not simply due to the fact that NSD-synthetic was collected in a\nseparate scan session, we trained and tested the encoding models using fMRI responses\nfrom separate NSD-core scan sessions (Supplementary Fig. 8). Second,\n8\nto ensure that the specific regression method that one chooses to use does not change\nour conclusions, we trained and tested the encoding models on image features from\nindividual ResNet50 or MoCo layers using either linear or ridge regressions\n(Supplementary Figs. 9-12). In both cases, OOD tests revealed differences between\nmodels not detected by ID tests.”\n“Supplementary Fig. 8 | In-distribution and out-of-distribution generalization\ndifferences between encoding models, controlling for session-specific\neffects. To control for session-specific effects, we trained and tested encoding\nmodels using fMRI responses from separate NSD-core scan sessions. For each\nparticipant, we trained the encoding models using image features and fMRI\nresponses from all but the last 5 NSD-core scan sessions. We then tested these\nmodels’ in distribution (ID) generalization on 284 image conditions that were only\npresented during NSD-core’s last 5 scan sessions, and their out-of-distribution\n(OOD) generalization on the 284 image conditions from NSD-synthetic. This figure\nshows results averaged across participants on flattened cortical surfaces. To reduce\nfalse positives, for each vertex we averaged results from only those participants with\na noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling\nabove 0.3 are not shown). a, Difference between AlexNet and vit_b_32’s noise-\nceiling-normalized explained variance. b, Difference between MoCo and\nResNet50’s noise-ceiling-normalized explained variance.”\n3. In the encoding analysis, image features are derived from PCA-reduced vectors (top 250\ncomponents) based on DNN activations from multiple layers, extracted from NSD-core\ntraining images. It is possible that this PCA reduction can emphasize effective information in\nID components while discarding low-level visual features that are potentially useful for\npredicting responses in early visual areas—particularly for synthetic stimuli that lack\nnaturalistic structure.\nTo evaluate this possibility, the authors can analyze the explained variance ratio or PCA\nreconstruction error for both NSD-core test images and NSD-synthetic images. This would\nhelp determine whether the PCA disproportionately suppresses information from OOD\nstimuli. Additionally, running encoding analyses using features from either lower or higher\nDNN layers independently, rather than combining them, would be better to isolate the\ncontributions of low- versus high-level features clearly.\nWithout such analysis, it remains uncertain whether the observed encoding\nperformance—and comparisons across models—truly reflect differences in model\ngeneralization, or are instead confounded by limitations in the feature representation itself. In\nits current form, the work does not sufficiently demonstrate that the observed effects are\nattributable to the models rather than artifacts of feature construction. As it stands, the work\nlacks the evidential clarity required to support the key claims.\nWe agree with the Reviewer that the current results might be influenced by the specific\ntype of image features used, or from the dimensionality of the image features being\nreduced through PCA. To confirm that our findings are not a spurious product of the type\nof image features used or the downsampling of image features with PCA, we carried out\ntwo additional analyses. First, we trained and tested the encoding models using image\nfeatures from individual deep neural network (DNN) layers, and mapped these image\nfeatures onto the brain using either linear regression (after downsampling the feature\nspace to 250 principal components), or ridge regression (on the full feature space, that\nis, not reduced with PCA). Second, we trained and tested the encoding models on image\nfeatures from several DNNs. In both cases this did not change our findings, namely that\nout-of-distribution (OOD) tests on NSD-synthetic result in lower generalization\nperformances than in-distribution (ID) tests on NSD-core (Fig. 4), and that NSD-\nsynthetic reveals differences between models that are not detected ID (Fig. 5).\n(Results (Fig. 4), page 13, lines 276-285): “To ensure that the lower OOD\nperformances are meaningful and robust, we performed three controls. First, to ensure\nthat the lower performance on NSD-synthetic is not simply due to the fact that it was\ncollected in a separate scan session, we trained and tested the encoding models using\nfMRI responses from separate NSD-core scan sessions (Supplementary Fig. 2).\nSecond, to ensure that the specific regression method that one chooses to use does not\nchange our conclusions, we trained and tested the encoding models on image features\nfrom individual AlexNet layers using linear or ridge regression (Supplementary Figs. 3-\n4). Third, to ensure that our results are not contingent on a specific choice of deep neural\nnetwork, we trained and tested the encoding models on image features from deep neural\nnetworks other than AlexNet (Supplementary Figs. 5-7). In all three cases, we\nobserved lower OOD performances for NSD-synthetic than for NSD-core.”\n10\n”Supplementary Fig. 3 | In-distribution and out-of-distribution generalization\ncomparison for encoding models based on individual AlexNet layers, trained\nwith linear regression. We trained separate encoding models for each of 4 AlexNet\nlayers: convolutional layer 1, convolutional layer 3, convolutional layer 5, and fully\nconnected layer 7. The encoding models consisted of linear regressions that map\neach layer’s image features onto the fMRI responses of each vertex. We tested\nthese models in-distribution (ID) on NSD-core as well as out-of-distribution (OOD)\non NSD-synthetic. This figure shows results averaged across participants on\nflattened cortical surfaces. a, ID explained variance normalized by the noise ceiling.\nTo reduce false positives, for each vertex we averaged results from only those\nparticipants with a noise ceiling greater than 0.3 for NSD-core (vertices for which no\nparticipant has a noise ceiling above 0.3 are not shown). b, OOD explained variance\nnormalized by the noise ceiling. For each vertex we averaged results from only those\nparticipants with a noise ceiling greater than 0.3 for NSD-synthetic. c, Difference\nbetween ID and OOD noise-ceiling-normalized explained variance. For each vertex,\nwe averaged results from only those participants with noise ceiling greater than 0.3\nfor both NSD-core and NSD-synthetic. The asterisk indicates a significant difference\nbetween the ID and OOD noise-ceiling-normalized explained variance scores (P <\n10-6, paired sample t-test, one-sided, N = 8 participants).”\n11\n”Supplementary Fig. 4 | In-distribution and out-of-distribution generalization\ncomparison for encoding models based on individual AlexNet layers, trained\nwith ridge regression. We trained separate encoding models for each of 4 AlexNet\nlayers: convolutional layer 1, convolutional layer 3, convolutional layer 5, and fully\nconnected layer 7. The encoding models consisted of cross-validated ridge\nregressions that map each layer’s image features onto the fMRI responses of each\nvertex. We tested these models in-distribution (ID) on NSD-core as well as out-of-\ndistribution (OOD) on NSD-synthetic. This figure shows results averaged across\nparticipants on flattened cortical surfaces. a, ID explained variance normalized by the\nnoise ceiling. To reduce false positives, for each vertex we averaged results from\nonly those participants with a noise ceiling greater than 0.3 for NSD-core (vertices for\nwhich no participant has a noise ceiling above 0.3 are not shown). b, OOD explained\nvariance normalized by the noise ceiling. For each vertex we averaged results from\nonly those participants with a noise ceiling greater than 0.3 for NSD-synthetic. c,\nDifference between ID and OOD noise-ceiling-normalized explained variance. For\neach vertex, we averaged results from only those participants with noise ceiling\ngreater than 0.3 for both NSD-core and NSD-synthetic. The asterisk indicates a\nsignificant difference between the ID and OOD noise-ceiling-normalized explained\nvariance scores (P < 10-6, paired sample t-test, one-sided, N = 8 participants).”\n12\n”Supplementary Fig. 5 | In-distribution and out-of-distribution generalization\ncomparison of vision-transformer-based encoding models. We built vision-\ntransformer-based encoding models and tested them in-distribution (ID) on NSD-core\nas well as out-of-distribution (OOD) on NSD-synthetic. This figure shows results\naveraged across participants on flattened cortical surfaces. a, Model explained\nvariance normalized by the noise ceiling. To reduce false positives, for each vertex\nwe averaged results from only those participants with a noise ceiling greater than 0.3\n(vertices for which no participant has a noise ceiling above 0.3 are not shown). b,\nDifference between ID and OOD noise-ceiling-normalized explained variance. For\neach vertex, we averaged results from only those participants with noise ceiling\ngreater than 0.3 for both NSD-core and NSD-synthetic. The asterisk indicates a\nsignificant difference between the ID and OOD noise-ceiling-normalized explained\nvariance scores (P = 10-7, paired sample t-test, one-sided, N = 8 participants). c,\nScatterplots of vertex-wise explained variance scores against their noise ceilings for\nID (NSD-core) and OOD (NSD-synthetic) generalization tests, color coded by the\nvisual stream that each vertex belongs to. For each vertex, we plotted results from\nonly those participants with noise ceiling greater than 0.3 for both NSD-core and\nNSD-synthetic (same as in panel b). Black crosses indicate median scores across all\nvertices.”\n13\n”Supplementary Fig. 6 | In-distribution and out-of-distribution generalization\ncomparison of ResNet50-based encoding models. We built ResNet50-based\nencoding models and tested them in-distribution (ID) on NSD-core as well as out-\nof-distribution (OOD) on NSD-synthetic. This figure shows results averaged across\nparticipants on flattened cortical surfaces. a, Model explained variance normalized\nby the noise ceiling. To reduce false positives, for each vertex we averaged results\nfrom only those participants with a noise ceiling greater than 0.3 (vertices for which\nno participant has a noise ceiling above 0.3 are not shown). b, Difference between\nID and OOD noise-ceiling-normalized explained variance. For each vertex, we\naveraged results from only those participants with noise ceiling greater than 0.3 for\nboth NSD-core and NSD-synthetic. The asterisk indicates a significant difference\nbetween the ID and OOD noise-ceiling-normalized explained variance scores (P =\n10-8, paired sample t-test, one-sided, N = 8 participants). c, Scatterplots of vertex-\nwise explained variance scores against their noise ceilings for ID (NSD-core) and\nOOD (NSD-synthetic) generalization tests, color coded by the visual stream that\neach vertex belongs to. For each vertex, we plotted results from only those\nparticipants with noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic\n(same as in panel b). Black crosses indicate median scores across all vertices.”\n14\n”Supplementary Fig. 7 | In-distribution and out-of-distribution generalization\ncomparison of MoCo-based encoding models. We built MoCo-based encoding\nmodels and tested them in-distribution (ID) on NSD-core as well as out-of-\ndistribution (OOD) on NSD-synthetic. This figure shows results averaged across\nparticipants on flattened cortical surfaces. a, Model explained variance normalized\nby the noise ceiling. To reduce false positives, for each vertex we averaged results\nfrom only those participants with a noise ceiling greater than 0.3 (vertices for which\nno participant has a noise ceiling above 0.3 are not shown). b, Difference between\nID and OOD noise-ceiling-normalized explained variance. For each vertex, we\naveraged results from only those participants with noise ceiling greater than 0.3 for\nboth NSD-core and NSD-synthetic. The asterisk indicates a significant difference\nbetween the ID and OOD noise-ceiling-normalized explained variance scores (P =\n10-7, paired sample t-test, one-sided, N = 8 participants). c, Scatterplots of vertex-\nwise explained variance scores against their noise ceilings for ID (NSD-core) and\nOOD (NSD-synthetic) generalization tests, color coded by the visual stream that\neach vertex belongs to. For each vertex, we plotted results from only those\nparticipants with noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic\n(same as in panel b). Black crosses indicate median scores across all vertices.”\n15\n(Results (Fig. 5), page 15, lines 335-342): “To ensure that the OOD differences not\ndetected by ID tests are meaningful and robust, we performed two controls. First, to ensure\nthat our results are not simply due to the fact that NSD-synthetic was collected in a\nseparate scan session, we trained and tested the encoding models using fMRI responses\nfrom separate NSD-core scan sessions (Supplementary Fig. 8). Second, to ensure that\nthe specific regression method that one chooses to use does not change our conclusions,\nwe trained and tested the encoding models on image features from individual ResNet50\nor MoCo layers using either linear or ridge regressions\n15\n(Supplementary Figs. 9-12). In both cases, OOD tests revealed differences between\nmodels not detected by ID tests.”\n”Supplementary Fig. 9 | In-distribution generalization differences between\nencoding models based on individual MoCo and ResNet50 layers, trained with\nlinear regression. We trained separate encoding models for each of the 4 MoCo and\nResNet50 layers: convolutional layer 1, convolutional layer 2, convolutional layer 3,\nand convolutional layer 4. The encoding models consisted of linear regressions that\nmap each layer’s image features onto the fMRI responses of each vertex. We then\ntested these models in-distribution (ID) on NSD-core. This figure shows results\naveraged across participants on flattened cortical surfaces. a, MoCo’s ID explained\nvariance normalized by the noise ceiling. To reduce false positives, for each vertex\nwe averaged results from only those participants with a noise ceiling greater than 0.3\nfor NSD-core (vertices for which no participant has a noise ceiling above 0.3 are not\nshown). b, ResNet50’s ID explained variance normalized by the noise ceiling. For\neach vertex we averaged results from only those participants with a noise ceiling\ngreater than 0.3 for NSD-synthetic. c, Difference between MoCo and ResNet50’s ID\nnoise-ceiling-normalized explained variance. For each vertex, we averaged results\nfrom only those participants with noise ceiling greater than 0.3 for both NSD-core and\nNSD-synthetic.”\n16\n”Supplementary Fig. 10 | Out-of-distribution generalization differences between\nencoding models based on individual MoCo and ResNet50 layers, trained with\nlinear regression. We trained separate encoding models for each of the 4 MoCo\nand ResNet50 layers: convolutional layer 1, convolutional layer 2, convolutional layer\n3, and convolutional layer 4. The encoding models consisted of linear regressions\nthat map each layer’s image features onto the fMRI responses of each vertex. We\nthen tested these models out-of-distribution (OOD) on NSD-synthetic. This figure\nshows results averaged across participants on flattened cortical surfaces. a, MoCo’s\nOOD explained variance normalized by the noise ceiling. To reduce false positives,\nfor each vertex we averaged results from only those participants with a noise ceiling\ngreater than 0.3 for NSD-core (vertices for which no participant has a noise ceiling\nabove 0.3 are not shown). b, ResNet50’s OOD explained variance normalized by the\nnoise ceiling. For each vertex we averaged results from only those participants with\na noise ceiling greater than 0.3 for NSD-synthetic. c, Difference between MoCo and\nResNet50’s OOD noise-ceiling-normalized explained variance. For each vertex, we\naveraged results from only those participants with noise ceiling greater than 0.3 for\nboth NSD-core and NSD-synthetic.”\n17\n”Supplementary Fig. 11 | In-distribution generalization differences between\nencoding models based on individual MoCo and ResNet50 layers, trained with\nridge regression. We trained separate encoding models for each of the 4 MoCo and\nResNet50 layers: convolutional layer 1, convolutional layer 2, convolutional layer 3,\nand convolutional layer 4. The encoding models consisted of cross-validated ridge\nregressions that map each layer’s image features onto the fMRI responses of each\nvertex. We then tested these models in-distribution (ID) on NSD-core. This figure\nshows results averaged across participants on flattened cortical surfaces. a, MoCo’s\nID explained variance normalized by the noise ceiling. To reduce false positives, for\neach vertex we averaged results from only those participants with a noise ceiling\ngreater than 0.3 for NSD-core (vertices for which no participant has a noise ceiling\nabove 0.3 are not shown). b, ResNet50’s ID explained variance normalized by the\nnoise ceiling. For each vertex we averaged results from only those participants with\na noise ceiling greater than 0.3 for NSD-synthetic. c, Difference between MoCo and\nResNet50’s ID noise-ceiling-normalized explained variance. For each vertex, we\naveraged results from only those participants with noise ceiling greater than 0.3 for\nboth NSD-core and NSD-synthetic.”\n18\n”Supplementary Fig. 12 | Out-of-distribution generalization differences\nbetween encoding models based on individual MoCo and ResNet50 layers,\ntrained with ridge regression. We trained separate encoding models for each of\nthe 4 MoCo and ResNet50 layers: convolutional layer 1, convolutional layer 2,\nconvolutional layer 3, and convolutional layer 4. The encoding models consisted of\ncross-validated ridge regressions that map each layer’s image features onto the fMRI\nresponses of each vertex. We then tested these models out-of-distribution (OOD) on\nNSD-synthetic. This figure shows results averaged across participants on flattened\ncortical surfaces. a, MoCo’s OOD explained variance normalized by the noise\nceiling. To reduce false positives, for each vertex we averaged results from only\nthose participants with a noise ceiling greater than 0.3 for NSD-core (vertices for\nwhich no participant has a noise ceiling above 0.3 are not shown). b, ResNet50’s\nOOD explained variance normalized by the noise ceiling. For each vertex we\naveraged results from only those participants with a noise ceiling greater than 0.3 for\nNSD-synthetic. c, Difference between MoCo and ResNet50’s OOD noise-ceiling-\nnormalized explained variance. For each vertex, we averaged results from only those\nparticipants with noise ceiling greater than 0.3 for both NSD-core and NSD-\nsynthetic.”\n4. A further important concern is that, although the stated goal of this study is to advance the\nmodeling of visual processing under out-of-distribution (OOD) conditions, the analyses rely\nsolely on simple linear encoding models.\nRecent work has increasingly raised concerns about the validity and interpretability of\nencoding analyses. For instance, Nonaka et al. reported that in feature-selection-based\nencoding models, only around 40% of model units contributed to prediction, suggesting that\n19\nmuch of the model’s representational richness may be ignored in the encoding analysis.\nMoreover, there is growing skepticism about the overreliance on encoding performance\nmetrics alone, particularly in the context of benchmarks such as brain-score. Malhotra and\nBowers (2024), for example, showed that encoding accuracy can sometimes depend more\non image background than on object content, raising questions about what these models are\nactually learning to predict. In the current study, it also remains unclear what aspects of the\nimage features are being used for prediction—whether they reflect ideal image content,\nsemantic content, trivial properties such as average luminance, session-specific noise, or\nother unexamined factors. While the recent work by Schaeffer et al. (2024) has several\nproblems, their claim about the fundamental concerns of frameworks that focus primarily on\nmaximizing encoding scores should not be neglected.\nTaking these recent discussions into account, this study that relies exclusively on standard\nencoding performance—without addressing potential confounding factors or validating the\ninterpretability of the predicted responses—falls short of demonstrating meaningful progress\nin the field. The authors should consider conducting more multifaceted evaluations—such as\ntesting whether the predicted responses can identify the presented stimuli—to better assess\nwhether the predictions genuinely reflect stimulus related information.\nWe appreciate (and agree) with the Reviewer’s recognition of the potential\nlimitations/pitfalls of encoding model metrics. Following the Reviewer’s suggestion, as a\ncomplementary way of assessing the encoding models’ generalization performance we\nused the encoding models’ predicted fMRI responses to identify, in a zero-shot fashion,\nthe stimulus images of the recorded test fMRI responses (for the corresponding results\ntext and figures, see Reviewer #1 comment 1, and Reviewer #1 comment 5).\nReassuringly, we found that the zero-shot identification yielded similar result patterns to\nthe standard encoding performance (i.e., NSD-synthetic image classes resulting in lower\nencoding accuracies, such as chromatic noise, also resulted in lower identification\nperformances). Together, this suggests that the model predictions genuinely reflect\nstimulus related information, and that the reported result pattern is not a bias of the\nevaluation metric used.\nBeyond this, to rule out several confounding factors related to our encoding modeling\nresults, we carried out extensive additional analyses (reported above) which show that\nour encoding results are not a spurious product of session-specific noise (see Reviewer\n#1 comment 2), nor of the type of image features or regression method used (see\nReviewer #1 comment 3).\nFinally, based on the Reviewer's comments concerning the interpretability of results from\nencoding models, together with several helpful discussions on this topic with other\nresearchers at the 2025 Cognitive Computational Neuroscience conference, we also\nadded a new Discussion paragraph where we elaborate on the interpretation of OOD\ngeneralization performances of computational models of the brain implemented through\nthe linear encoding approach (Discussion, page 23, lines 537-561):\n“To obtain valid inferences, the details, quantifications, and interpretation of OOD tests\nrequire careful consideration. Here, we comment on four specific aspects that\nresearchers should consider when performing OOD generalization tests.\n20\n[...]\nSecond, there are several approaches to implement and test computational models of\nthe brain, each with its unique strengths, and which approach should be best used\ntowards a particular scientific goal is a current topic of debate (Diedrichsen &\nKriegeskorte, 2017; Doerig et al., 2023; Schaeffer et al., 2024). Here we used the\nlinearizing encoding approach (Naselaris et al., 2011), which consists of two\nindependently trainable components: the image-computable model (typically a deep\nneural network trained on a computer vision task) used to extract the image features,\nand the linear mapping trained to map these image features onto brain responses. Thus,\nthe outcomes of our OOD tests reflect the combination of both the image-computable\nmodel and the linear mapping. We note that other researchers might wish to test only\nthe OOD generalization performance of the image-computable model. For this modeling\nphilosophy, a different analytic approach would be necessary, such as representational\nsimilarity analysis (with no feature re-weighting) (Kriegeskorte et al., 2008), or\ndetermining a one-to-one mapping between model features and neural units (Finzi et al.,\n2025).”\nMinor Comments:\n5. The current OOD set seems to be heavily skewed toward spiral gratings (~40% of stimuli).\nAs such, there is a possibility that aggregate performance in Figures 4 and 5 are dominated\nby this single stimulus type. To assess the encoding performance on OOD stimuli more\nclearly, I recommend reporting encoding performance separately by OOD category (e.g.,\nspiral gratings, hue-manipulated scenes, words, line drawings, etc.). Even as supplementary\nmaterial, such an analysis would help users understand whether performance drops\ngeneralize across stimulus types or are specific to particular OOD classes.\nWe thank the Reviewer for the suggestion, which led to a new Results section where\nwe tested the OOD generalization of encoding models independently for each NSD-\nsynthetic image class. Together, this new analysis showed that the degree of OOD–that\nis, the distance between the test and train fMRI responses–is informative of the model’s\ngeneralization performance to these test responses. Thus, the degree of OOD is a\nuseful index of the stimulus properties an encoding model might fail to generalize to, in\nturn providing explicit targets for model improvement (Results (Fig. 6), page 17, lines\n347-422):\n“The degree of out-of-distribution is informative of brain model failures\nWe trained vision-transformer-based encoding models on NSD-core, and tested\nthem independently for each NSD-synthetic image class. a, MDS embedding of trial-\naveraged fMRI responses (aggregated across participants) from NSD-core (gray\ndots) and from NSD-synthetic (colored dots, color-coded according to image class).\nb, Encoding models’ explained variance for each NSD-synthetic image class,\nnormalized by the noise ceiling. This figure shows results averaged across\nparticipants on flattened cortical surfaces. To reduce false positives, for each vertex\nwe averaged results from only those participants with a noise ceiling greater than 0.3\n(vertices for which no participant has a noise ceiling above 0.3 are not shown). c,\nScatterplot indicating the relationship between the degree of OOD–that is, the degree\nof distributional distance–of fMRI responses and the encoding accuracy scores,\nacross NSD-synthetic image classes. The x-axis represents the Euclidean distance\nin MDS embedding space between fMRI responses for each NSD-synthetic image\nclass and NSD-core (i.e., the distances in panel a). The y-axis represents the noise-\nceiling normalized encoding accuracy of encoding models tested on each NSD-\nsynthetic image class (i.e., the mean\n22\nscores in panel b). d, Encoding models’ zero-shot identification scores of the\nrecorded fMRI response stimulus images, for each NSD-synthetic image class, on\nviolin plots. The y-axis represents the correct image rank, that is, the position at\nwhich the correct image appears among the models’ choices (choosing among all\nidentification. The colored violin shapes indicate the distribution of observations,\nwith wider sections corresponding to higher density of values. Small transparent\ndots indicate the correct image rank for single images and participants.\nLarge black dots indicate the average correct image rank across all participants and\nimages within each NSD-synthetic image class, with the corresponding score reported\nabove the dot. Horizontal dashed lines indicate chance identification scores. e,\nScatterplot indicating the relationship between the degree of OOD of fMRI\nresponses and the zero-shot image identification scores, across NSD-synthetic\nimage classes. The x-axis represents the Euclidean distance in MDS embedding\nspace between fMRI responses for each NSD-synthetic image class and NSD-core\n(i.e., the distances in panel a). The y-axis represents the average correct image\nrank across all participants and images within each NSD-synthetic image class (i.e.,\nthe average scores in panel d). c,e, Large opaque and small transparent dots\nindicate participant-average and single participant scores, respectively. The\nasterisk indicates a significant correlation between response distance scores and\nencoding model performance (P < 0.001, one-sample t-test, one-sided, N = 8\nparticipants).\nThe neural responses on which a brain model is tested can exhibit various degrees of\nOOD–that is, various degrees of distributional distance–with respect to the responses on\nwhich the model is trained. Therefore, we ask whether there is a relationship between the\nextent to which the fMRI responses used to test an encoding model are differently\ndistributed from the train responses and the encoding model’s generalization performance\nto these test responses. If so, the degree of OOD could serve as an indicator of the\nstimulus properties that encoding models might fail to account for, which may help guide\nthe engineering of more robust models (Madan et al., 2024; Kay 2018). We operationally\ndefine the degree of OOD as the Euclidean distance between the fMRI response to a given\nstimulus of interest and the NSD-core train responses.\nUsing MDS we reduced the trial-averaged fMRI responses for NSD-synthetic (used to\ntest the encoding models) and for NSD-core (used to train the models), to two\ndimensions. Within this two-dimensional embedding space we then computed the\nEuclidean distance between the responses of each NSD-synthetic image and the\nresponses of each NSD-core image, and averaged these distance scores across all\nNSD-core images, as well as across NSD-synthetic images belonging to the same image\nclass. This resulted in distributional distance scores indicating, for each NSD-synthetic\nimage class, the degree of OOD of its fMRI responses from NSD-core’s responses (Fig.\n6a).\nNext, we compared these distributional distance scores with the noise-ceiling-\nnormalized encoding accuracy scores of vision-transformer-based encoding models\ntrained on NSD-core and tested on each NSD-synthetic image class (Fig. 6b). We\nfound that the distributional distance scores are anticorrelated with the encoding\naccuracy scores (Spearman’s ρ = -0.36, P = 0.0004, one-sample t-test, one-sided, N\n= 8 participants) (Fig. 6c), indicating that encoding models generalize\n23\nbetter to fMRI responses that have lower distributional distance to the responses on\nwhich the models are trained.\nAs a complementary way of assessing the encoding models’ generalization performance\nfor the different NSD-synthetic image classes, we used the models’ predicted fMRI\nresponses to identify, in a zero-shot fashion, the stimulus images of NSD-synthetic’s\nrecorded fMRI responses (Kay et al., 2008; Horikawa & Kamitani, 2017; Seeliger et al.,\n2018), choosing among all 284 NSD-synthetic images (Fig. 6d). We found that the ranks\nof the correct stimulus image (i.e., the position at which the correct image appears\namong the models’ choices) correlate with the distributional distance scores\n(Spearman’s ρ = 0.63, P = 10-6, one-sample t-test, one-sided, N = 8 participants) (Fig.\n6e), indicating that encoding models more reliably identify stimulus images when the\ncorresponding fMRI responses have lower distributional distance to the model’s train\nresponses.\nWe obtained quantitatively similar results when using encoding models based on other\ndeep neural network architectures (Supplementary Figs. 13-15), indicating that the\ncorrelation between distributional distance scores and encoding model performance is\nnot contingent on a specific choice of deep neural network.\nIn summary, these results show that the degree to which an encoding model’s test fMRI\nresponses are OOD with respect to the model’s train responses is informative of the\nmodel’s generalization performance to these test responses. Thus, the degree of OOD\nis a useful indicator of the stimulus properties that encoding models might fail to account\nfor.\nIn light of these new results, we made the following addition to the Discussion\n(Discussion, page 22, lines 503-513):\n“Beyond indicating that further model improvement is required to accurately describe\nvisual representations in the brain, NSD-synthetic also suggests directions for closing\nthe gap between ID and OOD generalization performance. One direction of model\nimprovement comes from consideration of the properties of NSD-synthetic’s stimuli.\nSince the stimuli consist of a multitude of diverse and parameterized images, they\nfacilitate discovery of the specific visual features to which computational models fail to\ngeneralize. Accordingly, we found that encoding models well generalize to test fMRI\nresponses that have lower degree of OOD (i.e., responses for naturalistic or manipulated\nscenes), whereas they fail to generalize to fMRI responses with a higher degree of OOD\n(i.e., responses for chromatic noise or spiral gratings). To optimize modeling\nperformance, it may be fruitful for future research efforts to improve robustness for\nstimulus properties with a high degree of OOD (Madan et al., 2024; Kay, 2018).”\n6. L 300: The phrase “ResNet-50 is a recurrent convolutional neural network” is inaccurate.\nResNet-50 is not a recurrent model and should be rephrased.\nWe thank the Reviewer for pointing out this incorrect model description, which we\namended (Results, page 15, lines 322-324):\n24\n“Specifically, we compared encoding models based on ResNet-50, a convolutional\nneural network with skip connections pretrained on task-supervised image classification\n...”\n7. L 524: The method for generating line-drawing stimuli is unclear. Compared to other\nstimulus types, the description feels incomplete. Please provide more detail on the algorithm\nor pipeline used.\nWe added more detail on the approach used to generate the line-drawing stimuli\n(Methods, page 28, lines 725-730):\n“Line-drawing scenes: Image subclass 7 (images 25–28). Line drawings were obtained\nfrom a previous study (Walther et al., 2011). In brief, trained artists at the Lotus Hill\nResearch Institute created line drawings by tracing the contours of color photographs\nusing a custom interface. The sequence and coordinates of each line stroke were digitally\nrecorded. The line drawings associated with the natural scenes used in image subclass 4\nwere square cropped, resized to 714 pixels × 714 pixels, and then prepared as black lines\n(0) on a gray background (0.5). (Manipulated scenes)”\nAlthough the manuscript introduces a useful dataset, it does not currently meet the journal’s\nstandards for presenting conceptually novel and methodologically rigorous work that offers\nsignificant advances for specialists in the field. For these reasons, I do not recommend this\nmanuscript for publication in this journal.\nWe thank the Reviewer for raising these important concerns. We believe that addressing\nthese review comments strengthened the rigour, novelty, and usefulness of NSD-\nsynthetic.\n**Reference:**\nMalhotra, G. & Bowers, J. Predicting brain activation does not license conclusions regarding\nDNN-brain alignment: The case of Brain-Score. in Cognitive Computational Neuroscience\n(Boston, USA, 2024).\nhttps://2024.ccneuro.org/pdf/309_Paper_authored_Neural_Predictivity_CCN2024.pdf.\nNonaka, S., Majima, K., Aoki, S. C. & Kamitani, Y. Brain hierarchy score: Which deep neural\nnetworks are hierarchically brain-like? iScience 24, 103013 (2021).\nSchaeffer, R. et al. Position: Maximizing Neural Regression Scores May Not Identify Good\nModels of the Brain. in UniReps workshop (Vancouver, Canada, 2024).\nhttps://openreview.net/pdf?id=vbtj05J68r\nShirakawa _et al._ Spurious reconstruction from brain activity. Neural Networks (in press)\nWe thank the Reviewer for these references.\n(2025). https://doi.org/10.48550/arXiv.2405.10078\nRemarks to the Author\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports. This\nis part of the Nature Communications initiative to facilitate training in peer review and to\nprovide appropriate recognition for Early Career Researchers who co-review manuscripts.\nWe thank Reviewer #2 for co-reviewing the manuscripts and for the helpful comments.\nReviewer #3\nRemarks to the Author\nA very valuable resource for the visual neuroscience community. The original Natural\nScenes Dataset of 7T fMRI responses to over 9,000 scenes (Nature Neuroscience, 2021) is\none of the largest and most used open datasets in visual neuroscience (557 citations). The\ncurrent paper announces and describes a supplement to this dataset, which seems to have\nbeen collected around the same time, but only released in March of this year. The\nsupplement consists of 7T fMRI responses from the same individual subjects, to an\nadditional 284 synthetic images (either manipulated, e.g. via contrast reduction or phase\nscrambling; or artificially generated e.g. colour fields or spatial-frequency-controlled spiral\npatterns). Although the new data supplement is much smaller than the original core dataset,\nit will be useful both for testing encoding models outside the distribution of natural images\n(the main purpose advocated by the authors) and for probing the cortical representation of\ncolour, orientation, and spatial frequency. The stimuli in the dataset are probably not as\nexhaustively parameterised for this second purpose as a researcher would desire (e.g.\ncolour hue space is sampled, but not the saturation and lightness axes), but the dataset will\nnevertheless be helpful for some carefully-chosen research questions (as well as serving as\nthe intended out-of-distribution test set).\nThe paper is very clearly written and figures and methods are excellently presented. The\nanalyses of the paper go beyond the \"minimum required\" to describe and quality-check the\ndata, and also evaluate a number of deep neural network based encoding models\nWe thank Reviewer #3 for the positive evaluation and helpful comments, which\nresulted in valuable edits and additions to the manuscript.\nMinor comments:\n26\n1. line 46 \"intelligence\" seems like an odd way to characterise the research that would be\ndone with this dataset. Why not \"visual neuroscience\" or similar?\nBased on the Reviewer’s suggestion, we changed \"intelligence\" to \"visual neuroscience\"\n(Introduction, page 2, lines 51-52):\n“However, OOD tests are critical for visual neuroscience research in three significant\nways.”\n2. Figure 1B \"Manipulates scenes\" should read \"Manipulated scenes\".\nWe thank the Reviewer for spotting this typo, which we fixed.\n3. lines 96-101: It would be good to already acknowledge in the main text two important\ndifferences between the NSD-core and NSD-synthetic datasets - the temporal design (2s vs\n3s on), and the task (mixture of fixation and same/different tasks, vs continuous old/new\ntask). These differences do limit the comparability of the main and supplemental dataset and\nshould be acknowledged early.\nWe added NSD-core’s differing trial and task information to the Introduction\n(Introduction, page 5, lines 103-107):\n“We presented these images in a rapid event-related design consisting of 4-s trials (2-s\nON, 2-s OFF; instead of NSD-core's 3s ON, 1-s OFF trials) while measuring fMRI\nresponses (7T, 1.8-mm resolution, TR 1.6 s) from each of the eight NSD participants\n(Figure 1C). To assess potential task dependence of neural responses, participants\nperformed a fixation task and a one-back task in alternating runs (instead of NSD-core's\nlong-term continuous recognition task) (Figure 1D).”\nFurthermore, we acknowledged the spurious distributional differences introduced by\ndistinct task and temporal design choices between NSD-core and NSD-synthetic in a\ndedicated Discussion section, together with suggestions on how to mitigate possible\nunwanted effects of these differences on modeling analyses (Discussion, page 23,\nlines 537-587):\n“To obtain valid inferences, the details, quantifications, and interpretation of OOD tests\nrequire careful consideration. Here, we comment on four specific aspects that\nresearchers should consider when performing OOD generalization tests.\n[...]\nThird, we interpreted the distributional differences between NSD-synthetic and NSD-\ncore’s fMRI responses as stemming from differences in the stimuli of the two datasets.\nHowever, in theory, distributional differences might also reflect other properties of the\ndata, such as differences in task (Kay et al., 2023) or trial timing (Zhou et al., 2017),\nsession-specific effects, and fluctuations in cognitive state. All these data properties\nrepresent diverse dimensions across which different datasets might fall in- or out-of-\ndistribution. Therefore, to improve transparency, reproducibility,\n27\nand comparison across studies using NSD-synthetic, we recommend researchers to\nexplicitly define the OOD operationalization used.\nFourth, properties of the data not relevant to the researcher’s question of interest might\nintroduce spurious distributional differences between NSD-synthetic and NSD-core’s\nfMRI responses. For example, the trial timing differed between NSD-core (3-s ON, 1-s\nOFF) and NSD-synthetic (2-s ON, 2-s OFF), with longer image onscreen times typically\nleading to higher fMRI responses (Zhou et al., 2017). Similarly, NSD-core and NSD-\nsynthetic differed in task, with tasks also affecting fMRI response magnitude (Kay et al.,\n2023). Additionally, since NSD-synthetic and NSD-core’s fMRI responses come from\ndistinct scan sessions, they are affected by session-specific effects, that is, differences\nin fMRI responses across sessions due to incidental session-specific factors (e.g., time\nof day, arousal, cognitive state, hardware state). To eliminate spurious distributional\ndifferences, researchers might consider z-scoring fMRI responses within each scan\nsession. However, by centering responses and transforming them to unit variance, z-\nscoring also alters the signal of interest, potentially eliminating distributional differences\nof interest. Our approach for compensating for differences in fMRI response gain due\nto trial timing was to quantify encoding model performance using Pearson correlation\n(which is insensitive to gain differences). Furthermore, to reduce session-specific\neffects we analyzed fMRI responses aggregated across participants, considered\nvertices with a signal-to-noise ratio score above a certain threshold (see Methods), and\nperformed control analyses where we trained and tested encoding models on fMRI data\nfrom separate NSD-core scan sessions.”\n4. line 128: It would be helpful to mention how the noise ceilings measured in this dataset\ncompare to those in the NSD-core dataset.\nWe elaborated on the comparison between NSD-synthetic and NSD-core’s noise ceiling\nsignal-to-noise ratio (NCSNR) through text and a new Supplementary Figure (Results,\npage 7, lines 129-144):\n“NSD-synthetic’s main purpose is to facilitate out-of-distribution (OOD) generalization\ntests aimed at improving the robustness of computational models of the brain. A\nprerequisite of these tests is that NSD-synthetic’s fMRI responses must reliably encode\nthe different stimulus images. To assess this, we calculated NSD-synthetic’s noise\nceiling signal-to-noise ratio (NCSNR), a measure of stimulus-related signal in the fMRI\nresponses, using methods introduced in previous work (Allen et al., 2022). For all\nparticipants, we found that NCSNR scores are high over visual cortex, ranging\napproximately from 0.75–2 (Fig. 2a). This indicates that 36–80% variance in single-trial\nresponses reflects signals driven by the stimulus. As a comparison, NSD-core’s\nNCSNR scores range approximately from 0.5–1.25, indicating that 20–60% variance in\nsingle-trial responses reflects signals driven by the stimulus (Supplementary Fig. 1).\nHence, stimulus-related information is indeed reliably encoded in NSD-synthetic’s fMRI\nresponses. Within visual cortex, NSD-synthetic’s NCSNR is highest in early areas\ncompared to intermediate, ventral, dorsal, and lateral areas. We believe this reflects the\nlimited visual and semantic complexity of NSD-synthetic’s stimuli, which leads to\nreduced signals in higher visual areas that\n28\npreferentially respond to more complex visual features. Supporting this intuition, the\nvisually and semantically more complex naturalistic images from NSD-core lead to\nNCSNR scores that are more uniform across visual cortex (Supplementary Fig. 1).”\n“Supplementary Fig. 1 | NSD-synthetic and NSD-core’s noise ceiling signal-to-\nnoise ratio (NCSNR). a, NSD-synthetic’s participant-wise noise ceiling signal-to-\nnoise ratio (NCSNR), plotted on flattened cortical surfaces. b, NSD-core’s\nparticipant-wise NCSNR, plotted on flattened cortical surfaces. a-b, White contours\nindicate regions based on the ‘streams’ ROI collection as provided in the NSD data\nrelease.”\n5. line 300: ResNet is a feedforward architecture with skip connections, in my understanding,\nnot a recurrent network (i.e. it has no temporal dynamics or memory over stimuli).\nWe thank the Reviewer for pointing out this incorrect model description, which we\namended (Results, page 15, lines 322-324):\n29\n“Specifically, we compared encoding models based on ResNet-50, a convolutional\nneural network with skip connections pretrained on task-supervised image classification\n...”\nRemarks on code availability\nI have skimmed but not attempted to use any of the code. It appears well documented and\nthorough. Code is provided to reproduce all figures in the paper, as well as sharing the\nimages and fMRI data.\nWe thank the Reviewer for checking the code.\nReviewer #4\nRemarks to the Author\nThe authors present NSD-synthetic, a companion dataset to the Natural Scenes Dataset\n(NSD), consisting of 7T fMRI responses from 8 subjects viewing 284 synthetic images. This\ndataset is designed to enable out-of-distribution (OOD) generalization tests for\ncomputational models of visual processing. The authors demonstrate that neural responses\nto these synthetic stimuli are reliably encoded and are indeed OOD relative to NSD-core.\nThey also present encoding model comparisons showing differences between\nself-supervised and task-supervised models in OOD settings, limited to early/mid visual\ncortex where the synthetic stimuli drive robust responses.\nStrengths\nThis is an exceptionally valuable resource for the computational neuroscience community.\nThe dataset addresses a critical gap in the field - namely, the ability to test whether models\nof visual processing generalize beyond the natural image distributions they are typically\ntrained on. The experimental design is rigorous, using the same 8 subjects from NSD-core\nand maintaining high data quality standards with 7T fMRI at 1.8mm resolution. The open\nrelease of this data will enable researchers to develop more robust models and better\nunderstand the limitations of current approaches.\nThe technical quality is, as always from Ale, excellent. Comprehensive validation shows that\nthe synthetic stimuli reliably drive neural responses, particularly in early visual areas. The\nuse of multidimensional scaling to confirm the OOD nature of the responses is convincing,\nand the variety of synthetic stimuli (noise patterns, manipulated scenes, gratings, words)\nprovides good coverage of different visual features.\nOf particular interest, but also a potential confound, is the choice of artificial stimuli. as deep\nneural networks are (often) surprisingly bad using these type of stimuli. Using stimuli like\nspiral gratings, phase-scrambled images, etc. etc., the dataset probes a known weakness of\n30\ncurrent computational models. This makes NSD-synthetic an excellent diagnostic tool for\nunderstanding where and why DNNs fail to capture human visual processing.\nWe thank Reviewer #4 for the very positive evaluation and helpful comments, which\nresulted in useful clarifications in the manuscript’s text and figures regarding the validity\nof our findings across both low- and high-level visual cortex, as well as in a new\ndiscussion and analysis related to the comparison of encoding models based on\nResNet50 and MoCo deep neural networks.\nMajor/Medium comments:\n1. The authors show that self-supervised models (MoCo) outperform task-supervised models\n(ResNet-50) in OOD generalization, and that vision transformers outperform CNNs. For early\nvisual cortex (V1-V4), where the synthetic stimuli drive robust responses, these findings\nappear valid and represent an important discovery about differential model architectures.\nThe high noise ceiling values in these areas (Figure 2A) support meaningful model\ncomparisons. We do not really know what happens in other parts of cortex, or if the models\nwould generalise better towards OOD stimuli for areas in higher Cortex (e.g. OOD stimuli\nthat the cortex would respond to). Not sure what to do about this, you could look at OOD\neffects vs SNR effects. It could also really be that they are restricted to early cortex. But in\nany case I think it is important to reframe the conclusion a little bit to reflect that the OOD\ngeneralization findings are strongest for early visual cortex, with important caveats about\ninterpretability in higher areas.\nWe agree with the Reviewer that the lower SNR over early compared to higher visual\ncortex could potentially limit the strength of the conclusions that one can make about\nOOD generalization on higher visual areas, and we do explicitly address this in the\nmanuscript. However, because the fMRI responses of NSD-synthetic's higher visual\nareas are not devoid of signal, we do not believe that NSD-synthetic prevents meaningful\nconclusion concerning higher visual areas. Following we lay out the arguments/analyses\nof why we think this to be the case.\nWhile in Fig. 2a we show that the noise ceiling is highest in early areas compared to\nintermediate, ventral, dorsal, and lateral areas, these results do not show that higher-\nlevel visual areas do not encode visual information at all. In fact, we show that the\nresponses of intermediate, ventral, dorsal, and lateral areas also consist of stimulus-\nrelated signal based on their noise ceiling (Fig. 2a), and based on the results of\nunivariate (Fig. 2b-e) and multivariate (Fig. 2f) analyses (Results (Fig. 2), page 6, lines\n115-186):\n31\n“Fig. 2 | Noise ceiling, univariate, and multivariate analyses reveal robust visual\nsignals in NSD-synthetic. a, Participant-wise noise ceiling signal-to-noise ratio\n(NCSNR), plotted on flattened cortical surfaces. White contours indicate regions\nbased on the ‘streams’ ROI collection as provided in the NSD data release. b, ROI-\nwise participant-average univariate responses for noise, natural/manipulated scenes,\ncontrast modulation, and phase-coherence modulation (groupings indicated by\nvertical white lines). c, ROI-wise responses for chromatic noise (Hue01–Hue16) and\ncorresponding achromatic noise (Hue00). d, ROI-wise responses for single words.\nFor visualization purposes, the images with the most peripheral words are not square\ncropped (Word4 Pos1, Word4 Pos5, Word6 Pos1, Word6 Pos5). e, ROI-wise\nresponses for spiral gratings. f, RSA scores indicating the similarity of multivariate\nfMRI responses between participants\n32\nand ROIs. Thin white lines separate groups of the eight participants within the same\nROI. Cyan boxes indicate cross-participant comparisons within the same ROI.”\nFurthermore, when comparing the in-distribution (ID) and out-of-distribution (OOD)\nencoding accuracies within the same model, or across models, to prevent our results from\nbeing biased by vertices that do not encode stimulus-related signal (i.e., noisy vertices),\nwe only considered vertices with noise ceiling above the threshold of 0.3 (i.e., vertices\nwith at least 30% of variance consisting of signal). Even after this thresholding, a large\nportion of ventral, lateral, and dorsal cortex vertices were retained for NSD-synthetic,\nindicating that our OOD findings are also valid for a significant amount of high-level visual\ncortex (Fig. 4c-d; Fig. 5). To make this more explicit, we edited the scatterplots in Fig. 4e\nto highlight the fact that, when tested OOD, the encoding models fail to account for large\nportions of explainable variance across both lower- and higher-level visual cortex\n(Results (Fig. 4), page 12, lines 268-275):\n“We further observed that decreases in OOD performance compared to ID performance\nare consistent across both lower- and higher-level visual areas (see red regions in Fig.\n4d). Importantly, these decreases in performance are not due to lack of explainable\nsignal in NSD-synthetic’s fMRI responses, but rather to model failures. When plotting the\nvertex-wise explained variance scores against their noise ceilings, we found that\ncompared to ID tests on NSD-core, the encoding models’ explained variance scores\nwere consistently lower relative to the noise ceiling for OOD tests on NSD-synthetic,\nacross both lower- and higher-level visual areas (Fig. 4e).”\n33\n“Fig. 4 | Brain encoding models exhibit reduced performance when tested out-\nof-distribution on NSD-synthetic. We built AlexNet-based encoding models and\ntested them in-distribution (ID) on NSD-core as well as out-of-distribution (OOD) on\nNSD-synthetic. This figure shows results averaged across participants on flattened\ncortical surfaces. a, Model explained variance (r2). b, Noise ceiling (r2). c, Model\nexplained variance normalized by the noise ceiling. To reduce false positives, for each\nvertex we averaged results from only those participants with a noise ceiling greater\nthan 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown).\nd, Difference between ID and OOD noise-ceiling-normalized explained variance. For\neach vertex, we averaged results from only those participants with noise ceiling\ngreater than 0.3 for both NSD-core and NSD-synthetic. The asterisk indicates a\nsignificant difference between the ID and OOD noise-ceiling-normalized explained\nvariance scores (P = 10-7, paired sample t-test, one-sided, N = 8 participants). e, Scatterplots of vertex-wise explained variance scores against their noise ceilings for ID (NSD-core) and OOD\n34\n(NSD-synthetic) generalization tests, color coded by the visual stream that each\nvisual stream that each vertex belongs to. For each vertex, we plotted results from\nonly those participants with noise ceiling greater than 0.3 for both NSD-core and\nNSD-synthetic (same as in panel d). Black crosses indicate median scores across all\nvertices.”\nOf course, even though in our encoding analyses we only considered vertices with noise\nceiling above the threshold of 0.3, since noise ceilings are higher for early visual cortex\nit could be that our findings might also be more robust for early visual cortex. We\ntherefore made this more explicit in the limitations paragraph of the Discussion section\n(Discussion, page 25, lines 628-635):\n“The results of our analyses indicate that stimulus-related information in NSD-synthetic’s\nfMRI responses is primarily encoded in early to intermediate visual cortical areas (i.e.,\nV1 to hV4). This presumably is due to the fact that NSD-synthetic stimuli consist mostly\nof simple visual features that well activate early to intermediate visual cortex (e.g.,\nvarious forms of noise and spiral gratings), but which are not well suited for driving\nactivity in higher visual cortical areas responsive to perceptually and semantically more\ncomplex stimuli. Because of this, the results and conclusions obtained with NSD-\nsynthetic might be most robust when testing the OOD generalization of computational\nmodels of lower visual cortical areas.”\n2. The paper lacks discussion of why this might occur. and why this occurs for early cortex.\nIs the dictionary of these networks just better? Or because contrastive learning explicitly\ntrains models to be invariant to various image transformations, and that this generalises\ntowards these artificial stimuli?\nIn this cortex it might be an idea to look at layers in MOCO in relation to effects?\nSome mechanistic insight would really help.\nOur guess is that MoCo outperforms ResNet50 in OOD generalization due to its\naugmented training diet from the self-supervised contrastive learning objective. We\nelaborated on this in the Discussion section (Discussion, page 22, lines 514-528):\n“Another direction of model improvement comes from benchmarking the performance of\ndifferent computational models on NSD-synthetic and isolating model properties leading\nto best OOD generalization (Golan et al., 2020; Ren & Bashivan, 2025). Because each\nmodel embeds a different hypothesis of visual processing, these OOD generalization\ntests can help adjudicate between competing hypotheses, therefore facilitating theory\nformation. We found that encoding models based on a self-supervised deep neural\nnetwork better generalize OOD than encoding models based on its task-supervised\ncounterpart, in line with recent work proposing self-supervision as a more plausible\naccount of coding in visual cortex (Konkle & Alvarez, 2022; Doerig et al., 2023;\nPrince et al., 2024). The improved performance of self-supervised deep neural networks might\nderive from their training diet, which beyond colored naturalistic images also includes\naugmented images such as grayscale or blurred images. Compared to the fully\nnaturalistic colored images typically used to train task-supervised deep neural networks,\nthese augmented images are more similar to the artificial images from NSD-synthetic,\ntherefore resulting in encoding models with better OOD generalization performances.\nFuture efforts that\n35\nsystematically benchmark a larger variety of encoding models (Conwell et al., 2024)\nmight confirm this finding, and also isolate further model properties that improve OOD\ngeneralization performance.”\nAs suggested by the Reviewer, we also compared the ID and OOD encoding\nperformances using individual ResNet50 and MoCo layers. This layerwise analysis\nrevealed a complex effect landscape, where MoCo does not better generalize OOD\ncompared to ResNet50 for all layers, but only for some layers or when all layers are jointly\nused to train/test the encoding models. Nevertheless, the layerwise analysis further\nconfirmed our effect of interest in that OOD tests reveal differences between models that\nare not detected with ID tests. For these new analyses and results, please see the\nresponse Reviewer #1 comment 3 above.\nRemarks on code availability\nI perused the code. It seems complete and is intelligible.\nWe thank the Reviewer for checking the code.\nPlease find the detailed point-by-point response below. Reviewer comments are highlighted\nin gray, and the corresponding responses indented. Within the responses, extracts from the\nmanuscript are “quoted in italics”, and extracts reflecting changes to the manuscript are\n“quoted in italics and red font”. Manuscript line/page/figure numbers are indicated in bold\nfont.\nReviewer #1\nRemarks to the Author\nThe authors have responded sincerely to the previous comments, and through clarifications\nadditional analyses, the manuscript has surely improved. I have only one remaining concern\nregarding the expression “less strict OOD generalization tests” used to describe the\nnaturalistic OOD split. As currently written, this phrasing may unintentionally undervalue the\npotential of NSD-core.\nWe thank Reviewer #1 for the positive evaluation, and for the dedication in reviewing\nthe manuscript which resulted in helpful revisions that clarified the manuscript text.\nI understood that preparing OOD test splits that additionally control for session-specific\neffects is challenging in the present analysis. Considering that the main purpose of this study\nis a proof-of-principle demonstration using NSD-synthetic, I will not request further additional\nanalyses. However, I would encourage the authors to refine the manuscript by more clearly\nacknowledging the limitations of the current naturalistic OOD split and by avoiding\nstatements that might be interpreted as fundamental limitations of NSD-core itself.\nIn particular, referring to the NSD-core–based OOD evaluation as a “less strict OOD\ngeneralization test” could be read as implying that NSD-core inherently permits only weak\nOOD tests. In practice, with more carefully designed train–test splits, stricter naturalistic\nOOD evaluations are possible even within NSD-core. For example, following the\nconventional stimulus-based OOD splitting approaches (Mitchell et al., 2008; Brouwer &\nHeeger, 2009; Madan et al., 2024; Shirakawa et al., 2025), we could construct a naturalistic\nOOD test set consisting of a specific semantic category (e.g., “zebra” images) presented\nonly in the last few sessions. The training set could then be built from earlier sessions while\nexcluding all zebra-containing stimuli. This would allow direct comparison between\nnaturalistic OOD tests (inherent to NSD-core) and synthetic OOD tests (inherent to\nNSD-synthetic) while simultaneously removing session confounds existing in the current\nnaturalistic OOD.\nA similar logic can also be applied to the fMRI-based splits. For instance, given the\ndistribution shown in Revision Fig. 1, we could design the red samples at the right side as a\ntest set and construct the training set from the remaining black points while excluding the\ncorresponding region of the MDS space.\nthat the manuscript be revised to (i) clarify that the “less strict” nature of the current\nNSD-core OOD tests reflects the specific data split used in this study, not a fundamental\nlimitation of the dataset itself, and (ii) acknowledge more explicitly that stricter OOD tests\nwithin NSD-core are, in principle, possible with suitably designed splits.\nWe addressed the Reviewer’s concern by (i) removing the statements that NSD-core\nallows for less strict OOD generalization tests compared to NSD-synthetic, and by (ii)\nacknowledging that more constrained train-test splits on NSD-core might allow for\nstricter OOD generalization tests.\nSpecifically, the following revised sections may merit careful reconsideration:\nL20-L22:\n“because the stimuli of these datasets typically live within a common naturalistic visual\ndistribution, they do not allow for strict OOD generalization tests”\n→ Suggest clarifying that strict tests are possible but require more deliberate split design.\nWe edited this segment (Abstract, page 1, lines 15-18):\n“because the stimuli of these datasets typically live within a common naturalistic visual\ndistribution, they make it challenging to implement out-of-distribution (OOD)\ngeneralization tests crucial for the development of more robust brain models.”\nL31-33\n“less strict OOD generalization tests can be usefully applied even within the domain of\nnaturalistic stimuli.”\nWe removed this segment since by Editorial request we had to reduce the length of the\nabstract.\nL555-556\n“although resulting in less strict OOD generalization tests due to the lower degree of OOD of\nfMRI responses for naturalistic compared to artificial stimuli.”\nWe removed the statement of NSD-core being a less strict testbed of OOD tests, and\nmade a minor edit to the preceding sentence (Results, page 14, lines 369-371):\n“The results also demonstrate that the concept of OOD is not restricted to artificial\nstimuli, but that through carefully-designed train-test splits it can be usefully applied\neven within the domain of naturalistic stimuli.”\nL633- L635\n2\n“from lower degrees of OOD and thus less strict generalization tests using fMRI responses\nfor NSD-core’s naturalistic stimuli, to higher degrees of OOD and thus stricter tests using\nNSD-synthetic”\nWe made an addition to this segment (Discussion, page 16, lines 438-446):\n“Our empirical results reinforce the fact that OOD generalization performance of brain\nmodels cannot be fully described by a simple summary number, but lives in a\ncontinuum of distances between the train and test data distributions. Combining\nNSD-core and NSD-synthetic provides a way to test several degrees of OOD\ngeneralization across this continuum, from lower degrees of OOD and thus less strict\ngeneralization tests using fMRI responses for NSD-core’s naturalistic stimuli, to higher\ndegrees of OOD and thus stricter tests using NSD-synthetic. (However, stricter OOD\ngeneralization tests are in principle also possible with NSD-core’s naturalistic stimuli,\nthrough a more constrained division of train-test splits than the one employed in this\nwork.)”\nThis revisions have considerably strengthened the manuscript. With the minor clarifications\nsuggested above, I am confident that the manuscript will be even clearer and more\ninformative, and that it will serve as a valuable resource for future studies on large-scale\nvisual neural datasets.\nWe thank the Reviewer for the positive evaluation of the revised manuscript, as well as\nfor the new comments, which helped improve the clarity of the manuscript.\nReviewer #2\nRemarks to the Author\nI co-reviewed this manuscript with one of the reviewers who provided the listed reports. This\nis part of the Nature Communications initiative to facilitate training in peer review and to\nprovide appropriate recognition for Early Career Researchers who co-review manuscripts.\nWe thank Reviewer #2 for co-reviewing the manuscripts and for the helpful comments.\n3"}]

Decision letter