Comparing Representations in Static and Dynamic Vision Models to the Human Brain

Authors: hamed  karimi
stefano  anzellotti
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 09:33:02

Abstract: We compared neural responses to naturalistic videos and representations in deep network models trained with static and dynamic information. Models trained with dynamic information showed greater correspondence with neural representations in all brain regions, including those previously associated with the processing of static information. Among the models trained with dynamic information, those based on optic flow accounted for unique variance in neural responses that were not captured by Masked Autoencoders. This effect was strongest in ventral and dorsal brain regions, indicating that despite the Masked Autoencoders' effectiveness at a variety of tasks, their representations diverge from representations in the human brain in the early stages of visual processing.

Main paper: 
 introduction: 
 The human visual system is organized into distinct processing streams [Ungerleider et al., 1982, Pitcher and Ungerleider, 2021] : a ventral stream that extends from early visual regions into the inferior portions of temporal cortex, and a dorsal stream that extends into lateral occipital cortex and branches into a lateral stream (along the superior temporal sulcus) and a parietal stream (reaching the inferior parietal lobule). This organization likely results from the computational requirements of visual perception. Therefore, understanding the representations encoded by different visual streams could offer insights about the human brain and also about more general principles of vision.The ventral stream has been proposed to encode static object identity [Grill-Spector and Weiner, 2014] , while dynamic information has been associated with the dorsal, lateral and parietal streams [Ganel and Goodale, 2003, Culham et al., 2003] . Indeed, static images of objects are known to drive responses in ventral temporal regions in macaques [Pasupathy and Connor, 2002 , Logothetis et al., 1995 , Tanaka, 1996 , Hung et al., 2005] and in humans [Edelman et al., 1998 , Haxby et al., 2001] . Moving stimuli drive stronger responses in dorsal and lateral regions [Zeki et al., 1991 , Tootell et al., 1995 , Saito et al., 1986] . In addition, disruption to lateral regions using TMS affects the processing of dynamic information [Beckers and Hömberg, 1992, Pitcher et al., 2014] as well as motion prediction [Vetter et al., 2015] .However, other studies have challenged the hypothesis that visual streams in the human brain differ based on whether they encode static or dynamic visual features. These studies suggested that both static and dynamic features are represented in multiple visual streams [Kourtzi et al., 2002 , Freud et al., 2017 , Cornette et al., 1998 , Sunaert et al., 1999 , Robert et al., 2023] . Here, we investigated the contribution of static and dynamic information to the representations encoded by different visual streams by quantifying the convergence between neural representations and representations learned by deep network models.Previous work compared neural responses to deep network models trained with static images [Yamins et al., 2013 , Khaligh-Razavi and Kriegeskorte, 2014 , Zhuang et al., 2021 , Konkle and Alvarez, 2022] . The present work studies the additional contribution of dynamic information during the observation of quasi-naturalistic videos by comparing neural responses to deep networks whose inputs are static images (e.g., convolutional ResNets [He et al., 2016] , image masked autoencoders [He et al., 2022] ) and to deep networks whose inputs are videos (e.g., hidden two-stream networks, video masked autoencoders [Zhu et al., 2019 , Tong et al., 2022 , Feichtenhofer et al., 2022] ). Critically, we include in our analyses a family of self-supervised models that are widely used in Computer Vision but that are understudied in Cognitive Neuroscience: masked autoencoders (MAEs) [He et al., 2022] .Previous work has indicated that models yielding more accurate categorization performance also typically offer a more accurate prediction of neural responses [Yamins et al., 2014] . MAEs are remarkably effective at categorization, making them promising candidate models of neural responses. Additionally, thanks to their transformer architecture, MAEs can capture larger-scale spatial dependencies in images, and masked video autoencoders can capture temporal dependencies. Human observers are able to represent such dependencies as well, thus in principle, MAEs could account for unique variance in neural representations compared to convolutional neural networks (CNNs). 
data: 
 BOLD fMRI responses (3×3×3 mm) to eight movie segments of 'Forrest Gump' were obtained from the publicly available studyforrest audiovisual dataset (http://studyforrest.org). Fifteen right-handed participants took part in the study (6 females; age range 21-39 years, mean 29.4 years). The data was acquired with a T2*-weighted echo-planar imaging sequence, using a whole-body 3 Tesla Philips Achieva dStream MRI scanner equipped with a 32 channel head coil.Data were first preprocessed using fMRIPrep (https://fmriprep.readthedocs.io/en/ latest/index.html): a robust pipeline for the preprocessing of diverse fMRI data. Anatomical images were skull-stripped with ANTs (http://stnava.github.io/ANTs/), and FSL FAST was used for tissue segmentation. Functional images were corrected for head movement with FSL MCFLIRT (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/MCFLIRT), and were subsequently coregistered to their anatomical scan with FSL FLIRT. Finally, the skull-stripped anatomical images were normalized to the MNI template using SPM. We denoised the data with CompCor Behzadi et al. [2007] using 5 principal components extracted from the union of cerebrospinal fluid and white matter. To identify the regions of interest (ROI), we used an atlas of probabilistic maps of visual topography in the human cortex from a previous study [Wang et al., 2015] . The atlas contains twenty-five cortical regions and spans multiple visual streams: ventral, dorsal, parietal, and lateral (Figure 1 ).A list of probabilities is associated with each voxel to reflect the likelihood of that voxel being part of each of the twenty-five brain regions (R i , i = 1, . . . , 25). We calculated the transformation from MNI space to each participant's native space and co-registered the probability maps with each participant's anatomy. To prevent overlap between the regions of interest in the participants' native space, we followed a procedure analogous to [Wang et al., 2015] . Specifically, we calculated the maximum probability map for each participant, using which we exclusively classified each voxel as either belonging to a specific ROI or as being outside of all the ROIs.The inclusion probability was computed as the probability of a voxel of being in any of the defined regions (P (∪ 25 i=1 v ∈ R i )), and The exclusion probability is the probability of a voxel not belonging to any of the ROIs (P(∩ 25 i=1 v ̸ ∈ R i )). If the exclusion exceeded the inclusion probability, we discarded the voxel. Otherwise, we classified the voxel as belonging to the region with the highest probability. The resulting ROIs were grouped into four sets associated with distinct visual streams. The ventral stream contains V1v, V2v, V3v, hV4, VO1, VO2, PHC1, PHC2; the dorsal stream V1d, V2d, V3d, V3a, V3b; the lateral stream LO1, LO2, hMT, and finally the parietal stream IPS0, IPS1, IPS2, IPS3, IPS4, IPS5, SPL1, and FEF. While often the term "dorsal stream" is used to refer to the combination of the regions we labeled as "dorsal stream" and the regions we labeled as "parietal stream", here we sought to distinguish between the initial branch of the dorsal stream and its parietal and lateral temporal continuations, without implying that the initial segment is disproportionately associated with one or the other.To study representations of quasi-naturalistic visual stimuli, we used a variety of vision models, including feed-forward convolutional neural networks, as well as state-of-the-art foundation vision models. The models vary in architecture, learning objective, and training data (Table 1 ). Here, we propose an overview of the models. Training details for the HAA-trained CNNs are presented in supplementary materials. The trained versions of all other models are adopted from their official implementation repository. For model details, refer to the original papers. Supervised (sup) static net is the spatial stream of the hidden two-stream convolutional neural network model [Zhu et al., 2019] . The sup static net has a resnet18 architecture and encodes static features of visual stimulus. Two versions of the model were included in the models' pool: one is trained on Image-Net [Deng et al., 2009] and predicts object identity, and the other is trained on HAA-500 action dataset [Chung et al., 2021] and predicts action label. Both versions take a single frame as input. Self-supervised (s-sup) dynamic net is the first part of the temporal stream (i.e., motion net) in the hidden two-stream convolutional neural network model [Zhu et al., 2019] . The self-supervised dynamic net takes 11 consecutive frames as input and infers the optic flow between each pair of consecutive frames. The network is trained to minimize an self-supervised learning objective obtained by combining three loss functions:1) a pixel-wise reconstruction error, 2) a smoothness loss addressing the ambiguity problem of optic flow estimation (also known as the aperture problem), and 3) a structural dissimilarity between the original and the reconstructed image patches (see Zhu et al. [2019] for details of loss functions). The models' pool contains one version of the self-supervised dynamic net, trained on the HAA-500 action dataset Chung et al. [2021] . Supervised (sup) dynamic net is the second part of the temporal stream in the hidden two-stream convolutional neural network model [Zhu et al., 2019] . The model has resnet18 architecture and takes optic flows from the self-supervised dynamic net as input. We used the HAA-500 dataset Chung et al. [2021] and trained the supervised dynamic net to predict action labels using optic flows. Masked Autoencoder (MAE) learn image representations, required to reconstruct original uncorrupted images from corrupted (masked) input through a series of transformer blocks He et al. [2022] . The models' pool contains two versions of the MAE model: 1) a pre-trained version, where the model is trained to reconstruct pixel values, and 2) a fine-tuned version, where the pre-trained model is further fine-tuned to predict object identities. Both versions were trained on Image-net Deng et al. [2009] . Video Masked Autoencoder (VMAE) learns a spatiotemporal representation of videos, required to reconstruct original uncorrupted videos, from corrupted (tube masked) input through a series of transformer blocks Tong et al. [2022] . We added two versions of the VMAE to our models' pool. The first is a pre-trained version, where the model is trained to reconstruct missing pixels of the input set of frames. The second version is the fine-tuned version obtained by fine-tuning the pre-trained version to predict action labels of input videos. Both models take a consecutive set of frames as input, and were trained on the Kinetics-400 action dataset Kay et al. [2017] . Masked Video Distillation (MVD) learns a higher-level spatial and spatiotemporal representation of the input video, required to reconstruct the representation of teacher MAE and VMAE while taking corrupted (tube-masked) videos as input [Wang et al., 2023a] . Unlike VMAE and MAE, the MVD model does not learn pixel-level features. Rather, it learns high-level features of the input video using pre-trained MAE and VMAE models' features as masked prediction targets. Using the Kinetics-400 action dataset Kay et al. [2017] , a pre-trained version was obtained and added to the models' pool.In order to compare the models and the fMRI data, we computed representational dissimilarity matrices (RDMs) for the models' layers with a multi-step procedure. First, since the temporal resolution of the models' representations (25Hz) is much higher than the temporal resolution of fMRI data, we down-sampled each layer's activation timecourses over time by selecting one data point every five time points(down to 5 Hz). Then, we convolved the layer's activations with a standard Hemodynamic Response Function (HRF). Given that the fMRI data's repetition time (TR) is 2 seconds, we took a layer's activation every 25 × 2 = 50 time points.Finally, for each layer we computed the dissimilarities between all pairs of timepoints, obtaining RDMs in which the entry at column j and row i contains correlation dissimilarity (1-Pearson's r) between the layer activations at time i and time j. We repeated this procedure for BOLD responses to all eight movie segments, resulting in eight RDMs.RDMs were constructed separately for each brain stream in the subject's native space. The voxels for each brain stream were obtained as the union of the region voxels for individual regions within that stream. For each brain stream, we calculated the correlation dissimilarity (1 -r where r is Pearson's correlation) of fMRI response patterns for all pairs of TRs. This yielded eight RDMs, corresponding to BOLD responses in eight video segments.To evaluate how well each model accounts for the activity in the brain streams, we used a crossvalidated linear regression to predict the left-out movie segment brain stream RDM and computed the correlation between the predicted and the true RDM in each brain stream. The correlation captures how well a model's layers can predict a brain stream's responses to the visual stimuli. First, we used each model's layers' RDMs corresponding to seven (out of eight) video segments to train a linear regression model that predicts the corresponding seven RDMs in each brain stream. Then, we averaged the linear regression model's coefficients along the seven segments and used the averaged coefficients to predict the brain stream RDM of the left-out segment, using the model layers' RDMs of the corresponding segment. Finally, we calculated the Pearson's correlation between the predicted and the true RDMs. We repeated the leave-one-out cross-validation process for all the segments and averaged over the obtained correlations.We sought to study whether a combination of features from two models can improve similarity with brain data. We followed the procedure in 2.7 and used RDMs of all the layers in a pair of combined models to estimate the coefficients of a linear regression model that best predicts the RDM of a brain stream in seven (out of eight) of the video segments. Using leave-one-out cross-validation, we predicted the brain stream RDM of the left-out video segment using the average of the coefficients obtained from the seven video segments during training. Finally, we measured the correlation between the predicted RDM and the actual brain stream RDM to measure the correspondence between the combined models' features and the brain activity.To evaluate how well unique and shared features among a pair of computational models correspond to the brain data, we used Pearson's r to measure the accuracy of a "target" model's layers prediction of a brain stream RDM while controlling for the variation of a "control" model layers. Using leaveone-out cross-validation, first, we estimated the coefficients of a linear regression model that predicts a brain stream's RDM from the control model's layers in training video segments (seven out of eight). Second, we subtracted the predicted from the actual brain stream RDM in the training and the left-out video segments to obtain training and left-out residuals. Third, we estimated the coefficients of a linear regression model that predicts training residuals of each video segment using the target model layers. Finally, we measured Pearson's correlation between the target model's prediction of the left-out video segment residuals and the residuals obtained from the prediction of the control model. 
results: 
 The human visual system does not consist of a single processing stream. Instead, it is organized into distinct neural pathways. To the extent that the structure of the human visual system is shaped by computational optimality, understanding the visual representations encoded in these pathways can offer insights into more general principles of vision. The contribution of this work is to quantify the similarity between the representations in the different visual pathways in the human brain and representations in models of vision that are widely used in Computer Science but understudied in Cognitive Neuroscience (e.g. masked autoencoders, two-stream networks, masked video distillation). In a first set of analyses 3, we leverage differences between models to reveal differences between the information encoded in different visual pathways in the brain. In a second set of analyses, we quantify the unique contribution of different deep network models to account for neural responses 3.2.While numerous research studies have been conducted on model-to-brain correspondence using static images [Rose et al., 2021 , Doshi and Konkle, 2023 , Tsao et al., 2006] , the impact of dynamic information on neural responses to naturalistic videos is understudied. To fill this gap, we tested the correspondence between neural representations in different visual pathways (ventral, dorsal, lateral, and parietal) and deep network models that can process dynamic information (two-stream networks, video masked autoencoders, and masked video distillation). Comparing the correspondence of neural responses with these models and their correspondence with models that only process static information (standard convolutional ResNets, masked autoencoders) made it possible to study the contribution of dynamic information independently of whether the learning objective is supervised (as in two-stream networks) or unsupervised (as in masked autoencoders). In addition, the study of the correspondence between neural representations and representations in masked autoencoders is of interest in its own right: masked autoencoders are effective and widely used, but little is known about their similarity to neural representations.Functional MRI responses recorded during the observation of naturalistic videos in the ventral, dorsal, lateral and parietal visual pathways were compared to the representations in feed-forward convolutional neural networks. The same dataset (HAA-500) was used to train the different branches of a hidden-two-stream network: the "supervised static" branch (a ResNet that takes as input individual frames of a video and computes as output the action category), the "unsupervised dynamic" branch (a convolutional network trained to compute optic flow by minimizing a self-supervised loss), and the "supervised dynamic" branch (a ResNet that takes as input optic flow and computes as output the action category). In addition, to facilitate parallels with prior work, we compared neural responses to a widely studied feed-forward model: a ResNet trained with Image-net.Comparing deep network models trained with the same dataset (HAA-500) showed that models including dynamic information correlated with neural responses more than the Spatial model, that does not use dynamic information (Figure 2 ). This effect was observed for all visual pathways. In addition, representations in the lateral and parietal pathways correlated more with the supervised dynamic model than with the unsupervised dynamic model (fisher-transformed t-values with Bonferronicorrected threshold). Lateral and parietal regions are located downstream compared to the dorsal regions, thus this result is complementary to earlier work that reported a correspondence between subsequent stages of processing in deep neural networks and in neural pathways in the case of static visual stimuli [Khaligh-Razavi and Kriegeskorte, 2014] and in the case of auditory stimuli [Kell et al., 2018] .Supervised CNNs trained with Image-Net performed well, achieving correspondence with neural responses that was close to that of HAA-trained models that included dynamic information. This could indicate that some of the variance in neural responses that correlates with dynamic models might also be accounted for by models trained exclusively with static information, as long as a suitable training dataset is used (in this case, Image-Net). However, an alternative possibility is that the supervised static model trained with Image-Net and the dynamic models trained with HAA might account for different portions of the variance in neural responses. We investigate these alternative possibilities in section 3.2.Masked Autoencoders (MAE, [He et al., 2022] ) and Video Masked Autoencoders (VideoMAE, [Tong et al., 2022 , Feichtenhofer et al., 2022] ) models are trained to reconstruct masked pixels of input (image or video) during pre-training and are further fine-tuned to predict object/action labels. MAE and VideoMAE models are very effective in learning visual representations and have been shown to outperform competing models in several visual tasks [He et al., 2022 , Tong et al., 2022 , Feichtenhofer et al., 2022 , Wang et al., 2023b, Venkatesh et al.] . However, it is still unknown whether the representations learned by models based on masked autoencoding are similar to visual representations in the human brain. Here we investigated this question, quantifying the correlation between neural responses measured with fMRI while participants watched naturalistic videos, and representations learned by models trained with masked autoencoding.We compared the correspondence between neural responses and MAEs trained with images (which learn spatial relationships between component of an image, [Wang et al., 2023a] ) as well as Video-MAEs (which learn temporal relationships in videos, [Wang et al., 2023a] ). Finally, we also compared neural responses to masked video distillation (MVD, [Wang et al., 2023a] ), which combines image MAEs and videoMAEs to better capture both spatial and temporal relationships. Unlike MAE and VideoMAE, the MVD model does not aim to reconstruct missing patches at the level of pixel values. Instead, MVD adopts a knowledge-distillation approach, reconstructing missing information at the level of features extracted from pre-trained MAE and VideoMAE teachers.As in the case of supervised models trained with the HAA dataset, models that included dynamic information (VideoMAEs) outperformed models using only static information (Image MAEs). This pattern was observed across all visual pathways. Image MAEs did not correlate well with neural responses, even compared to supervised models trained with static inputs. Overall, the representations learned by Image MAEs were very different from neural representations. By contrast, VideoMAEs showed greater correspondence with neural responses. In particular, fine-tuning with an action recognition task (Figure 2 , VideoMAE fine-tuned) improved the correspondence between Video-MAE representations and neural representations across all streams (fisher-transformed t-values with Bonferroni-corrected threshold). Across all the pre-trained models, pre-trained MVD showed the highest similarity to neural representations in all brain streams. Further, MVD showed comparable similarity with brain streams to that of fine-tuned VideoMAE.The results described in 3 show that representations from models trained with dynamic information are more correlated with neural representations compared to representations from models trained with static information. In particular, a mixed-effects model was conducted to examine the effect of model type (static vs. dynamic) on Fisher-transformed correlation values across subjects. Results indicated a significant effect of model type with models that process dynamic information having higher correlation (F(fixed effect df=1, residual df=502)=-13.487). This overall pattern is broken by the exception of ResNets trained with ImageNet, which performed on par with models trained with dynamic information. This raises the question of whether ResNets trained with ImageNet and dynamic models explain overlapping variance in neural responses or whether, instead, they are complementary, capturing non-overlapping portions of the variance. This question can be posed more generally for any pair of models studied in section 3. We investigated this first by combining layers from two models and measuring whether a combination of models can better predict the pattern of neural activity in visual pathways. Second, we measured the correspondence between a "target" model's representations and the representations in each brain stream while controlling for the representations encoded in a "control" model. To this end, we predicted neural representations using the representations of the control model and obtained the residuals. Then, we predicted the residuals using the representations in the target model (see Methods for details).Each matrix in Figure 3 .a shows how well a combination of models' layers can predict the pattern of neural activity in a brain stream. Each column of each row demonstrates the correlation between the neural response pattern of a brain stream and the combined models' layers' prediction of that brain stream's neural activity pattern. Model-to-brain-stream similarity increased in all brain streams when combining features from static models with features from dynamic models. Notably, the correspondence between combined models' features with both dorsal and ventral streams improved in two cases: 1) combined features from Image-net-trained static supervised models with dynamic features from self-supervised model and 2) features from the combination of the self-supervised dynamic model with either VideoMAE or MVD. These cases shows that, first, ventral and dorsal brain streams both represent static and dynamic visual features, and second, different types of dynamic information are represented in both of these human streams. learned by these models also capture non-overlapping variance with those learned by the unsupervised dynamic models: the VideoMAEs. This finding shows that despite VideoMAEs exhibit relatively high correlations with neural responses (outperforming Image MAEs), they nonetheless fail to capture some variance in human visual representations that is accounted for by self-supervised and supervised dynamic models.VideoMAEs and MVD accounted for additional variance in neural responses compared to MAEs (as expected given the results in Figure 2 ) but also compared to the HAA-trained static and selfsupervised dynamic models. However, they accounted for a minimal amount (if any) of additional variance compared to the supervised dynamic model, suggesting some degree of convergence on common representations across models trained with different learning objectives. 
limitations: 
 This study focused on a set of models selected to enable comparing the contribution of static and dynamic information and the impact of supervised and unsupervised learning objectives. The selection of models in this study includes only a subset of the existing models; future work will be needed to expand the set of models tested. In particular, future research should employ a larger space of models to disentangle the specific contributions of factors such as datasets, architectures, and loss functions to differences in brain predictability. In addition, the present work centered on the comparison between models and entire visual streams. A finer-grained analysis comparing models to individual regions within each stream will require further work. 
discussion: 
 In the current study, we focused on predicting brain responses to a quasi-naturalistic movie using two popular classes of models-CNNs and MAEs-that also differ in their representation of spatial and temporal information. This yielded three main findings. First, models including dynamic information outperformed models using exclusively static information, not only in the dorsal, lateral, and parietal streams but also in the ventral stream. This is in line with recent evidence of responses to dynamic features in ventral brain regions [Robert et al., 2023] . Patients with deficits for motion perception typically present with lesions affecting dorsal regions (such as area V5, [McLeod, 1996 , Vaina et al., 1990 , Zihl et al., 1983] ) or parietal regions [Battelli et al., 2003] . By contrast, patients with damage to ventral regions typically do not present with deficits for motion perception [Gilaie-Dotan et al., 2015] . This raises the question of what might be the use of dynamic information represented in the ventral stream. We hypothesize that this information might be used to support object segmentation, as proposed in recent computational models [Chen et al., 2022] inspired by classic work in Developmental Psychology [Spelke, 1990] .Second, Image MAEs showed little correspondence with neural representations, even compared to other models trained exclusively with static information. These results indicate that despite the effectiveness of Image MAEs for learning visual representations that can transfer to a variety of visual tasks [He et al., 2022] , these models do not converge on representations that are similar to those observed in the human brain, suggesting that human vision and image MAEs rely on different computational mechanisms.Third, models based on optic flow representations accounted for unique variance in all streams, even compared to video masked autoencoders that can make use of dynamic information. Fine-tuning video MAEs with an action classification task increased their correspondence with neural representations, but did not fully bridge the gap with neural responses compared to optic flow models, which still explained additional unique variance compared to the fine-tuned video MAEs. The additional contribution of optic flow models was particularly strong in ventral and dorsal regions, suggesting that representations based on optic flow exhibit greater correspondence with representations in early stages of visual processing in the human brain compared to both image and video MAEs.In complex videos such as a movie, perceptual features and higher-level features can be partly correlated, thus, it is difficult to establish with certainty whether observed correlations between models and neural responses are driven by the neural representation of perceptual properties of the input or higher level information (see [Grall and Finn, 2022] ). Nonetheless, it is possible to evaluate which models provide a better account of neural responses overall. Determining whether the correspondence is driven by perceptual or higher-level features will require targeted experiments using stimuli in which these different types of features are uncorrelated.The correlation values we obtained for our quasi-naturalistic stimuli (the movie Forrest Gump) are comparatively lower than those obtained in previous work for static images [Conwell et al., 2022] , likely reflecting the greater complexity of the quasi-naturalistic stimuli. It also needs to be noted that the overall magnitude of the differences between static and dynamic models' correlation with brain responses is relatively small (approximately r=0.05).The selection of models in this study includes only a subset of the existing models; future work will be needed to expand the set of models tested. In particular, future research should employ a larger space of models trained with a broader variety of datasets to disentangle the specific contributions of factors such as datasets, architectures, and loss functions to differences in brain predictability. Further comparing neural responses with models that compute motion energy will be an important direction for future research. In addition, research will be needed to test the ability of MAEs to account for responses in other regions, particularly the anterior portions of the temporal lobe (including the anterior superior temporal sulcus), which have been implicated in high-level and social perceptions.Preprint. 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Review
main_review: main_review: - Summary: 

  - The study characterizes the ability of a set of models to predict neural responses to natural videos as measured via fMRI. In particular the authors are interested in whether: models trained on video inputs are more predictive (in general, or in supposed motion processing regions) of neural activity, and whether masked autoencoders are strong models of neural responses (which thus far has not been systematically tested in the literature.
  - The initial experiment measures the neural predictivity of several models in several different brain regions. My high level takeaways are (1) there is surprisingly little variation in neural predictivity across ventral/dorsal/lateral/parietal regions (2) Image MAEs are poor models of neural responses to natural videos and (3) models trained on dynamic tasks/inputs are generally slightly more predictive than models trained on single frames/static tasks *given that the models have been trained using the same dataset and finally (4) the advantage of dynamic training is approximately the same size as the advantage of training on a large/diverse set of images rather than HAA-500 (i.e. the height differnce between the first and second columns of the left panel of fig 2 is similar to the height difference between 2 and 3). 
  - Next the author's conduct two experiments to determine the degree to which different models are explaining either unique or overlapping portions of the neural response. First they measure how well neural responses can be predicted from *pairs of model responses* (so, if two models are predicting non overlapping portions of variance their paired predictivity will be larger than either of their individual predictivity scores). And second, they train models to predict the remaining response variability after conditioning on a second model (essentially by having model A predict the residual response left after predicting the response with model B).  These analyses show that (1) the ImageNet static supervised model is explaining a mostly distinct portion of variance to the dynamically trained models and (2) VideoMAEs improve on MAEs, but mostly by explaining a similar part of the neural response signal to the other (supervised action recognition and self-supervised optic flow prediction) dynamic objective/models considered.

- Strengths:
  - Originality/Significance: The considered question is very interesting and, to my knowledge not systematically answered in the current literature. 
  - Rigor: Experimental details are explained in significant detail (likely enough to reproduce the core results). It is clear the author's made a significant effort to provide reasonably reliable estimates of predictivity and their associated uncertainties. 
  - Clever Design of Controls: I think the experiments to determine whether static and dynamic models are predicting overlapping portions of response variance were interesting and important. This is a good example of why the field should move away from simply reporting predictivity scores! 

- Weaknesses/Suggestions/Questions:
  - All of the models predictivities are quite low (mostly pearson's r varies betwen .05 and .15 in Fig. 2). Is there any reason we should expect this particular modality of fMRI to be much more difficult to capture with ANNs (for example I think the cRSA scores in [1] are most similar to what you are measuring and these typically have values ~0.3).
  - Many of the error bars are significantly overlapping in Fig 2. What gives? Also this is Std over subjects, but what about the variance within subjects (over CV splits)? In all, it would be good to provide more context for evaluating the statistical significance of some of the differences.

[1] Conwell, Colin, et al. "What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?." BioRxiv (2022): 2022-03.
title: title: Comparing Representations in Static and Dynamic Vision Models to the Human Brain
main_review: main_review: This work investigates how well embeddings from deep neural network models trained using static and dynamic inputs predict neural responses to a naturalistic, dynamic movie stimuli. This is an important area to research since it tests the generalization of insights gained in research settings to how the brain processes stimuli that are closer to daily life. The study also compares self-supervised and supervised models, including Masked Autoencoders. 

Pros:

- Good methods to compare brains and models, comparing to all three streams is well motivated by the cognitive neuroscience literature
- Compared across models trained with the same dataset

I liked the analysis where they controlled a model with another model to see what unique information each model provided to the neural prediction. It was comprehensive and clearly shows the inability of the MAE’s in predicting neural activity and how the optic flow models predict neural activity the best.

Cons: 

- Overall differences between static and dynamic stimuli are extremely small, with differences of about r = 0.05 (which corresponds to the dynamic model explaining just .25% (r^2=0.0025) more variance than the static model) between paired static and dynamic versions of models. This is fine (and interesting!) to report, but concluding that dynamic models are better predictors than static models seems too strong and premature.
- While I liked the analysis looking for unique variance, it is difficult to interpret some of the results since we don’t know what the differences are due to for some of the pairs of models (could be different datasets or the architectures or the loss functions for example).

It would have been nice for the authors to provide more motivation or a framing for why they thought each of these models would be a good neural predictor, especially for the Masked Autoencoders, which is their novel contribution. They gave good background on how these models do well on visual tasks and they state that they are understudied in Cognitive Neuroscience, but that is not quite a satisfactory reason to study them.

The results could be improved by comparing the optic flow models to pymoten (motion energy extracted using a using a pyramid of spatio-temporal Gabor filters) to see if these models learn specific dynamic visual representations that benefit predicting neural activity in each of the three streams.
title: title: An interesting study demonstrating the importance of studying dynamic vision
main_review: main_review: This is an interesting paper examining the correspondence between different DNNs and the human brain in dynamic, natural contexts. Most work in this domain has been done with static images, and extending to the temporal domain is an important area.

The results show that video models / temporal dynamics are important to match visual cortex resopnses. This corresponds with other recent work with video clips (Garcia et al. 2024 PsyArxiv). Interestingly, even a simple optic flow model can account for unique variance in the neural responses.

I think there are a couple of limitations that would help to acknowledge / discuss. First, a continuous Hollywood movie (Forest Gump) has inherent correlations between the perceptual and higher level features, as well as autocorrelations across the movie that can make the results difficult to interpret (for an interesting discussion see Grall and Finn 2022). Second, like many neuroAI papers only a handful of models are compared to the brain. While it is tempting to attribute these model differences to broad distinctions between models (e.g., dynamic vs static or training objective), they differ along many other factors such as architecture. The relatively small set of models considered can make it difficult to draw strong conclusions about the cause of model differences. One other small point is that more anterior regions of the lateral stream in the STS were not modeled.

Overall this is an interesitng paper that points towards the importance of benchmarking neuroAI in dynamic, naturalistic contexts.

Comments: comment: Thank you for bringing Grall and Finn (2022) to our attention. It is important to acknowledge that perceptual and higher-level features can correlate in complex movies such as the one we used in the present study. We added this sentence to the Discussion section to highlight this point:

“In complex videos such as a movie, perceptual features, and higher-level features can be partly correlated, thus, it is difficult to establish with certainty whether observed correlations between models and neural responses are driven by the neural representation of perceptual properties of the input or higher level information (see Grall and Finn, 2022).”

Despite this, it is possible to evaluate which models provide a better account of neural responses in different brain regions. We added this passage to reflect this:

“Nonetheless, it is possible to evaluate which models provide a better account of neural responses overall. Determining whether the correspondence is driven by perceptual features or higher level features will require targeted experiments using stimuli in which these different types of features are uncorrelated.”

The ability to only test a subset of models is a limitation of this study, and caution needs to be applied in the interpretation of the results. We added the following to the Discussion section:

“The selection of models in this study includes only a subset of the existing models; future work will be needed to expand the set of models tested. In particular, future research should employ a larger space of models trained with a broader variety of datasets to disentangle the specific contributions of factors such as datasets, architectures, and loss functions to differences in brain predictability.”

One aspect worth noting is that we made an effort to include models with similar architectures that differ in terms of their use of static vs dynamic information. For example, the supervised dynamic CNN and the action-trained static CNN both have ResNet architectures with the same number of layers, and they were both trained with the HAA dataset, but the static net used individual frames as input, while the supervised dynamic net used optic flow. Similarly, for masked autoencoders, we included both image-based and video-based MAEs. The pattern of results we observed showed a better performance of the dynamic models (the supervised CNN and the video-based MAEs) compared to their static counterparts. Nonetheless, we agree that testing a large pool of models with a larger space of datasets will be important to characterize more accurately what are the key elements that drive differences in the models’ correspondence to neural responses.

We also added this passage to highlight that it will be interesting to investigate the anterior STS in future work:

“In addition, research will be needed to test the ability of MAEs to account for responses in other regions, particularly the anterior portions of the temporal lobe (including the anterior superior temporal sulcus), which have been implicated in high-level and social perceptions.”
>*Overall differences between static and dynamic stimuli are extremely small, with differences of about r = 0.05 (which corresponds to the dynamic model explaining just .25% (r^2=0.0025) more variance than the static model) between paired static and dynamic versions of models. This is fine (and interesting!) to report, but concluding that dynamic models are better predictors than static models seems too strong and premature.*

Thank you for raising this point. We agree that the differences between static and dynamic stimuli are relatively small. We added the following passage to the Discussion to address this:
“It needs to be noted that the overall magnitude of the differences between static and dynamic models correlation with brain responses are relatively small (approximately r=0.05).”

This observation needs to be considered in the context of the overall correlation obtained with static models. For example, the correlation between the ImageNet-trained supervised static model and different brain streams is also relatively low (r~0.12), as noted in the previous point - possibly due to the complexity of the quasi-naturalistic stimuli. Therefore, a difference of 0.05 after adding dynamic information represents more than a 30% increase in correlation, while seemingly small, is meaningful in this setting. To further establish the robustness of this increase, we have added statistical analyses across participants. A linear mixed-effects model reveals a significant main effect of model type (temporal models > spatial models). We have included this new statistical analysis in the revised manuscript.

>*While I liked the analysis looking for unique variance, it is difficult to interpret some of the results since we don’t know what the differences are due to for some of the pairs of models (could be different datasets or the architectures or the loss functions for example).*

Thank you for raising this question. We think that answering this question would require testing a larger space of models. In this study, we focused on predicting brain responses using two popular classes of models—CNNs and MAEs—that also differ in their representation of spatial and temporal information. Further analyses to disentangle the specific contributions of factors such as datasets, architectures, and loss functions to differences in brain predictability will be left for future research. We have clarified this in the manuscript by adding the following passage to the Limitation section:

“This study focused on a set of models selected to enable comparing the contribution of static and dynamic information and the impact of supervised and unsupervised learning objectives. The selection of models in this study includes only a subset of the existing models; future work will be needed to expand the set of models tested. In particular, future research should employ a larger space of models trained with a broader variety of datasets to disentangle the specific contributions of factors such as datasets, architectures, and loss functions to differences in brain predictability. ”

>*It would have been nice for the authors to provide more motivation or a framing for why they thought each of these models would be a good neural predictor, especially for the Masked Autoencoders, which is their novel contribution.*

Thank you for raising this important point. We have added the following to the Introduction to clarify the rationale for testing Masked Autoencoders:
“Previous work has indicated that models yielding more accurate categorization performance also typically offer a more accurate prediction of neural responses (Yamins et al. 2024). MAEs are remarkably effective at categorization, making them promising candidate models of neural responses. Additionally, thanks to their transformer architecture, MAEs can capture larger-scale spatial dependencies in images, and masked video autoencoders can capture temporal dependencies. Human observers are able to represent such dependencies as well, thus in principle, MAEs could account for unique variance in neural representations compared to convolutional neural networks (CNNs).”


>*The results could be improved by comparing the optic flow models to pymoten (motion energy extracted using a using a pyramid of spatio-temporal Gabor filters) to see if these models learn specific dynamic visual representations that benefit predicting neural activity in each of the three streams.*

Thank you for mentioning this. We agree that motion energy models are an excellent candidate to be included in the model pool. We intend to extend the work and study these models in our future research. We have added a note on this point to the Discussion section:

“Further comparing neural responses with models that compute motion energy will be an important direction for future research”.
>*All of the models predictivities are quite low (mostly pearson's r varies betwen .05 and .15 in Fig. 2). Is there any reason we should expect this particular modality of fMRI to be much more difficult to capture with ANNs (for example I think the cRSA scores in [1] are most similar to what you are measuring and these typically have values ~0.3).*

Thank you for this feedback and for pointing us to Conwell et al. (2022) for comparison. We agree that our model-to-brain predictivity values are lower than those typically observed in studies like theirs. However, we believe this discrepancy likely arises because our study differs in the prediction task. While Conwell et al. (2022) measure models’ ability to predict fMRI responses to static images of scenes (Natural Scenes Dataset), our work evaluates models’ predictivity for fMRI responses to quasi-naturalistic, dynamic stimuli—in this case, the full-length movie Forrest Gump. Given these differences in stimulus type, a lower model-to-brain correlation may be expected in our study. Naturalistic, dynamic stimuli introduce complex temporal dependencies that likely add noise to both the fMRI signal and the prediction task, making it more challenging for artificial neural network models to capture these responses as accurately as they would in studies using static stimuli.

We have added the following passage to the discussion section to highlight this point:

“The correlation values we obtained for our quasi-naturalistic stimuli (the movie Forrest Gump) are comparatively lower than those obtained in previous work for static images (Conwell et al. 2022), likely reflecting the greater complexity of the quasi-naturalistic stimuli.”


>*Many of the error bars are significantly overlapping in Fig 2. What gives? Also this is Std over subjects, but what about the variance within subjects (over CV splits)? In all, it would be good to provide more context for evaluating the statistical significance of some of the differences.*

Thank you for pointing this out. We agree that our analysis did not account for within-subject variance, which is indeed an important consideration for evaluating model predictivity. While we intend to leave the matter to future research, we will add more statistical analyses at the group level to evaluate brain response predictivity.
Specifically, we have added the following analyses:

“A mixed-effects model was conducted to examine the effect of model type (static vs. dynamic) on Fisher-transformed correlation values across subjects. Results indicated a significant effect of model type with models that process dynamic information having higher correlation (F(fixed effect df=1, residual df=502)=-13.487)”