Dynamics Based Neural Encoding with Inter-Intra Region Connectivity

## Abstract

Abstract Extensive literature has drawn comparisons between recordings of biological neurons in the brain and deep neural networks. This comparative analysis aims to advance and interpret deep neural networks and enhance our understanding of biological neural systems.
However, previous works did not consider the time aspect and how the encoding of video and dynamics in deep networks relate to the biological neural systems within a large-scale comparison. Towards this end, we propose the first large-scale study focused on comparing video understanding models with respect to the visual cortex recordings using video stimuli. The study encompasses more than two million regression fits, examining image vs. video understanding, convolutional vs. transformer-based and fully vs. self-supervised models. Our study resulted in both, insights to help better understand deep video understanding models and a novel neural encoding scheme to better encode biological neural systems. We provide key insights on how video understanding models predict visual cortex responses; showing video understanding better than image understanding models, convolutional models are better in the early-mid visual cortical regions than transformer based ones except for multiscale transformers and that two-stream models are better than single stream. Furthermore, we propose a novel neural encoding scheme that is built on top of the best performing video understanding models, while incorporating inter-intra region connectivity across the visual cortex. Our neural encoding leverages the encoded dynamics from video stimuli, through utilizing two-stream networks and multiscale transformers, while taking connectivity priors into consideration. Our results show that merging both intra and inter-region connectivity priors increases the encoding performance over each one of them standalone or no connectivity priors. It also shows the necessity for encoding dynamics to fully benefit from such connectivity priors.

## 1 Introduction

There has been a recent increase in studies that compare how deep neural networks process input stimuli to the processing that occurs in the brain whether in humans Zhou et al. (); Conwell et al. (); Schrimpf et al. (); Cichy et al. (; ), non-human primates, or rodents Conwell et al. (); Schrimpf et al. (). The benefits of these studies are two-fold. First, such a comparison can be used to interpret and have a better understanding of black-box deep neural networks and even provide inspirations on how to improve them. Second, it can provide a better understanding and encoding of biological neural systems. Towards achieving this, recent benchmarks have been released to improve the capabilities of machine learning models in neural encoding and comparing them to biological neural systems Schrimpf et al. (); Cichy et al. (; ); Gifford et al. (). The current established benchmark from The Mini-Algonauts Project 2021 Cichy et al. (), has provided neuro-imaging data for the brain responses from participants watching short video clips. A further extension of the aforementioned dataset Lahner et al. () provided exhaustive analysis with additional cortical regions for the ventral and dorsal streams. It also provided empirical evidence that temporal information was captured in fMRI recordings of the visual cortex for participants watching video stimuli, through studying the effect of frame shuffling and analyzing the first and last second of these videos. The former showed that shuffling lead to degraded performance confirming that there are dynamics captured in the recorded fMRI, while the later showed that these fMRI recordings encode temporally distinct early and late video snapshots. These developments and benchmarks can enable neuroscientists to study how the brain understands dynamics (e.g., motion), which is critical to study in neural encoding.

Machine learning models have been investigated in encoding such neuro-imaging data Lahner (); Zhou et al. (); Lahner et al. (). However, these studies either focused on a small-scale comparison of few video understanding models or conducted their study on single image based models, neglecting the dynamics aspect. More importantly, previous studies did not conduct a systematic analysis of different model families. To address this gap, we propose the first large-scale comparative study of video understanding in neural encoding that encompasses more than two million regression fits (this refers to source models, their layers, target models, regions, and voxels count). Our study takes various properties into consideration where we study image vs. video understanding, convolutional vs. transformer based, single-stream vs. two-stream and fully supervised vs. self supervised ones. Furthermore, we not only study these video understanding models to encode cortical regions recordings, we also study them in a simulated setup where the target is another neural network, inspired by the single image understanding study Han et al. (). Our results show that video understanding models are better than image understanding ones in modelling the human visual cortex recordings. Specifically, two-stream convolutional models and multiscale transformers were the best ones. Interestingly, we are the first to demonstrate the effect of multiscale processing in transformers that improves its ability to capture low level cues (e.g. oriented gradients) consequently improving its performance in the early regions.

Finally, we devise a novel neural encoding mechanism that takes into account encoded dynamics and connectivity priors. Voxel connectivity started to get explored in neural encoding in few recent works Mell et al. (); Xiao et al. (). However, previous works used this approach in a limited scope in which they studied the connectivity between pairs of regions without considering the connectivity across all regions at once. They also did not consider the combined effect of the local connectivity within the same region (intra-region connectivity) and the global connectivity across the different regions (inter-region connectivity).

In summary, our contributions are three fold:

- •
We showcase the first large-scale study of deep video understanding models on two datasets for the human visual cortex where the models include convolutional vs. transformer-based, single vs. two stream and fully vs. self-supervised. Our study is comprehensive with more than 35 models from various families and more than two million regression fits, unlike the recent work Lahner et al. () that showed only three models with limited types.
- •
We establish a simulated environment setup using image and video understanding models as the target and a real environment setup with the target model as the human visual cortex.
- •
We propose a novel fully integrated encoding model that takes into account intra and inter-region connectivity priors in the visual cortex with features extracted from pre-trained video understanding models. We also show that encoding dynamics is an important aspect to enable the full utilization of such connectivity priors.

## 2 Related work

Biological neural systems encoding.
Brain encoding is concerned with mapping the input stimuli to the neural activations in the brain. Learning this mapping has been heavily investigated in the literature Zhou et al. (); Conwell et al. (); Lahner (), where most of the approaches conduct a form of deep regression. The study of brain encoding has been advanced by the release of naturalistic neuroscience datasets and benchmarks, with text, audio, image or video stimuli. One of the well established benchmarks that studied how deep networks compare to biological neural systems is the Brain-Score benchmark and framework Schrimpf et al. () which relied on grayscale image stimuli. Another recent well established dataset and benchmark, The Algonauts project Cichy et al. (; ); Gifford et al. (), released datasets and challenges that focused on stimuli as natural objects images Cichy et al. (), action videos Cichy et al. (); Lahner et al. () and natural scenes Gifford et al. (). In these benchmarks, fMRI responses were recorded from different subjects and used to study how the human brain encodes these different kinds of stimuli. In this work, we focus specifically on recorded fMRI data for participants watching short video clips from Mini-Algonauts 2021 Cichy et al. (). Other benchmarks were released with video stimuli focus Popham et al. (); Lahner et al. (), including the extension of the aforementioned dataset and the most recent benchmark, BOLD moments Lahner et al. (). BOLD moments provided an exhaustive analysis and an improved dataset, as such we also evaluate on it as part of our study. Recent works have also investigated the ability of deep networks to regress on the brain responses for different stimuli Conwell et al. (); Zhou et al. (), where one approach Zhou et al. () focused on video stimuli. However, they mainly worked with single-image deep neural network architectures. In this work, we follow this approach closely, but we focus on studying video understanding models to draw insights on how the brain understands actions and models dynamics. While some works in neuroscience studied the time aspect  Zhuang et al. (); Nishimoto et al. (); Khosla et al. (); Nishimoto (); Lahner et al. (); Güçlü & Van Gerven (); Shi et al. (); Sinz et al. (); Huang et al. (), they did not focus on large-scale comparison. Our work focuses on the first study of state-of-the-art deep video understanding models from a neuroscience perspective with more than two million regression fits.

Voxel connectivity encoding models. The brain is an interconnected system with local correlations within one region and global correlations across regions  Genç et al. (); Li et al. (). Few recent works explored the potential of using cortical connectivity in neural encoding models  Mell et al. (); Xiao et al. (). In Mell et al. (), they proposed a model that used predefined source voxels as input to predict a target voxel and compared it to the vanilla stimulus-to-voxels prediction models as the standard neural encoding scheme. Nonetheless, these voxels-to-voxels models are not designed to take stimulus as input and define source voxels in an ad hoc manner. Our work is inspired by that direction, yet we propose a fully integrated model that learns a two-stage architecture, stimulus-to-voxels and voxels-to-voxels. More importantly, our approach takes into consideration all voxels from all visual cortex regions and learns the weighting mechanism, instead of relying on ad hoc non learnable mechanism to define source voxels. Another work Xiao et al. () proposed an encoding approach to improve the neural predictions of high-level visual areas using the predictions of low-level visual areas. On the other hand, our method enables learning from all voxels in the same region and other regions at once within a learnable scheme to leverage inter and intra-region connectivity priors. Additionally, our scheme allows for learning from multiple regions and from the same region in one shot, while previous works mainly used connectivity to one paired region. Finally, we are the first to demonstrate the impact of encoding dynamics on utilizing these connectivity priors to improve neural encoding performance.

## 3 Method

In this section, we describe our environment design for the real and the simulated experiments. Then, we discuss the candidate video understanding models and our proposed neural encoding scheme.

### 3.1 Environment design

In biological neural systems encoding, we aim to study how different stimuli map to the recorded brain responses. It is usually studied within the framework of aligning and comparing deep network architectures and biological neural systems. In this case, the biological neural system is considered the target model, and the candidate deep network architecture, that extracts features from the stimuli to be mapped to the brain responses through deep regression, is considered the source model. However, our first goal is to answer the question: “Can we perform system identification for the underlying dynamics encoding scheme?”. We use the dynamics encoding to refer to the model’s ability to learn from dynamic information provided in an input clip and encode it within the learned representations. To answer that, inspired by previous work Han et al. (), we use the features extracted from known architectures as the target, on which we apply our regression, instead of the brain responses as an upper bound. This allows us to identify the dynamics encoding scheme, which serves as a form of ground truth for comparing different models. However, unlike previous work that focused only on studying the architecture (i.e., convolutional vs. transformer-based), we go beyond that to study whether models process standalone images or learn dynamics from the input video (i.e. image vs. video understanding models).

The second question we aim to study: “How do deep video understanding models families compare to biological neural systems?”. Towards this, we study the identification across families of models when encoding the brain responses. Specifically, families are defined based on: (i) the input, whether models learned from single images or videos encouraging them to learn dynamics and motion, (ii) the supervision, whether they are trained in a fully-supervised framework for a certain downstream task or in a self-supervised manner using unlabeled data, and (iii) the architecture, whether the model architecture relies on local convolutional operations or transformer-based global operations. While previous works focused on the architecture aspect, we argue it is even more important to look into whether the model is learning dynamics (e.g., motion) or simply using static information from a single image. Moreover, it is important to understand the impact of the supervision signal used to train the model. Throughout all the experiments we utilize Mini-Algonauts 2021 dataset Cichy et al. (), in addition to providing additional results on the BOLD moments dataset Lahner et al. (). In the following, we describe the design details of both the simulated environment, where the target is yet another deep network representation, and the real environment, where the target is the visual cortex responses.

Simulated environment.
We select four target models from both the image/video understanding models and convolutional-/transformer-based models. Specifically, we use ResNet-50 He et al. (), I3D ResNet-50 Carreira & Zisserman (), ViT-B Dosovitskiy et al. () and MViT-B Fan et al. (). ResNet-50 and I3D ResNet-50 are convolutional-based models, while ViT-B and MViT-B are transformer-based models. On the other hand, I3D ResNet-50 and MViT-B are video understanding models, while ResNet-50 and ViT-B are single image understanding models. For each target, we use the other candidate models as source. The layers representations are extracted from each target model as detailed in Appendix A, with input videos from Mini-Algonauts. A dimensionality reduction is conducted on these representations for computational efficiency reasons.

**Table 1: List of the candidate models and their families and configurations that were used during their training. We list the backbone/s, the training datasets, and the configuration as clip length $\times$ sampling rate. For the training datasets we use ImageNet Deng et al. () (IN), Kinetics-400 Kay et al. () (K400), Charades Sigurdsson et al. () (Ch) and Something-something v2 Goyal et al. () (SSV2).**
| Input | Supervision | Architecture | Network (Backbone/s - Dataset/s - Config.) |
| --- | --- | --- | --- |
| Video | Fully-supervised | Convolutional | C2D (R50-K400-$8\times 8$) |
| CSN (R101-K400-$32\times 2$) |  |  |  |
| I3D (R50-K400-$8\times 8$) |  |  |  |
| R(2+1)D (R50-K400-$16\times 4$) |  |  |  |
| SlowFast (R50,101-K400,Ch,SSV2-$8\times 8$,$4\times 16$) |  |  |  |
| 3DResNet (R18,50-K400,Ch,SSV2-$8\times 8$,$4\times 16$) |  |  |  |
|  | X3D (XS,S,M,L-K400-Matched Sampling Rate) |  |  |
|  | Transformers | MViT (B-K400-$16\times 4$,$32\times 3$) |  |
|  | TimeSformer (B-K400,SSV2-$8\times 8$) |  |  |
|  |  |  | OmniMAE finetuned (B-SSV2-$8\times 8$) |
|  | Self-supervised | Transformers | stMAE (L-K400-$8\times 8$) |
|  | OmniMAE (B,L-IN/SSV2-$8\times 8$) |  |  |
| Image | Fully-supervised | Convolutional | ResNet (R152,101,50,34,18-IN-$8\times 8$) |
| Transformers | ViT (B16,32,L16,32-IN-$8\times 8$) |  |  |
| Self-supervised | Transformers | DINO (B-IN-$8\times 8$) |  |
| MAE (B-IN-$8\times 8$) |  |  |  |

Real environment.
In the real set of experiments, our target is the brain responses. In this case, we use the public fMRI datasets from Mini-Algonauts and BOLD moments Cichy et al. (); Lahner et al. (). We mainly use the training set, and perform cross-validation over four folds throughout all our experiments. The dataset provides fMRI recordings of ten subjects who watched short video clips of three seconds average duration. Each video and voxel in the brain was represented by a single activation value. In Mini-Algonauts 2021 dataset, we use the brain responses from nine regions of interest of the visual cortex, these are across two levels: (i) early and mid-level visual cortex (V1, V2, V3, and V4), and (ii) high-level visual cortex (EBA, FFA, STS, LOC, and PPA). The early and mid-level visual cortex regions are concerned with lower-level features such as orientations and frequencies, while the high-level ones are concerned with semantics in terms of objects, scenes, bodies, and faces. We also conduct the same study on BOLD moments with a comprehensive 46 cortical regions that is described in the appendix B.1. The datasets we use is provided at TR one second and has shown sensitivity to the temporal ordering of information in the video stimuli Lahner et al. (). This motivates our choice for the two datasets when studying video understanding models and their relation to the visual cortex when encoding dynamics.

### 3.2 Candidate models

Here we describe the candidate models that are used in our experiments. We choose to run our experiments on more than 35 source models, listed in Table 1, with their model family and configurations. Video understanding models include C2D Li et al. (), CSN Tran et al. (), I3D Carreira & Zisserman (), R(2+1)D Tran et al. (), SlowFast, the Slow branch (3D ResNet-50) Feichtenhofer et al. (), X3D Feichtenhofer (), MViT Fan et al. () and TimeSformer Bertasius et al. (). Self-supervised video understanding models, stMAE Feichtenhofer et al. () and OmniMAE Girdhar et al. () are used as well. We mainly focus on the models that showed state of the art performance in action recognition. Single image understanding models include ResNets He et al. (), ViTs Dosovitskiy et al. () and the self-supervised DINO Caron et al. () and MAE He et al. (). Families of models are categorized based on the input, supervision and architecture type as discussed earlier. We detail the number of layers and which layers are sampled for each deep network used in Appendix A.

Figure: Figure 1: Architecture of our dynamics based neural encoding with inter-intra region connectivity priors. Our fully integrated model learns the connectivity from all regions voxels (inter-region) and all voxels in the same region (intra-region). We only show one target region, V1, as an illustration where we use the same mechanism across all regions. Note the arrows thickness, in our visual cortex regions illustration, indicate the degree of connectivity corresponding to the computed in Fig. 5(c).
Refer to caption: x1.png

### 3.3 Encoding technique

Inspired by the recent work Zhou et al. (), we use a layer-weighted region of interest encoding that takes the hierarchical nature of deep networks into consideration. Initially, we sample the frames from a video to obtain the input clip. For image understanding models we extract features per frame, while for video ones we extract features from the entire clip at once. Then, we pre-process the input features from the different layers of a candidate model through averaging the features on the temporal dimension. This is followed by performing sparse random projection Li et al. () for dimensionality reduction and computational efficiency reasons. Assume input features for layer, $l$, after dimensionality reduction as, $X_{l}\in\mathbb{R}^{C\times 1}$, with $C$ features. We learn the weights of one fully connected layer to provide the predictions of the voxels of one region of interest in the visual cortex as, $\hat{Y}_{l}=W_{l}X_{l}$. Where $W_{l}\in\mathbb{R}^{N\times C}$, $\hat{Y}\in\mathbb{R}^{N\times 1}$ and $N$ is the number of voxels in the region of interest. Instead of a simple ridge regression, we learn a weighted sum of the predictions of all layers and use the following loss to train our regression model,

$$ $\mathcal{L}=\lVert Y-\sum\limits_{l=1}^{L}\omega_{l}\hat{Y}_{l}\rVert^{2}_{2}+ \beta_{1}\sum\limits_{l=1}^{L}\lVert W_{l}\rVert_{2}+\beta_{2}\lVert\omega \rVert_{1},$ (1) $$

where $\omega_{l}$ is a learnable scalar weight for layer, $l$, and, $\omega$, is the vector of weights. Each $\omega_{l}$, controls the contribution of layer, $l$, to the final regression of the region of interest, and $\beta_{1},\beta_{2}$ are hyper-parameters of the regularization. We use L1 regularization for the layer weights to enforce sparsity. This encoding scheme avoids unnecessary assumptions that there is a one-to-one alignment between layers and visual brain regions of interest. Accordingly, this encoding scheme allows for more complex interactions among the layers and the brain regions of interest.

### 3.4 Inter-intra region connectivity priors

In this section, we present a novel encoding scheme on top of the best-performing video understanding models by fully integrating the neural encoding with inter- and intra-region voxel connectivity priors. An overview of our architecture is shown in Fig. 1. The input video stimuli goes through the source video understanding model to extract multiple layers features as described in Sec. 3.3, followed by the connectivity module which takes the concatenated voxels of the nine visual regions as input. This module consists of four layers: two fully-connected coupled with L2 regularization and two dropout ones. Finally, the output of the model is the predicted voxels of a single visual region. We train our model in a two-stage fashion, where we train the standard neural encoding scheme without connectivity, followed by training the connectivity module. In the training phase, the main target is to learn the connectivity between the voxels of all the regions and the target region including intra-connectivity between the voxels of the target region itself and the inter-connectivity between voxels of the target region and the other visual regions. Accordingly, in the training phase, the input to the model is the groundtruth voxel activations. However, in the inference phase, the input to the model is the predicted voxel activations.

## 4 Experimental results

### 4.1 Implementation details

In this subsection, we describe our experiment design and implementation details. The input clips to all models are constructed based on a sampling rate that corresponds to the sampling rate used during its training for video understanding models. As for image understanding models we use the default sampling rate of eight. The clip length is computed based on the sampling rate and the input video length and changes according to the video length.

Before training the regressor, a hyperparameter tuning is conducted using two-fold cross-validation on the training set of the first subject, following previous work Zhou et al. (). The two main hyper-parameters we tune are the $\beta_{1},\beta_{2}$, using grid search $\beta_{1}\in\{0.1,1,10\}$ and $\beta_{2}\in\{1,10,100\}$. Moreover, an early-stopping strategy is employed through the hyperparameter tuning and training phases. The main metric used throughout the experiments is the average Pearson’s correlation coefficient across all voxels within a specific region in the brain. All results are averaged over the subjects. We conduct experiments on four folds and report the average and standard deviation. We report the statistical significance across families of models using Welch’s t-test. The real setup, in which the human visual cortex is the target, is conducted on the Mini-Algonouts 2021 dataset Cichy et al. (), additional results on BOLD moments dataset is provided in Appendix B.1.
In each fold of the four folds we used in our experiments, the 1,000 videos are split into training and testing sets as 90% and 10%, respectively. In the simulated setup, we follow the same setup for the real environment. All the previous models are trained on A6000 GPU with training span of one day average per regression model.

Figure: (a)
Refer to caption: extracted/5923110/images/image_video_simulated_MViT_B_16x4_neurips.png

### 4.2 Can we perform system identification with respect to the dynamics encoding?

We first investigate the question of whether system identification of single image vs. video understanding models is possible. Figure 2 shows that for MViT and ViT target models, the correct family of models is better able to represent each. Specifically, we observe higher regression scores of the video understanding models for MViT and the single image one for ViT.
Note that we include video understanding models that are trained on three different datasets which are Kinetics, Charades, and Something-Something v2 to ensure the results are not dependent on a certain training dataset.

When inspecting ResNet-50 as a target, we observe the mean of image understanding models is higher, but is only statistically significant in three layers. When looking at I3D target model, we notice the mean of video understanding models is higher and is statistically significant except in early layers. In summary, we have demonstrated that system identification can be attained to a certain level with most of the layers showing statistical significance. Additional simulated experiments are provided in Appendix B.2. In the following section we investigate the same but for the unknown biological neural system.

### 4.3 How do deep video understanding models families compare to the human visual cortex?

In this section, we focus on the real experiments, where the target model is the biological neural system, to understand the underlying mechanisms of the visual cortex. We conduct three comparisons; single image vs. video understanding families of models, convolutional-based vs. transformer-based models, and fully-supervised vs. self-supervised models. Figure 3(a) clearly demonstrates that across most brain regions, video understanding models have better capability to model the visual cortex responses than single image architectures. We believe the reason behind this is that the brain is encoding the dynamics occurring in a video similar to the encoded dynamics in video understanding models more than what occurs in single image understanding models. Figure 3(b) shows the comparison between transformer-based and convolutional-based models. It shows that convolutional models have higher regression scores across early-mid regions in the visual cortex with relatively high statistical significance. This difference decrease as we go to higher level regions until it becomes insignificant. Interestingly, it has been noted in recent works that transformers lack the ability to capture high-frequency components Bai et al. (). On the other hand, early layers in convolutional models tend to capture high-frequency components by detecting oriented gradients. This also might relate to empirical results that demonstrated vision transformers with shallow convolutional stem (i.e., three convolutional layers) perform better than the ones that directly take patches as input Xiao et al. (). As such, brain modelling in the early and mid regions of the visual cortex relates better to convolutional-based models, since they better capture high-frequency components. Nonetheless, we notice the best model in the transformer based family, MViT, tends to behave similar to convolutional ones in early-mid regions unlike other transformer based architectures. Additional results in Appendix B.2 are provided to confirm the previous insight. Surprisingly, Fig. 3(c) shows that fully-supervised models are better able to predict most of the regions than self-supervised models. Appendix B.1 provides the results of the previous study conducted on BOLD moments dataset across 46 regions, which confirms the consistency of our main findings across different datasets. Moreover, Appendix B.3 shows the model subfamilies while focusing only on the video understanding models and excluding the single image models, in addition to comparing convolutional vs. transformer based models that are trained with full supervision only.

Figure: (a)
Refer to caption: extracted/5923110/images/image_video_brain_neurips_update.png

Figure: (a)
Refer to caption: extracted/5923110/images/single_two_stream_finegrained_stats_2.jpg

### 4.4 Fine-grained analysis

In this section, we conduct a fine-grained analysis that goes beyond families of models. We start with studying two stream vs. single stream architectures across three video understanding datasets. Figure 4(a) shows that the two stream architectures have better ability to model the visual cortex than single stream ones in the low level regions and are either better or on-par in the high level regions. We then discuss the self-supervised learning results that showed worse regression scores in comparison to full supervision. Towards this end, we investigate OmniMAE variants (i.e., self supervised and finetuned) and TimeSformer. Figure 4(b) shows that fully supervised models give better scores than the self-supervised ones across the nine regions. We leave the reasons behind this result as an open question for future research on why fully supervised models tend to align better with the cortical responses than self supervised ones. Furthermore, we investigate which models are better at predicting the visual cortex responses. Figure 4(c) shows that both SlowFast, a two-stream architecture, and MViT, a multiscale vision transformer, are the best in modelling the visual cortex across the nine regions. SlowFast which is a convolutional approach is comparable to MViT with difference that is statistically not significant across all regions.
Moreover, we see self-supervised models (i.e., stMAE and OmniMAE) lagging behind fully supervised ones. Additional analysis is provided in Appendix B.4 and B.5.

### 4.5 Inter-intra region connectivity

In this section, we show the results of our improved neural encoding mechanism that builds upon the best video understanding models (MViT-B and SlowFast) while incorporating intra- and inter-region connectivity.
Figure 5(a) shows the statistically significant enhancements in prediction accuracy in both MViT-B and SlowFast models after incorporating voxel-connectivity to their predictions. These results show that integrating the intra- and inter-connectivity across the visual regions is an important component for a better neural encoding. As an ablation study, we examine the performance enhancement in MViT-B in the case of intra-connectivity or inter-connectivity separately. Figure 5(b) shows that the full-connectivity (i.e., combining both) is either superior or on-par with intra-connectivity or inter-connectivity standalone.
To better understand the directional connectivity between the regions, we analyze the average learned weights of each region as contributors to the MViT-B accuracy enhancement of each target visual region as shown in Figure 5(c). It shows the following: i) the effect of one region on another is not symmetric but directional, ii) early-mid regions (V1, V2, V3, and V4) are the highest contributors to the accuracy enhancement of other early-mid regions, and the same for late-regions (LOC, EBA, FFA, STS, and PPA), iii) V4 is contributing to both early-mid and late regions, and iv) the contributions of late regions on early regions (V1, V2) are stronger than contribution of early regions on late regions which could be attributed to the top-down influence of feedback-pathways in the visual cortex Gilbert & Li ().

Finally, we perform an ablation of the improvement from intra- and inter-region connectivity prior when using an image understanding model vs. a video understanding one. Figure 6(a) illustrates the effect of inter-intra region connectivity priors on a single image understanding model (ResNet-50), where we show that it improves over the baseline with no connectivity across most of the regions with statistical significance. More importantly, we study the gain from the connectivity priors with that single image understanding model (ResNet-50) vs. a video understanding model that learns dynamics (MViT). Figure 6(b) shows that the gain from MViT is on average higher than the ResNet-50 across all the regions. When looking upon the statistical significance of these results, we notice almost half of the regions show a significant gain from encoding dynamics over the single image understanding baseline. Hence, we show in these results that the dynamics encoding in video understanding models reinforces the connectivity priors. We refer to additional ablations that compares our learned connectivity to a random or an identity one in Appendix B.6, again confirming on the benefit of our learned connectivity.

Figure: (a)
Refer to caption: extracted/5923110/images/connectivity.png

Figure: (a)
Refer to caption: extracted/5923110/images/resnet50_connectivity.png

### 4.6 Summary

In summary: (i) Simulated results show that system identification between image and video understanding models is attainable to a certain level. (ii) We show that video understanding models are better in regressing the visual cortex responses than image ones. (iii) We show that convolutional models predict better the early-mid regions than transformer based ones. (iv) We show that MViT, with its multiscale processing, tends to perform similar to convolutional models in early-mid regions. (v) We show that two-stream models perform better than single stream. (vi) We show that models trained with full supervision surpass the ones with self supervision. (vii) Finally, we demonstrate a better neural encoding scheme that utilizes both encoded dynamics and inter-intra region connectivity. We refer to Appendix C and D for a discussion on the limitations and broader impact.

## 5 Conclusion

This paper has provided a large-scale study of video understanding models from a neuroscience perspective. We have shown the feasibility of system identification for single image vs. video understanding models. Moreover, we have shown that modelling dynamics should be considered when comparing biological neural system responses to deep networks and provided insights on different families of models. Finally, we showcased the interplay of dynamics modelling and inter-intra region connectivity in neural encoding.

## 6 Acknowledgements

We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and Digital Research Alliance of Canada (RAC) in conducting this research work.

## Appendix A Candidate models details

In this section, we provide details on the layers extracted from each of our candidate models in Table 2.
Note that for transformer-based architectures instead of using all layers to be selected we rather group layers into blocks of four layers for efficiency reasons and we noticed it gave better results than learning the regression with all input layers at once. Also note that OmniMAE finetuned is trained for the action recognition task on SSV2 dataset Goyal et al. ().

**Table 2: Detailed description per architecture of the used layers.**
| Architecture | # Layers | Sampled Layers |
| --- | --- | --- |
| CSN, R(2+1)D, 3DResNet | 6 | 5 convolutional blocks and last fully connected layer |
| C2D, I3D | 7 | 5 convolutional blocks, max pooling and last fully connected layer |
| SlowFast | 12 | 5 convolutional blocks for each branch (slow & fast) and the last 2 layers combined |
| X3D | 6 | 5 convolutional blocks and last fully connected layer |
| ViT, TimeSformer | 4 | Grouped 3 blocks, each block 4 layers and last fully connected layer |
| Dino - B | 4 | 3 Grouped blocks (4 layers) and CLS token |
| ResNet | 7 | 5 convolutional blocks, average pooling and last fully connected layer |
| MViT | 5 | Grouped 4 blocks (4 layers) and last fully connected layer |
| OmniMAE, MAE | 3 | Grouped 3 blocks (4 layers) |
| stMAE | 6 | Grouped 6 blocks (4 layers) |

## Appendix B Additional results

### B.1 BOLD Moments Dataset (BMD) results

In this section, we evaluate on the newly released BOLD Moments Dataset (BMD) Lahner et al. (), with fMRI recording of ten subjects watching 1,000 training video stimuli. The dataset provides a preprocessed version (named “Version B”) with additional flexibility in the region of interest (ROIs) format. In “Version B” of BMD, 46 regions are defined including right/left hemispheres and ventral/dorsal streams. The defined ROIs include: early visual regions (V1v, V1d, V2v, V2d, V3v, V3d, hV4), a body-selective region (EBA), an object-selective region (LOC), face-selective regions (FFA, OFA, STS), scene-selective regions (PPA, RSC, TOS), additional ROIs (V3ab, IPS0, IPS1-2-3, 7AL, BA2, PFt, and PFop), and finally the motion selective MT ROI. All these ROIs were defined in both right and left hemispheres forming in total 46 ROIs. To further show the robustness of our results and insights, we evaluate all our models using this preprocessed version across the 46 ROIs using cross-validation on 4 folds as shown in Fig. 7. It confirms the consistency of the results previously shown in Fig. 3. Figure 7(a) confirms the statistically significant superiority of video understanding models compared to single-image models in the majority of cortical regions. Figure 7(b) confirms that convolutional-based models are better than transformer-based models in representing early-mid visual cortex regions. Finally, Fig. 7(c) confirms that fully-supervised models perform better than self-supervised ones in predicting activity of most of the visual cortex regions with statistical significance.

Figure: (a)
Refer to caption: extracted/5923110/images/video_image_47rois_fixed.png

### B.2 Additional simulated experimental results

Although our focus in the system identification is on the ability to differentiate image vs. video understanding models, we provide additional results for other families of models. In Fig. 8, we show the simulated results comparing convolutional vs. transformer based models for four target models MViT-B 16x4, ViT-B, ResNet-50, and I3D ResNet-50, respectively. For the target model I3D, the figure clearly shows that convolutional models are better predictors than transformer-based ones with statistical significance across blocks except the last one. The target model ResNet50 shows that convolutional models are significantly better than transformer-based ones in the early blocks, but the differences are not significant in the late blocks. For ViT-B, the transformer-based models are significantly better than the convolutional models in all the blocks except for the first block which has non-significant difference between the two families. This result is consistent with the results presented in Han et al. (). The target model MViT-B shows a surprising result, that is confirming with previous findings in the real experiments as well as detailed in Section B.4, where convolutional models are better in regressing the multiscale variant than transformer-based ones with significant differences in the first two blocks. This might be related to how the multiscale ViT tends to act similar to the convolutional models when predicting early-mid regions of the visual cortex.

In the MViT target model results showed in Fig. 2(a) and Fig. 8(a), we did not include the MViT-32x3 variant for fair comparison between model families given that MViT-32x3 share the same architecture as MViT-16x4. However, for further investigation and understanding of MViT target model results, we re-assessed the results after adding MViT-32x3 as a source model. Figure 9 shows the results of MViT-16x4 target model after including MViT-32x3 source model. Figure 9(a) shows consistent results as shown in Figure 2(a) in which video understanding models are significantly better than image understanding ones. Figure 9(b) also shows consistent results to Figure 8(a) in which convolutional models are better than transformer-based models as predictors of MViT target model but with insignificant results, after adding MViT-32x3 source model to the transformers family, as anticipated.

Figure: (a)
Refer to caption: extracted/5923110/images/cnn_vit_simulated_MViT_B_16x4_neurips.png

Figure: (a)
Refer to caption: extracted/5923110/images/image_video_simulated_MViT_B_16x4_mvit_32x3_neurips.png

### B.3 Additional real experimental results

We add a study of the model subfamilies in terms of (a) convolutional vs. transformer based models and (b) fully vs. self supervised models, but focused on video understanding models only excluding models trained with single images. As shown in Fig. 10(a) it shows consistently that convolutional based models perform better in early layers as found earlier, where the first three regions show statistically significant results. Moreover, Fig. 10(b) shows models trained with self-supervision tend to be worse than fully supervised ones. However, note that these results are using a relatively small number of self-supervised learning video understanding models. Thus, we leave it for future work to expand on this further. Finally, we study the convolutional vs. transformer based models with focus on fully supervised ones excluding self supervision. Figure 10(c) shows the consistency of our results where the convolutional models in the early-mid regions surpass the transformer based ones.

Figure: (a)
Refer to caption: extracted/5923110/images/cnn_vit_video_brain_rebuttal.png

Figure: Figure 11: Layers contribution of four models for the nine regions of interest in the visual cortex.
Refer to caption: extracted/5923110/images/i3d_r50_layers_contribution.png

Figure: Figure 12: Fine-grained analysis comparing instances of Convolutional (i.e., Slow) vs. Transformer (i.e., TimeSformer) based models with different training datasets (SSv2 and K400).
Refer to caption: extracted/5923110/images/conv_vit_finegrained_stats_2.png

### B.4 Additional fine-grained analysis

In this section, we provide additional fine-grained analysis for both real and simulated environments. First, we demonstrate the layer contribution of the studied video understanding models across different regions to have a better understanding of the hierarchical nature of biological neural systems. We show the layer contribution to the regression according to the regularized encoding technique in Section 3.3. Figure 11 shows the layer contribution for three video understanding models; MViT, I3D and SlowFast (Slow and Fast branches). It clearly demonstrates that across the four early layers in these networks, there is a higher contribution to the early and mid-level regions (V1-4), and the opposite occurs as we go deeper.

Second, in Fig. 12 we compare instances of convolutional (i.e., Slow) and transformer (i.e., TimeSformer) based models. It clearly shows that especially across the early-mid regions in the brain (i.e., V1-4), the convolutional models tend to provide better regression scores. This confirms our previous insight that convolutional models are better at capturing orientation and frequencies because of their early local connections. In later regions in the brain (i.e., LOC, EBA, FFA, STS, PPA) convolutional model (i.e., Slow) tends to act on-par or less than the transformer-based model (i.e., TimeSformer).

Third, we analyze the best and worst across the models trained on single images vs. videos across both the simulated and real experiments. For the simulated experiments in Fig. 2, Table 3,4,5 and  6 show the worst (Min) and best (Max) predictors from each category (Image vs. Video) for target models MViT-B 16x4, ViT-B-16, ResNet-50, and I3D ResNet-50, respectively. For I3D, ResNet-50 and ViT, the results convey a simpler message that the best regressors are built with features extracted from architectures that exhibit higher similarity to the target model (i.e., convolutional/transformer) from both Video and Image understanding families. However, for MViT looking at Table 3 we see the best in the Image understanding models family is ResNet-50 which aligns with our previous insight that MViT tends to behave similar to convolutional models especially in the early layers.

Additionally, we show the best and worst predictors in the real experiments in Fig. 3 with the visual cortex regions as the target in Table 7. It shows both SlowFast and MViT are the best predictors from the video understanding family across the brain regions, with the SlowFast better at early regions similar to our previous findings as a two-stream and convolutional variant. It also shows ResNets to be the best from the image understanding family.

Fourth, we conduct an experiment comparing a randomly initialized network with respect to a trained model. The motivation behind this experiment is to confirm that our neural encoding is indeed capturing the learned representations in deep networks beyond random weights. Figure 13(a) shows SlowFast trained weights vs. random weights to confirm that the random weights provide worse correlation coefficient than the trained ones. This is indicative that our results are not only emergent from the architecture, but rather the learned representations that depend on architecture, dynamics encoding, training dataset and training signal (full/self supervision). Finally, we conduct experiments where we show the centered kernal alignment scores following previous works Han et al. () in Fig. 13(b). It shows that SlowFast, a video understanding model, surpasses ResNet-50, a single image understanding one, even when using another metric beyond the correlation coefficient of the regressed output.

**Table 3: Fine-grained analysis of simulated experiments results in Fig. 2(a) with target model MViT-B 16x4 showing the worst (Min) and best (Max) source model from image & video understanding models. B1-5: Blocks in the target model.**
|  | Min (Video) | Max (Video) | Min (Image) | Max (Image) |
| --- | --- | --- | --- | --- |
| B1 | OmniMAE-B-Fine | MViT-B-32x3 | ResNet-34 | ResNet-50 |
| B2 | OmniMAE-B-Fine | MViT-B-32x3 | ViT-B-16 | ResNet-152 |
| B3 | Slow-R50-SSv2 | MViT-B-32x3 | ViT-B-16 | ResNet-101 |
| B4 | Slow-R50-SSv2 | MViT-B-32x3 | ViT-B-16 | ResNet-34 |
| B5 | Slow-R50-SSv2 | MViT-B-32x3 | ViT-B-16 | ResNet-101 |

**Table 4: Fine-grained analysis of simulated experiments results in Fig. 2(b) with target model ViT-B-16 showing the worst (Min) and best (Max) source model from image & video understanding models. B1-4: Blocks in the target model.**
|  | Min (Video) | Max (Video) | Min (Image) | Max (Image) |
| --- | --- | --- | --- | --- |
| B1 | MViT-B-32x3 | OmniMAE-B-Fine | ResNet-101 | ViT-L-16 |
| B2 | SlowFast-R50-Char | OmniMAE-B-Fine | ResNet-152 | ViT-B-32 |
| B3 | SlowFast-R50-Char | OmniMAE-B-Fine | ResNet-34 | ViT-L-16 |
| B4 | Slow-R50-SSv2 | TimeSformer-K400 | ResNet-18 | ViT-B-32 |

**Table 5: Fine-grained analysis of simulated experiments results in Fig. 2(c) with target model ResNet-50 showing the worst (Min) and best (Max) source model from image & video understanding models. B1-7: Blocks in the target model.**
|  | Min (Video) | Max (Video) | Min (Image) | Max (Image) |
| --- | --- | --- | --- | --- |
| B1 | MViT-B-32x3 | SlowFast-R50-SSv2 | ViT-B-16 | ResNet-34 |
| B2 | MViT-B-16x4 | C2D | ViT-B-16 | ResNet-152 |
| B3 | MViT-B-16x4 | C2D | ViT-B-16 | ResNet-18 |
| B4 | SlowFast-R50-Char | OmniMAE-B-Fine | ViT-B-16 | ResNet-34 |
| B5 | SlowFast-R50-SSv2 | TimeSformer-K400 | ViT-B-16 | ResNet-101 |
| B6 | Slow-R50-SSv2 | TimeSformer-K400 | ViT-B-16 | ResNet-101 |
| B7 | Slow-R50-SSv2 | TimeSformer-K400 | ViT-B-16 | ResNet-101 |

**Table 6: Fine-grained analysis of simulated experiments results in Fig. 2(d) with target model I3D ResNet-50 showing the worst (Min) and best (Max) source model from image & video understanding models. B1-7: Blocks in the target model.**
|  | Min (Video) | Max (Video) | Min (Image) | Max (Image) |
| --- | --- | --- | --- | --- |
| B1 | MViT-B-32x3 | CSN | ViT-B-16 | ResNet-34 |
| B2 | MViT-B-16x4 | SlowFast-R101 | ViT-B-16 | ResNet-18 |
| B3 | MViT-B-32x3 | C2D | ViT-B-16 | ResNet-18 |
| B4 | MViT-B-32x3 | C2D | ViT-L-32 | ResNet-18 |
| B5 | OmniMAE-B-Fine | X3D-L | ViT-B-16 | ResNet-50 |
| B6 | SlowFast-R50-SSv2 | C2D | ViT-B-16 | ResNet-101 |
| B7 | Slow-R50-SSv2 | SlowFast-R50-K400 | ViT-B-16 | ResNet-101 |

**Table 7: Fine-grained analysis of real experiments results in Fig. 3(a) with target model the visual cortex regions showing the worst (Min) and best (Max) source model from image & video understanding models.**
|  | Min (Video) | Max (Video) | Min (Image) | Max (Image) |
| --- | --- | --- | --- | --- |
| V1 | stMAE | SlowFast-R50-8x8-Char | MAE | ResNet-18 |
| V2 | OmniMAE-B-Pre | SlowFast-R50-8x8-Char | MAE | ResNet-50 |
| V3 | OmniMAE-B-Pre | SlowFast-R50-8x8-K400 | MAE | ResNet-50 |
| V4 | OmniMAE-B-Pre | SlowFast-R101 | ViT-L-32 | ResNet-50 |
| LOC | OmniMAE-B-Pre | MViT-B-32x3 | ViT-B-16 | ResNet-101 |
| EBA | OmniMAE-B-Pre | SlowFast-R101 | ViT-B-16 | ResNet-101 |
| FFA | OmniMAE-B-Pre | SlowFast-R50-8x8-Char | ViT-B-16 | ResNet-101 |
| STS | OmniMAE-B-Pre | MViT-B-32x3 | ViT-B-16 | ResNet-101 |
| PPA | OmniMAE-B-Pre | TimeSformer-SSv2 | ViT-B-16 | ResNet-50 |

Figure: (a)

### B.5 Statistical significance results

Tables 8,9(a) shows the pairs of models from Fig. 4(a) (single-stream vs. two-stream) and Fig. 4(b) (fully-supervised vs. self-supervised) that exhibited a statistically significant result compared to each other. Table 8 confirms that two-stream models surpass the single-stream counterpart, across eight of the nine visual cortex regions, with a statistically significant result. The table specifically shows that the superiority of the two-stream models is independent of the training dataset. In five of nine brain regions, the two-stream models were superior compared to single-stream models at two different model versions that were matched based on their training dataset (K400 and Charades). On the other hand, Table 9(a) confirms that fully-supervised (i.e., OmniMAE Fine and TimeSformer) surpass the self-supervised counterpart (i.e., OmniMAE Pre) across all the visual cortex regions with a statistically significant result when the three models share the same architecture base (ViT-B) and training dataset (SSv2). These results, in addition to Fig. 4(b), show that TimeSformer (trained solely using full-supervision) achieved the highest regression scores followed by OmniMAE Fine (trained using both self- and full-supervision) and finally OmniMAE Pre (trained solely using self-supervision). It demonstrates that full-supervision better reflects the visual cortex responses. Table 9(b) shows the significance results between video understanding pairs of models. As shown in Table 9(b) and Fig. 4(c), SlowFast is statistically better than I3D in seven of the nine brain regions, I3D is statistically better than stMAE in five brain regions including four early regions of the visual cortex (V1 to V4), while SlowFast is not statistically different than MViT in any of the regions.

### B.6 Additional results on inter-intra region connectivity

In this section, we provide additional ablations on the connectivity module. We compare our learned connectivity to a simple random or an identity connectivity in Tables 10 and 11. In the random connectivity model, we assign random weights (using Xavier initialization) to the fully-connected layers in the connectivity model. On the other hand, in the identity connectivity model, we assign identity weights (ones) to the fully-connected layers. The connectivity model with identity weights represents a model where each region is learning from all the regions equally. The results clearly show that our learned connectivity surpasses these lower baselines of random or identity initialized connectivity.

### B.7 Comparison to state-of-the-art methods on Algonauts benchmark

This work focuses on studying video understanding models from a neuroscience perspective through a large-scale comparison of state-of-the-art deep video understanding models to the visual cortex recordings. Additionally, we propose a novel encoding approach using inter-intra region connectivity on the top of pre-trained deep neural network models to predict visual cortex voxel activity. To study the effect of our novel encoding approach, we compared the accuracy achieved by our best model (MViT-B with connectivity priors) to the state-of-the-art (SOA) accuracies achieved per region  Zhou et al. (). In  Zhou et al. (), they similarly built their encoding models on pre-trained deep learning models using the layer-weighting approach and their best model varied per brain region. Accordingly, we compare our best model results with their best model per region as shown in figure 14. It shows that our novel encoding approach achieved on average higher results across all the regions except for PPA in which it is on par. This comparison shows the accuracy enhancement achieved compared to the state of the art by incorporating connectivity priors to features extracted from pre-trained deep learning models. Note that our approach has a stronger form of cross validation as we take the mean of different subjects and we compute statistics across four folds from different videos. As such, our comparison to the baseline with no connectivity that follows Zhou et al. () in Fig. 5(a), better reflects our improvement gains. In future work, we will further explore the effect of integrating connectivity priors with end-to-end fine-tuning of deep learning models and ensemble approaches.

**Table 8: Statistical Significant of Slow vs. SlowFast (Fig. 4(a), showing the pairs of models (.,.) that exhibited statistical significance. K400: Kinetics 400, Char: Charades that were used as training datasets.**
|  | Stat. Sig. |
| --- | --- |
| V1 | SlowFast-Char, Slow-Char |
| V2 | SlowFast-K400, Slow-K400<br>SlowFast-Char, Slow-Char |
| V3 | SlowFast-K400, Slow-K400<br>SlowFast-Char, Slow-Char |
| V4 | SlowFast-K400, Slow-K400 |
| LOC | SlowFast-K400, Slow-K400<br>SlowFast-Char, Slow-Char |
| EBA | SlowFast-K400, Slow-K400<br>SlowFast-Char, Slow-Char |
| FFA | SlowFast-K400, Slow-K400<br>SlowFast-Char, Slow-Char |
| STS | SlowFast-K400, Slow-K400 |

**Table 9: (a) Statistical Significant of Self-supervised vs. Fully Supervised (Fig. 4(b)), showing the pairs of models (.,.) that exhibited statistical significance. (b) Statistical Significant of video understanding models (Fig. 4(c)), showing pairs of models (.,.) that exhibited statistical significance.**
|  | Stat. Sig. |
| --- | --- |
| V1 | OmniMAE Pre, OmniMAE Fine<br>OmniMAE Pre, TimeSformer |
| V2 | OmniMAE Pre, OmniMAE Fine<br>OmniMAE Pre, TimeSformer |
| V3 | OmniMAE Pre, OmniMAE Fine<br>OmniMAE Pre, TimeSformer |
| V4 | OmniMAE Pre, OmniMAE Fine<br>OmniMAE Pre, TimeSformer |
| LOC | OmniMAE Pre, TimeSformer |
| EBA | OmniMAE Pre, TimeSformer |
| FFA | OmniMAE Pre, TimeSformer |
| STS | OmniMAE Pre, TimeSformer |
| PPA | OmniMAE Pre, OmniMAE Fine<br>OmniMAE Pre, TimeSformer |

**Table 10: MViT learnable connectivity results compared to the base model without connectivity and other lower baselines, i.e., random and identity connectivity.**
| Regions | Base Model | Learnable Connectivity | Random Connectivity | Identity Connectivity |
| --- | --- | --- | --- | --- |
| V1 | 0.287 | 0.297 | -0.001 | 0.090 |
| V2 | 0.288 | 0.299 | 0.001 | 0.109 |
| V3 | 0.270 | 0.283 | 0.000 | 0.121 |
| V4 | 0.263 | 0.272 | 0.000 | 0.108 |
| LOC | 0.293 | 0.316 | 0.000 | 0.163 |
| EBA | 0.349 | 0.369 | 0.000 | 0.223 |
| FFA | 0.281 | 0.306 | 0.002 | 0.164 |
| STS | 0.204 | 0.232 | 0.001 | 0.108 |
| PPA | 0.194 | 0.218 | -0.002 | -0.009 |

**Table 11: SlowFast learnable connectivity results compared to the base model without connectivity and other lower baselines, i.e., random and identity connectivity.**
| Regions | Base Model | Learnable Connectivity | Random Connectivity | Identity Connectivity |
| --- | --- | --- | --- | --- |
| V1 | 0.294 | 0.301 | 0.001 | 0.101 |
| V2 | 0.295 | 0.305 | -0.001 | 0.121 |
| V3 | 0.280 | 0.288 | -0.001 | 0.129 |
| V4 | 0.274 | 0.285 | -0.002 | 0.127 |
| LOC | 0.296 | 0.311 | -0.001 | 0.156 |
| EBA | 0.354 | 0.360 | -0.001 | 0.218 |
| FFA | 0.279 | 0.293 | -0.004 | 0.153 |
| STS | 0.199 | 0.227 | 0.002 | 0.097 |
| PPA | 0.188 | 0.204 | 0.000 | -0.019 |

Figure: (a)

## Appendix C Limitations

In this section, we discuss limitations of our work. Our current large-scale study is limited to ten subjects watching short video clips with 1,000 videos provided. We leave it for future work to explore the collection of broader datasets with longer videos up to 5-10 minutes towards the creation of foundation models for neural encoding.

## Appendix D Broader impact

Conducting large-scale study on neural encoding of biological neural systems in comparison to deep neural networks, has multiple positive societal impacts as it can be used to drive insights into the understanding and advancement of deep networks. Our study also improved our understanding of the human visual cortex and its connectivity using our learned inter-intra region connectivity model, which is useful in various applications. Applications for neural encoding have various benefits including providing a way for testing and understanding brain mechanisms and computations in addition to multiple neural interfaces applications. Since the neural encoding modelling and its capabilities is still being developed and relatively in its infancy, in addition to the datasets available being limited in size, we do not perceive any negative societal impact from such developed models.

## Appendix E Assets and licences

We use the Mini-Algonauts Project 2021 Cichy et al. () dataset that was provided in the challenge and its development kit which is licensed under a MIT License that allows its use without restriction. The Bold Moments Dataset (BMD) including the fMRI data and stimulus metadata we used in this study is available in the OpenNeuro database under accession code DS005165 and CC0 License  Lahner et al. (). Most of our image and video understanding models and their trained weights are used from Pytorch Paszke et al. () and Pytorch Video Fan et al. (), except for few models that were retrieved from their publicly released codes and model weights.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	The following reflects the evolution of my understanding while reading the paper: the authors study the ability of artificial neural network activations to predict biological signals in the visual cortex (the “target”) – they also include an ANN-based target – what does the artificial target add to the study? Section 3.1 seemingly attempts to explain this, but it may need to be edited for clarity. Is the idea that we know the structure of the target ANNs in the artificial setting, so we can test our ability to uncover mechanistic insights through encoding experiments, and therefore get a sense for how informative the system identification experiments with biological targets are likely to be? This becomes much clearer upon review of Han et al 2023, but it should also be explained more explicitly in the introductory sections of the present work. 

2.	In equation 1, in the first term, is a 1/L normalization needed for the summation $\sum_{l=1}^{L} \omega_l \hat{Y}_l$ ? Without the normalization, it seems mildly inconsistent with the statement above equation 1 that “we learn the weights of one fully connected layer to provide the predictions of the voxels of one region of interest in the visual cortex as, $\hat{Y}_l = W_l X_l$.” Or perhaps this normalization is absorbed by $\omega_l$?

3.	In section 3.4, the description of the connectivity module indicates that all voxels from all regions are used to predict voxels from a single region. Doesn’t this mean that the desired output of the model (voxel values from a single target region) are contained in the input of the model, making this a trivial mapping? It makes sense that dropout layers could partially alleviate this problem, but why does it make sense to set up the architecture in this way in the first place where at least some of the outputs are trivially accessible in the input? 

4.	Section 4.1: for four-fold cross validation where each fold has 90% of the videos in the training set and 10% in the test set – are the 10% test sets disjoint between the folds, or randomly selected? (Not a major issue, but 75:25 might have been a slightly more natural choice for four-fold cross validation) 

5.	In section 4.2, there is a statement that “we include video understanding models that are trained on three different datasets which are Kinetics, Charades, and Something-Something v2”. These datasets are not cited, and it is not clear how the results from the three datasets were combined to produce figure 2 – were the results pooled somehow?

6.	Might it be useful to discuss why the observed trends seem to hold for some visual cortical regions and not others? For example, video-computable models are no better than image-computable models for predicting FFA and PPA activity - it is interesting to speculate that perhaps recognition of faces (associated with FFA) or environmental scenes (associated with PPA) in the brain might be less dependent on temporal dynamics.

7.  For plots like those in figures 2 and 3, what does each dot represent? For example, is it one regression score from one type of “source” model averaged among cross-validation folds, or is it an accuracy from one of the folds only? Are results from different source models pooled together in these plots? Relatedly, it is not clearly specified how the Welch’s t-tests are performed – which sets of values specifically are used to compute the t-test? 

8.	Why are system identification results presented in the main text specifically for image vs video computable models, rather than other comparisons of interest like CNNs vs transformers or self-supervised vs fully-supervised?
soundness: 3
strengths: 1.	Neural encoding is an important area of research that can yield insights to both improve artificial neural networks and better understand biological neural systems. 

2.	There is a thorough review of recent related work that makes it clear how the current paper is situated given existing literature. 

3.	A wide variety of appropriately chosen “source” candidate models are employed for comparative encoding experiments, and they vary in several interesting ways as outlined by the authors (single-image vs video computable, CNN vs transformer, fully supervised vs self-supervised). The breadth of the study in this regard is unprecedented for encoding of visual cortical responses to video inputs. 

4.	The work demonstrates in a convincing way that video computable, fully supervised CNN models can generally enable more accurate predictions of visual cortical activity than models that are image computable, are based on transformer architectures, and/or are self-supervised. 

5.	The work also presents evidence that system identification of image-computable vs video-computable systems is plausible (figure 2). This provides additional support for conclusions drawn regarding biological targets. 

6.	A novel approach for incorporating inter and intra-region connectivity is presented, which significantly improves the voxel prediction performance.
summary: This work provides a comprehensive study of predicting fMRI signals in visual cortical areas while humans view short video clips, by mapping from the activations within a wide range of artificial neural networks that process the same videos or still images from them. The study compares models that differ along several interesting dimensions:  image-computable and video computable models, CNNs and transformers, and fully-supervised vs self-supervised training regimes. The results show that video-computable, convolution-based, fully-supervised models are generally the best predictors of visual cortical activity. The relevance of these results is supported by additional encoding studies where the encoding “target” is an artificial neural network of known architecture, showing that it is possible to distinguish between image-computable and video-computable models using a system identification approach. A novel strategy for incorporating inter and intra-region functional connectivity information is proposed, and this approach improves the encoding performance of the already best-performing models.
weaknesses: This paper is primarily limited by a frequent dearth of clarity in the writing, as well as a methodological concern about the way inter and intra-region connectivity priors are incorporated into the encoding models. This reader’s score could improve considerably with clarification of these issues. 

1.	Additional proofreading/editing is recommended to clarify and improve sentence structure in many parts of the paper. For example, the description of the “Real environment” (last part of section 3.1) is ambiguous – it states that two different datasets are used, but then says “we mainly use the training set, and perform cross-validation…”, “The dataset provides fMRI recordings of ten subjects…”, “each video and voxel in the brain was represented by a single activation value” without indicating which dataset these statements refer to (or is it both?). Also in this part of the paper, what does it mean that each video and voxel in the brain was represented by a single activation value? Aren’t there different values across time and between subjects? In several parts of the paper, there are similar ambiguities such as it being unclear which dataset is being referred to at what time. 

2.	It is not clearly specified how the inter/intra region connectivity module is combined with the rest of the encoding models to make voxel activity predictions. The current version of the paper claims that intra-region connectivity priors improve predictive performance – but seemingly by directly providing the voxel values that are being predicted as inputs to the overall model. The improvements in performance might be due not to a meaningful incorporation of connectivity priors, but rather undesirable leakage of label information into the input. 

3.	One of the claims is that inter/intra-region connectivity information improves performance to a greater extent in video-computable models compared with image-computable models – the evidence for this is presented in  figure 6b. However, there is a confound because connectivity-related performance improvements are compared between image-computable ResNet-50 and video-computable MViT: how can it be determined whether the apparent interaction effect is related to image computable vs video computable models rather than CNNs vs transformers?

Minor comments: 

4.	In the abstract and several other places, saying that “the study encompasses more than two million regression fits” does not seem especially relevant – the number of regression fits in itself does not seem to be a particularly informative way to quantify the scale of the study. 

5.	In describing the contributions of the paper, the authors seem to emphasize that prior works did not compare different neural network architectures - but in section 3.1, there is a statement that “while previous works focused on the architecture aspect, we argue it is even more important to look into whether the model is learning dynamics (e.g., motion) or simply using static information from a single image.” Yet, the comparison between single image vs video processing would seem to be an important emphasis of Lahner et al 2024. These claims about the relationship between prior works and the current paper should be clarified. 

6.	The terms “real environment” and “simulated environment” are not especially informative – perhaps something like “biological target” and “artificial neural network target” would be an improvement

7.	In first paragraph of section 4.1, “As for image understanding models we use the default sampling rate of eight.” Does this refer to 8 frames per second? This whole paragraph seems quite ambiguous – why are video understanding vs image understanding models being discussed here in relation to sampling rates?  

8.	The text is some of the figures is too small to be easily readable, especially figure 2, figure 3, figure 5, and supplementary figures 7, 8, 10, and 11.

9.	Most of the in-text citations would be best formatted with both author and year inside the parentheses. In LaTex, you can use \citep{} instead of \cite{}

10.	The goal of the paper seems to be stated in an overly general way in the abstract (“This comparative analysis aims to advance and interpret deep neural networks and enhance our understanding of biological neural systems”). 

11.	Typos: 

A.	In the introduction (end of page 1), “while the later showed that these fMRI recordings” (should be latter instead of later)

B.	Grammar issue in start of 2nd paragraph of section 3.1: “How do deep video understanding models families compare to biological neural systems” 

C.	End of section 3.1 “The datasets we use is provided at TR one second” (should be are, not is, and the TR acronym should be defined)

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: I am a bit confused about the intra- inter- connectivity priors, what their structure is, and if the results demonstrate anything beyond that complex connectivity as seen in human visual cortex increases predictivity?

Relatedly, the paper would benefit from even a minimal interpretation of the learned connectivity weights. What is their structure and how does this change with training? How do they contribute to improvement of the representational alignment? What do these learned weights imply biologically? Such an analysis would likely also address the previous point.

I challenge the authors in L435-437 on leaving the reasons behind the better alignment seen in fully supervised networks to future work. This is a major result of the paper and I challenge the authors to at least make some hypotheses or have a few sentences of discussion as to why this could be the case. The validation of such a hypothesis can be left to future work.

Minor syntax issues:
L151 “: e?”.”
L363: “This difference decrease as”
soundness: 3
strengths: Originality: To my knowledge, this paper uses an established approach, but is novel in the scale and types of comparisons between types of models compared. 

Quality: The paper is of good quality, with good statistical analysis and control conditions.
 
Clarity: The majority of the paper is written very clearly, especially the sections comparing image/video, convolutional/transformer, single/multi stream, and self-supervised/unsupervised networks to BOLD signal.

The scale of experiments in terms of number of models is commendable, and the authors were clever to choose models that enabled comparisons across many different dimensions.

Significance: This is an important emerging area in understanding alignment of models with human data, especially for new types of models, which show improved performance on visual tasks, but as the authors show, have worse representational alignment with human responses. This paper is significant in its contributions to understanding the quality of these alignments over multiple different axes of comparison.
summary: This paper compares a variety of image and video models to fMRI data, evaluating these models for representational alignment to human responses. The paper reports this analysis for multiple types of networks, comparing image/video, convolutional/transformer, single/multi stream, and self-supervised/unsupervised networks. The authors report the best predictors of human responses are convolutional, multi-stream, unsupervised video networks. Additionally, the paper reports experiments with a intra- and inter- region connectivity priors which improve encoding performance.
weaknesses: There is little explanation of the intra- and inter- connectivity priors, especially their structure, and interpretation of their results beyond their contribution to improved representational alignment, and the value of their learned weights in figure 5c.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Overall the reviewer feels that although some of the presentation problems could be corrected, there are overarching problems with the structure, i.e. disjoint two parts, and the soundness of this paper, i.e. correlational evidence, unclean comparisons, and rushed conclusions, that make it hard to change in a way sufficient for acceptance. 
The final rating is 5 in absence of the option for 4 (because 3 would be too low).
soundness: 2
strengths: The paper is original in the aspect of application (model-to-brain encoding) to a new domain (human video fMRI aside from image fMRI), and the writing style has good quality. Previous work is also sufficiently covered. The questions asked are important and in general this work is a step in the right direction.
summary: This work is developed in two parts, aiming to (1) compare the neural predictivity of video deep neural networks on video fMRI data, and (2) propose a method to improve prediction scores via utilizing brain region connectivity.
In the first part, it tests the brain predictivity of models with different architectural design and learning paradigm and contrasts to image models (in total, 26 video action recognition models and 11 image object recognition models are tested). The results indicate that video models are better predictors than image models, in early regions convolutional models are better than transformers, and supervised models are better than self-supervised. Additionally a system identification check is performed which shows that video models can predict other video models better than image models, and vice versa, based on four target models. In the second part, this work proposes training a module to consider all regions in the prediction of any single region and shows improvements on the prediction scores of two selected video models and one image model; the image model has a less pronounced improvement which the authors claim is due to the lack of dynamics.
weaknesses: However, there are several soundness and presentation related issues in its execution that negatively impact this reader’s opinion of the paper.

First of all, the paper reads as two rather disjoint parts that fail to come together as one clear proposition (see summary for the two parts). As such, it looks like this could be two unrelated papers, that each would need further work to be able to stand on its own. This is the case in many points throughout the paper, for example in Figure 1, the reader is lead to think that this is the main figure / main method proposed, but the extent of the results with this method are underwhelming, as are the technical details provided.

Second, there are several issues with the soundness of the methods.
* The reader is not convinced that the improvements observed in brain predictivity are caused by the learned connectivity, and not just by obtaining a more robust or less noisy embedding by considering other regions. There are also inconsistencies in the results regarding this aspect; in figure 5c it is observed that LOC does not connect with any other areas apart from itself - then how is it explained that in figure 5a performance is significantly improved with the connectivity module? Could it be that connectivity is not the reason at all for the improvement? For the learned connectivity itself, extensive comparison with findings in neuroscience literature on region connectivity is lacking.
* For the analysis in section 4.5 as a whole, it is very unconvincing that only two models are compared, instead of the 37 models initially examined in the first part. When comparing to the image models, only one image model is tested and from that the conclusion “shows the necessity for encoding dynamics to fully benefit from such connectivity priors.” is inferred, which is too strong.  Additionally in figure 6b, it is inconsistent that not even Slowfast is shown but only MViT. In the same figure, scores “in a different scale after multiplying by 100” are shown instead of percentage of improvement, not allowing to observe the improvement regardless of the models’ performance scale itself. 
* The motivation to use the Algonauts challenge’s training set for the main results without evaluating on the official test set, is unclear. An even better, more obvious choice for the main results would be to use the full Bold Moments Dataset which is more refined, higher quality, and also includes the test set.
* The supervised vs. self-supervised comparison is not clean, as all the self-supervised models are transformers - thus the relationship between convolutional models and transformers is also partly transferred in that result. The comparison is also between very different numbers of models, with only 5 SSL and more than 30 SL. Additionally, throughout the paper results are shown with models mixed from different training datasets, which makes the effects of each examined comparison (e.g. architecture) and the training dataset, entangled. See Conwell et al. (2022, 2024) for an example of clean comparisons.
* The reader finds that the fine-grained analyses of figure 4 are too fine-grained to be scientifically valid. What does evaluating a single design choice for two-stream processing tells us about either the networks or the brain? Why is the specific TimesFormer architecture chosen to compare against two other (specific) architectures as representatives of supervised, self-supervised, and fine-tuned self-supervised? Furthermore, quite a lot of emphasis is put on the multiscale component of MViT, however, without an explicit ablation it is not possible to know which factor of this model contributes to its good brain predictivity. Additionally, the authors reach the conclusion that early regions relate better to convolutional models “since they better capture high-frequency components” but this is not evidence drawn from their results and rather from other publications. It is not shown that this is the reason behind their results. Further, the model MViT does not conform to this so it seems that this conclusion is too strong.

Third, there are some major problems in the paper’s presentation.
* Figure 1 has several issues. (1) It does not show any details of the Inter-intra Region Connectivity Model (layers, loss function), which is the main component proposed. (2) It shows the pipeline only at the inference mode and does not mention neither which mode is shown or their (substantial) differences. (3) In the first stage of voxel prediction (encoding) it is not clear that the predicted voxels are the outputs of the encoding procedure (they either look like the video models’ input or at best as if they are their direct output which is also not the case). (4) Finally, a major point of this figure was to show that different degrees of connectivity between regions are predicted, but from the size of the visualization on the left it is near impossible to tell differences between the thickness of the arrows.
* In section 3.4, where the Inter-intra Region Connectivity Model is introduced, there are no equations to show the model layers, and more importantly the loss function to train this module is not described at all, in equation or text form.
* In section 4.1, it is not clear whether the hyper-parameter tuning is conducted on the Algonauts training set or the authors’ training set (90% of the Algonauts training set, different for each of the 4 folds). This is an important clarification to make because in the first case the encoding test set is seen during hyper-parameter tuning.
* Phrases like “we mainly use the training set” in section 3.1 are too vague - when exactly do the authors not use it? From section 4.1 it seems that only the training test is used in all cases.
* Video models used are described as “video understanding” models throughout the paper, which is inaccurate because they are all video action recognition models - video understanding is a vague umbrella task that could potentially include segmentation, reasoning, and many other more complex tasks than classification. By also describing the image models as “image understanding” models, authors obscure the fact that they are trained on a different task than the video models, which is object recognition rather than action recognition.

References:

Conwell, Colin, et al. "What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?." BioRxiv (2022): 2022-03.

Conwell, Colin, et al. "A large-scale examination of inductive biases shaping high-level visual representation in brains and machines." Nature Communications 15.1 (2024): 9383.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: There is almost no explanation of what the authors did here. The authors are using existing datasets and they do not need to copy and paste the entire methods section of the original papers. But they need to explicitly indicate what they have done here.
	(1) In terms of the experimental data: 
	How many participants?
	How many video clips? 
	What was the length of the video clips?
	How many repetitions of each video clip?
	What was the size of the image, the frame rate, and the volume of the audio? 
	What was the resolution of the eye tracker used?
	What kind of video clips? What was the content? Were there cuts in the video? 
	(2) In terms of the analyses,
	Saying “… we use the brain responses…” has little meaning. 
	What are the units of the measurements?
	How reliable are the measurements (e.g., reproducibility across repetitions of identical stimuli)?
	Which time interval was used for the analyses? There is a mention of one second but the temporal resolution is much slower than 1 second, it would be important to document that there is indeed information at 1 second time scales. 
	Which voxels were used for each region?
	Which hemispheres? 
	How were the 9 regions of interest defined? 
	(3) Data fitting
	What was the dimensionality of the predictors in each case?
	How was cross-validation performed?
soundness: 1
strengths: The authors have done extensive calculations with lots of computational models.
Understanding how the brain processes dynamic signals is an important question.
summary: The authors use multiple neural network models to attempt to fit signals from functional magnetic resonance imaging (fMRI) obtained while participants watched short video clips from two existing datasets (Cichy et al 2021, Lahner et al 2024).
weaknesses: - The manuscript is poorly written and it is very hard to decipher what the authors did. There is almost no methodological information whatsoever. I list an initial list of questions that would be essential to answer when the authors submit the work to some other venue. 
- It is recommended that a native speaker goes over the paper to fix grammar and clarity. 
- It is also recommended that someone reads the paper with reproducibility in mind. This can help the authors understand all the information they need to provide. 
- In this type of data fitting work, it is always useful to define a lower bound given by chance and an upper bound sometimes defined by computing split-half regressions across multiple repetitions of identical stimuli
- The temporal resolution of fMRI is very poor. The field has struggled with getting any meaningful temporal information about of videos. Given that the authors seem to be interested in timing, it would be useful to consider data with better temporal resolution. If the authors cannot use better data, then they should first document that there is any temporal information that can be obtained from the fMRI signals during videos. 
- The authors do not say anything about eye movements. Participants typically make 2 to 4 saccades per second while watching images or video. Presumably these saccades would radically change the activity in visual cortex. It is unclear how one can make any meaningful claims about visual cortex without documenting the neural responses to each saccade.  
- Before jumping into large neural networks with large numbers of free parameters, it would be useful to quantify how well the data can be explained by simple models like pixels, contrast, optic flow, etc. This is assuming that the authors first convince themselves and show that there is actual reproducible data that warrants explanation.

Meta-review: The paper presents a comprehensive study comparing image and video understanding models to visual cortex recordings using video stimuli. The study reveals insights into how different specs of deep networks predict neural responses. Moreover, the paper introduces a novel neural encoding scheme that leverages connectivity priors for improved performance.

The paper is easy to follow. The experiments benchmarking so many AI models and leveraging connectivity prior are well-motivated, original, and interesting. 

The paper has mixed reviews with two 6s, 5, and 1.

After reviewing the rebuttal and initiating intensive internal discussions with all the reviewers, the Area Chair (AC) and reviewers collectively reached a consensus on the following weaknesses of the paper, which ultimately led to its rejection:

[Inconsistencies in results, not well-controlled experiments, and results fail to support the claims] The experiments comparing models are not well-controlled, and the results fail to support the claims made in the paper, such as those in Fig. 16e. Even after the rebuttal, the paper continues to overstate the positive effect of connectivity priors, without acknowledging that this effect is observed only in certain brain regions, not universally. While the authors have made minor adjustments to qualify their claims in response to other reviewers' points, they remain reluctant to fully address the nuances of their findings.

[Missing experimental details] The paper lacks clear details regarding the models, training procedures, and the specifics of the connectivity priors. Despite reviewers requesting these clarifications, the revised version does not include the necessary information.

[Inadequate responses to concerns during rebuttal] Based on the points above, it is evident that the authors largely dismissed these well-founded critiques. All reviewers agree that the authors persistently refused to acknowledge or adequately address these issues, either in their responses or in the manuscript itself.

Decision: Reject