LinBridge: A Learnable Framework for Interpreting Nonlinear Neural Encoding Models

## Abstract

Abstract Neural encoding of artificial neural networks (ANNs) aligns the computational representations of ANNs with brain responses, providing profound insights into the neural basis underpinning information processing in the human brain. Current neural encoding studies primarily employ linear encoding models for interpretability, despite the prevalence of nonlinear neural responses. This leads to a growing interest in developing nonlinear encoding models that retain interpretability. To address this problem, we propose LinBridge , a learnable and flexible framework based on Jacobian analysis for interpreting nonlinear encoding models. LinBridge posits that the nonlinear mapping between ANN representations and neural responses can be factorized into a linear inherent component that approximates the complex nonlinear relationship, and a mapping bias that captures sample-selective nonlinearity. The Jacobian matrix, which reflects output change rates relative to input, enables the analysis of sample-selective mapping in nonlinear models. LinBridge employs a self-supervised learning strategy to extract both the linear inherent component and nonlinear mapping biases from the Jacobian matrices of the test set, allowing it to adapt effectively to various nonlinear encoding models. We validate the LinBridge framework in the scenario of neural visual encoding, using computational visual representations from CLIP-ViT to predict brain activity recorded via functional magnetic resonance imaging (fMRI). Our experimental results demonstrate that: 1) the linear inherent component extracted by LinBridge accurately reflects the complex mappings of nonlinear neural encoding models; 2) the sample-selective mapping bias elucidates the variability of nonlinearity across different levels of the visual processing hierarchy. This study not only introduces a novel tool for interpreting nonlinear neural encoding models but also provides novel evidence regarding the distribution of hierarchical nonlinearity within the visual cortex.

## 1 Introduction

In recent years, aligning the computational representation of artificial neural networks (ANNs) with brain activity through neural encoding models significantly advances our understanding of the neural basis underlying information processing in the human brain. Previous research primarily employs linear encoding models due to their interpretability  (Naselaris et al., ; Yamins & Dicarlo, ). However, the prevalence of nonlinear neural processes is well recognized  (Naselaris et al., ; Wang et al., ; Tang et al., ; Jain & Huth, ), which may limit the ability of linear models to fully capture the complex dynamics of brain activity. This limitation not only undermines the predictive performance of linear encoding models but also restricts their capacity to effectively interpret neural activity.

With the rapid advancement of deep neural networks, nonlinear encoding models are increasingly utilized  (Zhang et al., ; Li et al., ; Cui et al., ; ). These models incorporate activation functions or other nonlinear structures, allowing them to better capture the brain’s responses to complex stimuli and leading to improved predictive performance compared to linear models (Zhang et al., ; Li et al., ; Cui et al., ; ). However, as illustrated in Figure 1, nonlinear encoding models exhibit sample-specific characteristics, resulting in unstable structures that complicate the interpretation of underlying relationships.

Figure: Figure 1: Comparison of linear and nonlinear encoding models. In linear encoding models, the mapping relationship between the feature space and brain activity space is invariant across input samples. On the contrary, nonlinear encoding models exhibit sample-specific characteristics, resulting in an unstable structure that complicates the interpretation of the underlying relationships.
Refer to caption: extracted/5954451/Figure1.jpg

In this study, we propose LinBridge, a learnable and flexible framework based on Jacobian analysis for interpreting nonlinear encoding models. The Jacobian matrix, which quantifies the output change rates relative to input, enables the analysis of sample-selective mapping in nonlinear models  (Gale & Nikaido, ). LinBridge posits that the nonlinear mapping between the computational representations and neural responses can be factorized into two components: a linear inherent component that approximates the complex nonlinear dynamics, and a mapping bias that captures sample-selective nonlinearities. However, this factorization presents a dilemma: the sample-selective nonlinearities introduce substantial variability in the Jacobian matrices across different input-output pairs, complicating the extraction of consistent and interpretable mapping structures within nonlinear encoding models.

To address this challenge, we propose a self-supervised learning strategy based on contrastive learning, which has demonstrated superior capabilities in differentiating shared and distinctive attributes through paired-sample analysis  (Oord et al., ; Schneider et al., ). Within the contrastive learning framework, LinBridge maximizes shared linear component within the Jacobian matrices while minimizing the influence of nonlinear features (nonlinear mapping biases) on those shared components, leading to an effective delineation of the linear inherent component and the nonlinear mapping biases. In addition, this self-supervised learning strategy allows LinBridge to adapt effectively to various nonlinear encoding models. Furthermore, LinBridge incorporates a low-dimensional embedding module that facilitates dimensionality reduction while preserving the intrinsic structure of the feature space, providing a more intuitive tool for analyzing the brain’s linear and nonlinear responses to external stimuli. The contributions of this work are summarized as follows:

- •
We introduce LinBridge, a flexible framework designed to extract both the linear inherent component and the nonlinear mapping biases from nonlinear encoding models. This framework enables an interpretable analysis of the nonlinear mappings between computational representations and brain responses.
- •
The linear inherent component extracted by LinBridge exhibits activation patterns highly consistent with the original nonlinear encoding model, suggesting that the complex mappings in nonlinear neural encoding models can be effectively captured.
- •
We apply LinBridge to a neural encoding exploration of vision transformer models and reveal the variability in nonlinearity across different levels of the visual processing hierarchy.

## 2 Background and Related Works

### 2.1 Linear Neural Encoding Models

Due to their simplicity and interpretability, linear neural encoding models are widely applied across various domains to disentangle the neural basis underpinning information processing in the brain, including vision (Yamins et al., ; Khaligh-Razavi & Kriegeskorte, ; Güçlü & van Gerven, ; Eickenberg et al., ; Zhuang et al., ), audition (Zhou et al., ; Li et al., ; MILLET et al., ; Tuckute et al., ; Vaidya et al., ), and language (Liu et al., ; Caucheteux & King, ; Goldstein et al., ; Jain & Huth, ; Schrimpf et al., ; Abdou, ). However, the inherent nonlinear dynamics of neural activity limit the predictive power and interpretability of linear models, particularly when addressing more complex cognitive functions. This limitation is especially pronounced in higher-order cortical areas, which often involve complex and dynamic interactions, suggesting that their underlying neural mechanisms may not be adequately captured by linear representations (Naselaris et al., ).

### 2.2 Nonlinear Neural Encoding Models

Nonlinear encoding models emerge as a solution to the limitations of linear models by incorporating nonlinear structures that more effectively capture the complex patterns of brain activity. These approaches demonstrate superior predictive performance compared to linear models (Zhang et al., ; Li et al., ; Cui et al., ; ). However, the opaque nature of nonlinear encoding models presents significant challenges in understanding the mapping between the computational representations and brain responses. To address this challenge, most existing methods adopt the framework proposed by (Tank et al., ), leveraging various time-series prediction models to enhance the interpretability of these learned black-box mappings (Khanna & Tan, ; Bussmann et al., ; Suryadi et al., ). In particular, the Jacobian matrix has been employed to elucidate the local mapping relationships in artificial neural networks (ANNs)  (Zhou et al., ; Suryadi et al., ). Nonetheless, nonlinear models often exhibit sample-specific mappings, leading to substantial variability in the Jacobian matrices across different input-output pairs. This variability complicates the extraction of consistent and interpretable mapping structures within nonlinear encoding models.

## 3 Methods

### 3.1 Dataset and Preprocessing

We use the Natural Scenes Dataset (NSD)  (Allen et al., ) in this study. The NSD dataset contains fMRI data from eight subjects passively viewing 73,000 color natural scene images over 40 hours. These images are cropped from the MS-COCO dataset  (Lin et al., ), with each image displayed for three seconds and repeated three times across 30 to 40 scanning sessions, resulting in a total of 22,000 to 30,000 experimental trials. The fMRI data in the NSD are acquired using a whole-brain gradient-echo EPI (echo-planar imaging) sequence at 7T, with a resolution of 1.8 mm and a repetition time of 1.6 seconds. Single-trial beta maps are estimated using a customized general linear model and released alongside the raw fMRI data  (Wang et al., ; Tang et al., ). Similar to previous studies  (Wang et al., ; Tang et al., ), these beta maps are normalized (zero mean and unit variance) within each run and averaged across image repetitions to be used as functional brain activity measures. In our experiments, each image and its corresponding beta map are treated as a single sample. We divide the dataset into training, validation, and testing sets in a ratio of 8:1:1 (8000:1000:1000 samples). Notably, subjects 1, 2, 5, and 7 complete the full experimental protocol, and thus, their fMRI data are utilized in our experiments.
In the experiments on neural encoding of vision transformer models, we use the pre-trained CLIP-ViT  (Radford et al., )(^1^11https://huggingface.co/openai/clip-vit-base-patch32.) image encoder to derive the computational representation of the visual image stimuli.

### 3.2 Encoding Model

The general encoding model can be formulated as follows:

$$ $\hat{y}=f(x)$ (1) $$

where $x$ denotes the feature space of external stimuli, $\hat{y}$ is the brain activity space, and $f(\cdot)$ denotes the encoding function. Specifically, the computational representation of visual images spans the feature space, and the normalized beta maps serve as the brain activity space. To validate LinBridge, we construct both linear and nonlinear models in simplified form. Each model consists of two fully connected layers, with the nonlinear model additionally incorporating a ReLU activation function. Further details regarding the encoding models are provided in  A.1. The mean squared error (MSE)  (Huang et al., ) is used as the loss function in the encoding model.

### 3.3 Computation of the Jacobian Matrix

The Jacobian matrix captures the local linear mapping between feature space and neural responses. It reflects the sensitivity of the neural encoder to variations in the input. Upon completion of model training, we fix the parameters of the encoding model and input the test set data to obtain the corresponding predictions. Given the $k$-th sample $x_{k}\in\mathbb{R}^{d}$ in the test set and the prediction of the nonlinear neural encoder $\hat{y}_{k}=f(x_{k})\in\mathbb{R}^{p}$, the Jacobian matrix $\mathbf{JM}_{k}$ can be calculated by taking the derivative of the model’s output with respect to its input as follows:

$$ $\mathbf{JM}_{k}=\left(\frac{\partial\hat{y}_{k}}{\partial x_{k}}\right)^{T}= \begin{bmatrix}\frac{\partial\hat{y}_{k,1}}{\partial x_{k,1}}&\cdots&\frac{ \partial\hat{y}_{k,p}}{\partial x_{k,1}}\\ \vdots&\ddots&\vdots\\ \frac{\partial\hat{y}_{k,1}}{\partial x_{k,d}}&\cdots&\frac{\partial\hat{y}_{k ,p}}{\partial x_{k,d}}\end{bmatrix}\in\mathbb{R}^{d\times 1\times p}$ (2) $$

We denote the collection of Jacobian matrices of all testing samples as $\mathbf{JM}\in\mathbb{R}^{d\times N\times p}$, where $d=512$ is the dimensionality of the representation, $N=1000$ is the sample size of the test set, and $p$ is the number of voxels.

### 3.4 LinBridge

LinBridge leverages Jacobian matrices ($\mathbf{JM}$) to quantify the complex mapping relationships in nonlinear encoding models. It extracts the linear inherent component ($\text{JM}_{\text{inherent}}$), which captures consistent, interpretable mapping structure invariant to input samples, and the nonlinear mapping biases ($\Delta\text{JM}$) that reflect the unique nonlinear behaviors associated with distinct inputs.

#### 3.4.1 Extraction of the Linear Inherent Component

Similar to the dimensionality reduction architecture introduced in CEBRA (Schneider et al., )(^2^22https://github.com/AdaptiveMotorControlLab/CEBRA.), LinBridge employs a multi-layer convolutional neural network (CNN) to refine the input Jacobian matrices. Specifically, LinBridge progressively compresses the sample dimension of the Jacobian matrix using the CNN, enabling the extraction of the linear inherent component ($\mathbf{JM}_{\text{inherent}}\in\mathbb{R}^{d\times 1\times p}$) that captures the linear mapping between the input representations of the nonlinear neural encoding model and the corresponding voxel activations in the brain. This approach provides a more structured interpretation of neural activity.

Figure: Figure 2: Nonlinear Encoding Model and the LinBridge Framework. (a) Image representation extraction and the general neural encoding model structure; (b) LinBridge framework, which includes the computation of $\mathbf{JM}$, the extraction of $\mathbf{JM}_{\text{inherent}}$ based on CNN module, the calculation of $\Delta\mathbf{JM}$, and the implementation of a low-dimensional embedding module.
Refer to caption: extracted/5954451/Figure2.jpg

#### 3.4.2 Nonlinear Mapping Biases Δ ⁢ 𝐉𝐌 Δ 𝐉𝐌 \Delta\mathbf{JM} roman_Δ bold_JM

We directly subtract $\mathbf{JM}_{\text{inherent}}$ from $\mathbf{JM}$ to obtain the nonlinear mapping biases $\Delta\mathbf{JM}$. This operation ensures that both the linear inherent component and the nonlinear mapping biases retain within the original space of $\mathbf{JM}$. Therefore, the computation of $\Delta\mathbf{JM}$ follows:

$$ $\Delta\mathbf{JM}=\mathbf{JM}-\mathbf{JM}_{\text{inherent}}\in\mathbb{R}^{d \times N\times p}$ (3) $$

#### 3.4.3 Low-Dimensional Embedding

LinBridge incorporates a low-dimensional embedding module that linearly reduces the dimensions of $\mathbf{JM}_{\text{inherent}}$, $\mathbf{JM}$, and $\Delta\mathbf{JM}$ to a more compact representation. This linear dimensionality reduction effectively preserves the relative spatial distances of $\mathbf{JM}_{\text{inherent}}$ $\mathbf{JM}$, and $\Delta\mathbf{JM}$ in the latent space. On the one hand, lower dimensional features increase the effectiveness of contrastive learning, on the other hand, it reduces computational cost. Specifically, we use a fully connected (FC) layer to project each matrix into a low-dimensional space.

$$ $\mathbf{JM}_{\text{inherent}}^{\text{down}}=FC_{\text{Downsample}}(\mathbf{JM} _{\text{inherent}})\in\mathbb{R}^{1\times p}$ (4) $$

$$ $\mathbf{JM}^{\text{down}}=FC_{\text{Downsample}}(\mathbf{JM})\in\mathbb{R}^{N \times p}$ (5) $$

$$ $\Delta\mathbf{JM}^{\text{down}}=FC_{\text{Downsample}}(\Delta\mathbf{JM})\in \mathbb{R}^{N\times p}$ (6) $$

In the equations above, the superscript “down” in $\mathbf{JM}_{\text{inherent}}^{\text{down}}$, $\mathbf{JM}^{\text{down}}$, and $\Delta\mathbf{JM}^{\text{down}}$ indicates the dimensionality-reduced representations. Generally, they are the result of applying FC module to the original $\mathbf{JM}_{\text{inherent}}$, $\mathbf{JM}$, and $\Delta\mathbf{JM}$, reducing them to a more compact, low-dimensional space.

#### 3.4.4 Loss Function

The low-dimensional embeddings ($\mathbf{JM}_{\text{inherent}}^{\text{down}}$, $\mathbf{JM}^{\text{down}}$, and $\Delta\mathbf{JM}^{\text{down}}$) of these matrices are utilized for contrastive learning. Essentially, $\mathbf{JM}_{\text{inherent}}$ represents the shared linear components of $\mathbf{JM}$ in the latent space. The objective of the contrastive learning framework is to maximize the alignment of these components while minimizing the influence of the unique nonlinear characteristics represented by $\Delta\mathbf{JM}$. Consequently, it seeks to maximize the distinctiveness of $\mathbf{JM}_{\text{inherent}}$ in relation to $\Delta\mathbf{JM}$. To this end, LinBridge employs the InfoNCE loss function (Oord et al., ; Schneider et al., )(^3^33 https://github.com/RElbers/info-nce-pytorch.). Contrastive learning maximizes the similarity between the reference sample and positive samples ($\mathbf{JM}_{\text{inherent}}^{\text{down}}$ and $\mathbf{JM}^{\text{down}}$), while concurrently minimizing the similarity between the reference sample and negative samples ($\mathbf{JM}_{\text{inherent}}^{\text{down}}$ and $\Delta\mathbf{JM}^{\text{down}}$). Thus, the InfoNCE loss function in this study is formulated as follows:

$$ $\mathcal{L}_{\text{InfoNCE}}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(\text{ sim}(\mathbf{JM}_{\text{inherent}}^{\text{down}},\mathbf{JM}_{i}^{\text{down}} )/\tau)}{\exp(\text{sim}(\mathbf{JM}_{\text{inherent}}^{\text{down}},\mathbf{ JM}_{i}^{\text{down}})/\tau)+\sum_{j=1}^{N}\exp(\text{sim}(\mathbf{JM}_{\text{ inherent}}^{\text{down}},\Delta\mathbf{JM}_{j}^{\text{down}})/\tau)}$ (7) $$

where $\text{sim}(\cdot,\cdot)$ denotes the similarity metric (e.g., cosine similarity), and $\tau$ is the temperature parameter. Additionally, we introduce L1-regularization to prevent overfitting of $\Delta\mathbf{JM}$:

$$ $\mathcal{L}_{\text{Reg}}=\lambda\sum|\Delta\mathbf{JM}|$ (8) $$

where $\lambda=0.01$ is the regularization coefficient. The final loss function is as follows:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{InfoNCE}}+\mathcal{L}_{\text{Reg}}$ (9) $$

In summary, the structural block diagram is shown in Figure 2. LinBridge utilizes a Jacobian matrix-driven strategy, enabling its application to any neural encoding model.

Figure: Figure 3: Comparison of $R^{2}$ between linear and nonlinear encoding models, showing predictions significantly above chance levels ($P<0.05$, FDR corrected). (a) $R^{2}$ in the linear encoding model; (b) $R^{2}$ in the nonlinear encoding model; (c) The histograms of $R^{2}$ in the whole brain, the primary visual cortex (PVC), the secondary visual cortex (SVC), and the tertiary visual cortex (TVC). Results for other subjects are provided in A.3.
Refer to caption: extracted/5954451/Figure3.jpg

### 3.5 Evaluation Metrics

We use the coefficient of determination (${{R}^{2}}$) to evaluate the predictive performance of the neural encoding models on the test set. To assess the statistical significance of the predictions, we follow the method described in  (Wang et al., ), conduct 200 bootstrapped resampling iterations on the test set and calculating FDR-corrected $P$-value thresholds for various performance metrics  (Wang et al., ; Subramaniam et al., ).

## 4 Results

### 4.1 Prediction of Visual Cortex with Nonlinear Encoders

We compare the predictive performance of the linear and nonlinear encoding models. Figure 3 shows the $R^{2}$ values for both models for Subject 2 (all results reported in the main text are exemplified using Subject 2 unless otherwise stated). The results demonstrate that the nonlinear encoding model activates a broader range of brain regions compared to the linear model (24,490 voxels vs. 16,084 voxels), particularly in the visual cortex. Figure 3 (c) further illustrates that the nonlinear encoding model achieves significantly higher $R^{2}$ values across various visual areas, including the primary visual cortex (PVC: V1), secondary visual cortex (SVC: V2, V3, V4), and tertiary visual cortex (TVC: EBA, PPA, RSC, OPA, FFA-1, FFA-2). Notably, 9.30% of the voxels in the nonlinear encoding model exhibit relatively good predictive performance ($R^{2}>$ 0.05), in contrast to 6.65% in the linear model. These results suggest that the nonlinear encoding model outperforms the linear one.

### 4.2 Linear Inherent Component: Linear Interpretations of Nonlinear Mappings

Figure 4 (a) illustrates the stability of the linear inherent component extracted by LinBridge. We calculate the Pearson correlation coefficients between the linear inherent component obtained at various batch sizes ($\mathbf{batchsize\in[16,32,64,128,256,512]}$) and the linear inherent component obtained at a batch size of 1000, that is, the entire test set. The training strategies are detailed in  A.2. This analysis is repeated for 200 times to obtain means and standard deviations. As shown in Figure 4 (a), the inherent structure extracted by LinBridge is highly stable across all evaluated batch size, as evidenced by Pearson coefficients approaching 1. This high stability suggests that LinBridge can accurately capture the relationship carried by the encoding model even with small batch sizes.

Figure: Figure 4: Comparison of the linear inherent component extracted by LinBridge to the brain activation predicted by the nonlinear encoding model. (a) The stability of the extracted linear inherent component across different batch sizes. (b) Activation patterns of the linear inherent component extracted by LinBridge. (c) Comparison of the distribution of $R^{2}$ values between the linear inherent component extracted by LinBridge and the original nonlinear encoding models in the whole brain, PVC, SVC and TVC. Results for other subjects can be found in A.4.
Refer to caption: extracted/5954451/Figure4.jpg

The linear inherent component extracted by LinBridge closely aligns with the activation patterns in the nonlinear encoding model (Figures 3 (b) and 4 (b)). Additionally, we compare the $R^{2}$ values of the nonlinear encoding model and the linear inherent component extracted by LinBridge across the whole brain, PVC, SVC, and TVC (Figure 4 (c)). The $R^{2}$ values in the two conditions exhibit high correlations, with Pearson correlation coefficients of 0.9972, 0.9949, 0.9937, and 0.9984 at the whole brain level, and in PVC, SVC, and TVC, respectively. The linear inherent component achieves comparable or even superior performance in specific brain regions. All these results demonstrate that the linear inherent component extracted by LinBridge can accurately capture the complex relationship represented by the nonlinear encoding model.

### 4.3 Nonlinear Encoding in Visual Cortex

The variations of $\Delta\mathbf{JM}$ regarding to various samples are valuable to characterize the nonlinearity of voxel-wise encoding model, and hence the visual cortex.

However, the high dimension of $\Delta\mathbf{JM}$ complicates the quantification of its variations among samples. In this context, the low-dimensional embedding $\Delta\mathbf{JM}^{\text{down}}$ derived from LinBridge provides a novel perspective for depicting the nonlinearity of the visual cortex. To this end, we incorporate linear fitting  (Allen et al., ; Cohen, ; Hlinka et al., ) to assess how $\Delta\mathbf{JM}$ varies with different samples. Specifically, for each selected voxel, we first generate a sample index array and then extract the corresponding response values from $\Delta\mathbf{JM}^{\text{down}}$. We subsequently calculate the coefficients of a first-degree polynomial through polynomial fitting, with these coefficients representing the linear response weights of the voxel across different categories. It is hypothesized that a voxel responses to external stimuli linearly if the corresponding $\Delta\mathbf{JM}^{\text{down}}$ is invariant to different samples (i.e., the absolute value of the first derivative approaches 0). Conversely, the voxel is deemed nonlinear. Thus, we employ the absolute value of the first derivative (AFD) as a metric to evaluate the nonlinearity of a voxel.

We first perform dimensionality reduction for the CLIP-ViT features of all the 73,000 stimulus images using t-SNE  (Van der Maaten & Hinton, ). We then reorder the image samples in descending order according to their 1D-representation resulting from t-SNE. Figures 13-16 in A.5 show the sorted image samples, demonstrating an obvious transition from ”simple” to ”complex”, as well as that the samples with similar semantics cluster together. This distribution pattern remains consistent across subjects.

Figure 5 (a) illustrates $\Delta\mathbf{JM}^{\text{down}}$ in the test samples for two voxels, in which the  x-axis is the sorted image index according to their 1-D t-SNE representation in descending order. The left and right panels correspond to the voxels with the highest and lowest AFD values, respectively. In the left panel, the low-dimensional embedding of $\Delta\mathbf{JM}^{\text{down}}$ varies sharply with the sorted image samples, and the high AFD value indicates strong nonlinearity. Conversely, in the right panel, the low-dimensional embedding of $\Delta\mathbf{JM}^{\text{down}}$ is invariant to image samples, evidencing strong linearity of the voxel.

Figure 5 (b) shows the histograms of AFDs in PVC, SVC, and TVC. The number of voxels with relatively higher AFDs increases significantly from PVC to TVC. This observation may suggest a progressive nonlinearity within the hierarchy of visual cortex. Our findings closely align with previous research. For example, the primary visual cortex predominantly processes relatively simple visual features  (Wang et al., ; Glasser et al., ; Huff et al., ), and consequently exhibits lower nonlinear encoding. In contrast, the middle and higher visual cortices manage more complex visual scenes  (Wang et al., ; Glasser et al., ), such as spatial relationships and object recognition, exhibiting stronger nonlinear components in their neural activity.

Figure 5 (c) visualizes the nonlinearity measured by AFDs for significantly activated voxels across the whole brain. The PVC and SVC display more linear characteristics, whereas the TVC exhibits remarkable nonlinearity, further reinforcing the notion of hierarchical distribution of nonlinearity in the visual cortex. The probability density distribution of the AFDs in Figure 5 (d) indicates that the TVC demonstrates a higher probability of higher AFDs.

Further analysis reveals that, beyond the visual cortex, other brain regions associated with higher cognitive processing also exhibit nonlinearity (Figure 5 (c)), For instance, the temporoparietal-occipital junction (TPOJ) and prefrontal areas. The TPOJ, recognized as a key region for multimodal information integration  (Wang et al., ; Glasser et al., ), exemplifies the complexity of nonlinear processing and hierarchical information transmission in the brain. In parallel, the nonlinearity of the prefrontal cortex suggests that during higher cognitive tasks, such as decision-making and reasoning, the brain appears to rely on complex nonlinear mechanisms.

Figure: Figure 5: Distribution of nonlinear visual encoding in the brain. (a) The left and right images show voxel fitting results for the highest and lowest evaluation metrics, respectively; (b) Histogram comparison of evaluation metrics for significantly activated voxels in PVC, SVC, and TVC; (c) Visualization of evaluation metrics for significantly activated voxels across the whole brain; (d) Comparison of probability density functions for evaluation metrics of significantly activated voxels in PVC, SVC, and TVC. The sorted image samples can be found in Figure 13, and the results for other subjects can be found in A.6.
Refer to caption: extracted/5954451/Figure5.jpg

## 5 Conclusions, Limitations, and Prospects

In this study, we introduce LinBridge, a novel framework aimed at interpreting nonlinear neural encoding models through Jacobian analysis. We hypothesize that the intricate nonlinear mappings between ANN representations and neural responses can be decomposed into a linear inherent component and a sample-selective mapping bias. LinBridge effectively bridges the interpretability and nonlinearity divide, advancing our understanding of neural encoding and serving as a valuable tool for future investigations into complex neural computations across various brain regions.

The present study acknowledges several limitations. First, we validate LinBridge by using a simple nonlinear encoding model. In the future, it is interesting to conduct further validation studies using more advanced nonlinear encoding models. Second, while the Jacobian matrix elucidates the mapping relationships between samples and corresponding outputs, its computation remains resource-intensive when applied to large datasets. Given that pre-trained encoding models inherently contain gradient information, the incorporation of low-rank matrix decomposition for efficient computation of mapping relationships between samples represents a promising avenue for future work. Third, although our study focuses on the visual cortex, it is interesting to apply LinBridge to neural encoding models in other modalities such as acoustic, linguistic, and multimodal information.

## Appendix A Appendix

### A.1 Nonlinear Encoder and Linear Encoder

Figure: Figure 6: Structural comparison between nonlinear and linear encoding models.
Refer to caption: extracted/5954451/App_1_EncodingModels.jpg

To investigate the relationship between visual feature representations and brain responses, two ANN encoding models are constructed: a linear encoding model (Linear encoder) and a nonlinear encoding model (Nonlinear encoder).

Nonlinear Encoding Model

$$ $\hat{y}^{nonlinear}=W_{2}\sigma(W_{1}x+b_{1})+b_{2}$ (10) $$

where $x$ denotes the input representations, and $W_{1},b_{1}$ and $W_{2},b_{2}$ are the weights and biases of the first and second layers, respectively. $\sigma$ denotes the ReLU activation function, which introduces nonlinearity to enhance the model’s expressiveness.

Linear Encoding Model

$$ $\hat{y}^{linear}=W_{2}(W_{1}x+b_{1})+b_{2}$ (11) $$

This model omits the ReLU activation function, allowing only linear transformations and serving to investigate the performance of linear encoding. In this study, the bias terms $b_{1}$ and $b_{2}$ are both set to False. Their structural comparison is illustrated in Figure 6. Meanwhile, the detailed code implementations of their models are presented in Algorithms 1 and 2, respectively.

Training Strategy

The training process of the encoding model utilizes the Adam optimizer  (Zhang, ) for parameter optimization. To prevent overfitting, early stopping is employed, ceasing training if the validation loss fails to improve over 8 consecutive epochs. Additionally, the model parameters that demonstrate the optimal performance on the validation set are recorded and preserved.

Figure: Algorithm 1 Nonlinear encoding model

Figure: Algorithm 2 Linear encoding model

### A.2 Training Strategies of LinBridge under Different Batchsizes

Here we demonstrate how LinBridge is trained with different batchsizes. Specifically, we provide the code implementation for LinBridge under various batchsize settings. Meanwhile, the detailed code implementations are presented in Algorithms 3. The code demonstrates how LinBridge is trained with different batchsizes, ranging from 16 to large 512 batchsizes.

Figure: Algorithm 3 Training strategies of LinBridge under different batchsizes

### A.3 Comparison of R 2 superscript 𝑅 2 R^{2} italic_R start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT between Linear and Nonlinear Encoding Models in Other Subjects

Figure: Figure 7: Comparison of $R^{2}$ between linear and nonlinear encoding models in Subject 1.
Refer to caption: extracted/5954451/Figure3_S1.jpg

Figure: Figure 8: Comparison of $R^{2}$ between linear and nonlinear encoding models in Subject 1.
Refer to caption: extracted/5954451/Figure3_S5.jpg

Figure: Figure 9: Comparison of $R^{2}$ between linear and nonlinear encoding models in Subject 1.
Refer to caption: extracted/5954451/Figure3_S7.jpg

### A.4 Comparison of the Linear Inherent Component Extracted by LinBridge with Brain Activation of the Nonlinear Encoding Model in Other Subjects

Figure: Figure 10: Comparison of the linear inherent component extracted by LinBridge with the brain activation of the nonlinear encoding model in Subject 1.
Refer to caption: extracted/5954451/Figure4_S1.jpg

Figure: Figure 11: Comparison of the linear inherent component extracted by LinBridge with the brain activation of the nonlinear encoding model in Subject 5.
Refer to caption: extracted/5954451/Figure4_S5.jpg

Figure: Figure 12: Comparison of the linear inherent component extracted by LinBridge with the brain activation of the nonlinear encoding model in Subject 7.
Refer to caption: extracted/5954451/Figure4_S7.jpg

### A.5 Display of Test Images after Progressive Sorting

Figure: Figure 13: The images of the test set for Subject 2 after progressive sorting. Each image corresponds sequentially from top to bottom and left to right to the positions 1-1000 in the ”Sorted Sample Order” of Figure 5 (a). From this, it can be observed that there is a certain pattern present in the progressively sorted test images.
Refer to caption: extracted/5954451/App_3_Sub2.jpg

Figure: Figure 14: Display of test images for Subject 1 after progressive sorting.
Refer to caption: extracted/5954451/App_3_Sub1.jpg

Figure: Figure 15: Display of test images for Subject 5 after progressive sorting.
Refer to caption: extracted/5954451/App_3_Sub5.jpg

Figure: Figure 16: Display of test images for Subject 7 after progressive sorting.
Refer to caption: extracted/5954451/App_3_Sub7.jpg

### A.6 Distribution of Nonlinear Encoding in the Brain in Other Subjects

Figure: Figure 17: Distribution of nonlinear encoding in the Brain of Subject 1.
Refer to caption: extracted/5954451/Figure5_S1.jpg

Figure: Figure 18: Distribution of nonlinear encoding in the Brain of Subject 5.
Refer to caption: extracted/5954451/Figure5_S5.jpg

Figure: Figure 19: Distribution of nonlinear encoding in the Brain of Subject 7.
Refer to caption: extracted/5954451/Figure5_S7.jpg

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: I have described my questions and suggestions above in the Weaknesses section. Beyond merely validating that the assumptions of LinBridge hold and that the technique slightly raises R2 scores across some parts of cortex compared to linear mappings, I would strongly recommend that the authors perform additional analysis to convey that LinBridge can provide more rich and detailed forms of insight into visual representation, over and above a linear encoding model applied to the same dataset.
soundness: 3
strengths: - The authors take seriously the need to balance having mappings that are maximally predictive, and also interpretable. These two desiderata are naturally in tension, so I applaud that they developed a novel approach for tackling this challenge that yielded some sensible results.
- The authors do a solid job citing recent literature from the encoding model/mapping literature, with only a few key omissions. The work is certainly timely. 
- I applaud the authors' use of NSD in this study - it's the data resource most naturally suited to developing new mapping approaches like these, and is substantially higher quality and larger scale than the datasets that were used in the papers by Zhang, Li, and Cui in section 2.2. 
- The comparison between the 2-layer linear and non-linear models is a nice clear setup for validating their proposed approach. 
- Using the factorization of the Jacobian to study to which voxels' responses are linear vs. nonlinear (using the AFD metric) is the most interesting part of the paper in my opinion. It's a clever application of their framework that could be more broadly applicable. I would like to see the AFD analyses extended in a revised version of this work.
summary: The authors introduce LinBridge, a framework for non-linear mapping between neural network activations and fMRI data (here, the Natural Scenes Dataset), with the goal of enhancing interpretability by factorizing the mapping into (i) a linear component that approximates the nonlinear relationship between source and target, along with (ii) a bias that captures the idiosyncratic aspects of the mapping that are "sample-specific" (given that in a non-linear encoding model, distinct samples may be transformed differently depending on their features). The authors motivate the need for such an approach by pointing out that the brain computes complex non-linear functions of the input, and thus, it is likely the case that non-linear mappings could provide better predictivity than linear mappings. 

LinBridge works by leveraging Jacobian matrices, which capture the model's sensitivity to input variations. It separates the consistent linear component, JMinherent, from sample-specific nonlinear biases, ΔJM, using a CNN to compress the Jacobian matrix and extract meaningful structure. LinBridge then applies contrastive learning with InfoNCE loss to maximize alignment between consistent mappings and minimize nonlinear bias effects. 

Using this approach, the authors show that that a nonlinear encoding model applied to NSD yield a greater percentage of voxels with significant encoding levels than a linear encoding model with the same 2-layer architecture, but no relu non-linearity. LinBridge successfully extracts a stable linear component from nonlinear models, which achieves comparable performance across visual regions. 

Additionally, the authors use this framework to assess the degree to which a given voxel responds linearly with respect to the input samples.  They do this by measuring how the low-dimensional embedding ΔJMdown varies across different input samples. Then for each voxel, a first-degree polynomial is fit to the response values from ΔJMdown across an array of ordered samples. The coefficient of the first derivative from this linear fit reflects how much the voxel's response changes with different inputs. If the voxel's responses show minimal variation (i.e., the absolute first derivative - AFD - approaches zero), it indicates a more linear response; higher values imply stronger nonlinearity. Using this AFD metric, the authors present evidence that a greater proportion of visual voxel responses are non-linear along the visual hierarchy, from primary visual cortex to high-level visual cortex where category-selective regions are located.
weaknesses: I had several concerns with the paper's introduction and framing, which I will describe below. Beyond the framing of the paper and the way the authors discuss linear vs. nonlinear mappings, my main critique is that the validation and application of LinBridge did not provide much new scientific insight. I feel that the approach is mostly sound, but the impact of the paper is strongly limited by the fact that the authors do not do enough to show that LinBridge can provide new scientific insight into the nature of brain representations, to held build better theories. Relatedly, it also appears that the overall complexity and computational requirement of the approach is dramatically higher than linear encoding procedures. Based on the authors' results, it remains unclear to me whether (a) the added complexity of the procedure provides enough added predictive power to justify using LinBridge, and (b) whether factorizing the linear and non-linear components of the mapping actually provides the sort of "interpretability" that we seek in the field today. Folks who study explainable AI / mech-interp would use this phrase to refer more to understanding the nature of the features that are represented in DNNs and in the brain. The authors did not convince me that LinBridge gets us closer to this goal. For these reasons, I am unable to rate the paper higher than a 3, though I would be willing to revise my score upward with sufficient revision. 

Comments on framing: 

- The authors missed some important literature on linear vs. nonlinear mappings, such as [Ivanova et al. 2022](https://www.biorxiv.org/content/10.1101/2021.04.02.438248v3). This line of work should be cited. There's also an important recent [review](https://arxiv.org/abs/2310.13018) of mapping methods in neuroAI that the authors should cite. Their approach and findings should be meaningfully situated in the context of these two papers, since these more closely reflect the current state of thinking in neuroAI around this topic. 
 
- The logic in Section 2 was a bit clunky. The authors critique linear encoding models by saying "the inherent nonlinear dynamics of neural activity limit the predictive power and interpretability of linear models". And later: "especially ... in higher-order cortical areas, [the] underlying neural mechanisms may not be adequately captured by linear representations". But, the linear encoding papers they cite nearly always fit the weights on top of deep and highly nonlinear DNN backbones. In other words, the authors misrepresent the typical use of linear encoding, giving the impression that the entire mapping function is linear. The more targeted and appropriate way to introduce this dichotomy would be: assuming some frozen feature extractor backbone, the trainable weights that map to brain data can either be an additional linear layer or an additional multi-layer function with some non-linearity. 

- Another comment on section 2: the evidence the authors cite in favor of nonlinear models is insufficient to justify the blanket claim of "superior performance" compared to linear models. Beyond the Zhang, Li, and Cui papers, are there any that use more recent datasets (especially NSD) that directly compare nonlinear and linear mapping functions, and show the superiority of the former? 

- Relatedly, because nonlinear functions are so much more expressive, they require much more data to fit, and often, stronger forms of regularization. For many datasets in the field, linear models may actually provide superior performance (in terms of generalization to held-out data), because they are less prone to overfitting. NSD is a great dataset for the purposes of studying nonlinear mappings, but the conclusions derived from NSD may not generalize to scenarios where fewer datapoints are available for fitting complex nonlinear functions.  This is one of the many reasons that the appropriate mapping method necessarily depends on one's research goals (see Ivanova et al. 2022) and also on the specifics of one's dataset - the blanket claim that non-linear models are inherently superior because of their computational expressivity is not clearly justified given the limitations inherent in most neuro datasets.

Comments on clarity, figures, results: 

- The "sample-specific" idea was described in a vague way in the abstract and introduction - the authors should more clearly convey what this means. If I understand correctly, it refers to the following idea: say you have a vector of DNN activations in response to an image. No matter what that vector contains, the linear weights will apply the same transformation to it. However, in the presence of a nonlinearity and a multi-stage nonlinear mapping, the nature of the transformation post-nonlinearity will depend on the specific content of the original input vector. The logic here should be spelled out more clearly. 

- The authors repeatedly refer to an "LLM" in their figure schematics, but the input is always an image, and the brain data they are using is visual. This could be a typo?

- The figure 2 caption did not do an adequate job explaining the complex methods schematic. 

- Figure 3 shows that a greater proportion of voxels achieve p > 0.05 encoding, but that threshold appears to be quite low - the gains from LinBridge over a linear model seem to mostly involve the voxels with weakest signal. A positive interpretation is that the non-linear function somehow has a denoising effect, allowing us to predict some bits of cortex that were too noisy to be fit with a linear model. But, I am troubled by an alternative interpretation: what if these voxels with significant predictivity at the left tail of the blue distributions in Figure 3 arise because the nonlinearity is essentially just modeling noise structure in the data? With only 3 trials available, averaging will be insufficient to fully eliminate noise, and sources of signal and noise variability are possibly correlated in NSD and many other datasets. Is there any analysis that would convince the reader that these voxels with R2 < 0.1 but significant predictivity actually carry useful signal that can teach us something new about visual representation?  On this topic, I feel it would be more interesting if the LinBridge approach conferred better predictivity in the right tail of the distribution, suggesting that even in high-SNR voxels, there are non-linear components that linear models cannot explain. But, this does not seem to be the case based on Figure 3.  At minimum, the authors should provide a graph that quantifies the performance of linear vs non-linear mappings across a range of R2 thresholds spanning from 0.05 to 0.6 or so. 

- While the results presented in Figure 5 seem to align with past work and suggest that higher-level voxels show more nonlinearity in their responses than earlier voxels, I wonder if this is more of a sanity check than a relevant scientific finding? Wouldn't the null hypothesis be exactly this, under the assumption that feedforward visual processing implements successive nonlinear transformations of the representations that begin in V1?  The authors could do more to convey the importance of this finding - perhaps I am not fully understanding the implications.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See previous section.

Additional questions:
1. Figure 1 does not effectively illustrate the sample-specific characteristics and structural instability of the nonlinear encoding models, despite the authors' claims.
2. The method is not clearly described. For example, LLM in Figure 2 has not been defined in the Method. Is it CLIP-ViT? CLIP-ViT is not LLM model, but multi-modal visual-language model.
3. Figure 3 compared LinBridge nonlinear encoder with linear encoder in terms of fMRI prediction R^2. It is a non-fair comparison since nonlinear model has better fitting capacity. I would suggest comparing LinBridge with another nonlinear model with contrastive learning, for example, CEBRA, as well as the model in [1].
4. What are the features corresponding to the linear responses? Would the features change between using the linear model and LinBridge model? 
5. What are sample-selective nonlinear responses? Can you please visualize it? 

Ref
[1] Wang, A. Y., Kay, K., Naselaris, T., Tarr, M. J., & Wehbe, L. (2023). Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5(12), 1415-1426.
soundness: 2
strengths: Using Jacobian matrices (JM) to quantify the complex mapping relationships in non-linear encoding models is interesting. The JM_{inherent} and \Delta JM is conceptually reasonable.
summary: The paper presents a LinBridge, a flexible framework to extract both the linear inherent component and the nonlinear mapping biases from nonlinear encoding models. However, the performance of LinBridge in terms of fMRI prediction and interpretable visual feature extraction should be fully validated. And the hierarchical nonlinearity within the visual cortex is not convincing.
weaknesses: 1. The LinBridge relies on an assumption that the nonlinear mapping between ANN representations and neural responses can be factorized into a linear inherent component that approximates the complex nonlinear relationship, and a mapping bias that captures sample-selective nonlinearity. This assumption is rather strong. Any evidence? 
2. The computational cost of Jacobian is not reported. 
3. Experiments are not rich. More model comparisons are needed.
4. This paper claims that the variability of nonlinearity across different levels of the visual processing hierarchy. However, the evidence of the distribution of hierarchical nonlinearity within the visual cortex is rather weak and highly dependent on the nonlinear embedding. The method should be fully validated before applying it to neuroscience discoveries. Moreover, evidence from previous neuroscience studies/experiments is needed to support this statement.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. As far as I can tell, the non-linear encoding model doesn't do much better than the linear one in terms of R^2. It does predict more voxels though. Is the fact that you see more "non-linearness" in the mapping in later brain regions of the hierarchy due to the fact that the linear model cannot predict many of those voxels? Put another way, could a simple comparison between the R^2 given by a linear model vs the non-linear model, voxel-by-voxel, reveal similar profiles i.e. linear model does worse in certain brain areas and therefore the mapping needs to be non-linear?
2. What does the mapping being non-linear entail? You are using CLIP-ViT to encode responses of voxels across a huge chunk of the brain. Is it expected that a linear map should be possible in the first place? The brain presumably does way more things than just encode visuo-semantics the way CLIP-ViT does. I would've predicted that the mapping would be more linear in later visual areas as they are more semantic - turns out that's not the case. What gives, according to you? Q1 probably gives hints - does the linear model do much worse in high-level visual cortex? If yes, then possibly you're onto something super interesting.
soundness: 2
strengths: Intepreting non-linear encoding models is critical because they do tend to outperform linear encoding models as the paper shows. This paper seems to take a step in that direction.
summary: This paper seeks to interpret non-linear encoding models, showing that the "non-linearity" of the mapping CLIP-ViT representations to NSD responses in the brain increases with the brain regions' ranks in their hierarchy.

Disclaimer: I didn't understand the central use of J_inherent. I could not understand what exactly the authors mean by it - a formal definition is absent; Hereforth, I'm intuiting that it is somehow a measure of linearness in the mapping.
weaknesses: As I mentioned before, the most critical element J_inherent isn't formally defined - making it very hard for me to understand what's happening next (although I can follow the intuitions).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: See specific questions in weaknesses, in particular referring to Fig. 3-4, and to the details of computing the Jacobian.
soundness: 3
strengths: The problem addressed here is important. Learned nonlinear encoding models are necessary to predict brain activity well, but they are difficult to interpret and therefore offer limited understanding of brain computation. The proposed solution is conceptually simple and, to the extent tested in this manuscript, effective. The method combines well established ideas including the Jacobian to characterize local linear behavior, low-dimensional embeddings, and contrastive learning. The pseudo-code in the Appendix also highlights the simplicity of the method. With some exceptions noted below, the manuscript is written clearly.
summary: The manuscript proposes a method called LinBridge to interpret non-linear encoding models of brain activity (models that predict the output response, given an input stimulus), by distinguishing the global Jacobian matrix (the linear part of the model) and local deviations from it around each single input presented in an experiment (the ‘biases’ that characterize the nonlinearity). Given a pre-trained encoding model, LinBridge uses contrastive learning on low-dimensional embedding of the Jacobian matrices. The manuscript demonstrates the method by first training a simple nonlinear model (a 2 layer network with sigmoid nonlinearity, and also a control linear model), on a large fMRI dataset of brain responses to images; and then applying LinBridge to the trained model. 

The main results are 1) The non-linear model, despite its simplicity, predicts neural activity better than the linear model. 2) In the non-linear model, the linear global component characterized by LinBridge is almost as predictive as the full model. 3) The local ‘biases’ add more predictive power for higher visual cortex than lower visual cortex, consistent with general knowledge that higher visual cortex computes a more nonlinear representation of the image.
weaknesses: The nonlinear encoding model is very simple and, not surprisingly, its predictive power is relatively small (how does it compare to other, previously published non-linear models for this dataset?). Although maximizing predictive power is not the goal of this manuscript, if the nonlinearity of the model is too simple, it seems not that surprising that the simple decomposition of LinBridge will work well. 

In other words, how much does the nonlinearity really help this particular encoding model? Figure 3 is confusing in this regard: panels a,b look indistinguishable to me (sorry, I am not that familiar with fMRI data). And panel c shows that more voxels are activated for nonlinear, but the average r-square seems even lower for nonlinear than linear (linear histograms are shifted to the right). I suspect I am not reading the plots the right way, some guidance would help the reader. 
Then figure 4 shows that the encoding performance with the full nonlinear model is almost identical to the linear component extracted by LinBridge. Doesn’t this say that the nonlinear part of the model is not that useful? The analysis of Figure 5 partly addresses that concern, showing that the nonlinear biases are more predictive in higher visual cortex. 

Clarity: the computation of the Jacobian matrix JM should be explained clearly (not the linear and Delta terms, those are clear enough). It is the central idea of the paper, and the authors also list as a limitation that it is resource intensive, but it is not explained at all. It is conceptually simple, but I think the practical details might be important.
I found Figure 1 not informative. If the point is to convey that nonlinear mappings are characterized by the fact that they vary around different inputs, that is not at all conveyed by the image.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - The description of the encoding model is not complete. Where the same intermediate features of a CLIP-ViT used to predict all voxels? 

- in Fig. 2, the embedding extraction is denoted as "LLM" in the graphic. Is this supposed to represent the CLIP model?

- It is not clear why the contrastive training is needed. Could the authors provide further motivations why this strategy is meaningful here?
soundness: 2
strengths: The authors present an overall very clearly written paper. The problem statement, the approach and the performed analysis are well described. The approach is promising, but is severely limited by weak evaluations.
summary: The authors presents a novel encoding model architecture designed to disentangle linear from nonlinear relationships between visual representations and neural responses. The authors demonstrate the framework's utility through its application to fMRI data an successfully reveal hierarchical patterns of nonlinearity in the cortical visual processing stream.
weaknesses: Despite the clear presentation and interesting ideas, the paper is severely limited by the weak evaluations and missing baselines. The model performances of both the linear and non-linear model are not only very similar, but seem to be fairly low overall, casting doubt on the validity of the approach. 
Considered as a whole, there needs to be a substantial revision that addresses these weaknesses, which would improve this paper significantly.

*Major Concerns*:
- I'm not convinced of the validity of the overall approach. The authors claim that their approach allows disentangle the contributions of linear and nonlinear processing in the visual hierarchy. For this, they use a highly nonlinear feature extractor (CLIP-ViT) and then combine these features either linearly or non-linearly to predict responses in visual cortex. This method is severely limited in its explanatory power because it so crucially depends on the chosen non-linear feature extractor. A large model like a ViT very likely has an overcomplete basis, such that the differences in linearly or non-linearly combining these features will be very small. There is evidence for this in the nearly identical evaluation metrics between these two models. 

- The model performance overall seems very week, by visualizing the voxel-wise predictive performance, it seems that many voxels can't be more accurately predicted as compared to chance level. On fMRI data of the visual cortex, relatively simple LNP-models usually perform better than chance level.

- Because the models are so similar, it is crucial to compare that against a baseline with a different architecture and a different training paradigm to distinguish the contrastive training from a simple regression-based neuronal response fitting.

- The approach could further be studied more comprehensively by validating it on toy-data, such as combinations of gabor-like V1 simple and complex cells and non-linear combinations of these. There, it would be expected that LinBridge is able to disentangle the linear from non-linear components.


*Minor Concerns*:
- the authors write in the introduction: "We apply LinBridge to a neural encoding exploration of vision transformer models and reveal the variability in nonlinearity across different levels of the visual processing hierarchy". This does make it sound as if the approach was used to study intermediate representations of vision transformer models. However, such as analysis was not performed and is misleading.

Meta-review: This paper introduces a novel framework for characterizing and interpreting the relationships between artificial neural networks and fMRI imaging data.  The reviewers praised the paper for proposing a conceptually simple and effective solution to a timely and important problem. Unfortunately, however, they were not persuaded that it provided a rigorous or thorough enough set of evaluations and comparisons to existing baselines, and raised concerns about the reliance on a pre-specified nonlinear feature extractor and the ability to provide new scientific insights.    I regret that the paper cannot be accepted to this year's ICLR, but I wish the authors the best of luck in revising it for publication elsewhere.

Decision: Reject