CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding
Authors: Qiongyi Zhou
Changde Du
Shengpei Wang
Huiguang He
Venue: ICLR 2024 poster (2024)
Keywords: Multi-subject visual neural decoding
representational similarity analysis
CLIP
transformer
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose a multi-subject visual neural decoding method based on Transformer with subject-specific tokens, of which the shared neural representations are learned under the guidance of CLIP.

Abstract: The study of decoding visual neural information faces challenges in generalizing single-subject decoding models to multiple subjects, due to individual differences. Moreover, the limited availability of data from a single subject has a constraining impact on model performance. Although prior multi-subject decoding methods have made significant progress, they still suffer from several limitations, including difficulty in extracting global neural response features, linear scaling of model parameters with the number of subjects, and inadequate characterization of the relationship between neural responses of different subjects to various stimuli.
To overcome these limitations, we propose a CLIP-guided Multi-sUbject visual neural information SEmantic Decoding (CLIP-MUSED) method. Our method consists of a Transformer-based feature extractor to effectively model global neural representations. It also incorporates learnable subject-specific tokens that facilitates the aggregation of multi-subject data without a linear increase of parameters. Additionally, we employ representational similarity analysis (RSA) to guide token representation learning based on the topological relationship of visual stimuli in the representation space of CLIP, enabling full characterization of the relationship between neural responses of different subjects under different stimuli. Finally, token representations are used for multi-subject semantic decoding. Our proposed method outperforms single-subject decoding methods and achieves state-of-the-art performance among the existing multi-subject methods on two fMRI datasets. Visualization results provide insights into the effectiveness of our proposed method. Code is available at https://github.com/CLIP-MUSED/CLIP-MUSED.

Published as a conference paper at ICLR 2024
CLIP-MUSED: CLIP-GUIDED MULTI-SUBIECT VI- SUAL NEURAL INFORMATION SEMANTIC DECODING
Qiongyi !2, Changde Du', .hengpei Wang' and Huiguang He!
II Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences Uniiversity of Chinese Academy of Sciences Corresponding Author chougiongy1@hotmaiil. com, (changde du, sangshengpei2012 huiguang. he @ia.laa.aom ac.en
ABSTRACT
The study of decoding visual neural information faces challenges in generalizing single-subject decoding models to multiple subjects, due to individual differences. Moreover, the limited availability of data from single subject has constraining impact on model performance. Although prior multi-subject decoding methods have made significant progress, they still suffer from several limitations, including difficulty in extracting global neural response features, linear scaling of model parameters with the number of subjects, and inadequate characterization of the relationship between neural responses of different subjects to various stimuli. To overcome these limitations, we propose CLIP-guided Multi-sUbjeet visual neural information SEmantic Decoding consists of Transformer-based feature extractor to effectively model global neural representations. It also incorporates learnable subject-specific tokens that facilitates the aggregation of multi-subject data without a linear increase parameters. Additionally, employ representational similarity analysis (RSA) guide token representation learning based on the topological relationship of visual stimuli in the representation space of CLIP, enabling full characterization of the relationship between neural responses of different subjects under different stimuli. Finally, token representations are used for multi-subject semantic decoding. Our proposed method outperforms single-subject decoding methods and achieves state- of-the-art performance among the existing multi-subject methods on two fMRI datasets. Visualization results provide insights into the effectiveness of our proposed method. Code available https//httubbcom//CPPMMMDDDDDDD MUSED
method. Our method
INTRODUCTION
In recent years, researchers have made significant progress visual neural decoding tasks, allowing for the deciphering of semantic information from brain activities in response to visual stimuli (Akamatsu et all 2020: Li all 2022: Bagchi Bathula, 2022). However, due to individual differences, most decoding models are trained separately each subject. Single-subject models are susceptible to overfitting due to the limited data available for each subject as a result of constraints in data acquisition. Furthermore, single-subject models exhibit weak generalization performance new subjects. In contrast, multi-subject decoding methods can aggregate data from multiple subjects, mitigate overfitting issues, and achieve superior performance across different subjects. Thus, it is of great value to investigate multi-subject neural information decoding methods.
Individual differences manifest in both the anatomical structure and functional topography of the brain (Haxby et all 2020). While image registration techniques can eliminate anatomical differences. differences in functional topography persist, including variations in the size, shape, and location of

---

Published as a conference paper at ICLR 2024
Voxel space
Shared space
Stimulus representation space
Magpie
S1
My
R
Gridance
f Parrol
$2
My N3 Ka
f, Cat
Hyperalignment
Building
Category-hasel mothed
blocks
Our method
Figure Diagram of the different multi-subject functional alignment methods.
functional areas on the cortical surface. Therefore, aligning the functional topology of different subjects poses critical challenge in multi-subject decoding research.
Most multi-subjec decoding methods are based on hyperalignment (Haxby et all 2020: Chen all 2015), classic method for functional alignment. As shown in Fig. IL subjects S and S₂ view four stimuli (t, 14), which are represented as high-dimensional vectors in the voxel space of each subject. Hyperalignment transforms neural responses from the voxel space of each subject to shared space through the learning of mapping function Ry- It aligns the functional topology structure of different subjects by bringing together the neural representations of different subjects under the same stimulus in the shared space, illustrated by the same-colored balls in Fig. M. Nonetheless, hyperalignment cannot handle the common scenario where different subjects view different stimuli during data acquisition. To tackle this challenge, Li et all (2I21) proposed category-based functional alignment approach. As shown in Fig. M. this method pulls together the neural representations of different subjects in the shared space that relate the same category of stimuli (two bird images t and f22).
However, these multi-subject decoding methods still have three limitations:
1. The mapping function R; has restricted expressive power. Linear transformation discussed by Yousefnezhad & Zhang (2017) and MLP composed of stacked linear layers used by Yousetnezhad & Zhang (2017) are unsuitable for high-dimensional voxel responses. To overcome this limitation, Chen all (20T6 proposed CNN-based hyperalignment algorithm. However, CNNs face challenge in capturing global features that reflect the long-range functional connectivities between brain regions.
2. Current studies require learning distinct mapping function R, for each subject. As the number subjects increases, the number of model paramcters will also increase linearly, leading to a considerable increase in computational complexity.
3. Existing methods have not fully characterized the relationship between the neural responses of different subjects under similar stimuli. However, research has demonstrated that different subjects exhibit similar neural responses when presented with semantically analogous visual stimuli (Huth all 2012; Connolly all 2012: Carlson all, 2014 Zhang et all 2020). For instance, in Fig. II t and to are birds, while t. (cat) and t4 (building blocks) are not. Additionally, birds and cats both belong to the animal category, whereas building blocks are inanimate objects. Therefore, in the shared space, the representation of l3 should be more proximate to the representations t1 and t2 than that of t4.
To address the aforementioned issues, we propose the following solutions:
1. Transformers possess the ability capture long-range dependencies. Therefore, design Transformer based network to learn the mapping function and model the relationships between different brain regions.
2. Transformers can flexibly add tokens that learn specific knowledge. Prior research has introduced diverse inductive biases into Transformers through extra tokens (lauvron et all, 2021: Naseet stall 2021). Inspired by these studies, introduce subject-specifer tokens into the Transformer model in this paper. These learnable tokens encode individual differences, allowing other parameters to be shared across subjects.
2

---

Published as a conference paper at ICLR 2024
Hightlosel space
High-level Mar
Aggrepation sutjoct shured
Mi
Chasifier
regresentations
Teat eseoder
Insage crasdder
Guidknce by Mli
Last luyer
Last jayer
Low-Jevel
Midkle layass
Midille layers
space Title
Louslovd fenture
Anite
RSM Mi
L
Like
Fint hyer
Fint layer
Tranofomerr Casuur extracios
Transferner Gature outracie
by Mi
Highseeu Token of S1
Low-level Token ofS1
Highelevd
Text information
Visual stimuli
potches
Taken of S2
52 BOLD panches
Token of
Low-level und high-level fenture RSM visual stúmuli
IMRI representation learning guided by visual stimulus Feature RSM
obtained from CLIP
Figure 2:
The framework of the proposed method. Left: Low-level and high-level feature RSM of visual stimuli are obtained from CLIP first. Right: The Transformer-based encoder extracts multi-subject shared neural representations guided by the visual stimulus feature RSM.
3. Previous studies (Zhou all explanatory power for neural responses in the ventral visual pathway than single-modal DNNs, indicating high consistency between CLIP and cortical representations of visual stimuli. Thus, the topological relationships of visual stimuli in the CLIP representation space can serve as prior knowledge to guide neural representation learning the shared space.
202)) have shown that CLIP (Radiord et al. 4021 has stronger
In summary, we propose CLIP- guided Multi-sUbject visual neural information SEmantic Decoding method a Transformer-basee fMRI feature extractor to map the neural responses of each subject from the original voxel space to the shared space. We further divide individual differences into two categories: differences in the processing patterns of low-level features (such as shape and color) and high-level features (such semantic categories) of visual stimuli. To encode these two types of differences, we incorporate a low-level token and a high-level token for each subject into the Transformer architecture. The proposed method uses the topological relationships of visual stimuli in the shallower and deeper layers of CLIP to guide the representation learning of low-level and high-level tokens, respectively, through representational similarity analysis (RSA). To ensure that the low-level and high-level token representations of the same subject encode as much different information as possible, we impose an orthogonal constraint on two representations inspired by the previous study ct 2019).. Given that both low-level and high-level features play critical roles semantic classification, we concatenate the low-level and high-level token representations of each subject for semantic classification.
as illustrated in Fig. Q The proposed method leverages
The contributions are summarized follows.
1. We propose Transformer-bast fMRI feature extractor that can efficiently extract global features of neural responses.
2. CLIP-MUSED learns low-level and high-level tokens for each subject and shares the other parameters across subjects. The method can applied to multiple subjects without linear increase of parameters.
3. With the help of RSA, CLIP-MUSED utilizes the topological relationships of visual stimuli the CLIP representation space to fully characterize the relationship between neural representations under different stimuli for different subjects.
4. The experimental results on two fMRI datasets demonstrate that CLIP-MUSED surpasses Lingle-subjed decoding methods by aggregating more training data and reducing individual differences. Our method also achieves state-of-the-e art performance among the existing multi- subject methods.
3

---

Published as a conference paper at ICLR 2024
2 METHODOLOGY
2.1 OVERVIEW
Given neural dataset with neural activities of N subjects under visual stimuli, let 2 denote the voxel space. X(n) E Rd. is the neural responses of the n-th subject, where n, is the number of image stimuli and d, is the number of voxels. Let I and Y denote the pixel space and the label space of the image stimuli, respectively. Our goal is aggregate the neural responses of the N subjects to train classifier C' x()1 X x X²(N) + y. To achieve this goal, propose CLIP-MUSED that first maps the image stimuli from Z to the representation space of CLIP. Then, a Transformer feature extractor is used to map the neural responses from x² to a shared space Z. Finally, RSA is employed to guide the representation learning of Z using the topological relationships of the visual stimuli in F.
2.2
CLIP-BASED FEATURE EXTRACTION OF VISUAL STIMULI
In previous study (Zhou al, 2U2E it shown that CLIP (Radtord et all 20241) outperforms various single-modal DNNs in explaining cortical activity and that there is a hierarchical correspon- dence between CLIP and the human ventral visual pathway. Based these findings, we CLIP to extract features of visual stimuli in our method. However, it is worth noting that other DNN feature spaces could also be used in our approach.
CLIP comprises an image encoder and a text encoder, as shown in Fig. D. In our method, we input visual stimuli along with corresponding textual information (either textual descriptions or label names) into CLIP to obtain multi-modal features. Due to the hierarchical architecture of CLIP, we use the first-layer features of the image encoder as the low-level feature fit.. while the average of the image and text features from the last layer of CLIP is used as the multi-modal high-level feature faie. We compute representation similarity matrices (RSMs), MI. and M to quantify the similarity between B visual stimuli in low-level and high-level feature spaces, where B denotes the batch size in a mini-batch. Specifically, Ml. [i, j] and M ji represent the cosine similarity between the ith and jth images in the feature spaces Fule and Fatv, respectively.
2.3 TRANSFORMER BASED FMRI FEATURE EXTRACTION
The feature extraction process of the conventional Transformer can be formulated as follows:
Zo Xclassi x E; :xM + Eposs
(1)
z': MHSA (LN (zl-1)) Z1-1,
1= 1,2, ...
(2)
Z: MLP (LN (z'1)) +2'1.
[ = 1,2 ...
(3)
z = LN (2f))
(4)
First, the input data E Rd is divided into M equally sized patches, resulting in R.dd...
As shown in Eq. (., the patch embeddings of x are obtained by passing them through a linear embedding layer E Rd... Xdyut, The resulting patch embeddings are then concatenated with learnable class Token Xclass and fed into an T--layer Transformer encoder. To preserve the positional relationships in the original data, positional encoding Epos E R(M+1)) is used. As shown in Eqs. (2) and (E), each layer consists of multi-head self-attention (MHSA) module with residual connections and a feed-forward (MLP) module. The features are layer-normalized (LN) before being imput into the MHSA and MLP modules. As shown in Eq. (8), the model applies layer normalization to the variable zy, and generates the final representation
To ensure that our model applicable to different subjects, we design a Transformer- fMRI feature extractor with subject-specific tokens, depicted in Fig. &. In the input stage, we patchify the blood oxygenation level dependent (BOLD) signals at first. There are two ways to patchify the BOLD signals. The first method involves patchifying the data based regions of interest (ROIs), while the second method directly patchifies the 3-D BOLD volumes. Fig. ((a) illustrates the second method. Due to the high dimensionality of the BOLD volumes, directly patchifying them can result in large number of patches, which leads to high computation cost. To address this problem. first reduce the dimensionality of the BOLD volumes using 3D-CNN, and then patchify and
4

---

(n) level token Xil and high-level token Xite of the n-th subject. These tokens are then combined with Published as a conference paper at ICLR 2024
Semantic decoding
BOLD patches
Transformrt encoder
Flantn
LN
MI.P
Patchify
LN
Transformer encoder
Mult- Self Attentinn
Low-level Token
High-level
3D-CNN enooder
Linezr embsedding
LN
BOLD pasches
ROL ID volume
Patch embekkings
(n)
(b)
(c)
Figure 3: Transformer-based fMRI feature extractor of CLIP-MUSED, (a) Conversion of BOLD signals in Volume format to BOLD patches. (b) Flowchart of the feature extraction process. (c)
Network structure of the Transformer encoder.
flatten the feature volumes as BOLD patches. The framework of the Transformer-basee feature extractor is shown in Fig. Bbb). Distinct from the conventional Transformer, the model includes learnable subject-specific low-level and high-level tokens Xliv and Xhiv- The network structure of the Transformer encoder in Fig. (Kb) is illustrated in Fig. B(c). Each layer consists of residual MHSA module and a feed-forward module. The feature extraction process of the Transformer-based multi-subject fMRI feature extractor be formalized follows:
Z0 Xits Xitss x E; .... ; EE
poss
(5)
z'1 MHSA (LN (2-1)) + 21-1,
1 1 =,,,, 1,2,...,1.
(6)
2 = MIP (LN (2' ()) + 21-1,
1 = I.
(7)
Zite = LN (22).
(8)
Zhle LN (z))
(9)
As described in Eq. (6) and Eq. (7)) the low-level and high-level tokens interact with different BOLD patches, eventually forming low-level and high-level neural representations denoted as Zito and Zaie- To extract features from the neural signals (n) of the r-th subject, CLIP-MUSED calls the low-
the BOLD patches and fed into the model for feature extraction.
2.4 MULTI-SUBJECTT SHARED NEURAL RESPONSE REPRESENTATION
To capture the distinct processing patterns of visual stimuli across different subjects and encode them into the low-level and high-level token representations (Zute Znte), CLIP-MUSED leverages the topological relationships among visual stimuli in the feature spaces Fuo and Fuiv of CLIP to guide the representation learning process.
The guidance is realized by representational similarity analysis (RSA). First, B neural signals randomly sampled to obtain the low-level and high-level representations (Zites 2ntv). Next, we compute the representational similarity matrices (RSMs) M Me MX RBxB for the low-level and high-level representations Zite and Zutus respectively. Here, Mx.li,)) represents the cosine similarity between Zuo[i] and Zuw[i] During training, we shuffle the samples of all subjects and randomly sample from them. In mini-batch, Zuo [i] and Z [j] may come from different subjects. The topological relationships of visual stimuli in the multimodal feature space and the shared space are aligned by minimizing the squared F-norm of the difference matrix normalized by the matrix size, i.c..
Lue = || - Mx /B²,
(10)
Lute = || MX 1I²
(11)
5

---

Published as a conference paper at ICLR 2024
In addition to RSA, we also establish the association between visual stimuli and neural signal representations using a mapping network. However, RSA-based method is superior to the mapping- based method. We offer detailed explanation of why we chose RSA in Section E of the supplementary materials.
2.5 SEMANTIC CLASSIFIER
Both low-level visual features and high-level semantic features are crucial for semantic classification. To leverage both types of features, the low-level and high-level token representations concate- nated and fed into an MLP network for classification. The model outputs predicted probabilities y. The semantic classification process be formalized follows:
CONCAT (ZIe, Zhiv);
(12)
y MLP(z).
(13)
The cross-entropy loss function is used as the classification loss.
C
L.
My log((,)) (1 y,) log(1 -1,)
(14)
C
1=1
2.6 OPTIMIZATION OBJECTIVE
To encourage low-level and high-level token representations for each stimulus to differ much as possible, the proposed method applies an orthogonal constraint
min Ci = || 1 B
(15)
The optimization objective of the method
min C = L. + A_C1 + AutCCi + Antu
(16)
where ' Auts Anto are trade-off parameters.
3 EXPERIMENTS
3.1 DATASETS
HCP (Glasser caal 2013 Van Essen al. 2012): ): This dataset is part of the Human Connectome Project (HCP), containing BOLD signals from 158 subjects. To reduce computational demands, we randomly select nine subjects for subsequent experiments. The visual stimuli consist of four dynamic movie clips, each annotated with an 859-dimensional WordNet label (Miller. (1995). The top 53 categories with frequency higher than 0.1 are selected for subsequent experiments.
NSD (Natural Scenes Dataset) (Allen all 2022):: The dataset contains BOLD signals from eight subjects. The visual stimuli consist of natural images from the MSCOCO dataset et 2014), each with multiple labels from 80 categories. Each subject viewed total of 10,000 stimuli. Unlike subjects in HCP, subjects in NSD viewed different stimuli. Of the 10,000 stimuli, 9,000 have no overlap between subjects, while the remaining 1,000 stimuli were presented to all subjects. However, some subjects did not complete all sessions, and some trials are not publicly available. For this study, the number of stimuli per subject is approximately 9,000.
detailed description on how we preprocess the data to adapt the input form of Transformer, split the dataset, and obtain the multimodal features of each stimulus is provided in Section A of the supplementary materials.
3.2 BASELINE METHODS
To validate the effectiveness of the proposed method, we compare it with single-subject decoding methods and existing multi-subject decoding methods.

---

Published as a conference paper at ICLR 2024
Single-subject decoding methods: These methods using single-subject neural signals input and classification loss as the constraint. SS-MLP, SS-CNN, and SS- employ MLP, 3D-CNN, and ViT the backbone networks, respectively. Specifically, SS-MLP and SS-CNN are used for the NSD and HCP datasets, respectively.
Multi-subject data aggregation methods: These methods train single decoding model for all subjects by direct data aggregation. We refer to these methods as MS-SMODEL-MLLP MS- SMODEL-CNN, and MS-SMODEL-ViT, which use MLP, 3D-CNN, and ViT as the backbone networks, respectively.
MS-EMB (Chehab et all 20ZZ): MS-EMB trains single model for all subjects by direct data aggregation. In contrast to IS-SMODEL, MS-EMB learns token for each subject to encode their identity. This approach is based on a method proposed by Chehab el all (2022) and uses ViT as the backbone network.
Shared response model (SRM) (Chen alll, 2015): SRM is probabilistic generative hyperalign- ment method. However, SRM cannot handle the case where subjects view different stimuli, so is only used on the HCP dataset. We use the open-source code available BrainIAK. We search the optimal dimensionality of the shared latent space in intervals of 100 dimensions within the range of [100, 600]. The optimal dimensionality is 500.
3.3 PARAMETER SETTINGS
For the HCP dataset, we first extract volume features using six-layer 3D-CNN (Dai all 2022) as shown in Fig. B(a). This results in features of size 7 x 8 x 7 x 512, which are then reshaped into features of size 392 512. Each 5 512-dimensional vector serves a patch for the Transformer. Afterward, CNN and Transformer layers are alternated. CLIP-MUSED employs two-layer Transformer. Note that all CNN and Transformer layers are trained together. The learning rate is set to 0.001, and the batch size is 64, and the optimizer is Adam. We find the optimal values for the hyperparameters L Antu, and Åmr by grid-search within the range of [0.001, 0.01, 0.1] and the best values are À 0.001, Anto 0.001, Au 0.1. The models converge after approximately three hours of training on one NVIDIA A100 GPU.
For the NSD dataset, the network structure of Transformer is similar to that of VIT (Dosovisskry et all 2021), with a patch embedding dimension of 512, 24 layers, and 8 heads for multi-head self- attention. The learning rate is find the optimal values for the hyperparameters A1, Aste and Au by grid-search within the range of [0.0001, 0.001, 0.01] and the best values are ' 0.001, Anto 0.001, An 0.0001. The models converge after approximately two hours of training on one NVIDIA A100 GPU.
0.0001, the batch size is 64, and the optimizer is Adam. We
3.4 EVALUATION METRICS
We employ three commonly used evaluation metrics the field multi-label classification: mean Average Precision (mAP), the area under the receiver operating characteristic curve (AUC) and Hamming distance.
3.5 COMPARATIVE EXPERIMENTAL RESULTS
Table MI presents the results on the HCP dataset. Firstly, our method outperforms two single-subject decoding methods, SS-CNN and SS- ViT. SS-ViT and our method have the same backbone models, yet our method achieves significantly better metrics than VS-ViT, highlighting the superiority of the efficient data aggregation strategy of our method. The comparisons of SS-VIT and our method on different subjects are shown in Fig. C5 the supplementary materials. Secondly, our method also highly competitive when compared to the other multi-subject decoding methods (MS- SMODEL-CNN, MS- SUSS-MOOEL-YEE VIT, MS-EMB, and SRM). Despite having the same training data and backbone models, our proposed method outperforms MS- MS-SMODEL- EIT on all metrics. This indicates that the subject-specific tokens employed in our method can handle individual differences well and are superior the simple aggregation of multi-subject data used in AIS-SMODEL-WI VIT.
The method effectiveness is further validated on the NSD dataset, where the stimuli in the training set for each subject completely exclusive. Table A presents the results, which show that
7

---

Published as a conference paper at ICLR 2024
Table 1: Performance of different methods on the HCP dataset. All the improvement of our method compared to other methods is significant (f-test, p < 0.05) except for those underlined, where the p-values have been
Table 2: Performance of different methods on the NSD dataset. All the improvement of our method compared to other methods is significant (t-test, p < 0.05), where the p-values have been corrected with the Holm-Boc holm-Bonferroni method for multiple comparisons.
corrected with the Holm-Bonferron method for multiple comparisons.
Methods
mAP
A
Hamming
Methods
MAPt
AUC
Hamminn 4
SS-CNN
.347 + 0011
.531 + .006 .351 + .017
SS-VIT
360 + 002
.556
.004
.333 + .021
SS-MLP
.258 + .005 .8541 = .001 .033 .001
MS MS-SMODEL
.342 + .001
.525
.000
.328
.091
SS VIT
.238 1 .005 .815 1: .008
.032 - .000
MS MOSDEELE
.367 + 003
.576 + 001
.310 + .014
MS SE-SMMDELLLLLPP
.150 + .005 .767 :: .006
.039
.001
MS-EMB
.368 4 .044
.572
.004
.305 .003
MS MS-SMLOELLVVT
.156 + .006
.755 + .014
.038
.001
SRM
.348 0002
.534
.003
.204 + .000
MS-EMB
.220 + 014 .825 + .030 .U35 = .001
OURS
373 + .002 5581
.008
.283 + .001
OURS
.258 017 .877 .021
.002
Table 3: Results of the ablation study on the NSD dataset. Our method outperforms the methods with other loss configurations significantly (t-test, p < 0.05), where the p-values have been corrected with the
method for multiple comparisons.
T Anle
Alle
mAP 1
AUC 1
Hamming 4
.201 + .006
.794 + .010
.036 + .001
.209 + .002
.819 + .009
.036 + .002
.215 + .012
.820 + .018
.036 + .001
.209 + .016
.824 + .017
.037 +: .002
.25 .017
.877
.021 .030 + .002
method outperforms the single-subject methods. SS-MLP and ViT The comparisons of SS- SST and our method on different subjects are shown in Fig. CG in the supplementary materials. With the same amount of data and the same backbone models, the aggregation methods are far inferior to our method. This is mainly because the MS-SMODEL methods sharing all the model parameters across all subjects are hard to handle both inter-subject variability and differences in stimuli distribution. Although MS EMB performs better than the aggregation methods, it is still inferior to the proposed method.
3.6 ABLATION STUDY
We conduct an ablation study on the NSD dataset, and the results are shown in Table E. The model performance is unsatisfactory when only model guidance applied without orthogonal constraints or when only orthogonal constraints are applied without model guidance. Using either low-level or high-level features for guidance with orthogonal constraints results in a slight improvement in model performance, but there still gap compared the model's performance with all three constraints. These results confirm the necessity of multimodal model guidance and orthogonal constraints on primary and high-level token representations. We also investigate the guidance effect of CLIP on SS ViT and the guidance effect of different DNNs. The results are shown in Table DE and Table EG in the supplementary materials.
3.7 VISUALIZATION
We visualize the attention maps of low-level and high-level tokens on the last layer of CLIP-MUSED model. The visualization results on the left hemispheres of HCP dataset are presented in Fig. M where we randomly select four subjects for presentation. Fig. (a) shows the attention maps for the low-level tokens, which are concentrated in the occipital lobe. Fig. Bkb) shows the attention maps of the high-level tokens, which more dispersed across the cortex, with strong attention allocated to the frontal, parietal, and temporal lobes. These results are in line with our expectations, as previous studies have demonstrated that the processing of low-level visual features mainly occurs
8

---

Published as a conference paper at ICLR 2024
in the visual cortex, while the processing of high-level semantic features involves the temporal, parietal, and frontal lobes (De Benecictis et all 2014: ef all 2008). Fig. ((c) shows the attention maps of the embedding tokens of MS-EMB. In contrast the token attention maps of
CLIP-MUSED, the attention maps of MS-EMB are smoothly distributed across the entire cortical surface, making it difficult understand which information is encoded in the tokens.
Subject
Subject
Subject
Subject
8x10-=
(A)
1x10-2
(c)
Figure 4: On the HCP dataset, attention maps of (a) low-level tokens and (b) high-level tokens of our method, and (c) attention maps of subject embeddings of the MS-EMB method, were visualized on the cortical surface for 4 randomly selected subjects.
The visualized results the NSD dataset displayed in Fig. ES of the supplementary materials. The results provide explanations for the superior performance of the proposed method.
4 DISCUSSION AND CONCLUSION
To address the issue of individual variability, we propose multi-subject semantic decoding method, CLIP-MUSED, We design a Transformer-basse encoder that can extracts global features of neural responses. The method introduces subject-specifie low-level and high-level tokens to encode individual variability. Based on the RSA, CLIP-MUSED guides the representation learning of tokens by the topological relationship of visual stimuli in the representation space of CLIP. On two fMRI datasets, the proposed method outperforms the single-subject methods and achieves state- of-the-art performance.
All parameters of CLIP-MUSED, except for the subject-specific tokens, are shared among all subjects. This allows the method to be extended datasets with hundreds of subjects. Moreover, CLIP-MUSED is applicable to situations where multiple subjects have different stimuli. Even when the stimulus images of different subjects are completely mutually exclusive, our method outperforms single-subject decoding models. Therefore, the proposed method has the potential train a foundation model with aggregating multi-source neural datasets, following the research trends in the computer vision and natural language processing communities (Radlord et all 2021 Brown ef all (020).
In the future, we plan to extend our method to visual stimuli reconstruction (Chen el all 2024: Lin all 2022: Takagi & Nishimoto, 2022), a more challenging task than semantic classification. Besides. approach can be applied to subjects if we devise strategies to learn subject-spede tokens for new subjects and adapt the model accordingly. Since the experimental workload are substantial, plan to carry this work in the future.

---

Published as a conference paper at ICLR 2024
5 ACKNOWLEDGEMENTS
This work was supported in part by the National Key R&D Program of China (2023YFF1205500); in part by the National Natural Science Foundation of China under Grant 62206284, Grant 62020106015 and Grant 61976209; and in part by the CAAl-Huawei MindSpore Open Fund.
REFERENCES
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, and Miki Haseyama. Brain decoding of viewed image categories via semi-supervised multi-view bayesian generative model. IEEE Transactions Signal Processing, 68:5769-5781, 2020. doi: 10. (0.11099TSP.220.3022777
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince. Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116-1266 2022.
Elissa M Aminoff, Kestutis Kveraga, and Moshe Bar. The role of the parahippocampal cortex in cognition. Trends Cognitive Sciences, 17(8):379-39)) 2013.
Subhranil Bagchi and Deepti R Bathula. EEG-Conv Transformer for single-trial EEG-based visual EEGGConnTranformee stimulus classification. Pattern Recognition, 129:108757, 2022.
Richard T Born and David C Bradley. Structure and function of visual area MT. Annual Review of Neuroscience, 28: 157-189, 2005.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Virtual, 2020.
Thomas A Carlson, Ryan A Simmons, Nikolaus Kriegeskorte. and L Robert Slevc. The emergence of semantic meaning in the ventral temporal pathway. Journal of Cognitive Neuroscience, 26(1): 120-131, 2014.
Omar Chehab, Alexandre Défossez, Loiseau Jean-Christore Christophe, Alexandre Gramfort, and Jean-Remi King. Deep recurrent encoder: an end-to-end network to model magnetoonccephalography at scale. Neurons, Behavior, Data analysis, and Theory, 2022.
Po-Hsuan Chen, Xia Zhu, Hejia Zhang, Javier Turek, Janice Chen, Theodore I. Willke, Uri Hasson, and Peter J Ramadge. A convolutional autoencoder for multi-subject fMRI data aggregation. In Representation Learning in Artificial and Biological Neural Networks Workshop @ Advances in Neural Information Processing Systems, volume 29, Barcelona, Spain, 2016.
Po-Hsuan Cameron Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter Ramadge. A reduced-dimension fMRI shared response model. In Advances in Neural Information Processing Systems, volume 28, pp. 460-468, Montreal, Canada, 2015.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Masked modeling conditioned diffusion model for human vision decoding. arXiv preprint PrXiv:2211.06955 2022.
Andrew C Connolly, Swaroop Guntupalli, Jason Gors, Michael Hanke, Yaroslav O Halchenko, Yu-Chien Wu, Hervé Abdi, and James Haxby. The representation of biological classes in the human brain. Journal of Neuroscience, 3208):6088266 2012.
Wei Dai, Ziyao Zhang, Lixia Tian, Shengyuan Yu, Shuhui Wang, Zhao Dong, and Hairong Zheng. BrainFormer: A hybrid CNN-Transformer model for brain fMRI data classification. arXiv preprint arXiv:2208. 200.02008.00088.. 2022.
10

---

Published as a conference paper at ICLR 2024
Alessandro De Benedictis, Hugues Duffau, Beatrice Paradiso, Enrico Grandi, Sergio Balbi, Enrico Granieri, Enzo Colarusso, Franco Chioffi, Carlo Efisio Marras, and Silvio Sarubbo. Anatomo- functional study of the mapping evidence from a neurosurgical perspective. Journal of Anatomy, 225(2):132-151, 2014.
region: dissection, tractographic and brain
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, Vienna, Austria, 2021.
Matthew Glasser, Stamatios N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neurolmage, 80:105-124,
2013.
Kalanit Grill-Spectee Sriil--peetorr Zoe Kourtzi, and Naney Kanwisher. The lateral occipital complex and its role in object recognition. Vision research, A((0--11)1409-142)) 2001.
James V Haxby, Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment: Modeling shared information encoded idiosyncratic cortical topographies. Elife, 9te56601, 2020.
Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6):1210-1224, 2012.
Meenakshi Khosla, Gia Ngo, Keith Jamison, Amy Kuceyeski, and Mert R Sabuneu. A shared neural encoding model for the prediction of subject-specific fMRI response. In Medical Image Computing and Computer Assisted Intervention, volume 12267, pp. 539-548, Lima, Peru, 2020. Springer.
Dan Li, Changde Du, Haibao Wang, Qiongyi Zhou, and Huiguang He. Deep modality assistance co-training network for semi-supervised multi-label semantic decoding. IEEE Transactions on Multimedia, 24:3287-3299, 2022. doi: 10. (09/TMM..021..304999999
Weida Li, Mingxia Liu, Fang Chen, and Daoqiang Zhang. Graph-based decoding model for functional alignment of unaligned fMRI data. In AAAI Conference on Artificial Intelligence, volume 34, pp. (653-2660,, New York, USA, 2020. AAAI Press.
Sikun Lin, Thomas Sprague, and Ambuj Singh. Mind Reader: Reconstructing complex images from brain activities. In Advances in Neural Information Processing Systems, volume 35, pp. 19624-29633 New Orleans, USA, 2022.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, volume 8693, pp. 740-755, Zurich, Switzerland, 2014. Springer.
George A Miller. WordNet: a lexical database for English. Communications of the ACM, 38(11):
39-41, 1995.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. Science, 2005880):1111-11119 2008.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and ding-Hsuan Yang. Intriguing properties of vision transformers. In Advances in Neural Information Processing Systems, volume 34, pp. 23296-23308 Montreal, Canada, 2021.
Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. Multi-label co-regularization for semi- supervised facial action unit recognition. Advances in Neural Information Processing Systems, volume 32, pp. 909-919, Vancouver, Canada, 2019.

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, volume 139, pp. 8748-8763, Virtual, 2021. PMLR.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. BioRxiv preprint Biookiv..0.1012222.11....
2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, Virtual, 2021. PMLR.
David C Van Essen, Kamil Ugurbil, Edward Auerbach, Deanna Barch, Timothy EJ Behrens, Richard Bucholz, Acer Chang, Liyong Chen, Maurizio Corbetta, Sandra W Curtiss, et al. The human connectome project: data acquisition perspective. Neurolmage, 62(4):2222-2231, 2012.
Muhammad Yousefnezhad and Daoqiang Zhang. Deep hyperalignment. In Advances in Neural Information Processing Systems, volume 30, pp. 1604-1612, Long Beach, USA, 2017.
Yizhen Zhang, Kuan Han, Robert Worth, and Zhongming Liu. Connecting concepts in the brain by mapping cortical representations of semantic relations. Nature Communications, 11(1):1877, 2020.
Qiongyi Zhou, Changde Du, and Huiguang He. Exploring the brain-like properties of deep neural networks: A neural encoding perspective. Machine Intelligence Research, 19(5):439-455, 2022.
12

---

Published as a conference paper at ICLR 2024
A DATA PREPROCESSING
Preprocessing before input: The original BOLD volumes in the HCP dataset have dimensions of 113 x 136 x 113. The 3-D BOLD volumes are patchified by 3D-CNN as shown in Fig. Bla) during experiments. For the NSD dataset, we extract ROIs in the visual cortex based on brain parcellation, including VI-V3, V3ab, hV4, ventral occipital (VO), intraparietal suleus (IPS), lateral occipital (LO), middle temporal (MT), and parahippocampal cortex (PHC), covering both primary and high-level areas of the visual cortex. The number of voxels in each ROI varies across subjects, and the number of voxels for the same subject also varies across different ROIs. To enable model weight sharing across subjects, we align the dimensions of the BOLD signals in the same ROI across subjects. Additionally, to adapt the data to the Transformer model, we align the dimensions of the BOLD signals in the same subject across different ROIs. To achieve this, we employ principal component analysis to reduce the dimensionality of the BOLD signals to the minimum dimensionality value of 268 across all BOLD signals from all ROIs of all subjects. During the testing process, we can apply zero- prro-padding neural signals with dimensions lower than this value. In the methods using MLP as the backbone network, the patches of all ROIs are concatenated into a vector input into the model. In the methods using ViT as the backbone network, one patch is one token.
Data split: The HCP dataset is split in accordaace with the method described in Khosla al (2024)) The first three movies are used for training and validation, while the fourth movie is used for test. For the single-subject decoding task, the training, validation, and test sets consist of 2000, 265, and 699 samples, respectively. For the multi-subject decoding task, the training sets of nine subjects are combined and randomly shuffled for model training, while the validation sets of the same nine subjects are combined for model validation. To account for the randomness, we report the average results of three random runs. A hemodynamic delay of seconds estimated in Khosla et all (2020) is used in this paper. For the NSD dataset, the stimuli viewed by all subjects and their corresponding neural responses are used for test in the single-subject decoding task. A validation set consisting of 1,000 randomly sampled examples is used for hyper-parameter tuning and convergence monitoring during training. The remaining data is used for training. For the multi-subject decoding task, the training sets of eight subjects combined and randomly shuffled, and a validation set of 1, 1,000 examples is randomly sampled. The remaining data is used for training. Due to the randomness of data split, we report the average results of five random splits.
Multimodal feature extraction: We concatenate the WordNet annotations of each movie frame
to form the textual information for stimuli in the HCP dataset. In the NSD dataset, each stimulus image is associated with five captions. Following the approach in ctll (2(22), we compute the similarity between each image and its five captions in the multimodal feature space of CLIP. We use half of the maximum similarity score as the threshold and randomly select one caption from the selected candidates to serve as the textual information for the stimulus. We extract the text features by inputting the textual information into the CLIP text encoder. For both datasets, we truncate the image and text features with a threshold of 1.5 and normalize their 1.2 norms to 1. We then compute the average of the image and text features obtain the multimodal feature faiv. which guides the learning of Bite-
B THE CHOICE OF AN RSA-BASED LOSS
Using RSA to guide the representation learning of fMRI superior to directly mapping fMRI representations to the CLIP embeddings. There are three main reasons:
1. Mapping- methods tend to focus on learning local information when embedding fMRI representations into the CLIP space (e.g., minimizing the L2 norm between true and predicted values). However, they may overlook the global topological structure of the CLIP space, which can affect the interpretability and generalization capabilities of the embedding space. In contrast, using RSA loss ensures the global topological structure similarity between the embedding space and the CLIP space.
2. There is gap between the representation spaces of CLIP and fMRL. It is more challenging to directly learn the mapping from fMRI representations to CLIP representations than to learn the topological relationship. Foreing alignment may lead overfitting and poor generalization.
13

---

Published as a conference paper at ICLR 2024
Table B4: Results of the RSA-based method (CLIP-MUSED) and mapping-basee methods the
NSD dataset.
Method
mAP f
AUC
Hamming +
Mapping-Bann Basen
.247 + .026
.844 + .033 .035 + .002
RSA-Based
.258 .017 .877 + .021 .030 + .002
3. Mapping-basee methods introduce additional trainable parameters in the mapping network, and the architecture of the mapping network also needs be delicately optimized.
We also conduct a comparative experiment between RSA-based and mapping-based methods on the NSD dataset. The results Table BA demonstrate the superiority of the ASA-based loss. The coefficient (A = 0.0001) of the mapping- based loss item has been optimized.
C PEFORMANCE COMPARISON ON EACH SUBJECT
We compare the performance of SS-Vis VS- which is a single-subject decoding method with CLIP- MUSED, as they both utilize VIT the backbone network. Fig. C compares the mAP of our method and SS-ViT on each subject. As shown in the figure, our method outperforms SS-ViT on all subjects. Fig. CG show the comparison results the NSD dataset. It is evident that method consistently outperforms SS-ViT on the majority of subjects. These findings suggest that the neural representations shared among subjects learned by our method are superior those learned by the single-subject method.
0.40
55 VS-VT
0.39
0.38
MAP
0.37
0.35
0.34
0.32
Subjects
Figure C5: Performance comparison between SS-VIT and our method each subject of the HCP dataset.
0.30
SS-VIT
MAp
0.20
0.10
Subjects
Figure C6: Performance comparison between SS-ViT and our method on each subject of the NSD dataset.
D GUIDANCE EFFECT ON THE SINGLE-SUBJECT METHOD
In the experiments, we only employ CLIP guidance our proposed multi-subject model. To investigate the impact of CLIP guidance the single-subject model, modified the SS-VIT to
14

---

Published as a conference paper at ICLR 2024
Table D5: Results of two single-subject methods CLIP-SS-ViT and SS-ViT, with and without CLIP guidance, and CLIP-MUSED on the NSD dataset. All the improvement of CLIP-MUSED compared other cases significant (t-test, p 0.05), where the p-values have been corrected with the Holm- Bonferroni method for multiple comparisons.
Method
mAP
AUC 1
Hamming
SS-VIT .238 + .005
.815 + .008
.032 + .000
CLIP-SS-ViT .234 + .002
.822 + .006
.032 + .001
CLIP-MUSED .258 + .017 .877 .201
.030 .002
enable the learning of neural representations under the guidance of CLIP, i.e., CLIP-SS- VIT. The model structure and composition of the loss function for CLIP-SS-ViT are the same as our method, single-subject data. We carefully
with the only difference being that CLIP-SS-ViT is trained tuned the trade-off parameters for CLIP-SS- VIT, and the optimal parameters are t = 0.0001, Ante 0.0001, and Aute 0.0001. The results shown in Table D5
can be observed that, both guided by CLIP, our method outperforms the CLIP-SS-VIL VIT. This superiority can be attributed to the multi-subject aggregation strategy employed in method. On the single-subject model, the relatively weak effect of using CLIP as guidance may be due to the fact that the single-subject model does not need to handle individual differences. In other words, the original classification loss can implicitly learn the relationship between different fMRI representations of a single subject, and the guidance from CLIP is redundant for the model.
E GUIDANCE EFFECT OF DIFFERENT DNNs
Table HG presents the model performance when the neural representation learning is guided by the topological relationship of visual stimuli across different DNN representation spaces. CLIP- Img/CLIP- refer to the utilization of high-level image/text features extracted by the image/text encoder of CLIP, instead of the multimodal features, during the learning of high-level tokens. In summary, the results depicted in Table KR suggest that features extracted from CLIP exhibit a superior guidance effect when compared the baseline models (VIT and AlexNet). Notably, textual information provides a richer semantic understanding of the visual stimuli and integrating it with the image features can serve to augment the performance of the model guiding neural representation learning.
Table E6: Comparison of the guidance effect of different DNNs. All the improvement of CLIP compared to other cases is significant (t-test, < 0.05) except for those underlined, where the p-values have been corrected with the Holm- Bonferroni method for multiple comparisons.
HCP
NSD
Methods
mAP 1
AUC 1
Hamming
mAP 1
AUC 1
Hamming |
VIT
.362 + .003
.562 + .004
.383 = .00
.229 + .009
.849 + .014 .035 + .001
AlexNet
.362 + .002
.558
.005
.333 = .013
.225 .012
.855 + .013
.034 .002
CLIP-Img
.370 .001
.577 + .001
.291 .002
.238 + .006
.863 + .008 .033 + .001
Cext-Text
.370 +: .002
.579
+ .006
.286 + .003
.23 + .006
.841 + .008 .036 + .001
CLIP (OURS)
.373 + .002
.581
.008
283
.004
258 .017
.877
+ .021 .030 .002
F VISUALIZATION
We present the visualization of between-subjed representational similarity matrices (RSMs) of low- level and high-level tokens two datasets, depicted in Fig. E Notably, the similarity of tokens between subjects observed to be higher on the HCP dataset compared to the NSD dataset. This can be attributed to the fact that in the HCP dataset, all subjects viewed the same stimuli, and the distribution of stimuli across subjects is uniform, whereas the training set of the NSD dataset, the stimuli viewed by different subjects mutually exclusive. Evidently, different subjects process
15

---

Published as a conference paper at ICLR 2024
Low-level High-level
Low-level Token RSM
High-level RSM
0.90
2
3
0.15
0.50
4
6
5
0.05
6-
0.70
8
2
4
(a) HCP
NSD
Figure F7: RSM between low-level and high-level tokens across subjects of our method (a) the HCP dataset and (b) the NSD dataset.
Sublect
2ublect
Subject
Sublect
834
0.14
834
G.721
Ex3t
640-
833
0.00-
High-levll
Low-level Tilken
8:00
0.00
8:08
1:0
Subject
Subject
Scdject?
Subject
8.44
0:6-
922
031
0.50
600
0.5-
3.55
0000
Figure F8: Attention maps between the low-level and high-level tokens of our method and different brain region tokens at the last Transformer self-attention layer for subjects on the NSD dataset.
stimulus information in slightly distinct patterns even when viewing the same stimuli, and these differences are further amplified when presented with different stimuli. The tokens learned for each subject in CLIP-MUSED can encode these inter-subject variabilities, resulting in lower similarity of tokens between subjects on the NSD dataset with different stimuli for different subjects.
Fig. ES shows the attention maps of the low-level and high-level tokens on different ROIs of the NSD dataset. The low-level tokens exhibit a higher attention allocation towards the primary and intermediate visual cortex regions the left side of the bar chart, including V1-V4, while the high- level tokens exhibit more pronounced attention towards the higher visual cortex regions such as LO, MT, and PHC. Prior research has established that MT plays crucial role in depth perception (Born & Bradley, 200S), LO is involved in object recognition tasks Spector et all 2001), and PHC contributes visual perception related to memory and spatial scenes (Aminoff all 203).. In contrast, Fig. ES demonstrates the attention maps of the subject embeddings in the MS-EMB method, revealing more focused attention on the V3, V4, and VO brain regions, but less on the higher visual cortex regions such as LO, MT, and PHC. In contrast, our method leverages both low- level and high-level tokens allocate attention towards both low-level and intermediate visual cortex regions, as well as higher visual cortex regions. The difference in token attention maps between our method and the MS- EMB method partially elucidates the superiority of our method classification performance.
16

---

Published as a conference paper at ICLR 2024
Subject
Subject
Sublect
036-
8tm
0.16
41.0
0.0
0.19
0.00
0.22
0,315
sea-
0.00
8300
0.00
6.00
0.000
Subject
Subject
Subject
Subject
0175
61.m
0.40
8000
0/8
0.100
0.04
0.800
540
0.00
0.02
Figure F9: Attention maps between the subject embedding token of the MS-EMB method and different brain region tokens the last Transformer self-attention layer for 8 subjects on the NSD dataset.
17


Reviews:
[{"review_id": "FzHtpfrmD1", "reviewer": "Reviewer_mWYc", "summary": "The paper addresses the problem of alignment of the response to visual stimuli across multiple subjects. This is an important problem for visual neural decoding tasks. In general, models for visual neural decoding tasks are either trained on a single subject (leading to problems such as overfitting) or must correct for differences across anatomical structure and functional topography of the brain in different subjects. The paper proposes a new method for multi-subject functional alignment that goes beyond the SOTA, namely hyperalignment and category-based methods. This new method uses transformer-based models to capture long-range dependencies that exist in the functional connectivity between brain regions. Intersubject differences are modelled through extra subject-specific tokens and the CLIP representation space is used to achieve a high consistency between cortical representations of visual stimuli.", "questions": "- Why do extract low-level and high-level feature RSM seperately?\n\n- The HCP dataset contains more than 158 subjects. Why did you not use all subjects? How were the 158 subjects selected from the HCP? Randomly? Why is this further reduced to 9? What is the bottleneck here?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important problem in neurosciences and the proposed multi-subject alignment method is novel and interesting. Both the use of transformers to model long-range associations in brain regions and the use of the CLIP representation space to guide neural representation learning in a shared representation space is a nice idea! The experimental results on the two fMRI datasets are somewhat limited (esp. due to the small size) but demonstrate a proof-of-concept nicely.", "weaknesses": "I found that the abstract is not as clearly written as is the introduction. From the abstract, it is unclear what the paper sets out to achieve nor how this is done. The introduction does much better job at this. Perhaps the authors could try to formulate the problem addressed and the contributions more clearly (I appreciate that this is more difficult to do in the space available). The evaluation is rather limited, especially since the size of the datasets is quite small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oWk46Yh2mv", "reviewer": "Reviewer_qr5q", "summary": "The authors propose a transformer based cross-subject fMRI alignment method that uses learned special tokens (an approach also used in other ViT/language works which perform classification). \n\nSpecifically, the paper discusses the following issues in traditional hyperalignment approaches:\n1. Mapping functions are restrictive (I assume this is in reference to orthogonal procrustes, linear mappings, or kernelized mappings)\n2. Requirements for a per-subject mapping function, which can be expensive when deep networks are used\n3. Need for identical stimuli or stimuli from the same semantic category\n\nThey propose the following approach:\n1. Using a transformer to model long-range dependencies\n2. Using special tokens to model subject-wise information\n3. CLIP distances to guide the model, with the average of the text & visual branches used for the high level feature.", "questions": "Questions:\n1. For Figure 1 in the paragraph below you mention `two bird images t1 and t2` as well as `t1 and t2 are birds, while t3 (duck) and t4 (building blocks) are not`.\n\n    **a**. I encourage the authors to clarify this, or use other examples in the figure and paper. Ducks should be birds.\n\n2. In the paper, it is discussed that `Linear transformation and MLP are unsuitable for high-dimensional voxel responses`. However no justification on the MLP aspect is given, and the citation (Yousefnezhad & Zhang) explicitly uses a kernelized MLP approach. Could you clarify?\n\n3. In section 3.2, you never define what a backbone network is. Does it just take as input the fMRI BOLD responses and output a category? In that case, is the ViT a 3D model?\n\n4. It is not clear what exactly you are decoding. You mention that HCP has a wordnet label, and NSD has the 80 categories (I assume the binary 0/1 mask on if an object from a category is present in the image).\n\n5. Can you clarify if you use ROI based patching for NSD, and the CNN based patching approach for HCP? Is the CNN jointly trained with the transformer?\n\n6. How do you extract visual features for HCP given that the stimulus set consists of video clips?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "CLIP-MUSED is an interesting approach that shares model parameters and only varies a per-subject token. The use of CLIP for feature extraction and feature alignment is solid. The method allows for the use of multi-subject data without a linear increase in the number of model parameters, which is beneficial for scalability\n\nThe paper provides a detailed methods section that covers the technical aspects of the CLIP-MUSED, including the use of Transformers and CLIP and the design of an RSA based loss. The high level approach is clear.\n\nThe paper is well-organized, making it relatively easy to follow the arguments and understand the methodology.", "weaknesses": "**In my view there are two weaknesses of the paper:**\n1. The relatively weak decoding the authors perform, which is implied to be just category decoding. It is very strange to me that they use so much compute relative to traditional methods, and end up just doing category decoding. I believe the results would be strengthened by performing visual decoding in the form of full image reconstruction, at least for NSD (single image passive viewing task). If this is not possible, I think an alternative would be perform image retrieval (show top-5) on a test set, where the test set contains images not used for the training of any subject. This would likely require training the decoder on a single subject's stimuli only, then testing on other subjects.\n2. The fact that the subject-wise token is not computed via amortized inference (via an encoder conditioned on a few BOLD responses). But instead, they are fixed per subject. In my view this weakens the approach somewhat as you cannot take this approach and apply it to a new subject in a few-shot fashion, after the paper talks about how their method does not require subjects to view the same stimulus.\n\n**Math typos:**\n1. In both the paper eq 10, the authors discuss using the F-norm (frobenius norm), but in practice they are using the squared frobenius norm in the code (line 42 of `losses.py`). This is fine, as the squared F-norm is everywhere differentiable, but I ask the authors clarify/fix this.\n2. In the paper eq 15, the authors apply a orthogonal constraint such that matrix $z_\\text{llv}$ and matrix $z_\\text{hlv}$ are orthogonal to each other via the regularization loss. Same issue listed above applies. You should probably also clarify here that you want the low/high level representations for each stimuli to be orthogonal, otherwise it is not super clear.\n\n**Minor typos:**\n1. Page 3. Original `The contributes are summarized as follows`, should be `The contributions are summarized as follows`\n2. Page 3. It seems that $X^{(n)} \\in \\mathbb{R}^{n_i \\times d} $ should be $\\mathcal{X}^{(n)} \\in \\mathbb{R}^{n_i \\times d} $\n\nOverall I think the proposed method is interesting, but the experiments currently do not fully demonstrate the strength of the method. I would gladly reconsider if the authors can clarify my questions, and provide additional experiments to support their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4fbHPXelTK", "reviewer": "Reviewer_UuaC", "summary": "This paper proposes a CLIP-guided Multi-sUbject visual neural information SEmantic Decoding (CLIP-MUSED) method which generalizes single-subject decoding models to multiple subjects in visual neural decoding tasks. Different from other multi-subject decoding methods, CLIP-MUSED uses a Transformer-based fMRI feature extractor to effectively extract global features of neural responses. CLIP-MUSED uses low-level and high-level tokens to encode individual differences and uses the topological relationship of visual stimuli in CLIP representation space based on RSA to guide the representation learning of tokens.", "questions": "1. How to set labels for semantic classification of N subjects after concatenating low-level and high-level token representations of different image stimuli?\n2. In Section 2.1 \"X^{(n)}\" is recommended to be consistent with the previous notation. The rest of the article should also be consistent.\n3. The meaning of B in \"B visual stimuli\" in Section 2.2 should be explained.\n4. Is the underline in the AUC results of MS-SMODEL-ViT in Table 1 redundant? It should remain in the same format as other tables.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to read, and generally well written.\n2. The use of RSA is novel and contributes to the topological relationships of visual stimuli in the CLIP representation space as prior knowledge to guide neural representation learning in the shared space.\n3. The experiments have clearly verified the benefits of the proposed methods.", "weaknesses": "1. The use of letters in the article needs to be consistent. For example, the representation of X.\n2. In order to reduce the computational cost, using 3D-CNN to reduce the dimension will have an impact on the results? The results of this comparison are not shown.\n3. The high-level feature in RSMs uses the average of the image and text features from the last layer of CLIP. What are the advantages of using the CLIP image encoder or text encoder alone compared to directly using it alone? Can you provide corresponding results for an explanation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)