NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping
Authors: Yamin Li
Ange Lou
Ziyuan Xu
SHENGCHAO ZHANG
Shiyu Wang
Dario J. Englot
Soheil Kolouri
Daniel Moyer
Roza G Bayrak
Catie Chang
Venue: NeurIPS 2024 poster (2024)
Keywords: EEG-to-fMRI synthesis
EEG
fMRI
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose NeuroBOLT, a versatile deep-learning solution for projecting scalp EEG to BOLD fMRI signals.

Abstract: Functional magnetic resonance imaging (fMRI) is an indispensable tool in modern neuroscience, providing a non-invasive window into whole-brain dynamics at millimeter-scale spatial resolution. However, fMRI is constrained by issues such as high operation costs and immobility. With the rapid advancements in cross-modality synthesis and brain decoding, the use of deep neural networks has emerged as a promising solution for inferring whole-brain, high-resolution fMRI features directly from electroencephalography (EEG), a more widely accessible and portable neuroimaging modality. Nonetheless, the complex projection from neural activity to fMRI hemodynamic responses and the spatial ambiguity of EEG pose substantial challenges both in modeling and interpretability. Relatively few studies to date have developed approaches for EEG-fMRI translation, and although they have made significant strides, the inference of fMRI signals in a given study has been limited to a small set of brain areas and to a single condition (i.e., either resting-state or a specific task). The capability to predict fMRI signals in other brain areas, as well as to generalize across conditions, remain critical gaps in the field. To tackle these challenges, we introduce a novel and generalizable framework: NeuroBOLT, i.e., Neuro-to-BOLD Transformer, which leverages multi-dimensional representation learning from temporal, spatial, and spectral domains to translate raw EEG data to the corresponding fMRI activity signals across the brain. Our experiments demonstrate that NeuroBOLT effectively reconstructs unseen resting-state fMRI signals from primary sensory, high-level cognitive areas, and deep subcortical brain regions, achieving state-of-the-art accuracy with the potential to generalize across varying conditions and sites, which significantly advances the integration of these two modalities.

NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping
Yamin Li
Ange Lou'
Ziyuan Xu Shengchao Zhang' Shiyu Wang'
Dario J. Englot
Soheil Kolouri Daniel Moyer Roza G. Bayrak Catie Chang!
Vanderbilt University Vanderbilt University Medical Center
yamin. 1i Lidvanderbilt.edu
Abstract
Functional magnetic resonance imaging (fMRI) is indispensable tool in modern neuroscience, providing a non-invasive window into whole-brain dynamics at millimeter-seale spatial resolution. However, fMRI is constrained by issues such as high operation costs and immobility. With the rapid advancements in cross- modality synthesis and brain decoding, the use of deep neural networks has emerged as a promising solution for inferring whole-brain, high-resolution fMRI features directly from eleetroencephaalggaahy (EEG), a more widely accessible and portable neuroimaging modality. Nonetheless, the complex projection from neural activity IMRI hemodynamic responses and the spatial ambiguity of EEG pose substantial challenges both in modeling and interpretability. Relatively few studies to date have developed approaches for EEG-fMR translation, and although they have made significant strides, the inference of fMRI signals in given study has been limited a small set of brain areas and to single condition (i.e., either resting-state or a specific task). The capability to predict fMRI signals in other brain areas, as well as to generalize across conditions, remain critical gaps the field. tackle these challenges, we introduce a novel and generalizable framework: NeuroBOL.1 i.e., Neuro-to-BOll Transformer, which leverages multi-dimmesional representation learning from temporal, spatial, and spectral domains to translate raw EEG data to the corresponding iMRI activity signals across the brain. Our experiments demonstrate that NeuroBOLT effectively reconstructs unseen resting- state fMRI signals from primary sensory. high-level cognitive areas, and deep subcortical brain regions, achieving state-of-fe oiee----the-ttttt accuracy with the potential to generalize across varying conditions and sites, which significantly advances the integration of these two modalities.
Introduction
Functional magnetic resonance imaging (fMRI) and electroeneephalograhy (EEG) are the most commonly utilized non- non-invasivv neuroimaging techniques, providing crucial insights into brain functionality. These two modalities offer distinct advantages and limitations that can complement one another when combined 45 13 fMRI offers high spatial resolution imaging of whole-brain (BOLD) signal changes, which facilitates probing regional and network-level function. However, fMRI suffers from low temporal resolution and hemodynamic blurring, which limits ability to capture the accurate timing rapid neuronal activity dynamics. Additionally, its high cost, non-portability, and incompatibility with metal implants further restrict the utility of MRI certain contexts. Conversely, EEG stands out as a low-cost, portable neuroimaging modality with high temporal resolution [10 30 61.. However, EEG faces
activity by measuring dlodd-oxyen--levl--deeeeeeee
Project page: https://soupeelli https://soueelliggiuubiiioeeee hit.eeiiiiitbbbe///eBBBLLLL
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Multi-chanme EEG
Neuronal activity
Volume conduction
Neurovascular coupling
15g
HRF
NeuroBoLT
Temporalty alinnd
fMRI collection
fMRI course
Figure I: Overall illustration of EEG-to-BOLD fMRI translation using NeuroBOLT.
limitations due to volume conduction and from the superficial location of electrodes (on the scalp), making it difficult for EEG to accurately infer the origins of the measured electrical potentials, In this context, the high spatial resolution of fMRI becomes indispensable, especially for imaging deep brain regions.
One approach for investigating the relationship between EEG and fMRI signals is by analyzing data collected simultancously from both modalities. Such studies have demonstrated correlations between fMRI data and various features of EEG signals, such as the power in certain frequency bands 27 7 [28] which underscore the potential of EEG to inform fMRI features. However, factors such as the disparate biophysical origins of the signals, and differences in their spatial and temporal resolutions, can limit the accuracy, interpretation, and consistency of correlation-base
EEG-fM ffMII studies. These challenges are exacerbated in conditions during which a participant is resting passively (i.e., resting state), which is characterized by significant noise and randomness. Consequently, the mechanisms linking neuronal activity BOLD signals remain only partially understood, posing challenges in mapping EEG to fMRI.
With recent progress in cross modallty synthesis and brain decoding techniques using deep neural networks 8 9 [6]. EEG-to-IMRI synthesis has emerged as a promising yet largely untapped research area 4 Early pioneering work by 39 43) employed ridge regression EEG temporal-spectral
features to reconstruct fMRI signals from the visual cortex and sub-cortical regions. More recent studies have used deep neural networks for EEG-fMRI translation [3 35
31,, and those of (Seq-to-Seq) models to reconstruct fMRI time series of deep brain regions from EEG. However, these efforts have primarily focused on task datasets, [4l
25
(25 and (31) developed sequence-to-sequenee
54] 3) leaving the (fully eyes-closed) resting state condition largely unexplored. In addition, prior studies rely on subject-specific approach, wherein models are solely trained and tested on different sections of the same individual's 's scans. Further, only small number brain regions have been examined in current studies, leaving as an open question the predictive power of EEG for IMRI signals across broader areas (see Appendix B for further discussion of related work). These limitations underscore the need for developing more generalizable models for EEG-IMRO translation that are accompanied by more comprehensive evaluation.
including cued eye opening/closing 25 31
35
In this paper, we present NeuroBOLT: a multi-dimensional transformer-based EEG encoding frame- work for Neural-to-BOLD fMRI Translation (Figure 2). NeuroBOLT is designed to capture and integrate multi-dimensional representations across temporal, spatial, and spectral domains, offering generalizable approach for translating raw EEG waves to BOLDD fMRI time series from regions of interest (ROIs). Drawing inspiration from vision transformer (ViT) 15 and its adaptation for multi- channel biosignals [59] [22], our framework transforms raw EEG data into unified "EEG patches", enabling flexible and effective EEG data encoding. These EEG patches are then passed into two parallel sub-modules: (1) spatiotemporal and (2) spectral representation learning modules. The spatiotemporal module is propelled by a recently proposed EEG foundation model (LaBraM) [22]. which has been trained to learn effective representations on about 2,500 hours of various types of EEG signals. Moreover, as previous studies have emphasized the importance of leveraging spectral features in EEG representation learning [59
57 60 EEG-fMRI correlations 27 14] and EEG-to-fMRI
synthesis [3] 25), the spectral module incorporates multi-scall encoding on EEG spectrogram patches. Specifically, instead of employing a fixed window size for the Short-Tima Fourier Transform

---

we incorporate spectral features from windows of varying scales. This approach retains the advantages of high temporal resolution from smaller windows and high frequency resolution from larger windows, thereby enhancing the spectral analysis. The output embeddings from the above two encoding modules are then integrated, allowing NeuroBOLT to capture the complexity of neural dynamics and learn the projection from neural to BOLD signals. The key contributions of this work are summarized as follows:
(STFT) as commonly used [59 58 24
1) Generalizable EEG-10-MMRI translation framework. We introduce a novel approach, Neu- roBOLT, that utilizees multi sppettl representation to predict nigh-dimensional fMRI signals from raw EEG data without relying on predefined assumptions about the hemodynamic delay between fMRI and EEG signals. Additionally, our model is designed to accommodate any number of EEG channels, enhancing its versatility across various experimental setups.
2) Comprehensive evaluation of the predictive power. on subject-specific and cross-subject learning, across selected ROIs in primary sensory. high-level cognitive, and deep brain regions. Further, we probe the generalizability of our approach across data acquired in both resting state and task (auditory stimulus) conditions.
We performed comprehensive experiments
3) Successful resting-state fMRI reconstruction. To our knowledge, this is the first study to successfully reconstruct the eyes-closed resting- state fMRI signal from raw EEG data, with a relatively small number (26) of EEG electrodes.
2 Method
2.1 Task Formulation
While it is well-established that fMRI signals are coupled to neuronal activity, the details of this process are only partially understood and still under debate [19]. In practice, most studies model the relationship between neuronal activity and fMRI by convolving the assumed neural activity with the hemodynamic response function (HRF), which reaches its peak at about seconds and gradually returns to baseline over the next 12-15 seconds. However, the HRF is not uniform, varying significantly across different brain regions and between individuals. Moreover, although EEG can detect the electrical activity of neural populations millisecond timescales, it suffers from volume conduction, making it challenging to extract its neural origins. This spatial ambiguity also complicates the interpretation of EEG and its relation to fMRI data.
In this study, we aim to build neural network to learn this complicated projection from the clectrophysiological signal measured by EEG to the BOLD signal measured by IMRL. Our approach tries to overcome the above challenges in the following ways: 1) it captures multi-dimensional features of EEG across temporal, spatial, and spectral domains, learning representations that are crucial for accurate fMRI synthesis; 2) it does not assume a pre-defined delay between the two modalities, instead, we extract time windows of EEG data that span a duration approximating the length of HRF preceding each fMRI data point as the inputs to predict the corresponding fMRI data values (i.e.,& Sequence-to- One model), in order to accommodate potential variability across subjects, regions, and frequencies.
Functional MRI data are conventionally represented in four dimensions, and can be expressed as a temporal sequence = (V.,.., VK} with K observations, where each observation is a volume V E 1 with spatial dimensions z, y, and z. This high-dimensional IMRI signal is often summarized by representing it as a set of brain areas (i.e., P parcels), each with a corresponding fMRI signal that is averaged over its constituent voxels Uz,y.= V. Here, employ such functional parcellation, Dictionaries of Functional Modes (DiFuMo) [12], which was learned across millions of fMRI volumes (see details in [12 50)). We utilize the DiFuMo parcellation with P = 64 regions, and the resulting parcellated iMRI data Y e PxX contains the signals from each of these regions.
Since fMRI signals are delayed by several seconds compared to the corresponding neural activity, the relationship between an input EEG window and the corresponding fMRI prediction is defined as Yp.- = fo(X,- -xx-) where Yp,, E R the reconstructed fMRI value from the pth ROI at time index i,, and X E RCXT represents the multi-channel EEG signal inputs with C electrodes, T total timepoints. To estimate fe).), we formulate an optimization problem that minimizes the loss L
3

---

volume
Multi-dimension position encoding
(A) EEG Tokenization
Spatial embedding
Time embedding
Regieno of Internst (ROI)
EGG signals
1MRI ROI Time-Course
Fp1
Spatiotemporal Representation Learning
Temporal Conv
Tokenizaioo
200x1
1
Transformer Encoder
Ave Pooling
Fpi
Audu
Vintun
Multi-scale spectral feature Embedding
Transformer
Encoder
200k1
(C) Spectral Representation Learning
Figure 2: Overall architecture NeuroBOLT. Our method first divides the input EEG window into uniform patches (A), and has two modules that are trained simultaneously: the temporal-spatial representation learning module (B) and the spectral representation learning module (C), The output embeddings from the two modules are summed and used input to regression head, which generates the final output.
between the predicted fMRI signal
p.z and the true fMRI signal y, follows:
min Ex ((f((((((X)))a))) y,,)).
(1)
The function fe is NeuroBOLT, which includes a temporal-spatial representation module and spectral representation learning module (Figure 2).
2.2 Model Architecture
In this section, we introduce model: NeuroBOLT,, general architecture for translating EEG signals to fMRI. Our model is designed to accommodate input EEG signals with an arbitrary number of channels. As shown in Figure 2] we leverage the pre-trained EEG foundation model, LaBraM (checkpoint version: LaBraM-base) (22). and finetune on our dataset to obtain spatiotemporal representations of EEG signals. Additionally, propose multi-scale spectral feature fusion obtain comprehensive spectral representations. These two modules learn complementary attributes of EEG data. Finally, we integrate the spatiotemporal and multi-scale spectral representations and feed them into a regression head for fMRI prediction.
Spatiotemporal Representation We formulate the multi-channel EEG signals X RCT where C' represents the number EEG electrode channels and denotes the time length of the input EEG. To obtain the spatiotemporal representation of given set of EEG signals, leverage an operation from LaBraM (22), which segments the EEG signals into patches. Assuming the time window length (patch size) for tokenization of EEG signal is w and the stride is s, can be segmented into T-w + segments, with each segment denoted as x E RCx In this work, we segment each EEG channel into patches, obtaining
use window of length and a stride { & E R = 1,2, C, & = 2. 2 The total number of patches is C x These patches are then passed forward to temporal encoder 22 denoted as:
obtain the patch embeddings,
T
(ee,. I 1,2, .. 1,2,
where d is the dimension of the embedding.
To enable the model to capture the temporal and spatial information of the patch embeddings, we set up a list of trainable temporal embeddings and spatial embeddings, denoted as TE {te

---

EGG channel
thannel
666 channel
Multi-scale Spectral Feature Embedding
EEG Ammllnule
Sciee
Takas Lie
Frequancy Embedding
Time
Special Embed
m
m
smma
Srale
wW W
Temporal Embeddinn
Scale
m
NY
DorL.
MA
AUP
Leteed nander
Mr Sumber wf Progurree Pesture derted
Figure 3: Multi-scale spectral feature embedding.
= 1,2,, .,, 1 and SE = {se, j = 1,2,.... C), respectively. Thus, the final segment embedding Eseg can be represented the sum of the output embedding, temporal embedding, and spatial embedding, denoted as:
Eseg = (eey,h k lex + 88, 11=1,2 1,. C, 1,2,
}
The segment embedding Eseg directly fed into the Transformer encoder [52 to obtain the output embeddings. We then apply average pooling to these embeddings to obtain the spatial and temporal representation Tat E gd,t, where dat denotes the dimensionality.
Multi-scale Spectral Representation Compared to RGB images, EEG signals present several challenges, such low signal-to ratio, apparent stochasticity, nonstationarity, and nonlinear characteristics, making the reconstruction of the original signals difficult [41]. Previous research indicates that the frequency spectrum of EEG signals is crucial for understanding the brain's 's neuro- physiological activities (59 55 Therefore, in our work, we utilize the Short-Time Fourier Transform (STFT) achieve spectral representation of EEG signals. Unlike most recent state-of-the- of-the-art methods, we design a multi-scale spectral approach that captures both coarse and fine representations the temporal and frequency domains [56 36). Details of rationale can be found Appendix C). For the EEG signals X RCx' we set the base window size as wy and define the window size at level ( as Wi Wi 2 where = 0, l,..., represents the level number. Thus, the EEG patches with different window size can be denoted x; { R j 1,2, C, k 1,2...., I
wy
After obtaining the EEG patches, Fast Fourier Transform (FFT) is applied to each patch with the FFT size matches the window length calculate the magnitude spectrum with wy + frequency bins 2
each level I, which be denoted 81 Cx T
+1)
Similar to the operation in spatiotemporal module, we define list of trainable frequency embeddings for each magnitude spectrum at different levels, represented by:
FE = {fer E IR
Cx
| xd 1= 0, 1.....,) L}
(4)
These frequency embeddings. which vary in the number of temporal windows, are further mapped to of window embeddings as follows:
WE- = {wez E Cxxxd - 0, I,...,I)
(5)
Then, we calculate the sum of the we for each level to obtain an overall spectrum embedding as
Esp E IR. Cxnxd as shown in equation 6
Csp
we E Cxaxd
(6)
1=0
5

---

For each channel, we apply spatial embedding, similar to the spatiotemporal representation section, obtain the final embedding with dimensions Esp RCxnxd, where C, n and represent the channel number, window embedding dimension, and frequency embedding dimension, respectively. The original Transformer known to have quadratic complexity in both time and space, and EEG signals typically contain tens of channels. To learn these embeddings with lower complexity [59,, we feed spectrum embeddings Cep into a linear Transformer Encoder [53] 23]. Let the input embedding Eap E IR,xd, where = C n represents the number of tokens. The self-attention operation the linear Transformer Encoder can be formulated as follows:
H Attention(e,, wº Se.p w Feepp .FespWr)
stfaar (Cep W W
F.esp W
V
DXk
NxD
Here Wº WK W E Ba are the query, key and value matrices. The self-attention module uses rank-D approximation for the softmax attention (N N) by reduced-rank parameter matrices E E RDD F E IR DxN (where D < N). Main components of this module include one linear self-attention layer and one fully connected network. Layer normalization before each component (2). residual connection after each component [18]. and dropout right after the self-attention to enable stable training [49 [59]. The output is fed into average pooling along the token dimension to obtain the final spectral representation ''sp E TR where Tsp has the same dimension as the spatiotemporal representation Tst-
Projection Head
The hidden embeddings from the above modules then summed up and
fed into a regression head, which consists of Gaussian Error Linear Unit (GELU) (20) followed by single linear layer, to make the final prediction of fMRI in the ph ROI at time
Y p.r Linear(GELU(ra: rsp))
(8)
3 Experiments
3.1 Datasets
Resting-state dataset EEG and fMRI data were collected simultaneously from 24 healthy volun- teers in two sessions, each lasting 20 minutes. Scans with significant artifacts in the EEG or fMRI data were excluded for further analysis. The final sample contains 29 fMRI scans from 22 subjects, 7 of whom had two scans. During these iMRI scans, subjects rested passively with their eyes closed (resting state). Written informed consent was obtained, and all protocols were approved by the Institutional Review Board. BOLD IMRI data were collected on a 3T scanner using a multi-echo gradient-echo EPI sequence with repetition time (TR) 2100 ms. Scalp EEG was acquired simul- taneously with fMRI using a 32-channel (10-20 system) MR-compatible system with FCz as the reference (BrainAmps MR, Brain Products GmbH) a sampling rate of kHz, and was synchronized to the scanner's 's 10 MHz clock to facilitate MR gradient artifact reduction. Details of EEG and fMRI acquisition, preprocessing as well as artifact reduction can be found in Appendix D
Auditory Task Dataset To further evaluate the model generalization performance, we also include simultaneous EEG-IMRI data (16 scans from 10 subjects) collected during auditory tasks. Subjects were asked to keep their eyes closed the entire time and to make a right- hight-handed button press as soon as possible upon hearing a tone. This dataset was collected different site, different MR scanner (3T Siemens Prisma scanner) and using a slightly different EEG cap (32 channels but with partially different electrode settings). Please also see detailed information about data collection, preprocessing. and experiments in Appendix[]
Unless specified otherwise, the experiments and results below refer to the resting-state data.
3.2 Experimental Setup
(see Section 2.1 to extract measured BOLD signals within specific ROIs. We focus seven ROIs that span a range of spatial
Preprocessing We employ DiFuMo with n = dimensions 12

---

locations and functional roles, namely: primary sensory regions (cuneus and Heschl's gyrus), high-level cognitive areas (anterior preeuneus, anterior and middle frontal gyri), subcortical regions (putamen and thalamus), and the global (whole-brain average) signal. From these ROI time series, we regress out motion confounds, low-pass filter the signals below 0. 1.155 15H7, and use the P of the absolute amplitude to normalize the demeaned ROI time courses. To prepare the EEG data for input, we first exclude the ECG, EOG, and EMG channels (remaining 26 channels), and resample the EEG to 200 Hz to enhance computational efficiency while preserving meaningful frequency components (which are typically below 100 Hz). To predict the fMRI ROI signal, we extract EEG windows of 16 seconds before each fMRI data point. This window length is selected to encompass the peak and most of the variation in the hemodynamic response function. Additionally, this duration aligns with the maximum temporal encoding window length of the pre-trained LaBraM model (2). ensuring optimal fine-tuning. The normalization of the HEG also strictly follows (22), where EEG data values are divided by 100 that the resulting amplitude falls primarily between -1 to 1, since typical EEG amplitude range is from 100 μV to 100 μVV
Baselines and Evaluation Metrics The baseline models include state-of-the-art EEG encoding models from [59 and (22), as well as two EEG-to-fMR translation baselines [25] 31 Since the original EEG- baselines are tosuuennce----euueeee models, we adapted them by adding a final projection layer to their encoder enable sequence-to-one prediction. Additional details on baseline implementations are provided in the Appendix [E.2] We employed the following two metrics for comparison: 1) Pearson correlation coefficient (R), which measures the strength and direction of the linear relationship between prediction and ground truth; 2) Mean squared error (MSE), which measures the average squared differences between prediction and ground truth across samples. The main text highlights the correlation coefficient as the primary evaluation metric, while MSE results provided in the Appendix Table 6
Implementation Details For the spatiotemporal module, we initialize the model by loading the pretrained weights from LaBraM-base
(2], with token length 200, i.e., second, with no overlap. Please refer to [22 for more details of EEG pretraining. For the multi-scale spectral module, we set the smallest scale size lo = 100, i... 0.5 seconds without overlap. Experiments are conducted on a single RTX A5000 GPU using Python 3.9.12, Pytorch 2.0.1, and CUDA 11.7. To ensure consistency during model training, we set a fixed seed across all experiments. The batch sizes are set at 16 and 64 for intra-subject and inter-subject amalyses, respectively. AdamW is utilized as optimizer, and MSE as our training objective. The initial learning rate is set at 3e-4 with a weight decay of 0.05, and minimal learning rate of le-6. For subject- the same scan, we split the scan in an 1:1:1 ratio for training, validation, and testing, i.e., training on the first 80% of the data and testing the last 10%. Given that the fMRI signal exhibits significant autocorrelation, typically extending from about -10 to 10 seconds (6, we implement gaps of 20 seconds between the training and validation sets, as well as between the validation and testing sets, to prevent data leakage. For the inter-subject analysis in resting-state data, we randomly divided the datasets into rrininn//alidatiionnestngg sets by approximately 3:1:1 (18 scans : 5 scans 6 scans). For the task data, we split the scans by 9 scans: 3 scans: 4 scans. Since data from the same individual might have shared representations, the two scans from the same individual are ensured to be in the same set (either training, validation. or testing set) to avoid possible data leakage. All models are optimized on the training set and evaluated on the test set, with the best model and hyperparameters selected based on the validation set.
prediction, where training and testing occur on
3.3 Intra-subject Prediction
In this section, models were trained on approximately 16 minutes of EEG data, and used to forecast future interval of fMRI signal (about 2 minutes) within resting-state sean. Figure T illlustrats the distribution of correlation coefficients (R) between the predictions and the ground truth, along with examples that represent the average performance of our model. The quantitative results are detailed in the upper part of Table and Figure 7 in Appendix. NeuroBOLT outperforms the next two best EEG encoders by 12.26% and 9.71 % (respectively) in terms of the average correlation, exhibiting the best performance in reconstructing fMRI signals from all ROIs.
7

---

(B)
Primary sensory
2.56
0L540
High-orden cagnitive
K-10.500 598
grss
Subcortical
TImw
seturier
D522
CAFES
Cldad
Thas 3D1 signall
Figure 4: Intra-subject prediction results. (A) The parcellation of the chosen ROIs. (B) The distribution of prediction performance (Pearson's 's correlation coefficients), with example time-series reconstructions that represent performance levels near the mean (indicated by the small grey arrow in the histogram). The dashed lines in the histograms represent mean correlation values.
Table 1: Model performance (1t) in intra- and inter-subject experiments. Bold: the best performance; the underlined: the second-best performance
Model
Cunzus
Primary Senoory
Heschils & Gyns
Middie Frostal
High-level Cugnitive
Precuncus Anterice
Puramen
Subcortical
Thalamus
Global Signal
Arg. R
Tonra ocia
BIOT 50
0.531=0:2233
0,518:10 207
0:490::0 162
0,459:00 110
0:410:00 205
0,411::0233 231
0.493-0 133
0.473
LaHlraM
22
0,540.20.1
0.51920.10 19?
0.19310.153
0.490:101770
0.411
0,469.0.17
0,487 167
0.484
HEIRA
0.357 241
0.396
0.204+0
0.520+0.220
0 194
0.328-n
0.456-0 240
0,460.10221
0.515.10.20
0.376.10.16
0.57110.204
0.32410 183
0.398.10.11
0.583
NeuroBOLT (ours)
0.568:0.166
0.566:001883
0.502=0.168
0.559:0.141
0.437=0.184
0.480:20.213
0.507
FTCL 70
0.326.00004
0.412:10.039
0.32710.078
0.497.0.0091
0.243:0001 125
0.373:10082
0.512
0.376
Totez aobject
CNN
Transfommrr
0.218:10220 200
0,412:10:14
0.29110.007
0.310:10,133
0.252:00866
0,180110.10 106
0.282:00 185
0.273
STT Transfome
0.269-0010 197
0.1 188--00056
0.226+00.199 150
0.290++1.. 143
0,74++0 126
0,142--0110 101
0.347-0 124
0.218
BIOT 59
0.457.10.12 123
0.512
0.393 128
10.081
0.299
0.413.0.073
0.529 11010
0.415
LaBraM 22
0,177:0:116
0,211::0 105
0.153:10 132
0,17010.15
0.04::0.111
0,147::0122 122
0.150-0 152
0.151
BEIRA
0.421:0.112
0.4
0.384110.1 141
0.45210.149
0.241:10.135
0,41020.097
0.492:00
0.505-00.633
0.450-0.048
0415++0.11
0.416-0.076
0.217=0.13
0.424-00022
0.529-0 092
NeuroBOLTT
0,062.101100
0,561_0.:46
0.13310.155
0.86610.136
0.33510..44
3.4 Inter-subject Prediction
Predicting Resting-state Data from Unseen Subjects To assess the generalizability of NcuroBOLT across subjects, we first evaluated its performance on held-out recordings in the resting-state dataset, this time using EEG data to predict the fMRI ROI time series across entire scans (with length of about 20 minutes). As shown in the lower part of Table our model achieves the highest average correlation across ROIs, with a score of 0.473. Figure shows the correspondence between the correlation values and the appearance of the reconstructions. This visualization reveals that NouT successfully captures major features of fMRI dynamics in reconstructing the whole unseen fMRI scan, especially for primary sensory regions and global signals. Thus, even though both EEG and resting-state fMRI have relatively low signal-to-noise ratio, NeuroBOLTT still learns meaningful EEG representations and EEG-IMRI correlations that are crucial for EEG-to-fMRI translation. Additional results including performance distributions with statistical significance, MSE table and examples visualizing the best and the worst fMRI scan reconstructions, are provided in Appendix For comprehensively displaying the performance. Furthermore, in comparison to other sate-oo-the--s EEG-fMRI translation approaches [I 25 and EEG encoding models [59] 22 29 44 48). NeuroBOLT achieves lower mean squared error (MSE) and higher correlation, with at least an 8.74% improvement in average correlations. Similar to the scenario of within-subject prediction, NeuroBOLT shows the highest performance in predicting signals from global and primary sensory regions, achieving average correlations of 0.564 and 0.522, respectively, followed by its performance in high-level cognitive and subcortical regions.
Ablation Study In this section, we scrutinize the design of NeuroBOLT on the same unseen resting- state recordings. Sableeprovidde an ablation study on the contributions of each component in the NeuroBOLT model. The table reveals that the integration of the spectral learning module markedly
8

---

Primary sensory
High-level cognitive
Subcortical
Normod fIRR ROI signal
C-42
R
n-
R- 0-22
Time 60c
1304
These (ped)
Curous
Micdla frontal anterior
Putamen
Beach's gyne
Thalaaus
Global sigral
True ROI signal
2000
10.00
Time
Figure 5: Examples of reconstruction of the unseen scans. Scans with the median performance are shown. Dashed line: ground truth; colorful lines: model prediction
enhanced the average prediction performance by 0.285 in R, underscoring the pivotal role of spectral features in EEG-fMRI translation. Further, we examine the effectiveness of our multi-scale spectral representation learning module by incrementally increasing the number of included scales. Increasing the number of scales led to significant performance improvements most of the ROIs.
In our model, the number of FFT points matches the token length. Therefore, larger window sizes produce spectrogram with higher frequency resolution but at the expense of capturing more rapid changes in temporal dynamics (also see in Figure 3). We observe that fusing multi-scale spectral features leads to better performance in capturing the projection from EEG fMRL. Additionally, as shown in Table 2) different brain regions appear to prefer various numbers of scales, likely reflecting distinct temporal- -sporrtallptealll dynamics across regions. Overall, the NeuroBOLT configuration with four scales (lg) achieves the best average performance. We have adopted this setting for all of our experiments.
Table 2: Ablation study on the different components of NeuroBOLT and various spectral scale level settings. represents the Temporal-spar spatial module, while MS stands for the Multi-scale Spectral module. denotes the number of scale levels that are used for spectral feature integration, where lo means that only a single scale that equals the token size is employed.
Primary Sensory
High-level Cognitive
Subcortical
Model
Heschi's Gyrus R-
Frooral
Precineus Anterior
Putamon
Thalamus
Sipnal R1
Avg.
MSE.
R:
MSEJ
MSEJ
R-
MSE]
R
MSEJ
R:
MSEI
R:
MSEL
T 22
0.247
0.177
0.239
0.211
0.256
0.153
0.246
0.170
0.259
0.047
0.255
0.147
0.246
0.150
0.151
MS w/
0.208
0.404
0.193
0.436
0.222
0.384
0.193
0.489
0.245
0.274
0.222
0.422
0.181
0.534
0.432
T+MS w/ lg
0.215
0.405
0.190
0.452
0.234
0.314
0.185
T+MS w/i. i:
0,208
0.430
0.186
0.186
0.212
0.421
0.193
0.169
0.234
0.323
0.211
0.129
0.178
0.533
0.422
T+MS w/ 6
0.197
9.464
1.174
0.542
0.213
0.432
0.189
0.491
0.235
0.317
0.215
0.433
0.172
0.549
0.461
T-MS w/t
0.192
0.482
0.171
0,561
0.215
0.433
0
0.406
0.235
0L335
0.208
0.453
1.171
0.564
0.473
TiMS w/
0,210
0.432
0.524
0.213
0.200
0.502
0.235
0.323
0.206
0.455
0.179
0.558
0.400
To evaluate the generalizability of our model trained on cesting-state data to other conditions, we conducted the following experiments on the task dataset collected from a different site with different scanner: (1) zero-shot generalization, where we pretrained our model on resting-state data and tested its performance subject prediction, where models were trained and evaluated only on task fMRI data; (3) fine-tuning, where models trained on resting-state data were fine-tuned with task fMRI data, and (4) joint-training where we jointly trained (using both resting-statt and auditory task fMRI data) and evaluated the model on the respective held-out test sets of both datasets.
Generalization Task-related fMRI
task data; (2) intra- and inter-
As shown in Table 3 the performance of our pretrained model on zero-shot whole-scan task IMRI reconstruction achieved performances comparable to that of the resting-statt state data, with even better performance in several regions compared with the model that was trained only on task fMRL. Fine- tuning the model on the task dataset further improved the performance substantially. Moreover, carrying out joint training using both resting-state and auditory task fMRI datasets resulted in the best performance across 4 ROIs in task fMRI prediction. Our results also suggest that joint training not necessarily facilitating the prediction on resting-state fMRI (beyond training on resting-state data alonc), likely due to the smaller sample size of the task dataset and richer variability of brain
9

---

dynamics in the resting state condition. More results of task-scan prediction, such as the intra- prediction, comparison with baselines, and performance distribution, are provided in Appendix F.4
Table 3: Performance of NeuroBOLT in inter-subject prediction in resting-state and auditory task fMRL. Mean R values between prediction and g.t. are shown. RS: Resting- State, AT: Auditory Task, RS- RS-P+AT-f: Pretraining on RS and finetuning on AT, RS+AT: joint training of RS and AT.
Training
Testing
Cuncus
Primary Sensory
Heschi's Gryus
Middle Frontal
High-level Cognltive
Precuneus Anterior
Putamen
Subcortical
Thalamus
Signal
Aug.
AT
0.887:00.877
0,33==0.026
0419:00099
0:05:
0.340=02200
0.361=0
0.372
0,380
AT
AT
1422++0.141
0.479-0.064
0.47--0058
0.460-D.071
0.187-0253
0.362-0 166
0.287+0.120
0.373
RS P-AT-
AT
0.11610.003
0.547_ 0,000
0.437.10008
0.171100005
0.241.0.188
0.401_.0.177
0.385 1..05
0.118
RS+AT
AT
0.461100101
0.516:00044
0.43410.106
0.4761.0.041
0.248
0.401=02200
0.404
0.420
0.48210.100
0.561-00.46
0,423.001155
0.4961..136
0.335-0.144
0.453-0.106
0,56410.115
RS+AT
0.47840:110
0.500=00.44
0.477
0.49420:107
0.330=0.110
0.443:00004
0.540:0.1119
0.469
4 Discussion and Conclusion
Contributions In this study, we propose NeuroBOLT, versatile deep-learning solution for pro- jecting scalp EEG to BOLD IMRI signals. By learning a multi-dimensional EEG representation from spatial, temporal, and spectral domains, NeuroBOLT shows strong capability to reconstruct resting-state fMRI signals from EEG alone. In addition to subject-specific models, it can also predict fMRI time series from held-out subjects across entire 20-minute scans, which a first in this field. It is important to note that correlations of 0.4-0.5 are considered strong for predicting moment-by moment signal fluctuations in functional neuroimaging data, given the low signal-to-noise ratio of EEG/fMRI. It is also worth mentioning that our model reconstructs resting-state fMRI signals from deep subcortical regions like the thalamus (with accuracy comparable to, or exceeding, certain cortical ROIs), using raw EEG data with relatively small set of electrodes (26), enhancing the capabilities of EEG to map subcortical dynamics. This number of channels is typically insufficient to capture detailed neural activity from such regions using methods like EEG source localization 47 Moreover, NeuToBOLT supports an arbitrary number of EEG input channels, which allows for flexible application across different EEG setups and across different experimental and clinical settings. Overall, by advancing the field of EEG-IMRI IMRI translation, this work may ultimately open new possibilities for non- non-invasive brain research and cost-effective clinical diagnostics.
Limitations Currently, our model is trained separately for each brain region, which is inefficient for the ultimate goal of high-resolution fMRI reconstruction. Since functional modules in fMRI also co-fluctuate dynamically, our future work aims to develop an integrated training approach that leverages these co- co-fluctuations aiming to enhance the model's 's ability to reconstruct high-resolution fMRI signals more efficiently. Furthermore, although our model demonstrates promising zero- shot reconstruction performance on unseen dask-based scans when trained on resting-state data even with limited samples, the sample size of our resting-state and auditory task dataset remains limited compared to existing studies on single modalities, which may still introduce potential bias or overfitting. For certain comparisons, sizable variability in the performance of the proposed method as well as the baseline methods was also observed (see Appendix F and Figures 7 8 As our model is adaptable to various channel configurations, we plan to leverage other publicly available simultaneous EEG-IMRI datasets in our future work to train the model on larger, more diverse samples. Additionally, future evaluation with multiple random seeds could help mitigate potential biases especially under small sample size, enhancing the model's reliability and generalizability.
Acknowledgments and Disclosure of Funding
This work was supported by NIH grants R01 NS1 12252 and P50 MH109429.
References
[1] Philip J Allen, Oliver Josephs, and Robert Turner. A method for removing imaging artifact from continuous eeg recorded during functional mri. Neuroimage, 12(2):230-239, 2000.
[2] Jimmy Lei Ba. Layer normalization, arXiv preprint
2016.
[3] Paul Bricman and Jelmer Borst. Eeg2firri: Cross- modal synthesis for functional neuroimaging. 2021.
10

---

[4] David Calhas and Rui Henriques. Eeg to finr synthesis: Is deep learning candidate? arXiv preprint arXiv:2009.14133 14133, 2020.
[5] David Calhas and Rui Henriques. Eeg fmrr synthesis benefits from attentional graphs of electrode relationships. arXiv preprint arXiv:2203.0344 034481, 2022.
[6] Catie Chang, Moriah Thomason, and Gary H Glover. Mapping and correction of vascular hemodynamic latency in the bold signal. Neuroimage, 43( (3)900000,,, 2008
[7] Catie Chang, Zhongming Liu, Michael C Chen, Xiao Liu, and Jeff H Duyn. Eeg correlates of time-varying bold functional connectivity, Neuroimage, 72:227 236, 2013.
[8] Zijiao Chen, Jiaxin Qing. Tiange Xiang. Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22710-22720, 2023.
[9] Zijjao Chen, Jiaxin Qing, and Juan Helen Zhou, Cinematic mindscapes: High-quality vidco reconstruction from brain activity. Advances in Neural Information Processing Systems, 36, 2024.
[10] Michael X Cohen. Where does eeg come from and what does it mean? Trends in neurosciences, 40(4): 208. 2017.
[11] Heng Cui, Aiping Liu, Xu Zhang, Xiang Chen, Kongqiao Wang, and Xun Chen. Eeg-based emotion recognition using an end-to-end regional-asymmetrii convolutional neural network. Knowledge-Basee Systems,
2020.
[12] Kamalaker Dadi, Gaël Varoquaux, Antonia Machlouzarides-Shuli Krzysztof J Gorgolewski, Demian Wassermann, Bertrand Thirion, and Arthur Mensch. Fine-grain atlases of functional mades for fmri analysis. Neurolmage, 221:117126, 2020.
[13] Jean Daunizeau, Helmut Laufs, and Karl J Friston. Eeg-fmri information fusion: biophysics and data analysis. LEG-fMRI: Physiological Basis, Technique, and Applications, pages 511 2010.
[14] Jan C de Munck, Sonia Gonçalves, R Mammoliti, Rob M Heethaar, and FH Lopes Da Silva. Interactions between different eeg frequency bands and their effect alpha ffriffiiii correlations. Neuroimage, 47(1): 69-76, 2009.
[15] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.11990 2020.
[16] Yiqun Duan, Charles Chau, Zhen Wang, Yu-Kai Wang, and Chin-teng Lin. Dewave: Discrete encoding of eeg waves for eeg text translation. Advances in Neural Information Processing Systems, 36, 2024.
[17] Sarah E Goodale, Nañs Ahmed, Chong Zhao, Jacco A de Zwart, Pinar S Ozbay, Dante Picchioni, Jeff Duyn, Dario J Englot, Victoria L Morgan, and Catie Chang. fimri-based detection of alertness predicts behavioral response variability. elife, 10:e62376, 2021.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[19] David J Heeger and David Ress. What does fmri tell about neuronal activity? neuroscience, 3(2)142-151, 2002. 2002.
Nature reviews
[20] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:1606.0844 2016.
[21] Elizabeth MC Hillman. Coupling mechanism and significance of the bold signal: a status report. Annual review of neuroscience, 37:161-181, 2014.
[22] Weibang Jiang, Liming Zhao, and Bao liang Lu. Large brain model for learning generic representations with tremendous EEG data BCI. In The Twelfth International Conference on Learning Representations, 2024. URL http:/tt/////eeeeeeeeeee/ http://pppareeeiereeee/tttor/a//
[23] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rans: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 1165- PMLR, 2020.
11

---

[24] Min-Gu Kim, Hoon Ko, and Sung Bum Pan. A study on user recognition using 2d ecg based on ensemble of deep convolutional neural networks. Journal of Ambient Intelligence and Humanized Computing. 11: 1859-1867, 2020.
[25] Alexander Kovalev. Iii Mikheev, and Alexei Ossadtchi. fimri from eeg is only deep learning away: the of interpretable to unravel seg-finri relationships. arXiv preprint urXiv:2211.0022,, 2022.
[26] Helmut Laufs. Endogenous brain oscillations and related networks detected by surface eeg combined fmri. Human brain mapping. 29(7):622 769, 2008.
[27] Helmut Laufs, Andreas Kleinschmidt, Astrid Beyerle, Evelyn Eger, Afraim Salek-Haddadi, Christine Preibisch, and Karsten Krakow. Eeg-correlated fmri of human alpha activity. Neuroimage, 19(4):1463- 1476, 2003.
[28] Helmut Laufs, John Holt, Robert Elfont, Michael Krams, Joseph Paul, Karsten Krakow, and Andreas Kleinschmidr. Where the hold signal goes when alpha eeg leaves. Neuroimage, 1444))1408-1418 2006.
[29] Hongli Li, Man Ding, Ronghua Zhang, and Chunbo Xiu. Motor imagery eeg classification algorithm based on cnn-lstm feature fusion network. Biomedical signal processing and control, 72:103342, 2022.
[30] Yamin Li, Wen Shi, Zhian Liu, Jing Li, Qiang Wang, Xiangguo Yan, Zehong Cao, and Gang Wang. Effective brain state estimation during propofol-induce sedation using advanced cog microstate spectral analysis. IEEE journal of biomedical and health informatics, 25(4):978-98) 2020.
[31] Yamin Li, Ange Lou, Ziyuan Xu, Shiyu Wang, and Catie Chang. Leveraging sinusoidal representation networks to predict fmri signals from eeg. In Medical Imaging 2024: Image Processing, volume 12926, pages 795 795-800. SPIE, 2024.
[32] Thomas T Liu. Noise contributions to the fmri signal: An overview. Neurolmage, 143:141-151, 2016.
[33] Xueqing Liu and Paul Sajda. convolutional neural network for transcoding simultancously acquired eeg-fmri data. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), pages 477-482. IEEE, 2019.
[34] Xueqing Liu and Paul Sajda. Fusing simultaneously acquired eeg and fmrr via hierarchical deep transcoding. In International Conference on Brain Informatics, pages 57-67, Springer. 2023.
[35] Xueqing Liu and Paul Sajda. Latent neural source recovery via transcoding of simultaneous eeg-fmri. In International Conference on Brain Informatics, pages 318 318-34 -303. Springer, 2023.
[36] Ange Lou, Benjamin Planche, Zhongpai Gao, Yamin Li, Tianyu Luan, Hao Ding, Terrence Chen, Jack Noble. and Ziyan Wu. Darenerf: Direction-aware representation for dynamic scenes. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5031-5042, 2024.
[37] Han Lv, Zhenchang Wang. Elizabeth Tong, Leanne M Williams, Greg Zaharchuk. Michael Zeineh, Andrea N Goldstein-Piin Pickarskk, Tali M Ball, Chengde Liao, and Max Wintermark. Resting state functional mri: everything that nonexperts have always wanted to know. American Journal of Neuroradiology, 39(8): 1390-1399, 2018
[38] Dante Mantini, Mauro Perrucci, Cosimo Del Gratta, Gian L Romani, and Maurizio Corbetta. Elec- trophysiological signatures of resting state networks in the human brain. Proceedings of the National Academy of Sciences, 13112233:111--11177 2007.
[39] Yehudit Meir-Hasson, Sivan Kinreich, Ilana Podlipsky, Talma Hendler, and Nathan Intrator. An eeg finger-print of fmri deep regional activation. Neuroimage, 102: 102:128-14: 1411, 2014.
[40] Thomas M Mochlman, Jacco A de Zwart, Miranda G Chappel-Farley, Xiao Liu, Irene B McClain, Catie Chang. Hendrik Mandelkow, Pinar S Özbay, Nicholas L Johnson, Rebecca E Bieber, et al. All-night functional magnetic resonance imaging sleep studies. Journal of neuroscience methods, 316:83-98, 2019.
[41] Frank Moss, Lawrence M Ward, and Walter G Sannita. Stochastic resonance and sensory information processing: a tutorial and review of application. Clinical newrophysiology, 115(2):267 281, 2004.
[42] Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2):61--444 1928.
[43] Ayelet Or-Borichev, Guy Gurevitch, llana Klovatch, Ayam Greental, Yulia Lerner, Dino J Levy, and Talma Hendler. Neural and functional validation of fmri-informan eeg model of right inferior frontal gyrus activity. Neurolmage, 266:1 19822, 2023.
12

---

[44] Wei Yan Peh, Yuanyuan Yao, and Justin Dauwels. Transformer convolutional neural networks for automated artifact detection in scalp eeg. In 2022 44th Annual International Conference of the IEEE Engineering Medicine Biology Society (EMBC). pages 3599-3602. IEEE, 2022.
[45] Petra Ritter and Arno Villringer. simultaneous eeg fmri. Neuroscience Biobehhavoral Reviewss, 30(6): 223-838. 2006.
[46] Claude Elwood Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10-21, 1949.
[47] Jasmine Song, Colin Davey, Catherine Poulsen, Phan Luu, Sergei Turovets, Erik Anderson, Kai Li, and Don Tucker. Eeg source localization: Sensor density and head surface coverage. Journal of neuroscience methods, 256:9-21, 2015.
[48] Yonghao Song. Xueyu Jia, Lie Yang, and Longhan Xie. Transformer based spatial-temporal feature learning for eeg decoding. arXiv preprint 11X1v:2106.1110, 2021.
[49] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 2014.
[50] Armin Thomas, Christopher Ré, and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances in neural information processing sysiems. 35:21255-21269 2022.
[51] Kevin L Turner, Kyle W Gheres, Elizabeth A Proctor, and Patrick Drew. Neurovascular coupling and bilateral connectivity during nrem and rem sleep. elife, 9:e62071, 2020.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] Sinong Wang, Belinda Z Li, Madian Khabsa. Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 04XV:22006004788 2020.
[54] Huilin Wei, Amirhossein Jafarian, Peter Zeidman, Vladimir Litvak, Adeel Razi, Dewen Hu, and Karl J Friston. Bayesian fusion and multimodal dem for eeg and fini. Neuroimage, 211:116595, 2020.
[55] Di Wu, Siyuan Li, Jie Yang, and Mohamad Sawan. neuro2vec: Masked fourier spectrum prediction for neurophysiological representation learning. arXiv preprint arXiv:2204.1 12440, 2022.
[56] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet: Temporal 2d variation modeling for general time series analysis. arXiv preprint arXiv:2210002186, 2022.
[57] Chaoqi Yang, Danica Xiao, M Brandon Westover, and Jimeng Sun. Self-supervised eeg representation learning for automatic sleep staging. arXiv preprint arXiv:2110.15278 2021.
[58] Chaoqi Yang, Cheng Qian, Navjot Singh, Cao Danica Xiao, M Westover, Edgar Solomonik, and Jimeng Sun. Atd: Augmenting cp tensor decomposition by self supervision. Advances in neural information processing systems, 35:32039-32055 2022.
[59] Chaoqi Yang, M Brandon Westover, and Jimeng Sun. BIOT: Biosignal transformer for cross-data learning in the wild, seirth- Conference on Neural Information Processing Syxtems, 2023. URL. https: wwwp:/eeee//rurieee/eeeee ttttt/fooumidd-2221. c22yT22yyd
[60] Yi, Yansen Wang, Kan Ren, and Dongsheng Learning topology-agnoste eeg representations with
geometry-aware modeling. Advances in Neural Information Processing Systems, 36, 2024.
[61] Kexu Zhang, Wen Shi, Chang Wang, Yamin Li, Zhian Liu, Tun Liu, Jing Li, Xiangguo Yan, Qiang Wang. Zehong Cao, et al. Reliability of eeg microstate analysis at different electrode densities during propofol-inducees transitions of brain states. Neurolmage, 231:117861, 2021.
13

---

A
Notation Table
Table 4: Notations used NeuroBOLT
Symbols
Descriptions
S
the 4-D fMRI sequence
V
fMRI volume each time frame
Y, y
the real and predicted IMRI ROI time series the neural network
(.).
the input EEG time sequence the number of EEG electrodes the time length of input EEG
C
T
w
the time window length for tokenization of EEG signal the stride length the dimension of the latent embedding
d
the embedding of EEG patch at channel cy. kth time window in spatiotemporal module
teg
temporal embeddings of ERG patch kth time window in spatiotemporal module the spatial embeddings ERG patch channel ey in spatiotemporal module the temporal embedding set in spatiotemporal module
sec,
T
SE
the spatial embedding set in spatiotemporal module
Geeg
the final EEG patch embedding certain channel and time window spatiotemporal module the final representation of spatiotemporal module the dimension of final representation of spatiotemporal module
Tat
det
the window size level in spectral module
L
the number of window size level in spectral module
wh
base window size spectral module
the window size at level in spectral module the frequency embedding at level / in spectral module the window embedding at level in spectral module frequency embedding
fei
FE
spectral module
WE
the window embedding set in spectral module the overall spectrum embedding in spectral module the final representation spectral module the dimension of final representation spectral module the number of tokens
Cap
Tsp
dap
k
the new dimensions of tokens after self-attention module the query, key, and value matrices in self-attention the low-rank projection matrices
Wº, W W
E. F
H
output the self-atteniion the reduced rank of self-attention matrices, D & N
B
Related Work
Before the adoption of deep learning methods, research into the correlations between EEG and fMRI suggested the potential for inferring high-dimensional fMRI features based on only partial information from relatively low-dimensional EEG measures. Most of these studies investigate EEG-fMRI relationships by regressing the power in one more EEG frequency bands against the fMRI time series, after convolving the former with a BOLD hemodynamic response model comprehensive, definitive model that summarizes the complex translation from EEG to fMRI, possibly owing to complexities such as a potential state- dependence of neurovascular coupling [5I], and research into neural-BOLD relationships is still ongoing 21
38 26). However, there is currently
Early pioneering work by 39 applied ridge regression on EEG temporal-specteeeee features to recon- struet fMRI signals from the visual cortex during an eyes-open-eyes-olosee task, and to reconstruct fMRI signals from the amygdala during auditory neurofeedback sessions. Since this study attempted reconstruct fMRT signals only from task-specific brain regions, the ability of EEG to infer fMRI sig-
nals from other regions needs further investigation. Using deep neural networks, 35 5 3 25 3]
14

---

explored the capability of encoder- based frameworks for EEG-IMRI translation. Specif- ically, models developed by [33 5
mainly focus on reconstructing fMRI volumes, but their effectiveness in recovering fMRI temporal characteristics has yet to be quantitatively assessed. The work of (25] and [31] addressed this gap in evaluating reconstruction performance in the temporal grquence-to-squenee models to directly reconstruct fMRI time series in deep brain regions from raw EEG, analyzing the temporal correlation between predicted and actual fMRI signals. However, these studies are constrained by small sample sizes and rely on a subject-specific approach, wherein models are solely trained and tested on different sections of the same individual's scans. Additionally, most of the existing research has focused on EEG-to-fMRI synthesis during task
domain by employing
conditions, including cued eye opening/closing [25] 31 34 35 54 31. leaving the (fully eyes-closed) resting state condition largely unxxplored. This gap in the field may stem from the inherently noisier and more random nature of resting-state data [32], which poses challenges for stable and accurate translation between EEG and fMRI. Yet, resting state is widely used condition due to its simplicity and ease of applicability in patient populations (37; therefore, there is a crucial need for models that can successfully operate on resting-state data.
C Rationale of Designing Multi-scale Spectral Representation
Chort-Tima Fourier Transform (STFT) is the most widely used method to effectively evaluate how the frequency content of EEG changes over time. Given an multi-channel EEG input sequence X E CCXT with C electrodes and T total time points, X can be segmented into T- + 1 segments. Each segment denoted as x E Cx where represents the stride, and w is the time window length (patch size). Fast Fourier Transform (FFT) is then applied each segment to calculate the EEG frequency spectrum, where the number of FFT is set equal to the window length w. Thus, the number of frequency features derived from each patch N,- 2
w + 1. Consequently, the frequency
features of EEG sequence input X with stride equal to window length can be denoted as
SEEG E CC [ [
which serves as the input for spectral representation learning branch.
As noted from above, one of the limitations of the STFT is its fixed resolution. The width of the window affects the representation of the signal, influencing whether frequency resolution (the ability to distinguish closely spaced frequency components) or time resolution (the ability to detect when wider window w provides better frequency resolution (i.e., higher Nÿ) but results in poorer time resolution (i.e., fewer time windows), while a narrower window enhances time resolution but sacrifices frequency resolution, as illustrated in the bottom left of Figure 3 Moreover, different neural processes are associated with different frequency bands, which may require distinct time windows to capture accurately. Therefore, in this study, we propose a multi-level spectral representation learning approach that captures and aggregates range of rapid to smooth fluctuations with adaptive windows. By using varying time window lengths, the model can isolate low low-frequency, long- dungduurtionn oscillations with larger windows and tigh-frequency, short-duration oscillations with smaller windows. This flcxibility ensures that each frequency band is represented optimally, enhancing the model's 'ss ability map EEG to fMRI signals accurately shown in Table 2
frequencies change) is prioritized.
D Dataset and Preprocessing Details
In this section, we provide more details about the simultaneous EEG-IMRI datasets used in this study as well as the preprocessing steps. The resting-state data included in this analysis were collected from 22 healthy volunteers (mean age 35.23 = 16.86, 12 females). During these fMRI scans, subjects were instructed only to rest passively with their eyes closed (resting state), and to try and keep their head still. Participants were compensated 40 dollars. The auditory-task dataset consists of 16 scans from 10 healthy volunteers (mean age = 27.20 + 4.83, 4 females). During the scans, binaural tones were delivered with randomized inter-stimulus intervals (ISI), and there are two versions of the task that differed only in the timing of tone delivery: (1) fast ISI version (6 scans, 500 TR/scan, ISI mean = 5.6 + 0.7 sec); (2) sparse ISI version (10 scans, 693 TR/scan, ISI mean = 40.1 + 14.6 sec).
Written informed consent was obtained, and all protocols were approved by the Institutional Review Boards of Vanderbilt University and NIH. This non- non-invasive imaging study poses minimal risk, and individuals with contraindication to MRI were not eligible for participation.
15

---

Removal of Scanner Artifacts in EEG. For the EEG data, gradient and ballistocardiogram (BCG) artifacts were reduced using BrainVision Analyzer 2 (Brain Products, Munich, Germany). following the parameters and procedures described in more detail in [40 17 Gradient artifact reduction was performed using the average artifact subtraction technique([]. utilizing volume triggers. After
correcting for gradient artifacts, the EEG data were downsampled to 250 Hz.
Other EEG Preprocessing Details The EEG data were collected simultaneously with fMRI using 32-channel (10-20 system) MR-compatible system with FCz as the reference (BrainAmps MR, Brain Products GmbH). The full channel names for resting-state data ordered as below: 'Fpl', 'Fp2', 'F3', 'F4", 'C'', 'C4', 'P3', 'P4', '01'. '02', 'F7'. 'F8', 'T7', 'T8', 'P7', 'P8', 'FPz', 'Fz', 'Cz', 'Pz', 'POz', 'Oz', 'TT9' 'FT10', 'TP9', 'TP10', 'EOG1', 'EOG2', 'EMGI', 'EMG2', 'EMG3', 'ECG'. For the task condition, the EEG cap channel setting differed slightly: there was no 'FPz', 'FT9'. 'FT10', 'EOG1', 'EOG2', 'EMGI', 'EMG2', 'EMG3'. Instead, the cap included the following additional channels: 'FCI', 'FC2', "CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6'.
The EOG, EMG, and ECG channels are excluded from our analysis, resulting in the final channel number of 26 in resting-state and 31 in task-condition. Before sending into the model, the raw EEG data were first resampled to 200Hz for computational efficiency while maintaining the meaningful frequency components in EEG (typically below 100Hz) according to the Nyquist-Shannon sampling theorem 42] [6], and then normalized by dividing the EEG data values by 100, since the clean EEG data typically fall into the range between - 1000100 μV. In study, no additional low-pass or high-pass filter is applied to the EEG signal. We also test the model performance in resting-state state inter-subject prediction scenario with high pass filter (0.5Hz) applied to exclude the influence of low-frequency drift. The results with or without filtering are very close, with only a 0.006 absolute difference in average across all regions, which is also non-significant by t-test (T 0.1525, p 0.8814, two-sided). Therefore, we keep the original settings.
MMRI Collection and Preprocessing Resting-state MRI data were collected on 3T Philips scanner. A high-resolution, T1 T1-weighted structural image (TR = ms, 4.60 ms, flip angle 8 deg, 150 sagittal slices, 1 mm isotropic) was acquired for anatomic reference. BOLD-IMRI data were collected using a multi-echo gradient-echo EPI sequence with a repetition time (TR)= 2100 ms, echo times= 13, 31, and 49 ms, voxel size = 3 x x mm3 slice gap 1 mm, matrix size 96 x 96, 30 axial slices. Task MRI data were collected on a 3T Siemens Prisma scanner at a different site. The Tl-weighted structural images were collected with the following parameters: TR- 2200 ms. TE 4.25 ms, flip angle- 9 deg. mm isotropic. Multi-echo gradient-echo EPI sequence was also used for collection with TR- 2100 ms, echo times 13.0, 29.4, and 45.7 ms, voxel size = 3 x x mm? slice gap 1 mm, matrix size = 82 x 50, 30 axial slices. For each task scan, the first seven volumes were dropped to allow magnetization to reach steady state.
MRI scanner (volume) triggers were recorded together with the EEG signals for data synchronization. Slice-timing and motion coregistration were applied, followed by noise reduction using multi-echo ICA as implemented in tedana 0.0.9a2] Subsequent steps included alignment to an MNI152 standard template (matrix size- 91 109 91), removal of low-order trends (up to 4th-order polynomials). and spatial smoothing (to 3mm FWHM) using AFNI
Rationale for ROI Selection Although our model is designed to predict IMRI signals in any region of the brain, we choose 7 ROIs to report as representative examples that are located in primary sensory, high-level cognitive, and subcortical regions. Cuneus and Heschl's gyrus are responsible for primary visual and auditory processing; the anterior part of the precuneus is particularly involved in several cognitive and perceptual processes; the middle frontal gyrus anterior is part of the frontal lobe, serving an important role in high-level cognitive functions such as working memory, attention, cognitive control, executive function; as for the subcortical regions, the putamen is part of the basal ganglia, serving crucial role in brain's 's motor control; thalamus acts as the brain's 's relay station, transmitting sensory and motor signals to the cerebral cortex. It plays a crucial role in regulating consciousness, sleep, and alertness by processing and relaying information from various sensory systems and the spinal cord to appropriate areas of the brain for further processing.
Phttps//tedaa...attttheeeseii rewdthheddessssso///ttaeeee hhtps://aaiii nimh. nih, gov/afni
16

---

Table 5: Hyperparameters for NeuroBOLT
Hyperparameters
Values
Batch size
intra-sub:16 16, inter-sub:64
Peak learning rate
3e-4
Minimal learning rate Learning rate scheduler
le-6
Cosine
Optimizer
Adam W
Adam / Weight decay
(0.9.0.99)
0.05
Total epochs
30
Warmup epochs
5
Drop path
0.1
Layer-wise learning rate decay
0.65
E
Training Details
E.1
Hyperparameter Settings
For the spatiotemporal module in NeuroBOLT, we leverage the architecture and pre-trained weights from the state-of-the-art EEG foundation model LaBraM [22] (pre-trained weights version: LaBraM- base), and then fine-tune it on our dataset. To keep consistent with the setting of the pre-trained model, we also
token size of in 16 tokens per channel. Since we have EEG data with C = 26 channels, the final token number would be 16 26 + Icls token 417 before being sent into the transformer, please see other detailed settings of the model architecture in the "fine-tuning session" of (22).
(200 timestamps) without overlap for this module, resulting
For the multi-scale frequency spectral representation learning module, set the smallest scale to 0.5 seconds, corresponding to 100 points at our sampling rate of 200 Hz. For the multi-scale learning, we use Is throughout our experiments it achieves the best average performance across other settings, and it includes four window scales: 0.5 seconds, I second, 2 seconds, and seconds. The hidden features derived from these four scales are first projected the same frequency embedding size (equal to the final hidden embedding size: dpiddenembed = 200) and then projected to the same temporal embedding shape (drime 32, which is derived by scatefi where T = 3200 is the total input EEG length 3200 timestamps, and as mentioned scale) = 100). These projections are both achieved by using a single layer Linear projection. Therefore the shape of the embeddings before the transformer module is (32 x 26 + 1cls token, 200). In this study, we use a linear transformer (53) reduce the complexity and improve the speed. A previous study by (59) proves that the linear and the vanilla transformer yield very similar performance in biosignal representation learning, while the former one takes less time, so we also adopt the transformer with linear complexity in this study, with heads and 4 transformer layers. We obtain the final embedding from each module by an average pooling step. Then shape of the hidden embeddings from the spatiotemporal and multi-scale spectral modules are summed up, with final embedding shape (200, 1). Other training details are shown Table 5
T
instead as our encoder
E.2 Baselines
To ensure fair comparisons across all baselines, we apply the same seed and a consistent set of hyperparameters batch size, learning rate, optimizer, and number of training epochs as detailed in Tar each reconstruction scenario.
EEG-IMRI Translation:
Li et al. [I]] A Incoder-Decodee CNN that leverages effective spectral representation learning with sinusoidal activation function for Lequence-to-Sequene EBG-IMRI prediction in deep brain regions during syes-open-eyes-cloeee only using the encoder part and then using linear projection to make predictions similar to our approach.
(EOEC) task. We adapt this model by
17

---

An interpretable CNN Encoder-Decoder for deep brain region fMRI recon- struction during EOEC task. We also adapt this model by only using the encoder part and then using linear projection to make predictions similar to our approach.
BEIRA 25
EEG Encoding:
LaBraM [22]: A recent EEG foundation model trained over 2,500 hours of EEG data to learn the spatiotemporal information EEG signals, which achieved the state-of-the-art downstream classification task performance. We maintain the original model architecture with a 200-dimensional latent embedding and utilize the pretrained weights from this model (LaBraM-basse the only publicly available checkpoint for LaBraM) for fine-ture fine-tuning datasets.
BIOT 59]: transformer-based flexible bio-signal encoder architecture that learns the temporal-spectre information from EEG signals. Since BIOT and LaBraM share similar input processing, and both produce a one-dimensional latent representation for each sample, we set BIOT"s embedding size to match LaBraM and our model (dhiddenembed 200) to ensure a fair comparison, keeping other parameters consistent with the original paper.
FFCL [9] A model that combines CNN model and LSTM model for learning separate representations, which are merged to make the final prediction.
ST-Transformer 48] a transformer-based spatial-temporal feature learning neural network for EEG classification task.
CNN Transformer
44 A transformer convolutional neural network for automated artifact
detection in EEG.
For FFCL, ST-Transformer, and CNN Transformer, retain the original model architectures as used the baseline experiments in [5]] For detailed implementation, please refer to their GitHub repository
18

---

Additional Results
E1 Held-out Resting-state Scan Reconstructions Examples
this section, we show more examples that reflect the best and the worst reconstruction performance over all ROIs in Figure 6 The total length of the scans is 1207.5 seconds. NeuroBOLT shows the capability to reconstruct the whole resting-state fMRI scan over a large time range.
Best reconstruction
Worst reconstruction
0.591
0.357
200
400
600
R 0.637
0.498
200
400
6:00
E00
1000
200
400
600
8bc
1000
Normalized fMR RO! signal
R 0.556
0.294
400
600
R 0.647
R 0.250
R 0.516
R 0.455
R 0.606
0.316
205
60c
200
400
600
800
10000
R 0.684
R 0.414
200
400
600
860
1000
2010
400
800
abC
1000
Time (sec)
Time (sec)
Cuncus Heschi's
Middle frontal gyrus enterior
Putamen
Thalamus
Global signal
True RDI signall
Figure 6: More reconstruction results for inter-subject prediction (unseen subject) Normalized predicted and real fMRI ROI signals are shown.
F.2
MSE Results Resting-state Data
Here we provide the model performance assessed by MSE values for intra- and inter-subject experi- ments conducted resting-state data (see Table 6)
19

---

Table 6: MSE in intra- and inter-subject experiments on resting-state data.
Model
Caneus
Primary Sensory Heschi's Glyns
Micklle Frontal
High-level Cogaitive
Precuneuus Anterior
Putamen
Subcortical
Thalanus
Global
Avg. MSE]
lotza Rchn
BIOT
0.271
0.269
0.237
0.287
0.238
0.274
0.249
0.258
LaBraM
22
0.276
0.249
0.255
0.273
0.236
0.211
0.277
0.259
BEIRA
0.349
01.244
0.360
0.288
0.239
0.241
0.222
0.249
NeuroBOLT (ours)
0.272
0.242
0.263
0.274
0.225
0.221
0.245
FFCL CNN Transfonmnt
29
0.225
0.212
0.230
0.205
0.250
0.230
0.189
0,220
14
0.261
0.226
0.236
0.223
0.246
0.248
0.282
0.246
intar-zubjee
BIOT
0.217
0.186
0.220
0.201
0.239
0.220
0.196
0.21
LaBraM
0.247
0.259
0.256
0.246
0.259
0.255
0.246
0,250
BFIRA
0.231
0.194
0.242
0.197
0.250
0.230
0.201
0.221
[3]
NeuroBOLT (ours)
0.192
0.171
0.215
0.197
E3 Distributions of Resting-state Reconstruction Performance
This subsection presents statistical measures for resting-state fMRI sean reconstructions in comparison with other models (see Figure 7p. While the presence of statistical significance varies methods and ROIs, our proposed approach demonstrates consistently higher mean performance than the baselines for all (or all but one) of the ROIs in the intra/inter-scan comparisons.
Intra-subject prediction freating stato, N=29)
Out
Leeul
BOT
Lical
Inter-subject (unseen) whole-scan prediction (resting-state, N-G) Lietal
Oure
IROls
IRO
Figure 7: Intra- and inter-subject EEG-to-MMI synthesis performance in resting-state. Paired t-test significance between NeuroBOLT and each baseline: *pc0.05, **pc0.01, *** **0c<00011 Error bars are showing the standard deviation (S.D.).
F,4 Task-IMRI Prediction Performance
In this subsection, we show that, in addition to the resting-state condition, NeuroBOLT is also effective when trained and tested auditory-task condition in both intra- (Figure 8 ()) and inter-subject conditions (Figure BB)), achieving the best unseen scan reconstruction performance in out of ROIs. Figure (.C) C) present comparison example between the unseen task signal reconstructed by the model fully trained on resting-state data (zero-shot) and the signal reconstructed by the model trained resting-state data and further fine-tuned task data.
20

---

(A)
intra-subject preditiin performance distriution reating-stat staegg--tteeete and audio task IMRI
(B)
Intersubbjec (unseen) No4)
Hode
0.3-
CC 4-4-
ROb
(C)
Example of auditory task fMRI reconstruction (unseen scan) within Heschl's gyrus
Ground-Iruth
Zaro-shot (train on resting)
Train on rasting and finetune on task (MRI
Nomaltedd PMN
R 0.42
R 0.60
400
Figure 8: Task-condition EEG-tt-fMRI synthesis performance (A) Distribution of correlation coefficients between ground truth and signals predicted by NeuroBOLT for intra-subject predictions under resting-state and auditory-task conditions. (B) Inter-subject prediction performance (R value) in held-out auditory task scans compared with other baselines. Paired t-test significance between NeuroBOLT and each baseline: *pc0.05, **p<0.01, ***p<0.0011 Error bars are showing the standard deviation (S.D.). (C) Example scan of inter-subject task fMRI reconstruction: zero-shot testing with pretrained model on resting state (shown in light green) and subsequent fine-tuning on task condition (shown in dark green). Note that only partial time series shown here for better visualization. R values are calculated from full sequences.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper identifies the current challenges in EEG-IMRI translation tasks and proposes a method to address them. We summarize our contributions in the abstract and also highlight them using bullet points the end of the Introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section we discuss the limitations of our method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: Our paper does not include theoretical results. But we formulate our question, and method with detailed formulas.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the Methods section, we provide detailed information about the model architecture and the dimensions of the input and output. In the supplemental materials, we offer comprehensive details on preprocessing and training.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes. We have provided our project page link, where the data and code would be updated.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nipss wttpcc/pcccccccccccccooooo http:/gguuddes/ooeeuubbbiiiii ietbmmmsssinnn ssinnioicccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 3 and supplemental materials, we report how we prepared the data and provide all training and testing details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide plots with error bars and statistical test in the Appendix 7 and we provide S.D. in the main tables 33
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In section 3.2, report these information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:::/heurrips.cc/p/bll//////
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms, in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 4, summarize the broader impacts of our work, highlighting our efforts to address clinical diagnostic challenges through non-invasive and cost-cffective methods.
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited the source code paper we used properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
latasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code for the model and our analysis will be well documented, and will be public on github. Our data will also be well documented well and can be made public upon request.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: During our data collection, subjects rested passively with their eyes closed (resting state). Participants were compensated 40 dollars. Written informed consent was ob- tained, and all protocols were approved by the Institutional Review Board. This information is included in the appendix dataset session.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We state that we have IRB approval in section 3.1 and in the Appendix D and that the study poses minimal risk to participants. Individuals with contraindication MRI
were not eligible for the study.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "3CFFonOIT6", "reviewer": "Reviewer_EkS2", "summary": "In this work, the authors present a deep learning architecture for inferring functional magnetic resonance imaging (fMRI) signal from electroencephalography (EEG) data. The proposed model, named NeuroBOLT, utilizes transformer backbones to provide spatial, temporal, and frequency-based features from the EEG are utilized for reconstruction. The authors demonstrate the performance of their architecture on a small (N=22) data set using a propriatary data set of simultaneously measured EEG-fMRI.", "questions": "How well does the model trained on other ROIs transfer to reconstruction of completely different ROIs?\n\nHow might scanner model and particular parameters of the resting state sequence affect reconstruction?\n\nWhy do you not compare anywhere with Source Localization? Source Localization is only mentioned once offhand, and its limitations and efficacy as a reconstruction technique are not gone into in detail. I am surprised it was not included as a baseline method in fact.", "limitations": "The authors do provide some discussion of the limitations of their work; however, as I have noted above, there are some limitations which are ommitted from the main body of text which at least should have been acknowledged in this section.\n\nThe authors state they have IRB approval in section 4.1 of their paper. I see no reason for additional review.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "fMRI reconstruction from simultaneous EEG is a fascinating topic, and a difficult problem to tackle. The approach taken by the authors in this work is novel for the task at hand, i.e. using a multi-scale spectral feature embedding. Although the decision to use multi-scale spectral embeddings is not new in MRI analysis, as far as I could find the approach has not been utilized for this particular problem and the authors address novel problems for their application to simultaneous EEG-fMRI data in a deep learning architecture. At best this paper is a novel methodological tweak applied with state of the art architectures to see improvements over other deep learning baselines.\n\nThe breadth of the experiments attempted by the authors is promising; however, see my discussion below for more of a discussion of the limitations of the experiments performed. \n\nThe authors also perform an ablation study to explore how the inclusion of Multi-Scale Spectral features improves model performance, thus demonstrating the benefit of combining the multi-scale spectral features with the spatiotemporal. This is well appreciated.", "weaknesses": "The major weaknesses of this work come down to weaknesses in the empirical evaluation. I am afraid that in its current state, the evaluation does not lead to a convincing demonstration of this method for fMRI reconstruciton, and the claims in the introduction about novelty coming from the application to multiple brain regions and resting-state fMRI seem somewhat overemphasized. Currently this brings it to a full reject as the paper is otherwise sound but the limitations in the evaluation are significant enough to bring it well below the threshold, and cannot be easily addressed in the rebuttle I believe. \n\nFirst, I will highlight the lack of reported standard deviations or error bars in any of the results. No standard deviations are provided in tables 1 or 2, or in any of the figures providing results. In the checklist, the authors state \"Error bars are not reported at the current stage because it would be too computationally expensive to compute over all the brain regions and for all participants,also there is limited space in the paper to put all the statistics. But we could always add this information if reviewers think it’s important to know.\"\n\nWhile I appreciate the authors' acknowledgement of this exclusion, I do think error bars are absolutely necessary to demonstrate the efficacy of the proposed method. The demonstrated improvements are often quite small (e.g. improvement from 0.540 to 0.588 in table 1), and it is not clear if the purported improvements can be explained away from model noise. I could not find any information about controlling model initialization or seeds as well to ensure that random initializations played a less significant role between experiments even with the same architecture on different regions. I absolutely think error bars are necessary for this work, and the reasoning provided by the authors is not mitigated elsewhere or behind a more significant barriers other than training and evaluation time. Additionally, the authors could have mentioned this omission in the limitation section of their main paper since I had to go to the checklist to be sure the authors were aware of the issue.\n\nSecond, in the abstract the authors highlight the ability to \"generalize to other brain areas\" and \"other conditions (such as resting state)\".  Unless I am missing something, I cannot find any experiments by the authors that address these particular gaps. The authors do provide inter- and intra-subject predictions which is interesting; however, their model is still only trained on individual ROIs, and they don't include any experiments demonstrating transfer learning between models trained on other regions, and they do not include any experiments studying other tasks BEYOND resting-state fMRI. Thus, the paper falls into the same limitation as past works which were only focused on task, just in the other direction. This work would have been much more compelling if they could demonstrate a model which trained well both on task and rest related data, or even better, which could reconstruct task-related data despite only being trained on resting-state fMRI. The acknowledgement of the limitations in the literature is thus misleading as the proposed method still suffers from these same limitations. The choice to only demonstrate the results for several ROIs highlights this limitation - it would be okay if the authors did not seem to imply elsewhere that their model gets around the single-ROI training approach from past methods. \n\nClearly, the N in this study is quite small. This is to be expected as simultaneous EEG-fMRI is still quite rare as a sequence to collect; however, the authors seem to gloss over all of the myriad issues which will come with training their data over such a small data set. I am not penalizing this work for the small N in and of itself, but as I cannot find any mention of common obstacles such as overfitting, bias towards particular kinds of reconstruction errors, and other limitations that would inevitably arise. I am extremely surprised I could find no mention of pretraining anywhere in this work, which I almost imagine would be hugely necessary for these kinds of studies with very small data sets. Again, it's not necessarily a limitation in and of itself to not do these things, with how the paper currently reads these seem to be touted as benefits of the new approach which are not backed up by evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your excellent suggestions! We address specific concerns below. Please see the PDF in the general response for additional details and figures.\n\n> **Evaluation on task fMRI**\n\nOur motivation for focusing on resting-state fMRI stems from the rich information that can be gained from naturally evolving brain dynamics. We sought to address the question of reconstructing resting-state fMRI signals since the spontaneous nature of the signals could present challenges beyond reconstructing block- or event-related fMRI task data. \nBut we strongly agree that it would be more compelling if our proposed model is evaluated on task data, in addition to rest. In this rebuttal, we now include an auditory task EEG-fMRI dataset for additional evaluation, please kindly refer to the global response part 1 for details.\n\n> **Lack of error bars/standard deviations**\n\nWe have now revised our work to include error bars in figures and standard deviations in tables, and statistical significance. Please see the global response and the PDF attachment for details.\n\n> **Model initialization and seeds**\n\nTo ensure consistency during model training, we set a fixed seed. This clarification has been added to our revised manuscript. While this is certainly an important robustness test, we plan to focus on this point in a future study due to the large scope of the current analyses. We will raise this important point in the Discussion of the revised manuscript.\n\n> **Single-ROI training**\n\nWe now realize that the wording \"generalize to other brain areas\" can imply that a single trained model can predict fMRI signals from different brain areas. In fact, we had meant to convey that our modeling framework can be trained to reconstruct the fMRI signal from an arbitrary brain region (but, indeed, ROI-specific models are trained). We have now clarified the wording in an effort to avoid this misunderstanding. \n\n> **Small N**\n\nWe acknowledge the small sample size and agree that it is important to make this point clear to readers. We have carefully reviewed the entire manuscript and made sure this limitation is discussed more clearly in the main text. \n\nRegarding pretraining, we would like to clarify that we do leverage the pre-trained weights from the EEG foundation model (LaBraM [1], trained on 2500 hours of various types of EEG data from around 20 datasets) for our “Spatiotemporal Representation Learning module.” In the original manuscript, we had briefly discussed pretraining in the last paragraph of the introduction section and in section B.1 of the appendix. We have now added clarifications across the manuscript (specifically in the Methods and in Section 4, Experiments: Implementation Detail), describing the pre-training procedure and its importance.\n\nMoreover, the additional results on the task-based fMRI collected at a different site demonstrate that training our model on a relatively small sample of resting-state data shows promising potential to generalize to different task conditions and hardware. \n\n[1] Jiang, Wei-Bang, et al. \"Large brain model for learning generic representations with tremendous EEG data in BCI.\" arXiv preprint arXiv:2405.18765 (2024).\n\n> **Question 1**\n\nPlease also see our response to “Single-ROI training” above. The current framework takes multi-channel EEG as input and predicts the fMRI ROI signal from the region on which the model is trained. We would therefore expect the model trained for one ROI would tend to reconstruct other fMRI ROIs if they share correlated temporal fluctuations. In future work, we will extend the model to predict multiple fMRI ROIs signals at once. We are revising the main text accordingly.\n\n> **Question 2**\n\nWhile our new task analysis presents an initial probe into this question, showing promising performance on task data acquired with different hardware/parameters, answering this question directly would require future work that systematically investigates altering scan parameters (while keeping the task condition, and ideally the subject, consistent) and would involve collecting more data. We believe this is a useful avenue for future work.\n\n> **Question 3**\n\nThis is a great point! Source localization is indeed a valuable method. However, we did not include source localization for the following reasons:\n\n**(1) Limited number of EEG electrodes:** Our EEG data consist of 32 electrodes. After excluding the ECG, EOG, and EMG channels from the model input, this is reduced to 26 channels. However, larger numbers of channels (e.g., >64) may be needed for sufficiently accurate source localization [1-2]. \n\n**(2) Hemodynamic filter between source electrical signal and fMRI signal:** Even if we can map the EEG into source space, the output signals would be neural electrical signals. Our present study focuses on predicting fMRI hemodynamic signals, which are blurred/delayed relative to electrical signals. Although fMRI signals can be estimated by convolving neural signals with the hemodynamic response function (HRF), the shape and peak of the HRF vary across different brain regions and individuals, which would pose challenges for aligning between fMRI and source-reconstructed EEG signals.\n\nHowever, we do think that source localization is an excellent future direction.\n\n[1] Michel, Christoph M., and Denis Brunet. \"EEG source imaging: a practical review of the analysis steps.\" Frontiers in neurology 10 (2019): 325.\n\n[2] Michel, Christoph M., et al. \"EEG source imaging.\" Clinical neurophysiology 115.10 (2004): 2195-2222.\n\nWe genuinely appreciate your commitment to ensuring the rigor of our paper and the constructive suggestions. These insights have significantly improved this work. We hope our rebuttal addresses your concerns and please let us know if you have any additional questions/comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jZX99NmPZ9", "reviewer": "Reviewer_rcEv", "summary": "The manuscript proposes an EGG-to-fMRI synthesis model. The framework implements a transformer architecture and uses a multi-channel feature combination expanded across the temporal axis. To evaluate the proposed model, EGG and fMRI data from 22 participants were recorded while they were in the resting state with eyes closed.", "questions": "What is \"In-scan\" in Table 1? This is not explained in the manuscript.\n\nIt is unclear to me why all methods are not evaluated for both inter-subject and in-scan. For example, isn't it possible to evaluate BIOT [44] for inter-subject data?", "limitations": "The manuscript sufficiently discusses its current limitations. I think the limitations section should tap into the particular scenario that the model has been evaluated on (resting state) and whether the results will be generalisable to other scenarios remains to be shown.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The manuscript addresses an interesting problem and can open opportunities for multimodal neuroimaging analysis. Overall, this line of investigation is little explored, therefore, the present manuscript is novel and of interest to the community.\n\nThe present manuscript is quite complete, (1) the model and rationale behind are sound; (2) a dataset is collected which allows a faithful evaluation of the proposed translation (from EEG to fMRI); (3) it's rather easy to read and follow the manuscript, (4) the reported results are promising.", "weaknesses": "The biggest weakness is that the framework and the dataset are only addressing the resting state. While this is an important baseline to investigate, it would have been great to explore the fidelity of the proposed framework when participants are presented with some stimuli.\n\nIt is unclear whether the source code and dataset will be released publicly.\n\nThe stability of the results is fully guaranteed given no statistical analyses are performed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and encouraging feedback on our manuscript! We are delighted that the reviewer found our work novel and of interest to the community. We are very encouraged by the reviewers’ evaluation on this work opening up opportunities for multimodal neuroimaging analysis. We greatly appreciate the positive comments on the soundness of our model, the clarity of our manuscript, and the promise of our results. We have addressed their specific questions below and included additional details in our global response.\n\n> **Evaluation on task fMRI:**\n\nOur motivation for focusing on resting-state fMRI data stems from the rich information that can be gained from naturally evolving brain dynamics. We sought to address the question of reconstructing resting-state fMRI signals since the spontaneous nature of the signals could present challenges beyond reconstructing block- or event-related fMRI task data. \nBut we strongly agree with the reviewer that it is important to investigate the fidelity of our proposed model on fMRI data that contains a task. In this rebuttal, we now include an auditory task EEG-fMRI dataset for an in-depth evaluation, please kindly refer to the **global response part 1** for details.\n\n> **Data and code availability:**\n\nWe will release all datasets on OSF, code on GitHub, and model weights on HuggingFace upon acceptance. This will provide the necessary resources for the community to validate, further explore, and build upon our models.\n\n> **Statistical analyses:**\n\nPlease kindly refer to the global rebuttal response, where we provide additional results from our statistical analyses.\n\n> **Question 1:**\n \n“In-scan” stands for within-scan (i.e., intra-scan) prediction, where part of the scan is used for training and another part is used for testing. We clarified this terminology in the revised version.\n\n>**Question 2:**\n\nAll methods can be applied to both inter-subject and in-scan predictions. We would like to clarify that we evaluated BIOT for both inter-subject and in-scan (intra-subject) frameworks (as shown in Table 1 in our original manuscript). For the in-scan evaluation, we originally selected the state-of-the-art EEG encoders (LaBraM [1] and BIOT [2]), as representative baselines for comparison. However, in our analysis for this rebuttal, we have also evaluated the intra-subject prediction performance on the two EEG-to-fMRI baselines (Li et al.[3] and BERIA [4]) to provide a more comprehensive analysis. Please see the comparison below, which shows the means and S.D. of correlation between prediction and ground truth ( full MSE table will be included in the revised manuscript). The significance of the paired t-test between our model and other baselines is indicated as follows: *:_p_<0.05; **: _p_<0.01; ***: _p_<0.001. \n\n\n| Model | Cu | He | MF | PA | Pu | Th | GLB |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| Ours | **0.59±0.17** | **0.57±0.18** | **0.50±0.17** | **0.56±0.14** | **0.44±0.18** | **0.48±0.21** | **0.59±0.16** |\n| LaBraM\\[1\\] | 0.54±0.18 | 0.52±0.20\\*\\* | 0.49±0.15 | 0.49±0.18\\* | 0.41±0.18 | 0.45±0.18 | 0.49±0.17\\*\\* |\n| BIOT\\[2\\] | 0.53±0.22 | 0.52±0.21\\* | 0.49±0.16\\*\\* | 0.46±0.11\\*\\*\\* | 0.41±0.21 | 0.41±0.23 | 0.49±0.13\\*\\*\\* |\n| Li et al.\\[3\\] | 0.46±0.22\\*\\* | 0.51±0.21 | 0.38±0.17\\*\\*\\* | 0.46±0.20\\* | 0.32±0.18\\*\\* | 0.40±0.19\\* | 0.58±0.17 |\n| BERIA\\[4\\] | 0.36±0.24\\*\\*\\* | 0.40±0.24\\*\\*\\* | 0.29±0.23\\*\\*\\* | 0.32±0.22\\*\\*\\* | 0.23±0.19\\*\\*\\* | 0.24±0.10\\*\\*\\* | 0.46±0.24\\*\\* |\n\n> Cu: Cuneus; He: Heschl’s gyrus; MF: Middle Frontal Gyrus Anterior; PA: Precuneus Anterior; Pu: Putamen; Th: Thalamus; GLB: Global Signal \n\nAgain, we genuinely appreciate the reviewer’s encouraging feedback and constructive suggestions! Please feel free to let us know if you have any further questions or comments.\n\n[1] Jiang, Wei-Bang, et al. \"Large brain model for learning generic representations with tremendous EEG data in BCI.\" arXiv preprint arXiv:2405.18765 (2024).\n\n[2] Yang, Chaoqi, et al. \"Biot: Biosignal transformer for cross-data learning in the wild.\" NeurIPS 36 (2024).\n\n[3] Li, Yamin, et al. \"Leveraging sinusoidal representation networks to predict fMRI signals from EEG.\" Medical Imaging 2024: Image Processing. Vol. 12926. SPIE, 2024.\n\n[4] Kovalev, Alexander, et al. \"fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships.\" arXiv preprint arXiv:2211.02024 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "koHATLtwug", "reviewer": "Reviewer_bwvZ", "summary": "This paper introduces NeuroBOLT, a transformer-based model. NeuroBOLT utilizes multi-dimensional representation learning across temporal, spatial, and spectral domains to translate raw EEG data into comprehensive fMRI activity signals across the entire brain. Experimental results showcase NeuroBOLT's ability to effectively reconstruct resting-state fMRI signals across primary sensory, high-level cognitive areas, and deep subcortical regions.", "questions": "Please see the twelve weaknesses above.", "limitations": "Limitations are discussed in the section of Discussion and Conclusion.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles one of the most challenging and competitive topics in neuroscience.\n2. The motivation behind the paper is quite clear, and the experimental section is logically sound.\n3. The figures and tables in the article are clear and well-organized, making it highly readable.", "weaknesses": "1. The method abbreviation and the title are not closely related. It is unclear where 'BOLT' comes from in the title, and even after reading the abstract, it remains confusing.\n2. In fact, there has been a lot of work on fMRI-EEG in recent years, especially in 2023 and 2024, but the author's related work lacks a significant amount of relevant literature.\n3. In the abstract and introduction, the author's description of the method is inconsistent with the organization in the methodology section, resulting in a need for improved readability.\n4. The writing of the article needs to be further standardized. For example, 'FMRI data' is sometimes written with a capital 'F' and other times as 'fMRI data'.\n5. Although the layout and presentation of the tables are aesthetically pleasing, the font size is too small, making them difficult to read even when enlarged.\n6. The paper does not provide code or data to support the reproducibility of results. \n7. This paper lacks details on the parameter selection for the baseline methods. Although the authors state, 'The baseline models are from [44] and [16], where we choose the models with the best downstream classification task performance,' the datasets and tasks in references [16] and [44] are not entirely consistent with those in this paper. Therefore, the authors should specify the exact process of parameter selection.\n8. The equations and symbols in the article are not very standardized. The authors should provide notation to help readers understand.\n9. The authors should conduct statistical tests to validate the significance of their methods.\n10. For readers in the NeurIPS community, the theoretical contribution of this paper appears to be weak.\n11. I don't quite understand what the author means by the third point of contribution: 'Successful resting-state fMRI reconstruction To our knowledge, this is the first study to successfully reconstruct the resting-state fMRI signal from raw EEG data, with only 26 electrodes.' What is the significance of 26 electrodes?\n12. Equation 2 does not appear to be a complete equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable suggestions, which truly help us enhance the quality and readability of our paper from multiple aspects. Responses to your concerns are presented as follows:\n\n> **Consistency and presentation details (W3-5, W8, W12):** \n\n* **W3**: We restructured the model overview in the intro and abstract, and included summarizing sentences to ensure a coherent and clear presentation of our method throughout the manuscript.  \n* **W4**: We have carefully reviewed the manuscript and ensured that we use a capital \"F\" only when \"fMRI\" appears at the beginning of a sentence, following standard conventions.  \n* **W5**: We have now moved the MSE metric to the appendix to create more space for displaying the mean and standard deviation of R values. We will include an interactive table that allows readers to easily zoom in/out on our project page.  \n* **W8**: We have now reviewed all equations and symbols throughout the manuscript to ensure they are standardized. In the revised version of the manuscript, we will include a notation section (table) in the appendix that defines all symbols and terms used in the equations to help readers better understand the mathematical content and ensure consistency throughout the manuscript.  \n* **W12**: The expression in question is intended to define the set of patch embeddings (i.e., to define the notation), rather than to present an equation.\n\n> **Meaning of the model abbreviation (W1):** \n\n“NeuroBOLT” stands for “**Neuro**-to-**BOL**D-**T**ranslation.” We have added clarification of this abbreviation both in the abstract and introduction.\n\n> **Related work (W2):**\n\nWe have incorporated additional relevant literature and discussed their relevance to our work in the Introduction section of our revised manuscript, including the works from Calhas et al. [1-2] and Liu et al. [3-4].\n\n> **Data and code availability (W6):**\n\nWe will release all datasets, code, and model weights upon acceptance. This will provide the necessary resources for the community to validate, further explore, and build upon our models.\n\n> **Training details of baselines (W7):**\n\nWe ensured that all training hyperparameters (like batch size) were consistent with those used for our model across all the baselines. Our baseline models consist of (1) SOTA EEG encoders mentioned in [5-6] and (2) EEG-to-fMRI frameworks [7-8], as detailed in Appendix B.2. We strictly adhered to the model settings provided in their original papers when training. We have clarified this and added a table in our revised manuscript that specifies these parameter settings to ensure the results' replicability.\n\n> **Statistical tests (W9):**\n\nWe have now added statistical tests for all comparisons with other baseline models (please see **General Response**).\n\n> **Theoretical contribution (W10):**\n\nOur study aims to address a key limitation in previous methods for learning the effective EEG representation useful for EEG-fMRI synthesis. We not only incorporate the SOTA EEG encoding framework but also emphasize the importance of more comprehensive frequency representations, which are often neglected in previous methods [1-4,6-8]. We propose a multi-level spectral representation learning approach that captures and aggregates a range of rapid to smooth fluctuations with adaptive windows. As demonstrated in Table 2 (ablation study), NeuroBOLT benefits from this design by effectively capturing dynamic representations and improving synthesis accuracy and fidelity. Furthermore, additional experiments on an auditory task dataset (please see the **General Response**) validate the **robustness and applicability of our learned representations across neural data from different sites and scanners.**\n\nWe believe our approach not only facilitates the translation between two neuroimaging modalities but also highlights the need for multi-scale spectral features for more effective EEG encoding in future research within this community. We will include more detailed theoretical derivations and explanations of spectral representations in the appendix to strengthen the theoretical foundation of our work.\n\n> **Significance of 26 channels (W11):**\n\nFor EEG, having fewer than 32 electrodes (26 in our case) is typically considered a small channel set, which presents challenges in accurately estimating neural activity in cortical and subcortical regions using traditional methods like EEG source localization, which often requires >64 channels [9]. However, our experiments show that even with fewer electrodes, we were able to reconstruct the fMRI signals across multiple brain regions in both intra- and inter-subject prediction scenarios. This finding suggests that our method is practical and effective, potentially broadening its applicability to real-world scenarios, such as clinical applications, where electrode count may be limited.\n\nWe genuinely appreciate the suggestions, and believe our paper will be improved with your feedback. Please let us know if you have additional questions!\n\n[1] Calhas, David, et al. \"Eeg to fmri synthesis for medical decision support: A case study on schizophrenia diagnosis.\"\n\n[2] Calhas, David, et al. \"Eeg to fmri synthesis benefits from attentional graphs of electrode relationships.\"\n\n[3] Liu, Xueqing, et al. \"Latent neural source recovery via transcoding of simultaneous EEG-fMRI.\"\n\n[4] Liu, Xueqing, et al. \"A convolutional neural network for transcoding simultaneously acquired EEG-fMRI data.\" \n\n[5] Jiang, Wei-Bang, et al. \"Large brain model for learning generic representations with tremendous EEG data in BCI.\"\n \n[6] Yang, Chaoqi, et al. \"Biot: Biosignal transformer for cross-data learning in the wild.\" \n\n[7] Li, Yamin, et al. \"Leveraging sinusoidal representation networks to predict fMRI signals from EEG.\" \n\n[8] Kovalev, Alexander, et al. \"fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships.\"\n\n[9] Michel, Christoph M., et al. \"EEG source imaging.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and their insightful, constructive suggestions. We are excited that all reviewers find the topic of our paper important and fascinating. Reviewers found our study to be novel and well-motivated with promising results, and also noted that our manuscript is well-organized (Reviewer bwvZ) and highly readable (Reviewers rcEv, bwvZ).

We have addressed below two main concerns that are raised by the reviewers: absence of (1) evaluation on task-related fMRI, and of tests regarding generalizability of our model from resting-state to task-related data; and (2) statistical measures to demonstrate the model's efficacy and feasibility. In addition to this global response, we include point-by-point responses addressing the comments of each reviewer in the separate response sections below.

> #### **1. Generalization to task-related fMRI**

One concern that the reviewers shared involved the lack of task fMRI data in our experiments. Accordingly, we have now run experiments using **a simultaneous EEG-fMRI dataset collected during auditory tasks** (see details below **\*\*Auditory Task**). We conducted additional experiments using this dataset: **(1) zero-shot generalization,** where we pre-trained our model on resting-state fMRI data and evaluated the performance on task fMRI data; **(2) intra-subject prediction and inter-subject prediction,** where we trained and evaluated models using only task fMRI data; **(3) fine-tuning,** where models trained on resting-state data were fine-tuned with task fMRI data, and **(4) joint-training,** where we jointly trained (using both resting-state and auditory task fMRI data) and evaluated the model on the respective held-out test sets of both datasets. For these experiments, we used the same parameter settings for the model training and optimization as in the main paper. Specifically, in these experiments we addressed:


*  **Feasibility of our model framework on task fMRI (new experiments 2\)**: In the PDF attachment, Fig R1.1 shows the **intra-subject** prediction performance distribution, with average correlation \= 0.51±0.18 across 7 ROIs and 16 scans. Fig R1.2 displays the **inter-subject**, unseen whole-scan prediction (trained on 3+6 scans, validated on 1+2 scans, tested on 2+2 scans for Fast+Sparse auditory tasks) performance compared with other baselines. Our model exhibits superior performance across 6 ROIs.  
* **Generalizability of our model trained with resting-state data (new experiments 1, 3 and 4).** The performance of our pretrained model on *zero-shot* whole-scan task fMRI reconstruction (see in Table R1, the first row of data) achieved performances comparable to that of our original resting-state data, with even better performance in several regions compared with the model that was trained only on task fMRI. With further *fine-tuning* on the task dataset, the performance improved significantly. Moreover, carrying out *joint training* using both resting-state and auditory task fMRI datasets resulted in the best performance across 4 ROIs in task fMRI prediction. Our results also suggest that joint training is not necessarily facilitating the prediction on resting-state fMRI (beyond training on resting-state data alone), which might be due to the smaller sample size of task data and richer variability of brain dynamics in the resting state.   
   

The results from the above experiments demonstrate initial evidence that our model is indeed able to generalize across resting-state and task fMRI data, as well as across different sites and hardware settings. 


> #### **2. Statistical measures**  

Another concern raised by the reviewers is the lack of statistical measures to demonstrate the model's efficacy and feasibility. To address this, we added bar plots (with error bars showing S.D.) and indicated the statistical significance for resting-state intra- and inter-subject predictions. For statistical testing, the correlation values are firstly Fisher-z transformed and a paired t-test was employed to test the significance of improvement of our model compared with other baselines (\*: _p_-value\<0.05; \*\*: _p_-value\<0.01; \*\*\*: _p_-value\<0.001). Our task fMRI results shown in the PDF attachment also include the S.D. and error bars, and the statistical significance of the baseline comparisons.

> **3. Data and code availability.**  

We will release all datasets on OSF, code on GitHub, and model weights on HuggingFace upon acceptance. This will provide the necessary resources for the community to validate, further explore, and build upon our models.

Finally, we would like to express our deep appreciation for the reviewers' comments and their recognition of the promising potential of our work to inspire future advancements in multi-modal neuroimaging. All the additional results (experiments on task fMRI, R values, and MSE values with S.D.) will be properly incorporated into our revised manuscript, with detailed information about task-fMRI data collection/preprocessing and implementation details of the additional experiments.


> \*\* **Auditory Task details.** Specifically, binaural tones were delivered with randomized inter-stimulus intervals (ISI), and there are two versions of the task that differed only in the timing of tone delivery: (1) **a fast ISI version** (6 scans, 500 TR/scan, ISI mean(SD) \= 5.6(0.7) sec, TR \= 2.1 sec), (2) **sparse ISI version** (10 scans, 693 TR/scan, ISI mean(SD) \= 40.1(14.6) sec, TR \= 2.1 sec). Subjects were asked to keep their eyes closed the entire time and to make a right-handed button press as soon as possible upon hearing a tone. This dataset was collected at a different site, different MR scanner (3T Siemens Prisma scanner) and using a slightly different EEG cap (32 channels but with partially different electrode settings). Detailed information about data collection and preprocessing will be included in the revised version of the manuscript.

Meta-review: The paper introduces a model that adopts the transformer architecture to convert EEG data into fMRI. The model leverages spatial, temporal, and frequency-based features to reconstruct the fMRI signal. In our review process, while some considered the methodological advancements incremental, the majority of reviewers pointed out the significance of a thorough empirical evaluation on a simultaneous EEG/fMRI dataset. All reviewers recognized the importance of the tackled problem and the strong general interest of the field in it.

Addressing specific concerns, reviewer bwvZ was mostly satisfied apart from the issue of citing the relevant literature. This remaining concern was addressed in the rebuttal, with the proposed inclusion of the relevant literature in the final version of the manuscript. I disagree with the reviewer on the basis that this alone is a strong enough reason to reject the paper. The reviewer unfortunately failed to provide specific details regarding which "relevant literature" was omitted.

In summary, this paper displays promising features despite the following three areas of concern:

- A high variance in the results of empirical evaluation, which suggests the proposed advances may not be as groundbreaking as compared to existing methodologies.
- An evaluation performed on a proprietary dataset, eliciting potential concerns over reproducibility.
- A perceived limited innovation in methodology.

The significance of the problem addressed and the empirical results that convinced two out of three reviewers make this paper worthy of acceptance. However, due to the aforementioned limitations, it is more suitable to be presented as a poster. This gives the authors an opportunity to discuss these concerns with the wider research community, potentially strengthening their work in the process.
Decision: Accept (poster)