NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping

Authors: yamin  li
ange  lou
ziyuan  xu
shengchao  zhang
shiyu  wang
dario j englot
soheil  kolouri
daniel  moyer
roza g bayrak
catie  chang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:28:18

Abstract: Functional magnetic resonance imaging (fMRI) is an indispensable tool in modern neuroscience, providing a non-invasive window into whole-brain dynamics at millimeter-scale spatial resolution. However, fMRI is constrained by issues such as high operation costs and immobility. With the rapid advancements in cross-modality synthesis and brain decoding, the use of deep neural networks has emerged as a promising solution for inferring whole-brain, high-resolution fMRI features directly from electroencephalography (EEG), a more widely accessible and portable neuroimaging modality. Nonetheless, the complex projection from neural activity to fMRI hemodynamic responses and the spatial ambiguity of EEG pose substantial challenges both in modeling and interpretability. Relatively few studies to date have developed approaches for EEG-fMRI translation, and although they have made significant strides, the inference of fMRI signals in a given study has been limited to a small set of brain areas and to a single condition (i.e., either resting-state or a specific task). The capability to predict fMRI signals in other brain areas, as well as to generalize across conditions, remain critical gaps in the field. To tackle these challenges, we introduce a novel and generalizable framework: NeuroBOLT, i.e., Neuro-to-BOLD Transformer, which leverages multi-dimensional representation learning from temporal, spatial, and spectral domains to translate raw EEG data to the corresponding fMRI activity signals across the brain. Our experiments demonstrate that NeuroBOLT effectively reconstructs unseen resting-state fMRI signals from primary sensory, high-level cognitive areas, and deep subcortical brain regions, achieving state-of-the-art accuracy with the potential to generalize across varying conditions and sites, which significantly advances the integration of these two modalities.

Main paper: 
 introduction: 
 Functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) are the most commonly utilized non-invasive neuroimaging techniques, providing crucial insights into brain functionality. These two modalities offer distinct advantages and limitations that can complement one another when combined [45, 13] . fMRI offers high spatial resolution imaging of whole-brain activity by measuring blood-oxygen-level-dependent (BOLD) signal changes, which facilitates probing regional and network-level function. However, fMRI suffers from low temporal resolution and hemodynamic blurring, which limits its ability to capture the accurate timing of rapid neuronal activity dynamics. Additionally, its high cost, non-portability, and incompatibility with metal implants further restrict the utility of MRI in certain contexts. Conversely, EEG stands out as a low-cost, portable neuroimaging modality with high temporal resolution [10, 30, 61] . However, EEG faces Figure 1 : Overall illustration of EEG-to-BOLD fMRI translation using NeuroBOLT. limitations due to volume conduction and from the superficial location of electrodes (on the scalp), making it difficult for EEG to accurately infer the origins of the measured electrical potentials. In this context, the high spatial resolution of fMRI becomes indispensable, especially for imaging deep brain regions.One approach for investigating the relationship between EEG and fMRI signals is by analyzing data collected simultaneously from both modalities. Such studies have demonstrated correlations between fMRI data and various features of EEG signals, such as the power in certain frequency bands [27, 7, 28] , which underscore the potential of EEG to inform fMRI features. However, factors such as the disparate biophysical origins of the two signals, and differences in their spatial and temporal resolutions, can limit the accuracy, interpretation, and consistency of correlation-based EEG-fMRI studies. These challenges are exacerbated in conditions during which a participant is resting passively (i.e., resting state), which is characterized by significant noise and randomness. Consequently, the mechanisms linking neuronal activity to BOLD signals remain only partially understood, posing challenges in mapping EEG to fMRI.With recent progress in cross-modality synthesis and brain decoding techniques using deep neural networks [8, 9, 16] , EEG-to-fMRI synthesis has emerged as a promising yet largely untapped research area [4] . Early pioneering work by [39, 43] employed ridge regression on EEG temporal-spectral features to reconstruct fMRI signals from the visual cortex and sub-cortical regions. More recent studies have used deep neural networks for EEG-fMRI translation [33-35, 5, 3, 25, 31] , and those of [25] and [31] developed sequence-to-sequence (Seq-to-Seq) models to reconstruct fMRI time series of deep brain regions from EEG. However, these efforts have primarily focused on task datasets, including cued eye opening/closing [25, 31, 34, 35, 54, 3] , leaving the (fully eyes-closed) resting state condition largely unexplored. In addition, prior studies rely on a subject-specific approach, wherein models are solely trained and tested on different sections of the same individual's scans. Further, only a small number of brain regions have been examined in current studies, leaving as an open question the predictive power of EEG for fMRI signals across broader areas (see Appendix B for further discussion of related work). These limitations underscore the need for developing more generalizable models for EEG-fMRI translation that are accompanied by more comprehensive evaluation.In this paper, we present NeuroBOLT: a multi-dimensional transformer-based EEG encoding framework for Neural-to-BOLD fMRI Translation (Figure 2 ). NeuroBOLT is designed to capture and integrate multi-dimensional representations across temporal, spatial, and spectral domains, offering a generalizable approach for translating raw EEG waves to BOLD fMRI time series from regions of interest (ROIs). Drawing inspiration from vision transformer (ViT) [15] and its adaptation for multichannel biosignals [59, 22] , our framework transforms raw EEG data into unified "EEG patches", enabling flexible and effective EEG data encoding. These EEG patches are then passed into two parallel sub-modules: (1) spatiotemporal and (2) spectral representation learning modules. The spatiotemporal module is propelled by a recently proposed EEG foundation model (LaBraM) [22] , which has been trained to learn effective representations on about 2,500 hours of various types of EEG signals. Moreover, as previous studies have emphasized the importance of leveraging spectral features in EEG representation learning [59, 57, 60] , EEG-fMRI correlations [27, 14] , and EEG-to-fMRI synthesis [31, 39, 25] , the spectral module incorporates multi-scale encoding on EEG spectrogram patches. Specifically, instead of employing a fixed window size for the Short-Time Fourier Transform (STFT) as commonly used [59, 58, 24, 11] , we incorporate spectral features from windows of varying scales. This approach retains the advantages of high temporal resolution from smaller windows and high frequency resolution from larger windows, thereby enhancing the spectral analysis. The output embeddings from the above two encoding modules are then integrated, allowing NeuroBOLT to capture the complexity of neural dynamics and learn the projection from neural to BOLD signals.The key contributions of this work are summarized as follows:1) Generalizable EEG-to-fMRI translation framework. We introduce a novel approach, Neu-roBOLT, that utilizes multi-spectral representation to predict high-dimensional fMRI signals from raw EEG data without relying on predefined assumptions about the hemodynamic delay between fMRI and EEG signals. Additionally, our model is designed to accommodate any number of EEG channels, enhancing its versatility across various experimental setups.2) Comprehensive evaluation of the predictive power. We performed comprehensive experiments on subject-specific and cross-subject learning, across selected ROIs in primary sensory, high-level cognitive, and deep brain regions. Further, we probe the generalizability of our approach across data acquired in both resting state and task (auditory stimulus) conditions.3) Successful resting-state fMRI reconstruction. To our knowledge, this is the first study to successfully reconstruct the eyes-closed resting-state fMRI signal from raw EEG data, with a relatively small number (26) of EEG electrodes. 
task formulation: 
 While it is well-established that fMRI signals are coupled to neuronal activity, the details of this process are only partially understood and still under debate [19] . In practice, most studies model the relationship between neuronal activity and fMRI by convolving the assumed neural activity with the hemodynamic response function (HRF), which reaches its peak at about 6 seconds and gradually returns to baseline over the next 12-15 seconds. However, the HRF is not uniform, varying significantly across different brain regions and between individuals. Moreover, although EEG can detect the electrical activity of neural populations at millisecond timescales, it suffers from volume conduction, making it challenging to extract its neural origins. This spatial ambiguity also complicates the interpretation of EEG and its relation to fMRI data.In this study, we aim to build a neural network to learn this complicated projection from the electrophysiological signal measured by EEG to the BOLD signal measured by fMRI. Our approach tries to overcome the above challenges in the following ways: 1) it captures multi-dimensional features of EEG across temporal, spatial, and spectral domains, learning representations that are crucial for accurate fMRI synthesis; 2) it does not assume a pre-defined delay between the two modalities, instead, we extract time windows of EEG data that span a duration approximating the length of HRF preceding each fMRI data point as the inputs to predict the corresponding fMRI data values (i.e., a Sequence-to-One model), in order to accommodate potential variability across subjects, regions, and frequencies.Functional MRI data are conventionally represented in four dimensions, and can be expressed as a temporal sequence S = {V 1 , ..., V K } with K observations, where each observation is a volume V ∈ R x×y× z with spatial dimensions x, y, and z. This high-dimensional fMRI signal is often summarized by representing it as a set of brain areas (i.e., P parcels), each with a corresponding fMRI signal that is averaged over its constituent voxels v x,y,z ⊂ V . Here, we employ such a functional parcellation, Dictionaries of Functional Modes (DiFuMo) [12] , which was learned across millions of fMRI volumes (see details in [12, 50] ). We utilize the DiFuMo parcellation with P = 64 regions, and the resulting parcellated fMRI data Y ∈ R P ×K contains the signals from each of these regions.Since fMRI signals are delayed by several seconds compared to the corresponding neural activity, the relationship between an input EEG window and the corresponding fMRI prediction is defined as Ŷp,t = f θ (X t-T :t-1 ), where Ŷp,t ∈ R 1 is the reconstructed fMRI value from the pth ROI at time index t, and X ∈ R C×T represents the multi-channel EEG signal inputs with C electrodes, T total timepoints. To estimate f θ (.), we formulate an optimization problem that minimizes the loss L between the predicted fMRI signal Ŷp,t and the true fMRI signal Y p,t as follows: minEQUATIONThe function f θ is NeuroBOLT, which includes a temporal-spatial representation module and a spectral representation learning module (Figure 2 ).In this section, we introduce our model: NeuroBOLT, a general architecture for translating EEG signals to fMRI. Our model is designed to accommodate input EEG signals with an arbitrary number of channels. As shown in Figure 2 , we leverage the pre-trained EEG foundation model, LaBraM (checkpoint version: LaBraM-base) [22] , and finetune on our dataset to obtain spatiotemporal representations of EEG signals. Additionally, we propose multi-scale spectral feature fusion to obtain comprehensive spectral representations. These two modules learn complementary attributes of EEG data. Finally, we integrate the spatiotemporal and multi-scale spectral representations and feed them into a regression head for fMRI prediction.We formulate the multi-channel EEG signals as X ∈ R C×T , where C represents the number of EEG electrode channels and T denotes the time length of the input EEG. To obtain the spatiotemporal representation of a given set of EEG signals, we leverage an operation from LaBraM [22] , which segments the EEG signals into patches. Assuming the time window length (patch size) for tokenization of EEG signal is w and the stride is s, X can be segmented into T -w s + 1 segments, with each segment denoted as x ∈ R C×w . In this work, we use a window of length w and a stride s = w to segment each EEG channel into patches, obtainingx = {x cj ,k ∈ R w | j = 1, 2, . . . , C, k = 1, 2, . . . , T w }.The total number of patches is C × T w . These patches are then passed forward to a temporal encoder [22] to obtain the patch embeddings, denoted as:EQUATION)where d is the dimension of the embedding.To enable the model to capture the temporal and spatial information of the patch embeddings, we set up a list of trainable temporal embeddings and spatial embeddings, denoted as T E = {te k | k = 1, 2, . . . , T w } and SE = {se j | j = 1, 2, . . . , C}, respectively. Thus, the final segment embedding e seg can be represented as the sum of the output embedding, temporal embedding, and spatial embedding, denoted as:EQUATIONThe segment embedding e seg is directly fed into the Transformer encoder [52] to obtain the output embeddings. We then apply average pooling to these embeddings to obtain the spatial and temporal representation r st ∈ R dst , where d st denotes the dimensionality.Multi-scale Spectral Representation Compared to RGB images, EEG signals present several challenges, such as a low signal-to-noise ratio, apparent stochasticity, nonstationarity, and nonlinear characteristics, making the reconstruction of the original signals difficult [41] . Previous research indicates that the frequency spectrum of EEG signals is crucial for understanding the brain's neurophysiological activities [59, 55] . Therefore, in our work, we utilize the Short-Time Fourier Transform (STFT) to achieve a spectral representation of EEG signals. Unlike most recent state-of-the-art methods, we design a multi-scale spectral approach that captures both coarse and fine representations in the temporal and frequency domains [56, 36] . Details of rationale can be found in Appendix C). For the EEG signals X ∈ R C×T , we set the base window size as w b and define the window size at level l as w l = w b × 2 l , where l = 0, 1, . . . , L represents the level number. Thus, the EEG patches with different window size can be denoted as x l = {x l,cj ,k ∈ R w l | j = 1, 2, . . . , C, k = 1, 2, . . . , T w l }. After obtaining the EEG patches, Fast Fourier Transform (FFT) is applied to each patch with the FFT size matches the window length to calculate the magnitude spectrum with w l 2 + 1 frequency bins at each level l, which can be denoted ass l ∈ R C× T w l ×( w l 2 +1) .Similar to the operation in spatiotemporal module, we define a list of trainable frequency embeddings for each magnitude spectrum at different levels, represented by:EQUATIONThese frequency embeddings, which vary in the number of temporal windows, are further mapped to a set of window embeddings as follows:EQUATIONThen, we calculate the sum of the we l for each level to obtain an overall spectrum embedding as e sp ∈ R C×n×d as shown in equation 6EQUATIONFor each channel, we apply spatial embedding, similar to the spatiotemporal representation section, to obtain the final embedding with dimensions e sp ∈ R C×n×d , where C, n and d represent the channel number, window embedding dimension, and frequency embedding dimension, respectively. The original Transformer is known to have quadratic complexity in both time and space, and EEG signals typically contain tens of channels. To learn these embeddings with lower complexity [59] , we feed spectrum embeddings e sp into a linear Transformer Encoder [53, 23] . Let the input embedding e sp ∈ R N ×d , where N = C × n represents the number of tokens. The self-attention operation in the linear Transformer Encoder can be formulated as follows:EQUATIONHere W Q , W K , W V ∈ R d×k are the query, key and value matrices. The self-attention module uses a rank-D approximation for the softmax attention (N × N ) by reduced-rank parameter matricesE ⊤ ∈ R N ×D , F ∈ R D×N (where D ≪ N ).Main components of this module include one linear self-attention layer and one fully connected network. Layer normalization before each component [2] , residual connection after each component [18] , and dropout right after the self-attention to enable stable training [49, 59] . The output is fed into average pooling along the token dimension to obtain the final spectral representation r sp ∈ R dsp , where r sp has the same dimension as the spatiotemporal representation r st .Projection Head The hidden embeddings from the above two modules are then summed up and fed into a regression head, which consists of Gaussian Error Linear Unit (GELU) [20] followed by a single linear layer, to make the final prediction of fMRI in the p th ROI at time t.Ŷp,t = Linear(GELU(r st + r sp ))3 ExperimentsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In Section 5, we discuss the limitations of our method.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
datasets: 
 Resting-state dataset EEG and fMRI data were collected simultaneously from 24 healthy volunteers in two sessions, each lasting 20 minutes. Scans with significant artifacts in the EEG or fMRI data were excluded for further analysis. The final sample contains 29 fMRI scans from 22 subjects, 7 of whom had two scans. During these fMRI scans, subjects rested passively with their eyes closed (resting state). Written informed consent was obtained, and all protocols were approved by the Institutional Review Board. BOLD fMRI data were collected on a 3T scanner using a multi-echo gradient-echo EPI sequence with repetition time (TR) = 2100 ms. Scalp EEG was acquired simultaneously with fMRI using a 32-channel (10-20 system) MR-compatible system with FCz as the reference (BrainAmps MR, Brain Products GmbH) at a sampling rate of 5 kHz, and was synchronized to the scanner's 10 MHz clock to facilitate MR gradient artifact reduction. Details of EEG and fMRI acquisition, preprocessing as well as artifact reduction can be found in Appendix D.Auditory Task Dataset To further evaluate the model generalization performance, we also include simultaneous EEG-fMRI data (16 scans from 10 subjects) collected during auditory tasks. Subjects were asked to keep their eyes closed the entire time and to make a right-handed button press as soon as possible upon hearing a tone. This dataset was collected at a different site, different MR scanner (3T Siemens Prisma scanner) and using a slightly different EEG cap (32 channels but with partially different electrode settings). Please also see detailed information about data collection, preprocessing, and experiments in Appendix D.Unless specified otherwise, the experiments and results below refer to the resting-state data.Preprocessing We employ DiFuMo with n = 64 dimensions [12] (see Section 2.1) to extract measured BOLD signals within specific ROIs. We focus on seven ROIs that span a range of spatial locations and functional roles, namely: primary sensory regions (cuneus and Heschl's gyrus), high-level cognitive areas (anterior precuneus, anterior and middle frontal gyri), subcortical regions (putamen and thalamus), and the global (whole-brain average) signal. From these ROI time series, we regress out motion confounds, low-pass filter the signals below 0.15Hz, and use the 95th-percentile of the absolute amplitude to normalize the demeaned ROI time courses. To prepare the EEG data for input, we first exclude the ECG, EOG, and EMG channels (remaining 26 channels), and resample the EEG to 200 Hz to enhance computational efficiency while preserving meaningful frequency components (which are typically below 100 Hz). To predict the fMRI ROI signal, we extract EEG windows of 16 seconds before each fMRI data point. This window length is selected to encompass the peak and most of the variation in the hemodynamic response function. Additionally, this duration aligns with the maximum temporal encoding window length of the pre-trained LaBraM model [22] , ensuring optimal fine-tuning. The normalization of the EEG also strictly follows [22] , where EEG data values are divided by 100 so that the resulting amplitude falls primarily between -1 to 1, since a typical EEG amplitude range is from -100 µV to 100 µV .The baseline models include state-of-the-art EEG encoding models from [59] and [22] , as well as two EEG-to-fMRI translation baselines [25, 31] . Since the original EEG-to-fMRI baselines are sequence-to-sequence models, we adapted them by adding a final projection layer to their encoder to enable sequence-to-one prediction. Additional details on baseline implementations are provided in the Appendix E.2. We employed the following two metrics for comparison: 1) Pearson correlation coefficient (R), which measures the strength and direction of the linear relationship between prediction and ground truth; 2) Mean squared error (MSE), which measures the average squared differences between prediction and ground truth across samples. The main text highlights the correlation coefficient as the primary evaluation metric, while MSE results are provided in the Appendix Table 6 .Implementation Details For the spatiotemporal module, we initialize the model by loading the pretrained weights from LaBraM-base [22] , with a token length = 200, i.e., 1 second, with no overlap. Please refer to [22] for more details of EEG pretraining. For the multi-scale spectral module, we set the smallest scale size l 0 = 100, i.e., 0.5 seconds without overlap. Experiments are conducted on a single RTX A5000 GPU using Python 3.9.12, Pytorch 2.0.1, and CUDA 11.7. To ensure consistency during model training, we set a fixed seed across all experiments. The batch sizes are set at 16 and 64 for intra-subject and inter-subject analyses, respectively. AdamW is utilized as our optimizer, and MSE as our training objective. The initial learning rate is set at 3e-4 with a weight decay of 0.05, and a minimal learning rate of 1e-6. For subject-specific prediction, where training and testing occur on the same scan, we split the scan in an 8:1:1 ratio for training, validation, and testing, i.e., training on the first 80% of the data and testing on the last 10%. Given that the fMRI signal exhibits significant autocorrelation, typically extending from about -10 to 10 seconds [6] , we implement gaps of 20 seconds between the training and validation sets, as well as between the validation and testing sets, to prevent data leakage. For the inter-subject analysis in resting-state data, we randomly divided the datasets into training/validation/testing sets by approximately 3:1:1 (18 scans : 5 scans : 6 scans). For the task data, we split the scans by 9 scans : 3 scans : 4 scans. Since data from the same individual might have shared representations, the two scans from the same individual are ensured to be in the same set (either training, validation, or testing set) to avoid possible data leakage. All models are optimized on the training set and evaluated on the test set, with the best model and hyperparameters selected based on the validation set.In this section, models were trained on approximately 16 minutes of EEG data, and used to forecast a future interval of fMRI signal (about 2 minutes) within a resting-state scan. Figure 4 illustrates the distribution of correlation coefficients (R) between the predictions and the ground truth, along with examples that represent the average performance of our model. The quantitative results are detailed in the upper part of In our model, the number of FFT points matches the token length. Therefore, larger window sizes produce a spectrogram with higher frequency resolution but at the expense of capturing more rapid changes in temporal dynamics (also see in Figure 3 ). We observe that fusing multi-scale spectral features leads to better performance in capturing the projection from EEG to fMRI. Additionally, as shown in Table 2 , different brain regions appear to prefer various numbers of scales, likely reflecting distinct temporal-spectral dynamics across regions. Overall, the NeuroBOLT configuration with four scales (l 3 ) achieves the best average performance. We have adopted this setting for all of our experiments.Table 2 : Ablation study on the different components of NeuroBOLT and various spectral scale level settings. T represents the Temporal-spatial module, while MS stands for the Multi-scale Spectral module. l denotes the number of scale levels that are used for spectral feature integration, where l 0 means that only a single scale that equals the token size is employed. As shown in Table 3 , the performance of our pretrained model on zero-shot whole-scan task fMRI reconstruction achieved performances comparable to that of the resting-state data, with even better performance in several regions compared with the model that was trained only on task fMRI. Finetuning the model on the task dataset further improved the performance substantially. Moreover, carrying out joint training using both resting-state and auditory task fMRI datasets resulted in the best performance across 4 ROIs in task fMRI prediction. Our results also suggest that joint training is not necessarily facilitating the prediction on resting-state fMRI (beyond training on resting-state data alone), likely due to the smaller sample size of the task dataset and richer variability of brain dynamics in the resting state condition. More results of task-scan prediction, such as the intra-subject prediction, comparison with baselines, and performance distribution, are provided in Appendix F.4. Limitations Currently, our model is trained separately for each brain region, which is inefficient for the ultimate goal of high-resolution fMRI reconstruction. Since functional modules in fMRI also co-fluctuate dynamically, our future work aims to develop an integrated training approach that leverages these co-fluctuations, aiming to enhance the model's ability to reconstruct high-resolution fMRI signals more efficiently. Furthermore, although our model demonstrates promising zeroshot reconstruction performance on unseen task-based scans when trained on resting-state data even with limited samples, the sample size of our resting-state and auditory task dataset remains limited compared to existing studies on single modalities, which may still introduce potential bias or overfitting. For certain comparisons, sizable variability in the performance of the proposed method as well as the baseline methods was also observed (see Appendix F and Figures 7, 8 ). As our model is adaptable to various channel configurations, we plan to leverage other publicly available simultaneous EEG-fMRI datasets in our future work to train the model on larger, more diverse samples. Additionally, future evaluation with multiple random seeds could help mitigate potential biases especially under small sample size, enhancing the model's reliability and generalizability. Before the adoption of deep learning methods, research into the correlations between EEG and fMRI suggested the potential for inferring high-dimensional fMRI features based on only partial information from relatively low-dimensional EEG measures. Most of these studies investigate EEG-fMRI relationships by regressing the power in one or more EEG frequency bands against the fMRI time series, after convolving the former with a BOLD hemodynamic response model [38, 26] . However, there is currently no comprehensive, definitive model that summarizes the complex translation from EEG to fMRI, possibly owing to complexities such as a potential statedependence of neurovascular coupling [51] , and research into neural-BOLD relationships is still ongoing [21, 19] .Early pioneering work by [39] applied ridge regression on EEG temporal-spectral features to reconstruct fMRI signals from the visual cortex during an eyes-open-eyes-closed task, and to reconstruct fMRI signals from the amygdala during auditory neurofeedback sessions. Since this study attempted to reconstruct fMRI signals only from task-specific brain regions, the ability of EEG to infer fMRI signals from other regions needs further investigation. Using deep neural networks, [33-35, 5, 3, 25, 31] explored the capability of encoder-decoder based frameworks for EEG-fMRI translation. Specifically, models developed by [33] [34] [35] 5] mainly focus on reconstructing fMRI volumes, but their effectiveness in recovering fMRI temporal characteristics has yet to be quantitatively assessed. The work of [25] and [31] addressed this gap in evaluating reconstruction performance in the temporal domain by employing sequence-to-sequence models to directly reconstruct fMRI time series in deep brain regions from raw EEG, analyzing the temporal correlation between predicted and actual fMRI signals. However, these studies are constrained by small sample sizes and rely on a subject-specific approach, wherein models are solely trained and tested on different sections of the same individual's scans. Additionally, most of the existing research has focused on EEG-to-fMRI synthesis during task conditions, including cued eye opening/closing [25, 31, 34, 35, 54, 3] , leaving the (fully eyes-closed) resting state condition largely unexplored. This gap in the field may stem from the inherently noisier and more random nature of resting-state data [32] , which poses challenges for stable and accurate translation between EEG and fMRI. Yet, resting state is a widely used condition due to its simplicity and ease of applicability in patient populations [37] ; therefore, there is a crucial need for models that can successfully operate on resting-state data.Short-Time Fourier Transform (STFT) is the most widely used method to effectively evaluate how the frequency content of EEG changes over time. Given an multi-channel EEG input sequence X ∈ R C×T with C electrodes and T total time points, X can be segmented into T -w s + 1 segments. Each segment denoted as x ∈ R C×w , where s represents the stride, and w is the time window length (patch size). Fast Fourier Transform (FFT) is then applied to each segment to calculate the EEG frequency spectrum, where the number of FFT is set equal to the window length w. Thus, the number of frequency features derived from each patch N f = w 2 + 1. Consequently, the frequency features of EEG sequence input X with stride equal to window length can be denoted asEQUATION, which serves as the input for spectral representation learning branch.As noted from above, one of the limitations of the STFT is its fixed resolution. The width of the window affects the representation of the signal, influencing whether frequency resolution (the ability to distinguish closely spaced frequency components) or time resolution (the ability to detect when frequencies change) is prioritized. A wider window w provides better frequency resolution (i.e., higher N f ) but results in poorer time resolution (i.e., fewer time windows), while a narrower window enhances time resolution but sacrifices frequency resolution, as illustrated in the bottom left of Figure 3 . Moreover, different neural processes are associated with different frequency bands, which may require distinct time windows to capture accurately. Therefore, in this study, we propose a multi-level spectral representation learning approach that captures and aggregates a range of rapid to smooth fluctuations with adaptive windows. By using varying time window lengths, the model can isolate low-frequency, long-duration oscillations with larger windows and high-frequency, short-duration oscillations with smaller windows. This flexibility ensures that each frequency band is represented optimally, enhancing the model's ability to map EEG to fMRI signals accurately as shown in Table 2 .In this section, we provide more details about the simultaneous EEG-fMRI datasets used in this study as well as the preprocessing steps. Written informed consent was obtained, and all protocols were approved by the Institutional Review Boards of Vanderbilt University and NIH. This non-invasive imaging study poses minimal risk, and individuals with contraindication to MRI were not eligible for participation.Removal of Scanner Artifacts in EEG. For the EEG data, gradient and ballistocardiogram (BCG) artifacts were reduced using BrainVision Analyzer 2 (Brain Products, Munich, Germany), following the parameters and procedures described in more detail in [40, 17] . Gradient artifact reduction was performed using the average artifact subtraction technique [1] , utilizing volume triggers. After correcting for gradient artifacts, the EEG data were downsampled to 250 Hz.Other EEG Preprocessing Details The EEG data were collected simultaneously with fMRI using a 32-channel (10-20 system) MR-compatible system with FCz as the reference (BrainAmps MR, Brain Products GmbH). The full channel names for resting-state data ordered as below: 'Fp1', 'Fp2', 'F3', 'F4', 'C3', 'C4', 'P3', 'P4', 'O1', 'O2', 'F7', 'F8', 'T7', 'T8', 'P7', 'P8', 'FPz', 'Fz', 'Cz', 'Pz', 'POz', 'Oz', 'FT9', 'FT10', 'TP9', 'TP10', 'EOG1', 'EOG2', 'EMG1', 'EMG2', 'EMG3', 'ECG'. For the task condition, the EEG cap channel setting differed slightly: there was no 'FPz', 'FT9', 'FT10', 'EOG1', 'EOG2', 'EMG1', 'EMG2', 'EMG3'. Instead, the cap included the following additional channels: 'FC1', 'FC2', 'CP1', 'CP2', 'FC5', 'FC6', 'CP5', 'CP6'.The EOG, EMG, and ECG channels are excluded from our analysis, resulting in the final channel number of 26 in resting-state and 31 in task-condition. Before sending into the model, the raw EEG data were first resampled to 200Hz for computational efficiency while maintaining the meaningful frequency components in EEG (typically below 100Hz) according to the Nyquist-Shannon sampling theorem [42, 46] , and then normalized by dividing the EEG data values by 100, since the clean EEG data typically fall into the range between -100-100 µV . In our study, no additional low-pass or high-pass filter is applied to the EEG signal. We also test the model performance in resting-state inter-subject prediction scenario with a high pass filter (0.5Hz) applied to exclude the influence of low-frequency drift. The results with or without filtering are very close, with only a 0.006 absolute difference in average across all regions, which is also non-significant by t-test (T = 0.1525, p = 0.8814, two-sided). Therefore, we keep the original settings. Slice-timing and motion coregistration were applied, followed by noise reduction using multi-echo ICA as implemented in tedana 0.0.9a2 . Subsequent steps included alignment to an MNI152 standard template (matrix size = 91 × 109 x 91), removal of low-order trends (up to 4th-order polynomials), and spatial smoothing (to 3mm FWHM) using AFNI 3 .Rationale for ROI Selection Although our model is designed to predict fMRI signals in any region of the brain, we choose 7 ROIs to report as representative examples that are located in primary sensory, high-level cognitive, and subcortical regions. Cuneus and Heschl's gyrus are responsible for primary visual and auditory processing; the anterior part of the precuneus is particularly involved in several cognitive and perceptual processes; the middle frontal gyrus anterior is part of the frontal lobe, serving an important role in high-level cognitive functions such as working memory, attention, cognitive control, executive function; as for the subcortical regions, the putamen is part of the basal ganglia, serving a crucial role in brain's motor control; thalamus acts as the brain's relay station, transmitting sensory and motor signals to the cerebral cortex. It plays a crucial role in regulating consciousness, sleep, and alertness by processing and relaying information from various sensory systems and the spinal cord to appropriate areas of the brain for further processing. For the spatiotemporal module in NeuroBOLT, we leverage the architecture and pre-trained weights from the state-of-the-art EEG foundation model LaBraM [22] (pre-trained weights version: LaBraMbase), and then fine-tune it on our dataset. To keep consistent with the setting of the pre-trained model, we also use a token size of 1 sec (200 timestamps) without overlap for this module, resulting in 16 tokens per channel. Since we have EEG data with C = 26 channels, the final token number would be 16 × 26 + 1cls token = 417 before being sent into the transformer, please see other detailed settings of the model architecture in the "fine-tuning session" of [22] .For the multi-scale frequency spectral representation learning module, we set the smallest scale to 0.5 seconds, corresponding to 100 points at our sampling rate of 200 Hz. For the multi-scale learning, we use l 3 throughout our experiments as it achieves the best average performance across other settings, and it includes four window scales: 0.5 seconds, 1 second, 2 seconds, and 4 seconds. The hidden features derived from these four scales are first projected to the same frequency embedding size (equal to the final hidden embedding size: d hiddenembed = 200) and then projected to the same temporal embedding shape (d time = 32, which is derived by T scale0 , where T = 3200 is the total input EEG length 3200 timestamps, and as mentioned scale0 = 100). These projections are both achieved by using a single layer of Linear projection. Therefore the shape of the embeddings before the transformer module is (32 × 26 + 1cls token, 200). In this study, we use a linear transformer [53] instead as our encoder to reduce the complexity and improve the speed. A previous study by [59] proves that the linear and the vanilla transformer yield very similar performance in biosignal representation learning, while the former one takes less time, so we also adopt the transformer with linear complexity in this study, with 8 heads and 4 transformer layers. We obtain the final embedding from each module by an average pooling step. Then the shape of the hidden embeddings from the spatiotemporal and multi-scale spectral modules are summed up, with a final embedding shape (200, 1). Other training details are shown in Table 5 .To ensure fair comparisons across all baselines, we apply the same seed and a consistent set of hyperparameters -batch size, learning rate, optimizer, and number of training epochs -as detailed in Table 5 for each reconstruction scenario.• Li et al. [31] : A Encoder-Decoder CNN that leverages effective spectral representation learning with sinusoidal activation function for Sequence-to-Sequence EEG-fMRI prediction in deep brain regions during eyes-open-eyes-closed (EOEC) task. We adapt this model by only using the encoder part and then using linear projection to make predictions similar to our approach.• BEIRA [25] : An interpretable CNN Encoder-Decoder for deep brain region fMRI reconstruction during EOEC task. We also adapt this model by only using the encoder part and then using linear projection to make predictions similar to our approach.EEG Encoding:• LaBraM [22] : A recent EEG foundation model trained on over 2,500 hours of EEG data to learn the spatiotemporal information of EEG signals, which achieved the state-of-the-art downstream classification task performance. We maintain the original model architecture with a 200-dimensional latent embedding and utilize the pretrained weights from this model (LaBraM-base, the only publicly available checkpoint for LaBraM) for fine-tuning on our datasets. • BIOT [59] : A transformer-based flexible bio-signal encoder architecture that learns the temporal-spectral information from EEG signals. Since BIOT and LaBraM share similar input processing, and both produce a one-dimensional latent representation for each sample, we set BIOT's embedding size to match LaBraM and our model (d hiddenembed = 200) to ensure a fair comparison, keeping other parameters consistent with the original paper. • FFCL [29] : A model that combines a CNN model and LSTM model for learning separate representations, which are merged to make the final prediction. • ST-Transformer [48] : a transformer-based spatial-temporal feature learning neural network for EEG classification task. • CNN Transformer [44] : A transformer convolutional neural network for automated artifact detection in EEG.For FFCL, ST-Transformer, and CNN Transformer, we retain the original model architectures as used in the baseline experiments in [59] . For detailed implementation, please refer to their GitHub repository 4 .In this section, we show more examples that reflect the best and the worst reconstruction performance over all ROIs in Figure 6 . The total length of the scans is 1207.5 seconds. NeuroBOLT shows the capability to reconstruct the whole resting-state fMRI scan over a large time range. Here we provide the model performance assessed by MSE values for intra-and inter-subject experiments conducted on resting-state data (see Table 6 ). This subsection presents statistical measures for resting-state fMRI scan reconstructions in comparison with other models (see Figure 7 ). While the presence of statistical significance varies across methods and ROIs, our proposed approach demonstrates consistently higher mean performance than the baselines for all (or all but one) of the ROIs in the intra/inter-scan comparisons. In this subsection, we show that, in addition to the resting-state condition, NeuroBOLT is also effective when trained and tested on auditory-task condition in both intra-(Figure 8 Justification: The paper identifies the current challenges in EEG-fMRI translation tasks and proposes a method to address them. We summarize our contributions in the abstract and also highlight them using bullet points at the end of the Introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Yes. We have provided our project page link, where the data and code would be updated.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: In Section 3 and supplemental materials, we report how we prepared the data and provide all training and testing details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Yes, we provide plots with error bars and statistical test in the Appendix 7, 8, and we provide S.D. in the main tables 1, 3.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In section 3.2, we report these information.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: Yes, the research conducted in the paper conforms, in every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Section 4, we summarize the broader impacts of our work, highlighting our efforts to address clinical diagnostic challenges through non-invasive and cost-effective methods.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cited the source code paper we used properly. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The code for the model and our analysis will be well documented, and will be public on github. Our data will also be well documented as well and can be made public upon request.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: During our data collection, subjects rested passively with their eyes closed (resting state). Participants were compensated 40 dollars. Written informed consent was obtained, and all protocols were approved by the Institutional Review Board. This information is included in the appendix dataset session.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: We state that we have IRB approval in section 3.1 and in the Appendix D, and that the study poses minimal risk to participants. Individuals with contraindication to MRI were not eligible for the study.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Project page: https://soupeeli.github.io/NeuroBOLT 38th Conference on Neural Information Processing Systems (NeurIPS2024).https://tedana.readthedocs.io/en/stable/https://afni.nimh.nih.gov/afnihttps://github.com/ycq091044/BIOT/tree/main/model 
 
 Appendix:  


Decision: The paper introduces a model that adopts the transformer architecture to convert EEG data into fMRI. The model leverages spatial, temporal, and frequency-based features to reconstruct the fMRI signal. In our review process, while some considered the methodological advancements incremental, the majority of reviewers pointed out the significance of a thorough empirical evaluation on a simultaneous EEG/fMRI dataset. All reviewers recognized the importance of the tackled problem and the strong general interest of the field in it.

Addressing specific concerns, reviewer bwvZ was mostly satisfied apart from the issue of citing the relevant literature. This remaining concern was addressed in the rebuttal, with the proposed inclusion of the relevant literature in the final version of the manuscript. I disagree with the reviewer on the basis that this alone is a strong enough reason to reject the paper. The reviewer unfortunately failed to provide specific details regarding which "relevant literature" was omitted.

In summary, this paper displays promising features despite the following three areas of concern:

- A high variance in the results of empirical evaluation, which suggests the proposed advances may not be as groundbreaking as compared to existing methodologies.
- An evaluation performed on a proprietary dataset, eliciting potential concerns over reproducibility.
- A perceived limited innovation in methodology.

The significance of the problem addressed and the empirical results that convinced two out of three reviewers make this paper worthy of acceptance. However, due to the aforementioned limitations, it is more suitable to be presented as a poster. This gives the authors an opportunity to discuss these concerns with the wider research community, potentially strengthening their work in the process.

Reviews: review: paper_summary: paper_summary: In this work, the authors present a deep learning architecture for inferring functional magnetic resonance imaging (fMRI) signal from electroencephalography (EEG) data. The proposed model, named NeuroBOLT, utilizes transformer backbones to provide spatial, temporal, and frequency-based features from the EEG are utilized for reconstruction. The authors demonstrate the performance of their architecture on a small (N=22) data set using a propriatary data set of simultaneously measured EEG-fMRI.
strength_weakness: strength_weakness: The major weaknesses of this work come down to weaknesses in the empirical evaluation. I am afraid that in its current state, the evaluation does not lead to a convincing demonstration of this method for fMRI reconstruciton, and the claims in the introduction about novelty coming from the application to multiple brain regions and resting-state fMRI seem somewhat overemphasized. Currently this brings it to a full reject as the paper is otherwise sound but the limitations in the evaluation are significant enough to bring it well below the threshold, and cannot be easily addressed in the rebuttle I believe. 

First, I will highlight the lack of reported standard deviations or error bars in any of the results. No standard deviations are provided in tables 1 or 2, or in any of the figures providing results. In the checklist, the authors state "Error bars are not reported at the current stage because it would be too computationally expensive to compute over all the brain regions and for all participants,also there is limited space in the paper to put all the statistics. But we could always add this information if reviewers think it’s important to know."

While I appreciate the authors' acknowledgement of this exclusion, I do think error bars are absolutely necessary to demonstrate the efficacy of the proposed method. The demonstrated improvements are often quite small (e.g. improvement from 0.540 to 0.588 in table 1), and it is not clear if the purported improvements can be explained away from model noise. I could not find any information about controlling model initialization or seeds as well to ensure that random initializations played a less significant role between experiments even with the same architecture on different regions. I absolutely think error bars are necessary for this work, and the reasoning provided by the authors is not mitigated elsewhere or behind a more significant barriers other than training and evaluation time. Additionally, the authors could have mentioned this omission in the limitation section of their main paper since I had to go to the checklist to be sure the authors were aware of the issue.

Second, in the abstract the authors highlight the ability to "generalize to other brain areas" and "other conditions (such as resting state)".  Unless I am missing something, I cannot find any experiments by the authors that address these particular gaps. The authors do provide inter- and intra-subject predictions which is interesting; however, their model is still only trained on individual ROIs, and they don't include any experiments demonstrating transfer learning between models trained on other regions, and they do not include any experiments studying other tasks BEYOND resting-state fMRI. Thus, the paper falls into the same limitation as past works which were only focused on task, just in the other direction. This work would have been much more compelling if they could demonstrate a model which trained well both on task and rest related data, or even better, which could reconstruct task-related data despite only being trained on resting-state fMRI. The acknowledgement of the limitations in the literature is thus misleading as the proposed method still suffers from these same limitations. The choice to only demonstrate the results for several ROIs highlights this limitation - it would be okay if the authors did not seem to imply elsewhere that their model gets around the single-ROI training approach from past methods. 

Clearly, the N in this study is quite small. This is to be expected as simultaneous EEG-fMRI is still quite rare as a sequence to collect; however, the authors seem to gloss over all of the myriad issues which will come with training their data over such a small data set. I am not penalizing this work for the small N in and of itself, but as I cannot find any mention of common obstacles such as overfitting, bias towards particular kinds of reconstruction errors, and other limitations that would inevitably arise. I am extremely surprised I could find no mention of pretraining anywhere in this work, which I almost imagine would be hugely necessary for these kinds of studies with very small data sets. Again, it's not necessarily a limitation in and of itself to not do these things, with how the paper currently reads these seem to be touted as benefits of the new approach which are not backed up by evidence.
questions: questions: How well does the model trained on other ROIs transfer to reconstruction of completely different ROIs?

How might scanner model and particular parameters of the resting state sequence affect reconstruction?

Why do you not compare anywhere with Source Localization? Source Localization is only mentioned once offhand, and its limitations and efficacy as a reconstruction technique are not gone into in detail. I am surprised it was not included as a baseline method in fact.
limitations: limitations: The authors do provide some discussion of the limitations of their work; however, as I have noted above, there are some limitations which are ommitted from the main body of text which at least should have been acknowledged in this section.

The authors state they have IRB approval in section 4.1 of their paper. I see no reason for additional review.
paper_summary: paper_summary: The manuscript proposes an EGG-to-fMRI synthesis model. The framework implements a transformer architecture and uses a multi-channel feature combination expanded across the temporal axis. To evaluate the proposed model, EGG and fMRI data from 22 participants were recorded while they were in the resting state with eyes closed.
strength_weakness: strength_weakness: The biggest weakness is that the framework and the dataset are only addressing the resting state. While this is an important baseline to investigate, it would have been great to explore the fidelity of the proposed framework when participants are presented with some stimuli.

It is unclear whether the source code and dataset will be released publicly.

The stability of the results is fully guaranteed given no statistical analyses are performed.
questions: questions: What is "In-scan" in Table 1? This is not explained in the manuscript.

It is unclear to me why all methods are not evaluated for both inter-subject and in-scan. For example, isn't it possible to evaluate BIOT [44] for inter-subject data?
limitations: limitations: The manuscript sufficiently discusses its current limitations. I think the limitations section should tap into the particular scenario that the model has been evaluated on (resting state) and whether the results will be generalisable to other scenarios remains to be shown.
paper_summary: paper_summary: This paper introduces NeuroBOLT, a transformer-based model. NeuroBOLT utilizes multi-dimensional representation learning across temporal, spatial, and spectral domains to translate raw EEG data into comprehensive fMRI activity signals across the entire brain. Experimental results showcase NeuroBOLT's ability to effectively reconstruct resting-state fMRI signals across primary sensory, high-level cognitive areas, and deep subcortical regions.
strength_weakness: strength_weakness: 1. The method abbreviation and the title are not closely related. It is unclear where 'BOLT' comes from in the title, and even after reading the abstract, it remains confusing.
2. In fact, there has been a lot of work on fMRI-EEG in recent years, especially in 2023 and 2024, but the author's related work lacks a significant amount of relevant literature.
3. In the abstract and introduction, the author's description of the method is inconsistent with the organization in the methodology section, resulting in a need for improved readability.
4. The writing of the article needs to be further standardized. For example, 'FMRI data' is sometimes written with a capital 'F' and other times as 'fMRI data'.
5. Although the layout and presentation of the tables are aesthetically pleasing, the font size is too small, making them difficult to read even when enlarged.
6. The paper does not provide code or data to support the reproducibility of results. 
7. This paper lacks details on the parameter selection for the baseline methods. Although the authors state, 'The baseline models are from [44] and [16], where we choose the models with the best downstream classification task performance,' the datasets and tasks in references [16] and [44] are not entirely consistent with those in this paper. Therefore, the authors should specify the exact process of parameter selection.
8. The equations and symbols in the article are not very standardized. The authors should provide notation to help readers understand.
9. The authors should conduct statistical tests to validate the significance of their methods.
10. For readers in the NeurIPS community, the theoretical contribution of this paper appears to be weak.
11. I don't quite understand what the author means by the third point of contribution: 'Successful resting-state fMRI reconstruction To our knowledge, this is the first study to successfully reconstruct the resting-state fMRI signal from raw EEG data, with only 26 electrodes.' What is the significance of 26 electrodes?
12. Equation 2 does not appear to be a complete equation.
questions: questions: Please see the twelve weaknesses above.
limitations: limitations: Limitations are discussed in the section of Discussion and Conclusion.

Comments: title: Thank you!
Thank you for the comments - Part 2
Thank you for the comments - Part 1
Thank you for increasing your score
Score Adjustment
comment: Thank you very much for the review! Our volunteers are all healthy controls, consisting of 12 females and 10 males, with an average age of 35.22 ± 16.86 years. Additional detailed demographic information, along with data sources and IRB details, will be provided in the updated manuscript.
Thank you very much for the review! We will provide detailed information on data sources and IRB in the updated manuscript.
Thank you once again for your valuable and encouraging feedback! We sincerely appreciate the time and effort you put into the review process.
I thank the authors for responding to my questions. I do not have any further questions.
### **Here are the responses to <Point 3, Point 4 and references>** (For responses to **Point 1 and Point 2**, please see the **[comment block above](https://openreview.net/forum?id=y6qhVtFG77&noteId=ssbQ088SCz)**)

>**Point 3:**

If we understand correctly, the reviewer suggests running a grid search for each baseline model to find the parameters that perform optimally on the current dataset and with the current evaluation metrics. In our study, we had adopted the parameter settings recommended by the authors. When evaluating baselines, it is common practice to use the parameters suggested by the original authors, due to the computational burden associated with a comprehensive parameter search across the search space for each application. Moreover, we would like to clarify that we **used the same evaluation metrics as in \[7,8\], i.e., correlation coefficient (i.e., R values).** For the other EEG encoding models, the tasks in the original papers are different (mostly classification tasks like detection of seizures, event type classification, etc.), so metrics like accuracy, AUC, Cohen’s Kappa, and weighted F1 were used in these papers.

To further address your concern regarding differences in dataset, we conducted additional experiments on the ***same dataset*** as in refs. \[7,8\] (eyes open/close task fMRI), predicting the ***same brain regions***, using the ***same settings and objective*** (i.e., intra-subject prediction). The results (R ± S.D) are shown below, with the best values in bold. **We find that our model still outperforms \[7,8\].**

| Model | Pallidum | Caudate | Putamen | Accumbens | Average |
| :---- | :---- | :---- | :---- | :---- | :---- |
| Ours | **0.57 ± 0.18** | **0.58 ± 0.11** | **0.57 ± 0.16** | **0.60 ± 0.11** | **0.58 ± 0.02** |
| \[7\] | 0.43 ± 0.15 | 0.49 ± 0.12 | 0.51 ± 0.14 | 0.43 ± 0.13 | 0.47 ± 0.04 |
| \[8\] | 0.37 ± 0.04 | 0.47 ± 0.14 | 0.48 ± 0.16 | 0.42 ± 0.05 | 0.44 ± 0.05 |


>**Point 4:**
* **4.1)**: We apologize for the confusion, but our intention here for saying *"validate the robustness and applicability of our learned representations across neural data from different sites and scanners"* was to convey that our model shows promising applicability to another dataset, even if it is collected in a different condition (task) and on a different scanner (see **[Global response](https://openreview.net/forum?id=y6qhVtFG77&noteId=VLcizb9WHq)** and **Part 1** in the one-page PDF). We now notice that the word ”robustness” could cause confusion so we will omit this word in our revised manuscript. Evaluating the influence of noise requires simulating different noise levels, which is a bit out of the current scope of this paper, but it is a very interesting future direction.

* **4.2)**: We had planned to provide further details on the building blocks of our model, including specific details of the multi-scale spectral representation learning module. Multiscale temporal representations and Short-Time-Fourier-Transform (STFT) are proving to be highly effective in representing neural data \[9,10\]. While the components of this module have well-established theoretical foundations in signal processing, our key contribution lies in integrating these components to work effectively together, resulting in a novel method and application (shown through ablation studies in *Section 4.4 Table 2*). Our planned additions include the following:

1. Formally describe (i.e. provide mathematical notations) how we utilized STFT to convert EEG time sequences into time-frequency features.  
2. Discuss how the dimensions of frequency features depend on the time-window length with equations illustrating this relationship, and emphasize the rationale for selecting specific window lengths.  
3. Introduce the trade-off between temporal and frequency resolution, further clarifying the reasoning behind using an approach that considers multiple window lengths. 

Thank you once again for your valuable comments. We hope our responses fully address your concerns.


**References:**

[1-6]: see refs. [1-6] in the Reviewer’s previous comment

[7] Li et al. “Leveraging sinusoidal representation networks to predict fMRI signals from EEG”

[8] Kovalev et al. “fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships.” arXiv preprint.

[9] Van De Ville et al. “When makes you unique: Temporality of the human brain fingerprint." 

[10] Samiee et al. “Epileptic seizure classification of EEG time-series using rational discrete short-time Fourier transform”
Thank you very much for your additional comments. Please note that this year, **authors are not permitted to upload the revised full manuscript or include links to external pages during this rebuttal period.** We were only allowed to upload a one-page PDF (please kindly refer to our [Global Response](https://openreview.net/forum?id=y6qhVtFG77&noteId=VLcizb9WHq)). Therefore, we are unable to provide the revised manuscript here (in response to your **Points 1 and 4.2**), but will upload it when permitted. Responses to your other new concerns are below: 


>**Point 2:**

We appreciate the reviewers' thorough examination of related work. Our rationale for including particular baselines, and further discussion of your reference list ([1-6]), is provided below:

**Ref. \[2\]** is an excellent review article on simultaneous EEG-fMRI, and we will add it to the other simultaneous EEG-fMRI articles that we have cited in our original manuscript (including Ritter et al., 2006; Laufs et al., 2003, 2006; Chang et al. 2013; de Munck et al., 2009). With regard to the DCM model in **Ref. \[4\]**, we had not selected it as a baseline since it requires specifying stimulus onsets along with the neuroimaging data. Since no stimuli are presented during resting state, this model can not operate on resting-state data without major modifications. In discussing their future directions, the authors of **Ref. \[4\]** state: “Finally, the expansion of the current task-based analysis to the corresponding resting-state methodology, where an equivalent canonical microcircuit formulation for cross spectral data features, will be needed.” **Ref. \[6\]**, aims to infer the effective (directed) connectivity of a brain network based on the complementary information provided by EEG and fMRI. This is interesting work too, but does not appear to provide a framework for inferring fMRI time courses from EEG.   

As we mentioned in our first rebuttal, we indeed plan to cite **Ref. \[3\]** (which is the preprint version of *\[3\]* in the previous response) and **Ref. \[5\]** (which uses the same model as *\[1,2\]* in the previous response). For **Ref. \[3\]**, we were unable to find the code, and some experimental settings are not specified in their paper, which would make a direct comparison potentially unfair and unreliable, so we did not include it as a baseline. For **Ref. \[5\]**, the model only accepts 64-channel EEG as input (according to the authors’ GitHub repo and PyPi sites). Since we are using EEG data from 32-channel caps, we did not include this model as a baseline here, but look forward to doing so in future studies with 64-channel data. **Ref \[1\]** explored three conventional architectures: FCNN, CNN, and Transformer. However, the paper did not provide code/model parameters and offered limited information about the training process. In our baselines, we have included models that are similar to or more advanced versions of these architectures (see Appendix B.2 in the original manuscript: CNNs *\[19, 23, 24, 32\]* and Transformers *\[16, 44, 36, 32\]*). 

| Baselines | If&nbsp;included | Rationale |
| :---- | :---- | :---- |
| [1] | ✗ | - Source code is not available; - Model parameters are not specified; - We already included baselines with similar or advanced versions of these architectures |
| [3] | ✗ | - Source code is not available; - Model parameters are not specified |
| [4] | ✗ | Not applicable without major modifications |
| [5] | ✗ | Only supports 64-channel inputs |
| [7] | ✓ | Provided in the manuscript. |
| [8] | ✓ | Provided in the manuscript. |

We hope that our response helps to clarify the rationale behind the baseline models selected for this submission. While the simultaneous EEG-fMRI field is indeed large, the area of EEG-to-fMRI synthesis is currently a niche but rapidly emerging subfield. We will also include the referenced papers in our revised manuscript, which will appear on this forum when we are permitted to submit.

### **Please see our responses to _<Point 3, Point 4 and references>_ in our [next comment block](https://openreview.net/forum?id=y6qhVtFG77&noteId=wkIJndrOr2)**
Thank you so much for your prompt response and raising the score. We are happy that our additional experiments have improved this work, and sincerely appreciate your feedback, which helps to improve the quality of our paper.

We acknowledge that the error bars are relatively large across all methods. While variability in the performance across individuals presents a challenge, we find it promising that our method achieves consistently higher mean performance than the baselines for all (or all but one) of the ROIs in the intra/inter-scan comparisons (Figs. R1.2, R2.1, R2.2 in the one-page PDF), though indeed not all of these differences are statistically significant. By contrast, considering only the baseline methods, these appear to vary across ROIs in terms of which model achieves the highest mean performance. Additionally, we believe this is the first study to incorporate and adapt these state-of-the-art EEG Encoding baselines in this specific application of EEG-to-fMRI translation.

We believe that your comment is highly valuable, and will include a discussion of this limitation in our revised manuscript. In addition, we are planning further in-depth exploration of the factors that may drive performance variability across individuals in our future work, as this is indeed an important issue.

Thank you once again for your valuable comments!
Thank you for the author's response.

1. Some specific modifications by the author were not observed, such as W3, W5, W6.

2. Regarding W2, [1-6].... Aside from the paper titled "EEG-to-fMRI Synthesis," many papers titled "simultaneous EEG/FMRI" also align with the direction of this work. Among the articles cited and compared by the author, only 19 and 24 are related to EEG-fMRI translation, while most focus on EEG Encoding. The insufficient comparative experiments also stem from a lack of thorough research in the related work.

3. Regarding W7, the author stated, "We strictly adhered to the model settings provided in their original papers when training." However, the datasets and evaluation metrics used by the author may differ from those in the original paper, which may not be entirely fair.

4. Regarding question 10, the author mentioned "validate the robustness and applicability," but based on the appendix results, it appears that the author did not conduct an in-depth study of robustness, such as the method's robustness under different fMRI and EEG noise levels. Additionally, "We will include more detailed theoretical derivations and explanations of spectral representations in the appendix to strengthen the theoretical foundation of our work." These specific details should also be clarified.

[1] Bricman P, Borst J. EEG2fMRI: Cross-Modal Synthesis for Functional Neuroimaging[J]. 2021.

[2] Trujillo-Barreto N J, Daunizeau J, Laufs H, et al. EEG–fMRI Information Fusion: Biophysics and Data Analysis[M]//EEG-fMRI: Physiological Basis, Technique, and Applications. Cham: Springer International Publishing, 2023: 695-726.

[3] Liu X, Tu T, Sajda P. Inferring latent neural sources via deep transcoding of simultaneously acquired EEG and fMRI[J]. arXiv preprint arXiv:2212.02226, 2022.

[4] Wei H, Jafarian A, Zeidman P, et al. Bayesian fusion and multimodal DCM for EEG and fMRI[J]. Neuroimage, 2020, 211: 116595.

[5] Calhas D. EEG-to-fMRI Neuroimaging Cross Modal Synthesis in Python[J]. 2023.

[6] Tu T, Paisley J, Haufe S, et al. A state-space model for inferring effective connectivity of latent neural dynamics from simultaneous EEG/fMRI[J]. Advances in Neural Information Processing Systems, 2019, 32.

.........
The rebuttal provided by the authors addresses much of the concerns raised here. The requested error bars somewhat lower the impact of the model compared to baselines, as there is high variance and many baselines perform fairly comparably; however, there is demonstrative improvement. I have raised my score to a borderline accept.