Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion
Authors: Dongyang Li
Chen Wei
Shiying Li
Jiachen Zou
Quanying Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: EEG decoding
Visual reconstruction
BCI
Visual neural decoding
Primary area: neuroscience_and_cognitive_science

Abstract: How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of visual decoding and reconstruction based on functional Magnetic Resonance Imaging (fMRI). However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for visual decoding based on electroencephalography (EEG). In this study, we present an end-to-end EEG-based visual reconstruction zero-shot framework, consisting of a tailored brain encoder, called the Adaptive Thinking Mapper (ATM), which projects neural signals from different sources into the shared subspace as the clip embedding, and a two-stage multi-pipe EEG-to-image generation strategy. In stage one, EEG is embedded to align the high-level clip embedding, and then the prior diffusion model refines EEG embedding into image priors. A blurry image also decoded from EEG for maintaining the low-level feature. In stage two, we input both the high-level clip embedding, the blurry image and  caption from EEG latent to a pre-trained diffusion model. Furthermore, we analyzed the impacts of different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. The experimental results indicate that our EEG-based visual zero-shot framework achieves SOTA performance in classification, retrieval and reconstruction, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. Our code is available at https://github.com/ncclab-sustech/EEG_Image_decode.

Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion
Jiachen Zou' Quanying Liult Department of Biomedical Engineering, Southern University of Science and Technology,
Dongyang Li
Chen Wei * Shiying Li
Shenzhen, China
(1idy2023, veic3) sustech, edu. on liuqyOsustech. edu. cn
Visual Perception Sequence
Brain Encoding
Seen Images
Relax
Reet
Vituull
Classification
Our Reconstructed Images
Decade
Retrieval
Generation
Brain Embedding
Figure 1: EEG/MEG- zero-shot brain decoding and reconstruction. Left: Overview of three visual decoding tasks using EEG/MEG data under natural image stimulus. Right: Our reconstruction examples.
Abstract
How to decode human vision through neural signals has attracted long-standing interest in neuroseience and machine learning. Modern contrastive learning and generative models improved the performance of visual decoding and reconstruction based functional Magnetic Resonance Imaging (fMRT). However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting high need for visual decoding based electroen- cephalography (EEG). this study, we present an end-to-end ERG-bbsed visual reconstruction zero-shot framework, consisting of tailored brain encoder, called the Adaptive Thinking Mapper (ATM), which projects neural signals from different sources into the shared subspace as the clip embedding, and two-stage multi-pipe EEG-to-image generation strategy. In stage one, EEG is embedded to align the high- level clip embedding, and then the prior diffusion model refines EEG embedding into image priors. A blurry image also decoded from EEG for maintaining the low- blurry
level feature. In stage two, we input both the high-level clip embedding, image and caption from EEG latent to pre-trained diffusion model. Furthermore, we analyzed the impacts of different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magne- toencephalogram (MEG) data modality. The experimental results indicate that our EEG-based visual zero-shot framework achieves SOTA performance in classifica- tion, retrieval and reconstruction. highlighting the portability, low cost, and high temporal resolution of EEG, enabling wide range of BCI applications. Our code is available at httph:://// //::::/gittt thu.. com/ncclab-gusteehh
sustech EEG, Iwwge_mmaaeeeeeeeeeeeeeeeeeeeeee _ecooe
*D. and C. Wei contributed equally. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
key technical challenge in BCIs is to decode/reconstruct the visual world seen by humans through non-invasive brain recordings, such as IMRI, MEG or EEG. These highly dynamic brain activities reflect human perception of the visual world, which is influenced by properties of the external visual stimulus, our internal states, emotions and even personal experiences. Thus, visual decoding and reconstruction based on neural signals can uncover how the human brain processes and interprets natural visual stimulus, well as promote non-invasive BCI applications.
Contrastive learning and generative models have greatly advanced IMRI-based visual decoding in both decoding tasks (e.g.. image classification and retrieval) and generative tasks (e.g.. image reconstruction). By combining pre-trained visual models, existing fMRI decoding models can learn highly-refined feature embeddings in limited data I 2). Using these embedded IMRI features, generative models such as diffusion models can reconstruct the image one is seeing 2 3 However, despite many advances in fMRI-based visual decoding, fMRI equipment is unportable, expensive, and difficult to operate, largely limiting its application in BCIs. Alternatively, EEG is portable, cheap, and universal, facilitating wide range of BCI applications. EEG has higher temporal resolution and can effectively capture rapid changes in brain activity when processing complex, dynamic visual stimulus.
EEG has long been considered incomparable to fMRI in natural image decoding/recoonstrucion tasks, as EEG suffers from low signal-to-none ratio, low spatial resolution, and large inter-subject variability, Recent advances in multimodal alignment have made MEG/EEG visual decoding possible, although the performance is still inferior to fMRI 4 5 6 Yohann Benchetrit et al. used the CLIP model to extract the latent representation of the image and trained the MEG encoder to align it with the image representation extracted by CLIP. It achieved excellent retrieval and reconstruction performance on MEG and fMRI datasets, demonstrating the potential for real-time visual decoding and reconstruction using EEG//ME signals. Recently, Song et al. (5] used an EEG encoder based on ShallowNet [ and performed representation alignment through contrastive learning, achieving excellent decoding performance on the THING-EEG dataset [8) These two studies provide preliminary evidence of the potential of EEG/MEG- performance compared to the fMRI-level performance. This gap is largely caused by the fact that the framework of EEG visual decoding and reconstruction have not yet been thoroughly explored.
visual decoding. However, there is a significant gap in their
To fill this gap, we have developed visual decoding framework based on EEG/MEG, including novel EEG encoder and two-stage image generation strategy. Our work has three main contributions:
1. We present brain decoding framework, which is the first work allows zero-shot image classification, retrieval, and reconstruction via EEG data. Experimental results demonstrate that our framework applicable to various common EEG encoder architectures.
2. By extensively studying the existing EEG encoder modules, construct tailored EEG encoder ATM, which achieves state-of-the-art performance in three downstream visual decoding tasks.
3. We report two-stage EEG-tt-imaag generation strategy, which separately extracts high- level and low-level visual features from EEG and refining these features with an additional lightweight prior diffusion model, enabling reliable reconstruction of images using less than 500ms EEG.
2 Method
To learn high-quality latent representations of EEG data, it is crucial to consider the spatial position of EEG channels and the Temporal-Spatial properties of EEG signals. Let '' represent the length of the time window of the data, C' the number of EEG channels, and N the total number of data samples. Our objective is to derive EEG embeddings Z f(E) R N from the brain activity data E E R NxCXT where f is the EEG encoder and F is the projection dimension of the embeddings. Concurrently, we use the CLIP model to extract image embeddings Z, E NxF from images Our goal is to effectively align the EEG representation with the image representation,
illustrated in Fig. 2 In the training phase, the EEG encoder is trained with EEG and image pairs using a contrastive learning framework. In the inference phase, the EEG embeddings from the trained EEG projector

---

Visual stimulus
Training
Image Embedding
Contrastive Learning
Trainable
Frozen
Image Datasel
CLIP Encoder
Classification
antelope peor antelope Rettle Sousage
zero-shot
EEG Embedding
Retrieval
EEG
Encoder
Generatilon
High-level
MSE Loss
Reconstruction
EEG Emberiding
U Adapter
EEG sories
EEG Encoder
VAE image encoder
VAE image decoder
low level
SDXL- turbo
Low-level
VAE Latent
high level
Inference
EEG Embedding
EEG Embedding
Image Embedding
Noise
"The dense gummies look colorful"
EEG
img2texd
BLF2
Encoder
Difusio U-Net
Semartic level
Figure 2: EEG/MEG-based visual decoding and generation framework. The EEG encoder is designed as a flexible replacement component. After aligning with image features, the EEG features are used for zero-shot retrieval and classification tasks, and the reconstructed images are obtained through a two-stage generator.
can be used for a variety of zero-shot tasks, including BEGebased image classification, retrieval, and image reconstruction.
2.1 ATM for EEG Embedding
Inspired by advanced time series models 10 we develop an EEG encoder called ATM, for aligning the original EEG signals to its feature representation space (Fig. B). ATM is based on the channel Transformer encoder, Semporal-Spatian convolution and multilayer perceptron (MLP) architecture. In contrast to other conventional practices, the original EEG does not need to be segmented, and each sequence acts as a patch. After sinusoidal position embedding, these patches are processed through channel-attention module to integrate the information of different series. Subsequently, through the Samporal--Sattall aggregation, we project the output with a MLP to get rational shape representations. The Temporal-Spatin convolution module is effective way to represent EEG data with a small number parameters [5]. prevent overfitting in training. The difference is our components is plug-and-play and-play and can be flexibly replaced with different types of Temporal-Spatte convolution components as needed to adapt to various EEG/MEG datasets. Finally, MLP projectior consists of M simple residual components and fully connected layers, with LayerNorm applied in the output to ensure the stability of training. addition to entering the original series, we provide an identification input for known subject and can specifically use this token for downstream tasks. For unknown subjects, we use shared tokens or average all tokens equally directly into the MLP projector.
2.2 Image Embedding
Many previous studies have explored various training strategies to train deep neural networks for image embedding, such as VGG-19 and ResNet trained with supervised learning, CLIP, DINO trained with contrastive learning, and VAEs with self- self-supervised learning [1] 12
They have reported that DINO and CLIP models pre-trained using the Vision Transformer (VTT) architecture perform better in a range of downstream tasks, including image decoding and reconstruction, compared to models trained using supervised learning methods (such as VGG, ResNet) and self-supervised VAE frameworks. Thus, in this study, we use CLIP for image embedding, denoted as Z, E N instead of Z, E N x687x768 with the BEG embeddings. Before formal training, all images undergo the standard preprocessing procedure [13].
5 6
3

---

Flatten
Embedding
MLP Projectior
Add LayNorm
2t
Temporal Spatial Aggregation
Linear Projection
Temporal Spatial Conv Module
GELU
Channel-wise Attention Layer
CNN & Norm
Linear Projection
Gemasting
Input Embedding
Variate
Embedding Layer
Subject Takens
Shared Token
FPz
02
1,
Figure 3: The structure ATM. The original EEG sequences of different variates are independently embedded into tokens. Channel-wise attention is applied to embedded variate tokens with enhanced interpretability revealing electrode correlations. And representations of each token are extracted by the shared feedforward network (FFN). Then Tepporal-Spaatal convolution can prevent overfitting and enhance the ability Temporal-Spatte modeling.
2.3 EEG Guidance Image Generation
In this study, we present a two-stage pipeline for generating images that serve as visual stimulus for EEG recordings, shown in the bottom right of Fig. 2 In the left of Fig. 3 we have obtained the EEG embeddings 2g for each image by the EEG encoder ATM. Now our goal is to use these EEG embeddings generate the corresponding images. The joint distribution images, EEG embeddings, and image embeddings can be expressed as =E,, 21) = p( [e ) corresponding to the prior diffusion and CLIP--guide generation, respectively. diffusion stage. Inspired by DALL--F [14 and Mind's Eyes (2) we train diffusion model conditioned the EEG embeddings 2 to learn the distribution of CLIP embeddings p(z7 (= In this stage, we construct a lightweight U-Net: 6prio 4, ZE), where 2 represents the noisy CLIP embedding at diffusion time step f. We train the prior diffusion model using EEG and CLIP embeddings. Through this diffusion model, we can generate corresponding CLIP embeddings 2 from EEG embeddings
Stage we first focus on the prior
prior for stage II. In Stage II, employ the pre-traie mre-trained SDXL [15] and IP-Adapter [16) models to model the generator p( (,y)) thereby sampling image according to 2T. In addition, introduce the low-level features here using img2img( [7]] Further details provided in Appendix
2.4 Loss Function
Following the methodology outlined by Benchetrit et al. [1] we adopt a dual approach to loss functions, serving distinct objectives. For the classification and retrieval tasks, we only utilize the CLsIP which is inspired by the contrastive learning approach described in Radford et al. [13]. This loss function aids in aligning the EEG data E' with corresponding image data I, thereby facilitating the identification EEG-image pairs and maximizing the boundaries of EEG representations. For the generation tasks, besides the CLIP loss, we add a Mean Squared Error (MSE) loss to facilitate consistency learning in regression. Thus the overall loss function for our model a combination of these two distinct loss types, expressed as:
Loss = x LCLIP + (1-A) LMSE
Here, À is hyperparameter that balances the contribution of each loss type.

---

Clussification 9atjst
Classification
Classification
Classification
Retrieval (In tope
Classification Soljes
Retrieval
Classification On Silecttop-5
(1t Saljer suge
Retrieval Inn Subreet top-51
Classification (Ie Subject top-1)
Retricval Subjes kop-3
Classification Lin Subject top-1)
Retrieval
Generation Hep-S)
Retrieval
Generation
Retrieval
Generation (M9-1)
Retrieval
Generation
(Acros tabyarttop-))
(Armo: Sabjec up-d)
Uop-1)
BEOTTN
B.D.
I.D.
NLLP
ATTNNe
Shalion
Figure 4: EEG/MEG- decoding and reconstruction performance. Left: Comparisons of nine encoders on the THINGS-ER dataset, including within-subject and cross-subjes performance. Right: Comparisons the THINGS -MEG dataset, similar to left. Our method achieves the highest performance compared to other competing encoders EEG/MEG based visual decoding tasks.
3 Experiments
3.1 Training and Computational Considerations
We conducted our experiments on the THINGS-EEG dataset's training set 8 6). To verify the versatility of ATM for embedding electrophysiological data, we tested it on MEG data modality using the THINGS-MEG dataset [18). All experiments can be completed in a single NVIDIA RTX 4090 GPU. We used the Adam optimizer [19] to train the across-subject model on set of approximately 496,200 samples, and the within-subject model on a set of about 66, 160 samples, with initial learning rate of 3 10 and batch sizes of 16 and 1024. Our initial temperature parameter was set to 0.07. We splited the last batch of the original training set as the validation set and selected the best model based the minimum validation loss over 40 epochs. For faimess, all models' hyperparameters were kept consistent. In our study, we compared the performance of different encoders the within-subject test and cross-subject (leave-one-uujjectouul) test set (see Appendix
3.2 EEG Decoding Performance
Our method obtains the EEG embedding for the classification task. We output the category of EEG with the highest cosine similarity with text embeddings(Fig.. (ig. Fig.5p presents the average accuracy across different methods in the subjects, and shows that method outperforms others. More details of the EEG-based image classification are in Appendix B
In Fig. 5 we test the effectiveness of EEG embeddings in the image retrieval task. We calculate the cosine similarity between the EEG embeddings and the CLIP embeddings instead of text embeddings in the image dataset (with 200 images). Fig. 5 shows the average results for all subjects. We take the highest test accuracy in the evaluation process as the statistical result. Fig. (bb shows the Top-5 retrieved images corresponding
Bb
the real visual stimulus previous models, the Top-1 accuracy of our model significantly improved, and the Top-5 images all maintain high degree of similarity with the original images. See the Tab. 8 in Appendix for more detailed averages of test accuracy subjects.
by subjects. Compared with the
Ablation Study on ATM We systematically deconstructed and analyzed each layer of our EEG projector. We conducted an ablation study for each component in ATM (i.e., the MLP projector, the Temporal-Spatiin convolution module and the channel-wise attention block). two different convolution architectures, ShallowNet (ATM-S) and EEGNetV4 (ATM-E), our convolution backbone. Appendix B showed the results obtained under different experimental configurations.
specified
5

---

EEG Embedding
Retrieval
Image Dataset
Classification
Text Image Embedding
0.-
Chance Level
Clasefrcatoo arcuney by Conditiinn
87-
D. MonTEEEEEEe
Retieerel Aocuracy by Conditon
9.
BOGCeefomen
36-
0.2
2:409
6-600)
10
Top-5 KCC
106- aCC
Figure 5: EEG-based image retrieval and classification. (a) The paradigm of EEG-based image retrieval and classification. (b) Samples of the top-5 accuracy in EEG-image retrieval tasks. See Appendix G for additional images results. (c) Average in-subject classification accuracy across different methods. (d) Average in-subject retrieval accuracy across different methods.
3.3 Image Generation Performance
Fig. 66 shows the process of generating images under the guidance of EEG embedding and evaluating the quality of the generated images. To evaluate the generation performance, conducted image retrieval task. Specifically, we extract the CLIP embedding of the generated images and compare the similarity between the CLIP embeddings of all images retrieve the generated image.
Fig. 66 shows the similarity of distribution. Fig. 6C shows the generated samples. The generated images have high semantic similarity with the seen images and have good diversity in low-level visual features, which can be manipulated by the guidance scale hyperparameter (Fig. od). We also report the reconstruction performance for EEG, MEG, and fMRI across various metrics from different methods and datasets in the Tab.
Table Quantitative assessments of the reconstruction quality for EEG, MEG, and IMRI in Subject 8. For detailed explanations of the metrics.
Low-level
High-level
Dataset
PixCorr
SSIM t AlexNet(2)
AlexNeu(S) f Inception
CLIP
SwAV
NSD-IMRI 12
0.305
0.366
0.962
0.977
0.910
0.917
0.410
NSD-IMRI 20
0.254
0.356
0.942
0.962
0.872
0.915
0.423
NSD-EMRI 21
0.130
0.308
0.917
0.974
0.936
0.942
0.369
THINGS-
0.058
0.327
0.695
0.753
0.593
0.700
0.630
THINGS-MER (averaged)
0.076
0.336
0.736
0.826
0.671
0.767
0.584
THINGS-MEC (Ours)
0.104
0.340
0.613
0.672
0.619
0.603
0.651
THINGS-EEG (Ours)
0.160
0.345
0.776
0.866
0.734
0.786
0.582
3.4 Temporal Analysis
To investigate the effects of different EEG time window on visual decoding, we calculated the average top-1 classification accuracy for sliding and growing time windows: [0, t]. including the entire period

---

Embadding
b
Distribution of Similarity
Generation
Generated Sin.
15-
Image
Image
Embeodding
30-
Cosine
Similanty
0.00
0.05
5.00
0.13
0.00
025
0.32
03s
Simierity
d
Similarity and diversity for
Seen
Our Reconstructed Images
different guidance scale
0.20
0.23
06
45
04
0.2
0.1
022 0.3
1.0 20
50
Scale
Figure 6: EEG guidance image generation. (a) The paradigm of image generation. (b) The similarity between random visual objects and the EEG embeddings, and the similarity between generated visual objects and the target EEG embeddings. (c) Comparison between the original image and the image generated using the corresponding EEG data. (see Appendix for details). (d) The similarity between visual objects and target EEG embeddings the guidance scale changes, and the diversity of visual objects as the guidance scale changes. See Appendix for additional results.
from the onset of visual stimulus to time point and [t-100, t], only including the data 100ms before time point f. We compared the accuracy with randomly selected baseline (0.5% chance level) to test predictive performance (Fig. 7 Our results show that within 500ms after visual stimulus, the accuracy reaches upper limit of about 30%, after which the accuracy no longer improves (Fig. 77)) The MEG decoding shows a similar profile as the time window expands (Fig. ( We exhibit the generated images under different EEG time windows, [0, (] in Fig. 7e. The similarity low when the time window is less than 150ms, and this similarity gradually increase as the time window expands. After 500 milliseconds, EEG-guided image generation can reliably reveal the semantics the images seen. Interestingly, we find differences in the optimal reconstruction time windows for different categories of images, for example, jelly beans (200ms) are faster than aircraft carrier (500ms), implying that the human brain may process different visual objects at different speeds. This finding highlights the advantage of EEG's high temporal resolution in studying fast visual processing compared with the lower temporal resolution fMRI.
3.5 Spatial Analysis
To investigate the contribution of different brain regions to visual decoding, divided the EEG electrodes from the THING-EEG data into five distinct brain regions (i.e., Frontal, Temporal, Center, Parietal, Occipital regions in Fig. (l), and then conducted ablation experiments
retrieval task (Fig. (D) and the reconstruction task (Fig. (B). The results showed that using information from all brain regions is optimal, for both retrieval and generation tasks. The occipital had the highest retrieval accuracy and reconstruction performance compared to other regions. Parietal and temporal regions contain semantie information, whereas frontal and central regions contribute the least useful information to the visual decoding.
4 Related Works
Visual decoding using neural signals: Decoding visual information from our brain has been long-standing pursuit in neuroscience and computer science 222 23 Some progress has been
7

---

Average top-1 accuracy across subjects (EEG)
50ma
300mma
150ma
aboms
500ms
Sc0mm
230
b
Average top-1 accuracy subjects (MEG)
End Tre.eme/e (111)
Figure 7: Performance of differrnt EEG/MEG time windows on EEG-guided visual retrieval and reconstruction. (a) The retrieval accuracy of the expanding EEG windows at intervals [0. t] and intervals [t- (00, t] respectively.
(b) The retrieval aceuracy of the expanding MEG windows. (c) Samples reconstructed as the EEG window expands. When the EEG time window is greater than 200ms, the reconstructed image is stable. See Appendix Hor more detailed explanations.
Division of electrode regions
Seen
Frontal
Central
Temporal
Parietal
Occipital
Perial
50-
Deeoo Lord 000%
Cherce Lord for 01.9%
Frodaal Certial Tampcial Patical Occigtal Region lleaved)
Figure 8: EEG-guided retrieval and reconstruction using EEG from different brain regions. (a) The EEG electrodes assigned to five brain regions. (b) Top-1 and top-5 retrieval accuracy, using only the EEG channels in each leaved region and all channels. (c) Reconstructed images obtained using only the electrode channels in each individual region and all channels.
made in decoding steady-state visual stimulus. However, accurately and rapidly decoding semantic information in natural images remains a challenge [24]. fMRI has been widely used to estimate semantic and shape information in visual processing within the brain [25] (26) However, the demand for high-speed and practical applications brain-computer interfaces calls for alternative approaches. EEG, due to its high temporal resolution and portability, emerges promising option [27]. Yet, the overall performance across different subjects and biological plausibility remains unresolved [2 Furthermore, previous approaches often relied supervised learning methods with limited image categories, overlooking the intrinsic relationship between image stimulus and brain responses [
30
Neural decoding using FEG//EG data: Previous studies have shown the efficacy of Temporal- Spatial modules in representing neural data 31]. For example, lightweight convolutional neural

---

networks such as EEGNet and ShallowNet 7 have achieved considerable performance in small EEG and MEG datasets. Using contrastive learning. has been shown that merely using convolutional neural networks and projection layers can yield satisfactory results on neural datasets [32] More recently, Benchetrit et al proposed a method towards real-time MEG--ased reconstruction of visual perception [1]] Song et al. presented an EEG encoder using ShallowNet Temporal-Spatial convolu- tion module with a large convolution kernel with few parameters for EEG embedding, resulting in favorable performance on BEG-based visual decoding [5].
Limitations of previous studies: Previous EEG studies are primarily oriented toward understanding visual perception in the human brain rather than maximizing EEG decoding performance. Thus the visual decoding performance is far from optimal. Specifically, previous studies have trained linear models to (1) classify a small set of images from brain activity [33 34). (2) to predict brain activity from the latent representations of images 4, or (3) to quantify the similarity analysis between these
two patterns with representational similarity 4 33 g 35, While these studies also utilize image embeddings, their linear decoders are limited classifying a small group of object categories or distinguishing image pairs. Moreover, several deep neural networks have been applied to maximize classification of speech (36), cognitive load [37], and images [38] 40 in EEG recordings. (38] proposed deep convolutional neural network for classifying natural images using EEG signals. Unfortunately, the experiment presented all images of the same category in a single block, probably misleading the decoder to rely autocorrelated noise rather than the hidden informative patterns of brain activity [30]. Also, these EEG studies only classify a relatively small number of image categories.
5 Discussion and Conclusion
In this study, we proposed a novel and feasible BEG-based zero-shot image reconstruction framework. Although it utilizes existing machine learning techniques, we demonstrate for the first time that EEG-based zero-shot visual decoding and reconstruction can be competitive with MEG and fMRL.
Our technical contributions are mainly on the EEG encoder and the two-stage zero-shot visual reconstruction framework (Fig. 2 First, we developed the ATM, an EEG encoder which can efficiently represent EEG/MEG features for three tasks. Our comprehensive experiments of the EEG encoder (Fig. 3). compared to various architectures and training methods, achieves SOTA performance across various metrics and tasks (Figs. 4p, 5) Second, our two-stage EEG guidance image reconstruction framework achieves performance close to fMRI using only EEG data (Figs.
Technical Impact:
14
Tab. 6). and this method is compatible with MEG data (Figs. (D).
Neuroscience Insights: Our results offer insights into the relationship between brain activity and visual perception. We analyzed EEG-based visual decoding within different time windows to examine when visual information is perceived in the brain (Fig. 7). Our results revealed that visual information in EEG data is predominantly contained within the 200-400ms range (Fig. 7 7a), consistent with previous EEG studies
[1 6 5 Interestingly, the visual information in MEG data last up to 800ms, much longer than EEG (Fig. 7)), in line with the results reported by previous MEG study [12] []. We also found that EEG performs better than MEG in visual tasks (See Appendix for Tab. 6). which is different from other fields, such as speech decoding (36). In addition, through ablation experiments of spatial information, we found that visual information is mainly encoded the occipital and parietal areas (Fig. g
Interesting Phenomena and Future Directions: First, there are non-negligible performance dif- ferences between cross-subject and within-subject settings. This performance gap arises from inter-subject differences in EEG signals 4T 42). likely attribute heterogeneity in individual brain, differences in visual perception between individuals, and even shifts in noise distribution during EEG recording. So it calls for more efforts on EEG encoder, such as more flexible neural network Transfer learning and
architectures or better weight initialization of pre-trained models [43 44
meta-learning are also future directions worth exploring [45] 47 Moreover, how to unify various electrode montages of different EEG datasets when pre-training large EEG models is a challenge. EEG source localization, which converts senor-level EEG signals into the standard brain source space [48] 49), might be potential solution.

---

Acknowledgements
This work is supported by the National Key R&D Program of China (2021 YFF1200804), Shenzhen Science and Technology Innovation Committee (2020095515595704 XCXF22020122117440000
SGDX20201 10309280100),
This work originated as a course project in the BI&AI 2023 course at SUSTech, and the author would like thank course project members, Yuanhao Fan and Zhihong Wu, for their contributions the early stages of this work. Thank Haoyang Qin for his time and advice.
References
[I] Yulong Liu, Yongqiang Ma. Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainelip: Bridging brain and vinual-linguistic representation via clip for generic natural visual stimulus decoding from fmrri arXiv preprint 129711, 2023.
[2] Paul S Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, et al. Reconstructing the mind's eye: fmri-to- image with contrastive learning and diffusion priors. arXiv preprint arXiv:2305. 18274, 2023.
[3] Tao Fang, Qian Zheng. and Gang Pan. Alleviating the semantic gap for generalized fmriito-imaae reconstruction. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[4] Radoslaw Martin Cichy and Dimitrios Pantazis. Multivariate pattern analysis of meg and ceg: A comparison of representational structure in time and space. Neurolmage, 158:441 2017.
[5] Yonghao Song, Bingchuan Liu, Xiang Li, Nanlin Shi, Yijun Wang. and Xiaorong Gao. Decoding natural images from ceg for object recogmition. arXiv preprint arXiv:2308. 13234, 2023.
[6] Tijl Grootswagers, Ivy Zhou, Amanda Robinson, Martin N Hebart, and Thomas A Carlson. Human cog recordings for 1,854 concepts presented in rapid serial visual presentation streams. Scientific Data, 91):3, 2022.
[7] Robin T. Schirreiittr. Jost Tobias Springenberg. Lukas Dominique Josef Fiederer, Martin Glasstetter. Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, 38(11):5391-55 -4200, 2017.
[8] Alessandro T Gifford, Kshitij Dwivedi, Gemma Roig, and Radoslaw M Cichy. A large and rich eeg dataset for modeling human visual object recognition. Neurolmage, 264: 264:119754, 2022.
[9] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng I Long. itrans- former: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310006625, 2023.
[10] Shanghua Gao. Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. Units: Building unified time series model. arXiv preprint arXiv:2403.00) 2024.
[11] Lina Teichmann, Martin N Hebart. and Chris I Baker. Multidimensionaal object properties are dynamically represented in the human brain. bioRxiv, 2023.
[12] King J R. Benchetrit Y. Banville H. Brain decoding: toward real-time reconstruction of visual perception. arXiv, 2310:19812, 2023.
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMIR, 2021.
[14] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):, 2022.
[15] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high resolutoot image synthesis. arXiv preprint arXiv:2307.01955 2023.
2023.
10

---

[16] Hu Ye, Jun Zhang. Sibo Liu. Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image difflusio models. arXiv preprint arXiv:2308.06721, 2023.
[17] Chenlin Meng, Yutong Hc, Yang Song, Jiamiigg Song, Jiajun Wu, Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arxi: 2 2021.
[18] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12:882580, 2023.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: I:Xivv:412.698)) 1412.6980, 2014.
[20] Fatma Ozcelik and Rufin VanRullen. Natural seene reconstruction from fmrr signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023.
[21] Paolo Scotti, Arpan Banerjee, Jessica Goode, et al. Reconstructing the mind's eye: fmuri-t-immage with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.
[22] Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C Taaaaee Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, 60(5):155 129999 2008
[23] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[24] Nanlin Shi, Xiang Li, Bingchuan Liu, Chen Yang, Yijun Wang, and Xiaorong Gao. Representative based cold start for adaptive ssvep-bci. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:1521-1531, 2023.
[25] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pages 14463, 2023.
[26] Jun Kai Ho. Tomoyasu Horikawa, Kei Majima, Fan Cheng. and Yukiyasu Kamitani. Inter-individual deep image reconstruction via hierarchical neural code conversion. Neurolmage, 271:120007, 2023.
[27] Francis Willett, Donald Avansino, Leigh R Hochberg, Jaimie Henderson, and Krishna V Shenoy. perforrrformacce prain-to-text communication via handwriting. Nature, 593(7858):249- 204, 2021.
[28] Hamad Ahmed, Rounie Wilbur, Hari M Bharadwaj. and Jeffrey Mark Siskind. Object classification from randomized eeg trials. pages 3845-3854, 2021.
[29] Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End-to-end deep image reconstruction from human brain activity. Frontiers computational neuroscience, 13:21, 2019.
[30] Ren Li, Jared S Johansen, Hamad Ahmed, Thomas V Ilyevsky, Ronnie B Wilbur, Hari M Bharadwaj. and Jeffrey Mark Siskind. The perils and pitfalls of block design for eeg classification experiments. IEFE Transactions on Pattern Analysis and Machine Intelligence, 43(1):316-33,, 2020.
[31] Vernon J. Lawhern, Amelia Solon, Nicholas R. Waytowich, et al. Eegnet: a compact convolutional neural network for based brain computer interfaces. Journal of Neural Engineering. 1555)0056013, 2018.
[32] Ting Chen, Simon Kornblith, Mohammad Norouzi, al. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[33] Tijl Grootswagers, Amanda K Robinson, and Thomas A Carlson. The representational dynamics of visual objects in rapid serial visual processing streams. Neurolmage, 188:668-679, 2019.
[34] Jcan-Rémi King and Valentin Wyart. The human brain encodes a chronicle of visual events at each instant of time through the multiplexing of traveling waves. Journal of Neuroscience, 41(34);:224-7233 2021.
[35] Brett B Bankson, Martin N Hebart, Iris IA Groen, and Chris I Baker. The temporal evolution of con- ceptual object representations revealed through models of behavior, semantics and deep neural networks. Newrolmage, 178:172- 182, 2018.
11

---

[36] Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1077-1107 2023.
[37] Zhicheng Jiao, Xinbo Gao. Ying Wang. Jie Li, and Haojun Xu. Deep convolutional neural networks for mental load classification based on eeg data. Pattern Recognition, 76:582-595, 555, 2018
[38] Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, Joseph Schmidt, and Mubarak Shah. Decoding brain representations by multimodal learning of neural activity and visual features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1 (((11))3833-388 2020.
[39] Ben McCartney. Barry Devereux, and Jesus Martinez-del Rincon. zero-shot deep metric learning approach to brain-computerr interfaces for image retrieval. Knowledge-Bant Basedd Systems. 246:108556, 2022.
[40] Subhranil Bagchi and Deepti Bathula. teg-convtransfommer for single-trial eeg-hased visual stimulus classification. Patiern Recognition, 129:108757, 2022.
[41] Erin Gibson, Nancy J Lobaugh, Steve Joordens, and Anthony R McIntosh. Eeg variability: Task-driven or subject-driven signal of interest? Neurolmage, 252:1 19034, 2022.
[42] Lichao Xu, Minpeng Xu, Yufeng Ke, Xingwei An, Shuang Liu, and Dong Ming. Cross-dataset variability problem in eeg decoding with deep learning. Frontiers in human neuroxcience, 14:103, 2020.
[43] Wenhui Cui, Woojae Jeong. Philipp Thölke, Takfarinas Medani, Karim Jerbi, Anand A Joshi, and Richard M Leahy. Neuro-gpt: Developing a foundation model for eeg. arXiv preprint arXiv:231 1.. 1.03764, 2023.
[44] Xia Chen, Xiangbin Teng, Han Chen, Yafeng Pan, and Philipp Geyer. Toward reliable signals decoding for clectroeneephalogram: A benchmark study to cegnex. Biomedical Signal Processing and Control, 87: 2024.
[45] Dongrui Wu, Yifan Xu, and Bao-Liang Lu. Transfer learning for eeg-based brain-computer interfaces: A review of progress made since 2016. IEEF Tronsactions on Cognitive and Developmental Systems, 14(1):4-19 19, 2020.
[46] Dongrui Wu, Xue Jiang, and Ruimin Peng. Transfer learning for motor imagery based brain-computer interfaces: A tutorial. Neural Networks, 153:235-253, 2022.
[47] Yuting Xie, Kun Wang, Jiayuan Meng, Jin Yue, Lin Meng, Weibo Yi, Tzyy-Ping Jung, Minpeng Xu, and Dong Ming- Cross-dataset transfer learning for motor imagery signal classification via multi-task learning and pre-training Journal of Neural Engineering, 20(5):056037, 2023.
[48] C. Wei, K Lou, 7. Wang, al. Edge sparse hasis network: A deep learning framework for eeg source localization. In 202/ International Joint Conference on Neural Networks (IJCNN), pages 1-8. IERF, 2021.
[49] Song Wang, Chen Wei, Kexin Lou, Dongfeng Gu, and Quanying Liu. Advancing ceg/meg source imaging with geometric-informnt basis functions. arXiv preprint arXiv:2401. 2024.
[50] Matthias Guggenmos, Philipp Sterzer, and Radoslaw Martin Cichy. Multivariate pattern analysis for meg:
A comparison of dissimilarity measures. Neuroimage, 173:434-447, 2018.
[51] Yang Song, Jascha Sohl- Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[52] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. Advances in Neural Information Processing Systems, 04-26565-267777 2022.
[53] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311. 17042, 2023.
(54] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-ter transformer for vision and language. arXiv preprint arXiv:2205.1 14100, 2022.
[55] Zhou Wang, Alan C Bovik, Hamid R Sheikh, et al. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-61 612, 2004.
[56] Mathilde Caron, Ishan Misra, Julien Mairal, et al. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-999 29242,99222222 2020.
12

---

[57] Edgar Schonfeld. Sayna Ebrahimi, Samarth Sinha, et al. Generalized zero-and few-shot learning via aligned variational autoencoders. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 8247-8255, 2019.
[58] Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-superrised learning. Advances neural information processing systems, 31, 2018.
[59] Y Shi, B Paige, and P Torr. Variational mixture-of-experts autoencoders for multi-modal deep generative models, Advances in neural information processing systems, 32, 2019.
[60] Thomas M Sutter. Imant Daunhawer, and Julia E Vogt. Generalized multimodal elbo. In International Conference on Learning Representations. 2020.
[61] Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. Decoding visual neural representations by multimodal learning arain-visuull-ie linguiiii features. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[62] Chen Wei, Jiachen Zou, Dietmar Heinke, and Quanying Liu. Cocog: Controllable visual stimuli generation based on human concept representations. arXiv preprint arX1v22404.16482, 2024.
13

---

Supplementary Material:
Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion
A Datasets for experiments
A.1 EEG dataset
We conducted our experiments on the THINGS-EFA dataset's training set 6 This dataset includes a large EEG corpus from 10 human subjects during the visual task. The experiment employed the Rapid Serial Visual Presentation (RSVP) paradigm for orthogonal target detection tasks to ensure that participants attended to the visual stimulus. All 10 participants completed 4 equivalent experiments, resulting in 10 datasets with 16,540 training image conditions repeated 4 times, and 200 testing image conditions repeated
repetitions) + (200 testing image conditions x 80 repetitions) 82, 160 image trials. Original data were collected using a 64-channel system at a sampling rate of 1000 Hz. After signal denoising, epoch data were downsampled to 100 Hz, selecting 17 channels covering the occipital and parictal cortex. Instead of using the raw dataset, we chose to filter it to [0.1, 100] Hz, retaining 63 channels of the original EEG data at sampling rate of 1000 Hz. For preprocessing, ms after the stimulus onset into trials. Baseline correction was performed using the mean of the 200 ms pre-stimulus data. All electrodes were retained and downsampled to 250 Hz for analysis, and multivariate noise normalization was applied to the training data [50]. To improve signal-to-nois ratio, we averaged across the 80 EEG trials from the same image in the test set, while keeping each
times, totaling (16,540 training image conditions
segmented the EEG data from 0 to 1000
EEG trial in the training setting. We compared the effects of averaging across EEG trials and found it indeed improved the performance.
A.2 MEG dataset
To verify the versatility of ATM for embedding electrophyysological data, we tested on MEG data modality using the THINGS-MEG dataset [18]. It includes 27 -channeel MEG data from 4 subjects with 12 MEG sessions. The training dataset has 1854 Concepts x 12 images x repetition, and the test dataset has concepts X 1 image 12 repetitions for 200 times. Here, we discarded 200 testing concepts from the training set to construct the same zero-shot task as with the THINGS -EEG..SEEG....E Each image in the THINGS-MEG was displayed for 500 ms. There 1000 + 200 ms. Continuous MEG data from -100 ms to 1300 ms was segmented into trials after the stimulus onset from 0 1000 ms. Preprocessing was performed using bandpass filter of 1, 40] Hz and haseline correction after downsampling to 200 Hz. Note that due to the small number of participants, no statistical analysis was performed on the MEG dataset. We compared our approach with advanced methods i.c. NICE (5) and B.D. [12 for classification and retrieval tasks on the MEG dataset. We directly used the stimulus images to match the template, rather than other images belonging to the concept.
fixed time for each image of
B More Implementation Details
B.1 Evaluation metric implementation
Classification accuracy As CLIP has been designed to align text and image modalities, we also leverage its text encoder for EEG classification using the text embeddings of categories. This approach utilizes CLIP's text encoding capabilities facilitate EEG classification. We conducted zero-shot classification tests on the THINGS EEG dataset. We employed Top-K accuracy as metric for performance evaluation. Specifically, we assessed performance based on the Top-k (where k=1, 5) predictions. We conducted tests for both within-subject and leave-one-sine -ere--onn-subee-oouus classification accuracy, enabling comprehensive evaluation of the model's performance across different scenarios. Additionally, for each test instance, we extracted embeddings of N-1 unrelated samples from the test set as inputs. This means, apart from the entire test set, the model evaluated by N-Way accuracy (where N=2, 4, 10 in our experiments) on the test set. We report these results in Appendix H
14

---

Retrieval accuracy Similar to the classification task, in the retrieval task, the objective is to retrieve the Top-K images most related given stimulus image via its corresponding EEG signal. This implies that by changing the text embeddings of image labels to image embeddings, we can transition the task from classification to image retrieval. Given that contrastive learning known to be sensitive to batch size, we also compared the performance improvement of different methods under varying batch sizes (batch size=16, 16, 1024) (Appendix H).
Generation accuracy The generation task presents more challenges than the other tasks. For each image condition in the test set, we generate different images from subjects based on the corresponding EEG. Subsequently, image retrieval is performed for each generated image. The Top-1 and Top-5 accuracies are calculated. helps in evaluating the semantic alignment between the generated images and their original counterparts.
B.2 Computing methods implementation
In the upstream EEG encoder part, we compared various methods. For the B.D. method 12 we replicated the network structure as described in the original work, with the difference being in the shape of the input data due the original study's focus on MEG. worth mentioning that we used modify its subject-wise layer an linear
the leave one for subject method in the testing process layer for modeling the time dimension. To ensure fairness, we did not use the same hyperparameters as in the original paper. Instead, we chose settings yielded excellent results upon reproduction. Across all methods, we used identical hyperparameters. apart from the network structures. These included batch size, optimizer, initial learning rate, and temperature parameters.
B.3 Architecture details
Table 2: Brain module configuration
Layer
Input shape
Output shape
parameters
Channel-wise attention layer
(N, C, T)
(N, C, D)
553,078
Temporal-Spatte Conv module
(N. C, D)
(N. HI, H2)
103,680
Temporal-Spatie aggregation
(N, HI, H2)
(N, H1*H2)
0
MIP projector
(N, H1*H2)
(N, 1024)
2,527,232
Total
3,183,990
Table 3: Ablation study on the ATM model's different components for THINGS-EEG retrieval.
Module
MLP
TSCony
CAL
TOP-1
TOP-5
X
8.11+1.74
26.83+4.78
21.65+6.22
51.3449.83
ATM-S
V
X
V
23.73+7.62
52.71 49.71
28.6446.39
58.47+8.97
AY
X
X
8.11+1.74
26.83+4.78
V
17.95+5.95
ATM-E
X
V
23.73+7.62
52.71 19.71
24.9246.00
54.78+8.49
B.4 Model configuration
To validate the efficacy of our EEG encoder, we experimented with variety of empirical setups aimed at optimizing the model's efficiency. we leverage joint subject training to adapt to new subjects. Once a model trained, can be used both reasoning about known subjects (subject-specific tokens) and reasoning about unknown subjects (shared tokens). In the context of channel-wise attention layer, we explored four distinct approaches for enhaneing downstream retrieval capabilities: leveraging subject-specific tokens for retrieval, averaging all tokens for more generalized retrieval,
15

---

flattening the entire token set for direct retrieval, and preserving token dimensions to feed into the Cemporal-Spatial convolution module for feature integration. Notably, the strategy of preserving dimensions and using the Temporal-Spatial convolution module emerged as the most effective.
In our quest optimize token embeddings, we experimented with a variety of approaches, including using 1 x convolution and linear layers. Under our framework, using linear layers performs better than convolutions. Moving beyond, we delved into the efficacy of diverse Feed Forward Networks (FFNs) within the Transformer encoder layer. Our findings indicated that an FFN tailored to the temporal dimension emerged as the superior option. We also conducted a comparative analysis of various positional encoding techniques. Interestingly, for Temporal-S Spatial convolution utilized in retrieval tasks, the significance of positional encoding diminished.
In a further exploration, we examined the impact of deploying convolutions at various stages and even contemplated the complete removal of the convolution module. It was discovered that situating convolutions post the Transformer encoder layer yielded the most favorable outcomes. Conversely, a shift to a MLP or the removal of the convolution module led to notable degradation in performance. Our assumption is though the convolution 's inherent translational invariance is compromised when the context of the time dimension is disrupted, the efficiency of parameters it possesses may confer resistance to overfitting, thereby maintaining its effectiveness.
Table 4: Impact of each module the result in different configurations. The reported results represent the mean performance metrics of the ATM-S, calculated over the final 10 training epochs across all 10 subjects.
Module
Config
Top-1 (std)
Top-5 (std)
w/ mean token
7.29 (3.01)
23.11 (6.62)
w/ flatten token
14.85 (5.46)
37.56 (8.92)
Channel-wise attention layer
w/ keep dim
28.64 (6.39)
58.47 (8.97)
conv ld
24.81 (7.29)
55.68 (9.08)
Token embedding
w/ linear
28.64 (6.39)
58.47 (8.97)
w/ temporal dim
28.64 (6.39)
58.47 (8.97)
Feed Forward Network
w/ spatial dim
19.92 (7.74)
47.59 (11.5)
w/o
26.45 (7.73)
57.00 (9.95)
w/ sinusoidal
27.96 (6.54)
58.16 (8.44)
Position encoding
w/ learnable absolute
26.66 (6.56)
56.95 (7.95)
w/o
28.64 (6.39)
58.47 (8.97)
w/ pre
14.23 (4.20)
37.44 (6.06)
w/ post
28.64 (6.39)
58.47 (8.97)
Temporal spatial convolution
w/ both
25.37 (5.16)
57.07 (6.13)
w/o
23.73 (7.62)
52.71 (9.71)
B.5
Training details
In our EEG projector module, integrated two distinct strategies for steering model predictions: text embedding and image embedding. Given the variance in feature granularity, we observed that alignments that prioritize image embedding excel tasks of image retrieval and classification. Throughout the training phase, our experiments revealed that a batch size of 16 is judicious selection for all models. Conversely, batch size of 1024, which implies substantial number of samples are processed in each training iteration, necessitates the model to exhibit a heightened capacity for noise resistance. In order to enhance the signal-to-noise ratio within EEG data, we implemented an averaging technique on 80 repeated instances within the test set. This approach mirrors the methodology employed in identifying Event-Relater Potentials (ERP). To maximize the utilization of the available training data, we refrained from averaging the 4 repetitions in the training set. Instead, we opted to input the complete set of EEG data into the model, thereby facilitating comprehensive learning.
16

---

a
0.30
0.30
Aoweage tedt Brsmmey
Anerage Teat Arauney
Confomer
0.25
0.25
ATM-E
ATM-5 [Ours]
6.15
0.15
MLF
BEGNetvl
0.1
6.10
6.05
0.00
0.00
tooch
Epoch
Figure Test accuracy during each training epoch. (a) Training of different within-subject models. (b) Training of different across-subject models. We compared 7 different EEG encoding models, including EEGconformer, MLP, EEGNetv4, B.D., NICE, ATM-S (Ours) and ATM-E (Ours).
C Details of EEG guidance image generation
Here, we provide a concise overview the conditional diffusion model framework used in EEG- guided image generation, following the presentation of continuous-time diffusion models in [51 52
Diffusion models Diffusion Models (DMs) engage a generative process by transforming high- variance Gaussian noise into structured data representations. This transformation is achieved by gradually reducing noise levels across sequence steps. Specifically, we begin with a high-variance Gaussian noise ZM ~ (0, max) and systematically denoise it through a series of steps to obtain It ~ p(z;; t), where og < 0t+1 and σM = max- For a well-calibrated DM, and with oy = 0, the final To aligns with the original data distribution.
Sampling process The sampling in DMs is implemented by numerically simulating a Probability Flow ordinary differential equation (ODE) stochastic differential equation (SDE). The ODE is represented as:
dx - (t) V. log p(æ;
(1)
where z log p(a; t) is the score function, and (t) is pre-defined schedule with its time derivative (t). The SDE variant includes Langevin diffusion component and is expressed as:
dx (t) V. log p(2; t)dt
(()) log pog t)dt
(2)
+ (()
where du, is the standard Wiener process.
Training of DMs The core of DM training is to learn model for the score function. This is typically achieved through denoising score matching (DSM), where is learnable denoiser. The training process can formulated as:
[[leo(xo o,t,,c) -
where is Gaussian noise with variance o². and e represents condition.
C.1 Stage I EEG-Conditioned Diffusion
The initiation of the IEG-conditioned diffusion phase is paramount in our DEG-based image gen- eration framework, leveraging the classifier-fres guidance strategy alongside data pairs of CLIP embeddings and EEG embeddings (21, 2g). Adapting from state-of-the-an generative techniques, our diffusion process is specifically conditioned on the EEG embedding zæ to adeptly capture the distribution of CLIP embeddings p(2r(zm). The CLIP embedding 2ı, procured during this phase, establishes the groundwork for the ensuing image generation stage. Our architecture incorporates streamlined U-Net, labeled as Eprior (2), , ZE), where 2/ signifies the perturbed CLIP embedding at
17

---

given diffusion timestep t. The training utilizes pairs from the ImageNet database, consisting of over million images, to fine-tune the LEG-Condiiioned Diffusion model. This model meticulously trained using the classifier-free guidance approach, effectively balancing the conditioning signal's fidelity with the generative output's diversity.
Classifier-free guidance method The Classifier-Free Guidance technique is crucial in guiding the iterative refinement of Diffusion Model (DM) under specific EEG condition zg. achieves this by synchronizing the outputs of both a conditional and an unconditional model. The model's formulation, "prior f 1, Zg), is as follows:
(4) where w 2 represents the guidance scale. This mechanism facilitates concurrent training of the conditional and unconditional models within singular network framework, periodically substituting the EEG embedding zg with a null vector to promote training variability, i.e. 10% of the time. The primary objective of this method is to enhance the sample quality produced by DMs while maintaining output diversity.
20 Eprior (iiir(2);t, zg) = (1 + () zg) (2):
C.2
Stage II CLIP-Embedded Image Synthesis
In Fig. 10 we compare the effects of one-stage and two-stage EEG-guided image generation. We show images generated using EEG embeddings directly (One-stage) and images generated using image embeddings obtained via prior diffusion (Two-stage). It can be seen that the two-stage EEG- guided image generation can more accurately reconstruct the semantic and low-level visual features of the original image, and the style is more realistic.
Seen
Stage
Stage
2 4.777
10
No. PL
Figure 10: Comparison between one-stage and two-stage EEG guidance image reconstruction. (a) We present the images that subjects seen (Seen), our reconstructed images directly using EEG em- beddings (One-stage), and the reconstructed images from low level and high level image embeddings obtained by the prior diffusion (Two-stage). These results indicate that the strategy of our two wtaet generation can better reconstruct the seen visual stimulus. (b) We employed ATM-S compare the generated images with the original images in retrieval task. Our result indicates that the images generated in two stages significantly enhance the performance of the original model on the retrieval task.
In the second stage of our EEG-based image generation approach, the CLIP embedding z; derived from the EEG-conditioned diffusion acts as the precursor for synthesizing visual objects based on 21. This is achieved by harnessing the synergies of advanced pre-trained models, namely SDXL and IP-Adapter [15] [6]. facilitating the creation of high-caliber images.
The cornerstone of our synthesis process is the SDXL framework, acclaimed for its proficiency in text- to-image conversion. The integration of the IP-Adapter introduces dual cross-attention mechanisms, allowing the CLIP embedding 2, to within the U-Net structure. The synthesis model is denoted as (sD(2,, f, 21), where 2 denotes the SDXL Variational Autoencoder's (VAE) disturbed latents.
directive input and guide the denoising trajectory
SDXL-turbo for accelerated processing To augment the efficiency of our framework, we addition- ally explore the SDXL-Turbb (53). a refined iteration of SDXL optimized for swift image synthesis. This variant proves especially beneficial in scenarios demanding quick generation of high-fidelity visuals.
18

---

IP-Adapter's efficacy The Adaaeer, with its compact design, has proven to be effective in enhancing image prompt adaptability within pre-trained text-to-image models. Its compatibility with text prompts for multimodal image generation extends the versatility of our EEG-based image synthesis approach.
C.3 Low-level pipeline
Compared with pure vision pre-training models such (ViT, ResNet, DINO, etc.), the CLIP model lacks low-level visual features. Therefore, in order to make up for this shortcoming, our framework introduces a low-level visual reconstruction pipeline. We hope to restore basic such contour, posture, orientation and other pixel-level information from EEG by aligning with the latent of VAE.
Past work [25] has found that in the denoising stage of the early diffusion model, signals (corre- sponding to the VAE latent in our framework) dominated prediction of fMRI signals. And during the middle step of the denoising process, 2e predicted activity within higher visual cortex much better than 2. However, note that this is only an analysis based on decoding accuracy. These analyses do not have strong neuroscience causal relationship. We still cannot conclude that the low-level features of neural data are modeled by VAE.
Table 5: Latent VAE retrieval performance
Condition
Top-1 (%)
Top-5 (%)
ideal chance
0.5
2.5
ATM-S (Ours)
10.14
29.55
We trained the low-level pipe for 200 epochs, trying latent mean squared error (MSE) loss, along with contrastive learning loss, and a variational autoencoder (VAE) image reconstruction loss to align the 4 x 64 x 64 EEG latents obtained from projection layer and an upsampled CNN with the VAE latents. Nevertheless, reconstruction loss or contrastive learning loss performs worse than only applying the loss in latent space and also requires significantly more GPU memory. In addition, we found that using low-level visual model for distillation learning in the low-level pipeline is not only unhelpful for VAE latent training, but also leads to overfitting. Similar conclusions were reached in MindEye 2) Our results suggested that the low-level zeroshot reconstruction in EEG is not stable enough and may mislead the model results. When using the low level pipeline, we usually set the inference steps of SDXL to 10 (or SDXL-turbo to 4) and the image-to-image denoising strength to 0.5. We give several reconstruction examples in Fig. [o compare the impact before and after using low level pipeline. Moreover, low level alignment was validated through retrieval performance tests, as depicted in Tab. 5 Our findings indicate that retrieval using EEG latents can also achieve excellent performance. This further substantiates the fcasibility of aligning the low-level consistency through VAE latents.
C.4 Semantic-level pipeline
In addition to using ERG latent and low-level pipelines during reconstruction, we also add a cor- responding semantic level pipeline guided by text captions during the image reconstruction. We input the 1 x 1024 EEG features output by prior Diffusion into the trained image projector to obtain 256 x 1024 image features. Using the GIT m54e we can directly generate caption from the latent features of the image. IP- [P-Adapter accepts such a caption as text prompt to guide the semantic level reconstruction of the image. It should be noted that due to the difficulty of the zeroshot task itself and the low dimensionality of the EEG features, the caption generated from the latent may be unstable, thereby interfering with the original correct EEG semantics. Considering that the image features extracted by the CLIP model itself are already high-level visual features and do not require the introduction of more semantic information, this framework retains the entry of the text prompt, and the reconstructed image presented does not force the use of the semantic level pipeline.
19

---

CLIP Semantic
CLIP low level
CLIP
Seen image
Only CLIP
Only low level
Semantic
K
Figure 11: Example of our reconstructions for Subject 8 output from different pipelines. From left to right: reconstruction using only CLIP (i.e., Only CLIP), using only CLIP the semantic pipeline (i.e., CLIP + Semantic), using only the low level pipeline (i.e., Only low level), using only CLIP and the low level pipeline (i.e., CLIP + low level), using joint CLIP, low level, and semantic pipelines.
D Performance comparison
Comparison metrics Our study uses various metrics to evaluate how well we can recreate visual stimulus from brain data (EEG, MEG, IMRI) (Tab. T and Tab. 77. These metrics include PixCorr (pixelwise correlation, between ground truth and reconstructions). SSIM (structural similarity in- dex (etric)(55), SwAV (SwAV-ResNee55, refer to average correlation distance) [56], and two-way identification using neural networks (AlexNet(2/5), Inception, CLIP. Here AlexNet(2/5) the 2nd and 5th feature layers of AlexNet) for both low-level and high-level image features. Here two-way identification can be seem two-way retrieval task described in (20). In Tab. [ our results showed that on the THINGS dataset, we could achieve performance over MEG on EEG reconstruction using ATM. Tab. shows the decoding performance of different data sets (IMRL, MEG, EEG) on visual
20

---

stimulus tasks, and we even achieved the same or better performance than fMRI and MEG. Our results suggest that suitable neural representation plays a decisive role in the downstream task.
Table 6:
The classification performance of various methods are discussed. Due to differences in datasets and data modalities, have specified unified metrics objectively assess the performance of each method.
50-way
100-way
200- 200-way
Dataset
Model
top-1
top-5
top-1
top-5
top-1
top-5
CADA- VAE (V&T) 57)
10.02
40.37
MVAE (V&T) [58]
10.04
39.60
GOD-WiL (IMRI)
MMVAE (V&T) [59]
11.68
43.29
MoPoE- VAE (V&T)
60
12.90
51.78
BraVI. (V&T) [6]
13.99
53.13
THINGS (MEG)
ATM (Ours)
15.63
41.38
11.75
29.25
5,88
19.25
THINGS (EEG)
BraVL 61
14.33
40.28
5.82
17.45
ATM (Ours)
17.40
39.40
11.50
28.50
7,40
20.60
Table 7: Quantitative comparison results of image reconstruction Subject via our framework using different encoders.
Low-level
High-level
Dataset
PixCorr
SSIM f AlexNe((2) f AlexNeu(5)
Inception
CLIP
SwAV
NSD-AMRI 12
0.305
0.366
0,962
0.977
0.910
0.917
0.410
THINGS-MER
12
0.058
0.327
0.695
0.753
0.593
0.700
0.630
THINGS- MEG (averaged)
12
0.090
0.336
0.736
0.826
0.671
0.767
0.584
THINGS-MEG (Ours)
0.104
0,340
0.613
0.672
0.619
0,603
0.651
THINGS-EER EEG (NICE) 5
0.142
0.276
0.739
0.832
0.659
0.722
0.612
THINGS-FFE (EFGNeIV4) 31
0.140
0.302
0.767
0.840
0.713
0.773
0.581
THINGS-EEG (Ours)
0.160
0.345
0.776
0,866
0.734
0.786
0,582
E Representational analysis
As depicted in Fig. 12] we showcase the representational similarity matrix and visualization in the latent space. To investigate the relationship between the representations obtained from BEG and those images, we conducted a representational similarity matrix. We focused on Subject who exhibited the highest retrieval accuracy. By applying clustering algorithm to the image embeddings corresponding to 200 images in the test set, we observed distinct within-category clustering. We generated similarity matrices based on both image and text embeddings, which were then compared with EEG representations. As shown Fig. [2] clear within cattinycategory clustering is observable in the representational similarity matrix with image, whereas this phenomenon is present in the representational similarity matrix with text.
F Concept analysis
We have adopted the concept embedding encoder proposed by Wei et al. (62). which encodes the clip embedding of the original image into a 42-dimensional vector, with each dimension representing distinct concept. Utilizing the ATM for direct projection on the EEG data of 200 categories from the test set, we obtained EEG embeddings that were then fed into the concept encoder with frozen weights, yielding 200 concept embeddings, as depicted Fig. [3] An analysis of representational similarity at the concept level indicates that the EEG embeddings derived from our EEG projector effectively align with the conceptual space. This ensures semantic consistency at a high level of alignment, providing compelling evidence for the reconstruction of images from EEG data.
21

---

b
Image
Representation Simila
Chasters
Glanmr
6.35
chaner
0.35
6.20
1-SNF of EPG Features
0.30
Clusters
Chatoe
6.56
Clanse
TTG
Representation on Similant
-10
0.15
Data Type
0.12
t-SNE of Text Features
Ghane Chener
0.075
charse
0,023
-10
Dato
LEG
Figure 12: Visualization of the representation of EEG, image and text modality. (a) Representa- tional similarity matrix between EEG features and image/text features. (b) Visualization in the latent space of EEG/image/txxt by t-SNE.
22

---

a
ground animals
mascurme
EEG embedding
hame: tools
cotton clothing
wheeled ychicles
Reconstruction
household furniture
0
home tools
Seen
wheeled vehicles
flying fechnology
ground
activities
0.0 0.5
baked food
home tools
household furniture
pointy tools
containers for liquids
paper like
ice
Similarity Matrix
Clustered Similarity Matrix
Encoded con ception embedding
Encoded conception embedding
40
nn sicl
0.2
-0
-10
150
-0.90
10
120
190
Conception embedding
Conception embedding
Figure 13: Visualization the conceptual representation analysis. (a) Conceptual representations were obtained from eeg embeddings using concept encoder. (b) The similar matrix between EEG embeddings and real concept embeddings. (c) Concept embedding similarity matrix after cluster rearrangement (k=5).
23

---

G Additional images results
We visualize the best, medium and worst generated images in Fig. 14 We randomly selected the EEG data of subject viewing 100 images, and extracted EEG embeddings to guide image generation. By calculating the cosine similarity of the CLIP embedding between the generated image and the original image, we found 12 images each with the best, medium and worst generation effects. It can be seen that the best group, the generated image not only highly consistent with the semantics of the original image, but also well retains the low-level visual features. in the medium group, the generated image maintains the semantic features of the original image, and the low-level visual features are well preserved. Visual features were altered. in the worst group, both semantic features and low-level visual features were altered.
Generated Seen
Seen
Median
Generated
Generated Seen
Wost
Figure 14: Examples of EEG-guided visual reconstruction. From top to bottom, exhibit the best, median, and worst 12 generated images, respectively. We show the images subjects seen and the generated images by our two-stage image generator.
G.1 Additional retrieval results
24

---

Seen
To
Top
Top
Top
Top 5
Top
Top
Top
Isp 9
Top 10
Figure 15: Additional retrieval results
25

---

G.2
Additional generated images
Seen
Best
Generation
Figure 16: Additional generated results with the best alignment to original images
26

---

Seen
Median
Generation
Figure 17: Additional generated results with the median alignment to original images
27

---

Seen
Worst
Generation
Figure 18: Additional generated results with the worst alignment to original images
28

---

G.3
Additional generated images for each subject
Seen
Best
Generated
Seen
Median
Generated
Seen
Generated
Figure 19: Part of subject generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared to the original stimulus pictures.
Seen
Median
Generated
Figure 20: Part of subject 2 generates images. We do batch generation of the subjects and then calculate the best, medium, and worst performers compared to the original stimulus pictures.
H Additional evaluation results
29

---

Best
Generated
Seen
Median
Generated
Seen
Worst
Generated
Figure 21: Part of subject 3 generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared to the original stimulus pictures.
Generated
Generated Seen
Median
Seen
Generated
Figure 22: Part of subject 4 generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared to the original stimulus pictures.
30

---

Median
Worst
Generated
Figure 23: Part of subject 5 generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared the original stimulus pictures.
Seen
Generated
Generated Seen
Median
Seen
Generated
Figure 24: Part of subject 6 generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared the original stimulus pictures.
31

---

Generated Seen
Median
Generated
Seen
Generated
Figure 25: Part of subject generates images. We a batch generation of the subjects and then calculate the best, medium, and worst performers compared to the original stimulus pictures.
Generalta
Median
Generated
Seen
Generated
Figure 26: Part of Subject generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared to the original stimulus pictures.
32

---

Seen
Generated
Seen
Median
Generated
Seen
Generated
Figure 27: Part of subject 9 generates images. We do a batch generation of the subjects and then calculate the best, medium, and worst performers compared the original stimulus pictures.
Generated
Median
Generated
Seen
Generated
Figure 28: Part of subject 10 generates images. We do batch generation the subjects and then calculate the best, medium, and worst performers compared the original stimulus pictures.
33

---

H.1 Accuracy for time windows
According to our results in for time windows, which shows that for all embedddings, clear peak can be observed for windows ending around 200-250 ms after image onset. Comparing (29 and 30 we can see that, unlike [[2), our time window results on THINGS-EEG dataset do not have a second peak, which may be mainly affected by the experimental paradigm. In |
we can see that in the first 50ms-200ms, the image reconstructed 50 ms is completely messy and has no semantics; the semantics of the reconstructed image after 250 ms is basically correct and gradually stabilizes, and after 500ms, due to the lack of additional visual response, the content of the image reconstruction more stable. Similar results are also shown in Figure 4 A of [1] their method can also reconstruct high-quality images at 100ms. This just shows that our reconstruction results are line with the neuroscience prior. However, this does not mean that the EEG data after the absence of visual response (200ms) loses its contribution to decoding, because the processing of high-level visual features (corresponding to the visual features of CLIP) may be involved over time.
Subjeree
20-
Chance 10 29u
Chanze
B
susject
Average subjects
50-
60-
30-
29000 2000000
Figure 29: Accuracy for growing windows. We use an EEG time window of 100ms, sliding 100ms each time. (a) Top-1 accuracy. (b) Top-5 accuracy.
34

---

TPp- accuracy for each subject
Avenae top-1 accuracy across subjects
15.0-
12.5-
10.0
10
Charce 130
Chonce
0.0
End IMI
End FIne OYS)
Top 5 accuracy for each subject
Average top accuracy across subjects
Sudjects
32
22-
Change
150
500
1030
500
109
Figure 30: Accuracy for sliding windows. We use an EEG time window of 100ms, sliding 100ms each time. (a) Top-1 accuracy. (b) Top-5 accuracy.
The-1 subject
40-
Subaects
wrmmc th
Chance (2.2%)
Endtmme mnet
accurecy each subject
subjects
70
SuMjets
60
1000
1000
Figure 31: Accuracy for growing windows. The MEG time window grows from 50ms to 1000ms. (a) Top-1 accuracy. (b) Top-5 accuracy.
35

---

subject
11.5
Seigets
13.0
100-
Taxk
50-
Choece 990
Chence 510
60
3200
400
lime
End STA
accenwcy
Aueerage
wibjects
18-
13
20
100
End time
Fed
Figure 32: Accuracy for sliding windows. We use an MEG time window of 100ms, sliding 100ms
each time. (a) Top-1 accuracy. (b) Top-5 accuracy.
Table 8: Overall performance of zero-shot Retrieval on THINGS-EEG dataset. We divided the last batch from the original training set as the validation set, selected the best model according to the minimum validation loss in 40 epochs, and finally evaluated the performance on the test set. We showed the performance of in-subject and cross-subject retrieval task (Ave Std. %) under condition of batch size=1024. We compared the 2-way, 4-way, 10-way, the Top-1 and Top-5 accuracy of 200-way from different EEG embedding methods.
Subject dependent train and test on one subject (buich sizze (024)
Methods
2-Way
4-Way
10-Way
Top-1
Top-5
12.97
+
EBGConformer
76.17 2 13.13
56.20
16.70
34.72
14.79
3.5 2 2.30
17.10 4: 9.21
74.32 12.14
53.97
15.81
33.48
14:35
6.10
16:53 2: 9.94
EEGNNEI44
4.20
19.51
37.97
7.81
NICE
03.60 03.60 2.1.
84.27
5.27
69.91
8.21
21.53 5.00
51.57 2: 10.97
MLP
84.88 - 3.42
67.39
5.71
46.29
6.23
7.34 1344.22.14
24.39 4: 5.18
ATM-S (Ours)
(Ours)
92.09
4.46
68.87
2: 9.59
Subject independent one subject test size-1004)
2-Way
4-Way
10-Way
Top-1
Top-5
ERGNNVV4
82.85
42.35
3.56
20.95 5:73
EEOConformer
56.54
4.00
31.80
3.20
13.89
2.01
0.87
4.12 1: 1,20
ShalkoFFBCSPNat
75.76
2.49
53.63 =: 3.58
31.43
4.30
2.51 2:11::0 1.31
12.03 1: 2.78
LBGNeIV4
82.00
3.17
5.44
42.24
6.10
6.13
21.23 4: 5.19
NICE
83.88
5.30
45.13
5.85
0.342:30 3.00
23.76
5.09
MLP
75,50
2.45
55.08
3.07
34,05
2,83
4,46 0:0.1
15.26 4: 2.34
ATM- (Ours)
83.72
3.01
66.91
5.41
46.53
6.24
8.4 2.73
25.36 1: 6.24
ATM-E
8065
36

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, accurately explain the drawbacks of current use of IMRI for image reconstruction and some past EEG decoding methods the field of brain-computer interfaces. We propose new EEG encoder and image reconstruction framework, and achieve impressive performance on zero-shot decoding and reconstruction tasks. Our work provides valuable guidance toward practicality in EEG decoding and image reconstructing.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We dedicate a paragraph in Section 5 to explain the shortcomings of the current work and future improvement directions in more detail.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
37

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Our theory mainly involves the theory related to the diffusion model, and have relatively fully proofed it in Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We not only describe the experimental settings in detail in the experimental part of the main paper, but also provide comprehensive supplementary instructions in the Appendix of supplementary materials. Of course also provide code for reviewers to review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
38

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide an anonymous code repository link at the end of the abstract to facilitate review by reviewers. We have detailed how to reproduce experimental results step by step in the readme document of the code repository. We guarantee that all our results are reproducible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/
bobllic/guides// wwtps//ooooonniiiioooooolooo...e
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/pips..ccppulic//uuu/ / https///ooooeuuoienioooodoooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the configuration during preprocessing, training. and testing of our data in detail in the experimental section of the paper. Full details provided with the anonymous code repository, with detailed supplementary instructions in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
39

---

Justification: In the box plots and line charts in this article, we provide accurate error bars to describe the performance differences between different settings and different subjects.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Section 3.1 of this article, we dedicate section to describe in detail the environment configuration for running our experiments. We ensure that important results can be reproduced intact.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///uuripps.cppplt////eaeaaa
Answer: [Yes]
Justification:
We fully adhere to the NeurIPS Code of Ethics throughout the entire process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
40

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As work on decoding and reconstructing brain activity from neural data, our method uses authorized and publicly available EEG and MEG datasets. All our experiments produced no other negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We did not crawl the dataset online.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the datasets we used and the methods used to evaluate performance.
Guidelines:
41

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We open up the code and model weights of our method for reproducible results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We used publicly available datasets containing EEG and MEG from human subjects studies. We strictly adhere to the relevant requirements of neuroscience ethics.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
42

---

Justification: Our work does not involve crowdsourcing or human subjects research. We are using publicly available datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "rkEJ0zkcjT", "reviewer": "Reviewer_asin", "summary": "This paper proposes a learning framework to decode images from EEG signals. It introduces a tailored brain encoder, the Adaptive Thinking Mapper, which projects neural signals to the clip embedding space. Subsequently, a two-stage image generation strategy is applied to produce images, progressing from blurry to high-quality reconstructions. This is an interesting and innovative work.", "questions": "* Is the spider plot in Figure 4 showing normalized performance? Please clarify.\n\n* Please address my queries in the Weakness section:", "limitations": "I agree with the limitation regarding the cross-subject performance drop. Future work could consider incorporating existing cross-subject generalization efforts from the EEG domain to address this issue.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper is easy to understand, and it is an interesting work exploring EEG to image decoding.", "weaknesses": "* Some experimental settings were not very clearly illustrated. For instance, in Section 3.1, it is mentioned that the experiments were trained on an EEG dataset and tested on an MEG dataset. How did you align the channel heterogeneity? Is it a zero-shot approach?\n\n* Table 1 is a bit confusing to me. It would be helpful to clarify and explain the experimental settings of these additional datasets.\n\n* Although EEG has a fast response, the design of the proposed framework does not seem to leverage motivations related to EEG data. It would be better to demonstrate the effectiveness of the proposed framework compared to existing fMRI-image decoding methodologies. For instance, you could replace the EEG encoder with an fMRI encoder and test whether the framework can outperform existing methodologies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thorough comments! Below please find our point-to-point responses to your comments.\n\n**Q1. How did you align the channel heterogeneity? Is it a zero-shot approach?**\n\n**Regarding Section 3.1:** ”To verify the versatility of ATM for embedding electrophysiological data, we tested it on MEG data modality using the THINGS-MEG dataset\" : \n\nWhat we mean in the text is that in order to prove that our framework is universal for general embedding electrophysiological data, we also test the performance of our framework on the THINGS-MEG dataset. As can be seen in **Figure 4**, the MEG dataset and the EEG dataset are different evaluation systems because they are trained and evaluated separately. Therefore, the issue of channel heterogeneity is not involved here. Similar to the THINGS-EEG dataset, we also performed zero-shot testing on the THINGS-MEG dataset.\n\n**Q2. “It would be helpful to clarify and explain the experimental settings of these additional datasets. ”**\n\n**Regarding Table1:** In order to intuitively demonstrate the advantages of our framework, we compared the performance of our method with different methods on EEG, MEG and even fMRI dataset. Under the zero-shot setting, our framework makes the image reconstruction performance surpass the performance of [1] on MEG data, and is closer to the performance of current methods [1][2][3] on fMRI data, which highlights the superiority of our framework and makes image decoding and reconstruction more practical.\n\n**Q3. “It would be better to demonstrate the effectiveness of the proposed framework compared to existing fMRI-image decoding methodologies. ”**\n\nPlease kindly see the relevant answer in **Global rebuttal: Q1 and Q2.** We believe that the differences in the characteristics of fMRI and EEG themselves are the premise that the existing fMRI decoding technology cannot be applied in actual brain-computer interface tasks. Therefore, it is necessary to design a framework that is suitable for EEG real-time decoding and online image reconstruction applications. For EEG data with low signal-to-noise ratio, non-stationarity, and possible time window offset, it is impossible for EEG to be as simple as fMRI encoder. It requires special design to meet the requirements of replacing manual features. In terms of the superiority of our framework, compared to fMRI-based decoding that can only be attempted on datasets, our proposed EEG encoder can directly process EEG signals and respond quickly to complete a series of decoding and image reconstruction tasks. \n| Dataset                                 |  PixCorr  |  SSIM   | AlexNet (2) | AlexNet (5) | Inception |   CLIP   |   SwAV   |\n| --------------------------------------- | :-------: | :-----: | :--------: | :--------: | :-------: | :------: | :------: |\n| NSD-fMRI (Benchetrit_2023)              |   0.305   |  0.366  |   0.962    |   0.977    |   0.910   |   0.917  |   0.410  |\n| THINGS-EEG (NICE, Song Y et al.)                       |   0.142   |  0.276  |   0.739    |   0.832    |   0.659   |   0.722  |   0.612  |\n| THINGS-EEG (EEGNetV4, Lawhern et al.)                   |   0.140   |  0.302  |   0.767    |   0.840    |   0.713   |   0.773  |   0.581  |\n| THINGS-MEG (Benchetrit_2023)            |   0.058   |  0.327  |   0.695    |   0.753    |   0.593   |   0.700  |   0.630  |\n| THINGS-MEG (averaged) (Benchetrit_2023) |   0.090   |  0.336  |   0.736    |   0.826    |   0.671   |   0.767  |   0.584  |\n| **THINGS-EEG (ATM-S, Ours)**                     |   **0.160**   |  **0.345**  |   **0.776**    |   **0.866**    |   **0.734**   |   **0.786**  |   **0.582**  |\n| **THINGS-MEG (ATM-S, Ours)**                      |   **0.104**   |  **0.340**  |   **0.613**   |   **0.672**    |   **0.619**   |   **0.603**  |   **0.651**  |\n\nIn order to demonstrate the competitiveness of our framework, we used different neural encoder to test the reconstruction performance on the THINGS-EEG and THINGS-MEG dataset and the results are shown above.\n\n**Q4. Is the spider plot in Figure 4 showing normalized performance?**\n\nAs shown in **Figure 4**, in each metric direction, we use the accuracy calculated by the best performing method as the benchmark and draw a normalized radar chart to more intuitively show the superiority of our method. Its absolute performance can be seen in **Table 7** in the **Appendix H.1**.\n\n**Reference**\n\n[1]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.\n\n[2]Fatma Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023. \n\n[3]Paolo Scotti, Arpan Banerjee, Jessica Goode, et al. Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.\n\n[4]Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14453-14463.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mpgAfhiZGV", "reviewer": "Reviewer_zVcP", "summary": "The study proposes an end-to-end EEG\\MEG-to-image reconstruction framework, consisting of a tailored brain encoder ATM to project neural signals into the shared subspace as the clip embedding, and a two-stage image generation block. The model achieves successful cross-subject EEG\\MEG decoding and SOTA performance in classification, retrieval, and reconstruction.", "questions": "1. For low-level metrics, in Mindeye, Ozcelik et al.’s, and other fMRI-to-image reconstruction works, they usually provide PixCorr values. I wonder why this metric wasn't included here.\n2. For Figure 4, the plots are impressive, but it would be helpful to include numerical indications of the downstream task performances, such as 0.7 or 0.8. This would provide a clearer understanding of how well the model is performing.\n3. Could the author elaborate more on the functioning of the shared token and subject tokens?", "limitations": "The authors have adequately discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and nicely written.\n2. The experiments are comprehensive and convincing.", "weaknesses": "1. I think some parts of the model and implementation have not been clarified very well:\n* I think it hasn’t been clarified in the main text whether the main results (starting from 3.2 to 3.5) show within-subject or cross-subject performance, unless I missed something. Adding some notices in figure\\table captions or some summarizing sentences at the beginning of each section might be helpful.\n* What model is inside the frozen “VAE image encoder” for low-level image generation in Figure 2? Also, what diffusion model is this study conditioning on? Is it built and retained from a pre-trained model (like stable diffusion), or was it trained from scratch by the authors?\n* I think the authors didn’t introduce the “image2text” component (BLIP2), unless I missed it.\n2. There are two recent EEG-to-image reconstruction works that this study has not discussed or compared. While it is understandable that the authors did not compare them, as they are still preprints and use different datasets, it might be beneficial to discuss them in the related work section given the limited literature in this field:\n* Bai, Yunpeng, et al. \"Dreamdiffusion: Generating high-quality images from brain eeg signals.\" arXiv preprint arXiv:2306.16934 (2023).\n* Lan, Yu-Ting, et al. \"Seeing through the brain: image reconstruction of visual perception from human brain signals.\" arXiv preprint arXiv:2308.02510 (2023).\n\nFor other minor comments please see Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and comments. Below please find our point-to-point responses to your comments.\n\n**Q1. “I think it hasn’t been clarified in the main text whether....”**\n\n**Regarding 3.2 to 3.5:** We present the bar graphs of performance in the **Section 3.2 EEG Decoding Performance**, which are the absolute results of the within-subject. The detailed tables of retrieval performance are in **Appendix H.1**. In the **Section 3.3 Image Generation Performance**, we used the EEG data of subject 8 as a representative for image reconstruction and calculated mertics. \n\nThere are statistical images reconstructed by a single subject in **Appendix G**. The experiments related to ‘Section 3.4 Temporal Analysis’ and ‘Section 3.5 Spatial Analysis’ were all conducted on the data of subject 8. We will follow your suggestion and add clear sentences in the titles of the figures and tables for explanation.\n\n**Q2. What model is inside the frozen “VAE image encoder” in Figure 2?** \n\nThe details of this part are similar to [1]. We now make complementary explanations: The VAE model from the pre-trained StableDiffusion-v2.1-base is used for low-level image reconstruction, as shown in **Figure 2**. We use the latent obtained by VAE to align the latent of EEG and then reconstruct the low-level image from EEG. First, the preprosessed EEG passes through an MLP and upsampling CNN to obtain a 4x32x32 latent, and then passes through MSE loss, contrastive loss, reconstruction loss, and aligns the latent from the original image 3x256x256 after VAE encoding. We pass EEG latent through the VAE decoder to get a blurred 3x256x256 image, and then input the image into SDXL-turbo as low level guidance. \n\n**Q3. The authors didn’t introduce the “image2text” component (BLIP2).**\n\nThank you for pointing out a typo, and we didn't make it clear here. The BLIP2 model [7] is indeed used in this work, but it is only used to extract text features from the original image for EEG classification. In the image reconstruction stage, to provide guidance from text features, we adopted the same approach as [5], using the GIT model [6] to obtain text descriptions from image embeddings in the reconstruction stage, which is then input into the SDXL-turbo decoder as a condition. \n\n**Q4. Why are these two papers not compared? [2][3]**\n\nIn [2] and [3], both were conducted on a controversial dataset [4]. The experimental paradigm of this dataset was once accused of having a block design error, resulting in the contribution of decoding not entirely coming from the category itself. The experimental paradigm of the THINGS-EEG does not have this problem, because the training set of this dataset has 1654 categories and the test set has 200 categories, and the categories of its training set and test set are completely different.\n\n**Q5. Regarding the ‘PixCorr’ metric:** \n\nDue to the limited length of the table and the fact that the low-level metric already has SSIM, this is omitted in the manuscript. The performance on different benchmarks is summarized as follows: \n| Metric \\ Method| Benchetrit Y et al. [9] (fMRI) | Fatma Ozcelik et al. [10] (fMRI) | Paolo Scotti et al. [1] (fMRI) | Benchetrit Y et al. [9] (MEG) | Ours (EEG) | Ours (MEG) | \n|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| PixCorr   | 0.305 | 0.254 | 0.130 | 0.058 | **0.160** | **0.104** |  \n\t\n**Q6. For Figure 4, the plots are impressive, but ....**\n\nThank you for your suggestion, for **Figure 4**, our original intention was to make the radar chart as simple and intuitive as possible. Detailed performance tables for decoding and reconstruction are given in **Appendix H.1 Table 7** and in the rebuttal to the **reviewer (asin)** for more detailed results. \n\n**Q7. The functioning of the shared token and subject tokens.**\n\nSimilar to [8], we leverage joint subject training to adapt to new subjects. Once a model is trained, it can be used for both reasoning about known subjects (subject-specific tokens) and reasoning about unknown subjects (shared tokens).\n\n**Reference**\n\n[1]Scotti P, Banerjee A, Goode J, et al. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2]Bai, Yunpeng, et al. \"Dreamdiffusion: Generating high-quality images from brain eeg signals.\" arXiv preprint arXiv:2306.16934  (2023).\n\n[3]Lan, Yu-Ting, et al. \"Seeing through the brain: image reconstruction of visual perception from human brain signals.\" arXiv preprint arXiv:2308.02510 (2023).\n\n[4]Spampinato, C.; Palazzo, S.; Kavasidis, I.; Giordano, D.; Souly, N.; and Shah, M. 2017. Deep learning human mind for automated visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6809–6817.\n\n[5]Ferrante M, Boccato T, Ozcelik F, et al. Multimodal decoding of human brain activity into images and text[C]//UniReps: the First Workshop on Unifying Representations in Neural Models. 2023.\n\n[6]Wang J, Yang Z, Hu X, et al. GIT: A Generative Image-to-text Transformer for Vision and Language[J]. Transactions on Machine Learning Research.\n\n[7]Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//International conference on machine learning. PMLR, 2023: 19730-19742.\n\n[8]Zhou Q, Du C, Wang S, et al. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding[C]//The Twelfth International Conference on Learning Representations.\n\n[9]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.\n\n[10]Fatma Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent 342 diffusion. Scientific Reports, 13(1):15666, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wp4NXZyoEY", "reviewer": "Reviewer_4xL4", "summary": "The paper presents a end-to-end EEG-based visual reconstruction zero-shot framework, featuring the Adaptive Thinking Mapper (ATM) and a two-stage EEG-to-image generation strategy. This method achievies state-of-the-art performance in classification, retrieval, and reconstruction tasks, and significantly advancing the field of brain-computer interfaces.", "questions": "I have no quentions, please see the weaknesses.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1.Comprehensive EEG experiments, encompassing retrieval, classification, and visual stimulus reconstruction. \n2.Cross-subject considerations.", "weaknesses": "1.The manuscript has several significant deficiencies. First, its motivation is based on the signal differences between EEG and fMRI, concluding that EEG's performance limitations are due to constraints in decoding and reconstruction frameworks. However, the proposed EEG encoder merely adds a Channel Attention layer compared to NICE [1], with no detailed explanation provided. Additionally, the loss function in Section 2.4 is taken directly from [2], demonstrating a lack of originality. Furthermore, the visual reconstruction framework shows no substantial difference from existing fMRI methods [3,4], as it also utilizes pre-trained stable diffusion models and their variants, with minor differences, such as the incorporation of Sdedit [5], being tricks to enhance generation quality. Thus, the manuscript fails to substantiate its claims and contributions convincingly.\n2.The processing of visual information in the brain involves multiple stages and typically takes around 100-150 milliseconds to reach higher visual areas where complex processing occurs. EEG signals at 50 milliseconds are likely still within the retina and optic nerve stages. Thus, the generation of images from 50ms EEG signals, as shown in Figure 7, contradicts established neuroscience principles. This suggests that the visual stimulus reconstruction framework heavily relies on the image generation model, which may have limited significance for the field of neuroscience.\n\n[1] Song, Yonghao, et al. \"Decoding Natural Images from EEG for Object Recognition.\" arXiv preprint arXiv:2308.13234 (2023).\n[2] Benchetrit, Yohann, Hubert Banville, and Jean-Rémi King. \"Brain decoding: toward real-time reconstruction of visual perception.\" arXiv preprint arXiv:2310.19812 (2023).\n[3] Chen, Zijiao, et al. \"Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[4] Lu, Yizhuo, et al. \"Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion.\" Proceedings of the 31st ACM International Conference on Multimedia. 2023.\n[5] Meng, Chenlin, et al. \"Sdedit: Guided image synthesis and editing with stochastic differential equations.\" arXiv preprint arXiv:2108.01073 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and comments. Below please find our point-to-point responses to your comments.\n\n**Q1. “First, its motivation is based on the signal differences between EEG and fMRI, concluding that EEG's performance limitations are due to constraints in decoding and reconstruction frameworks......”**\n\nPlease kindly see the relevant answer in **Global rebuttal: Q1 and Q2**. This paper has two main motivations: \n\nFirst, existing fMRI-based visual decoding and image reconstruction methods [1][2] are very advanced, but they are limited by the temporal resolution, portability and cost of fMRI, so these technologies cannot be applied in practice. \n\nSecond, the idea of the NICE [3] and B.D. [2]’s method are simple but very general. Previous work on visual decoding using fMRI often used ridge regression as the encoder. However, using EEG is different from fMRI, as it is full of noise and the original signal-to-noise ratio is low. It requires a sufficiently strong prior and a more complex model structure to extract the feature maps. We carefully studied the shortcomings of different approaches in EEG decoding [5][6]. Inspired by NICE [3], and based on the theory of treating channels as patches in [4], we introduced joint subject training, which far aheads of NICE, and gained the ability to adapt to new subjects like [7][8]. \n\nThe contribution of this paper is not to improve decoding performance too much only by stacking tricks. The main purpose is to provide evidence for the neuroscience and machine learning communities: using our framework, even compared with fMRI, EEG can provide competitive performance.\n\n**Q2. “The generation of images from 50ms EEG signals, as shown in Figure 7, contradicts established neuroscience principles. This suggests that the visual stimulus reconstruction framework heavily relies on the image generation model, which may have limited significance for the field of neuroscience.”**\n\nPlease kindly see the relevant answer in **Global rebuttal: Q3**. We may not have explained it clearly in the article, but there are similar expressions in [2]. According to our results in **Appendix H.1 Accuracy for time windows**, **Figure 29** shows that for all embeddings, a clear peak can be observed for windows ending around 200-250 ms after image onset. Comparing **Figure 28** and **Figure 29**, we can see that, unlike [2], our time window results on THINGS-EEG dataset do not have a second peak, which may be mainly affected by the experimental paradigm. \n\nPlease pay attention to the caption in Figure 7 and the reconstruction results from 0 to 50 ms and 0 to 250 ms. It can be seen that the image reconstructed at 50 ms is completely messy and has no semantics; the semantics of the reconstructed image after 250 ms is basically correct and gradually stabilizes. Similar results are also shown in **Figure 4 A of [2]**: their method can also reconstruct high-quality images at 100ms. This just shows that our reconstruction results are in line with the neuroscience prior. \n\nIn addition, we provide three different tasks including image classification, image retrieval and image reconstruction. The excellent performance in image classification and image retrieval tasks shows that the decoding of visual stimuli is reasonable. Since the submission is to NeurIPS, the top machine learning conference, it is necessary for us to use the latest technology to ensure the quality of image reconstruction on this basis, which reflects the latest technological progress.\n\n**Reference**\n\n[1]Scotti P S, Tripathy M, Torrico C, et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data[C]//Forty-first International Conference on Machine Learning.\n\n[2]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.\n\n[3]Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.\n\n[4]Liu Y, Hu T, Zhang H, et al. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting[C]//The Twelfth International Conference on Learning Representations.\n\n[5]Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces[J]. Journal of neural engineering, 2018, 15(5): 056013.\n\n[6]Song Y, Zheng Q, Liu B, et al. EEG conformer: Convolutional transformer for EEG decoding and visualization[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 31: 710-719.\n\n[7]Xia W, de Charette R, Öztireli C, et al. UMBRAE: Unified Multimodal Decoding of Brain Signals[J]. arXiv e-prints, 2024: arXiv: 2404.07202.\n\n[8]Zhou Q, Du C, Wang S, et al. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding[C]//The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4eozSV5kHO", "reviewer": "Reviewer_gPm4", "summary": "This paper presents an end-to-end EEG-based visual decoding framework that includes two stages: a brain encoder for EEG feature extraction and a \"generator\" for producing reconstructed images. The experiments demonstrate effective results in retrieval, classification, and reconstruction tasks across two datasets, suggesting potential applications for real-time brain-computer interfaces.", "questions": "1. The paper mentions selecting the highest test accuracy during the training process as the statistical result. It would be more rigorous to test the model only once after it is fully trained.\n\n2. In the framework illustrated in Figure 2, which input to the diffusion process is most critical for image generation—the reconstructed output after VAE, the image embedding transferred from EEG, or the generated caption?\n\n3. How does the model ensure that it captures low-level features after the VAE encoder?\n\n4. What is the impact of the pre-trained language and generative models on the final performance?\n\n5. Could you provide more details about the cross-subject settings described in P3L91? Specifically, what roles do the subject token and shared token in Figure 3 play, and do they enhance cross-subject capability?\n\n6. Figure 11a seems to show no significant correlation between text and EEG features. Could you comment on this observation?", "limitations": "Yes. The authors adequately addressed the limitations and future works.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The research demonstrates the feasibility of using non-invasive EEG signals for image decoding as using fMRI techniques. The analysis provides a valuable reference for EEG feature extraction.\n\nThe paper highlights three primary contributions: (1) an EEG-image decoding framework, (2) a novel EEG encoder, and (3) a two-stage EEG-to-image generation strategy that leverages both low- and high-level visual information.", "weaknesses": "The paper's novelty in the context of brain-image decoding methodologies is not distinctly clear.\n\n1. The pipeline, which includes a contrastive learning-driven encoder and a diffusion-based decoder, is already well-established in the field.\n\n2. The ATM encoder, which incorporates channel-wise attention, spatial convolution, and temporal convolution, does not significantly differ from previous studies, such as mentioned in the introduction, Benchetrit et al. and Song et al.\n\n3. The method for disentangling low- and high-level features within the framework is unclear, particularly how the VAE is supposed to provide low-level image features.\n\nThe evaluation section contains many conclusions that conflict with scientific validity, which may cause serious misleading.\n\n1. How can we get clear reconstruction results with only 50 ms signals after the onset as in Figure 7c? The visual stimuli haven’t reached V1 in such a short time. \n\n2. We know the temporal cortex is strongly related to object recognition. But in Figure 8b, the temporal channels contribute very small. Based on that, should we think the retrieval task was based on some low-level information? \n\n3. The visual response is very quick after the onset, finished before 200 ms. It cannot be thought of as visual responses at 500 ms after the onset on Page 7 Line 182, and contained within 200-400 ms on Page 9 Line 246. \n\n4. On page 9 Line 251, It can’t be concluded that EEG is better than MEG in visual tasks where the paradigms used for these two datasets were different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and comments. Below please find our point-to-point responses to your comments.\n\n**Q1. “The pipeline is already well-established in the field.”**\n\nPlease kindly see the relevant answer in Global rebuttal: Q1 and Q2. \n\n**Q2. How the VAE is supposed to provide low-level image features.**\n\nThanks to the reviewer for pointing out this part, and we may not have expressed it clearly in the manuscript. Due to limited space, please refer to the reply to **reviewer (zVcP)**. \n\n**Q3. “How can we get clear reconstruction results when only 50 ms? ”**\n\nPlease kindly see the relevant answer in Global rebuttal: Q3. \n\n**Q4. Should we think the retrieval was based on some low-level information?**\n\nWe consider the high-level visual features should play a major role in the visual retrieval task. \n\nOn the one hand, there is much noise in the scalp EEG, and the dynamic activities of the temporal cotex may not be accurately captured by the EEG, while the occipital cortex continues to have a strong response.\n\nFurthermore, when we perform the retrieval task, we use the high-level visual features after the alignment of CLIP and EEG. So the semantic-level visual features should play a major role in the visual retrieval task. \n\n**Q5. The visual response is very quick after the onset, finished before 200 ms....**\n\nPlease also kindly see the relevant answer in Global rebuttal: Q3. \n\n**Q6. On page 9 Line 251, It can’t be concluded that...**\n\nIn THINGS-MEG, each image in the THINGS-MEG was displayed for 500 ms. There was a fixed time for each image of 1000 ± 200 ms. \n\nHowever, in THINGS-EEG, each of image was presented for 100 ms. For preprosessing and training, we segmented the EEG data from 0 to 1000 ms after the stimulus onset into trials. We speculate that our excellent performance in the EEG visual decoding task is due to the larger data size of the THINGS-EEG (the more trials, the better the decoding performance). That is, in the visual decoding task, scaling low still works. \n\nDue to the different experimental paradigms of THINGS-EEG and THINGS-MEG and the characteristics of the two types of data, we only report the phenomena we observed. We hope that these results can trigger further discussion and promote the development of the community. \n\n\n**Q7. It would be more rigorous to test the model only once after it is fully trained.**\n\nAs described in the manuscript, what we used in Figure 5c and Figure 5d is the result of the highest test accuracy of all methods in the process of training. In Table 7 in the Appendix G, we present the results after 10 tests and take the average results. Regardless of the test strategy, our method performs far ahead of other methods.\n\n**Q8 .Which input to the diffusion process is most critical for image generation ?**\n\nThis is also an interesting question. In fact, after our tests, the EEG embedding aligned with CLIP is the most necessary. The output after VAE reconstruction also comes from EEG and the reconstructive image is very blurry, so it contributes very little. On the other hand, the text obtained from the generated caption is very similar to the image embedding, so it also contributes very little. \n\n**Q9. How does the model ensure that it captures low-level features?**\n\nPlease kindly find the related explanations in reviewer (zVcP) in Q2. The low-level and high-level here refer to the representation that obtained from the pixel-level or the semantic-level after CLIP alignment with [1][2]. So it is not the low-level and high-level areas of visual cotex in neuroscience. \n\n**Q10. What is the impact of the pre-trained models on the final performance?**\n\nDuring the experiment, we found that the pre-trained language model seemed to have little impact on the final results, whether in the representation learning stage or the generative model stage. This may be because CLIP itself is a pre-trained model after the image and text are aligned. So the contribution of the text is not so important. \n\nOn the other hand, the generative model only reconstructs the semantically correct category. Therefore, it does not improve the decoding accuracy, but only guarantees the quality of the reconstructed image.\n\n**Q11. Provide more details about the cross-subject settings, and do they enhance cross-subject capability?**\n\nPlease find the explanations in reviewer (zVcP) in Q7. The objective is to retain both subject-independent joint EEG representations and subject-specific independent EEG representations during training. \n\n**Q12. Could you comment on this observation in Figure 11a ?**\n\nAs explained in Q10, this may be due to the fact that CLIP is a pre-trained model with aligned image and text features. We consider that these works [3][4] that use CLIP to align neural signals will also have this property.\n\n**Reference**\n\n[1]Scotti P, Banerjee A, Goode J, et al. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2]Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14453-14463\n\n[3]Du C, Fu K, Li J, et al. Decoding visual neural representations by multimodal learning of brain-visual-linguistic features[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10760-10777.\n\n[4]Zhou Q, Du C, Wang S, et al. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding[C]//The Twelfth International Conference on Learning Representations.\n\n[5]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.\n\n[6]Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Area Chairs and Reviewers,

We express our profound gratitude for the comprehensive feedback and comments on our manuscript. This paper receives relatively serious differentiation of Weakly Accept, Weakly Accept, Reject, and Reject during the review period. We are excited about the consensus among the reviewers regarding the innovative (asin) and convincing (zVcP) impact of our work. However, we are saddened that two reviewers (4xL4 and gPm4) have some concerns about the novelty and ‘conflict ’ of our manuscript to neuroscience priors. So we hope to do our best to clarify what we did not explain clearly in the manuscript to allay the concerns of the reviewers (4xL4 and gPm4).
In the following , we provide a general response to several reviewers' questions and concerns.

**Q1. What is the motivation for this paper?**

With the increase of public data sets and the rapid development of generative models, visual decoding and reconstruction methods [1][2] based on fMRI datasets [3] have become very advanced, but few methods based on EEG datasets have been formally published in top conferences or journals [5][6]. fMRI has a natural advantage in spatial resolution, but for well-known reasons, fMRI-based work is almost impossible to apply in practice, and this have hindered the development of the brain-computer interface (BCI). This motivates us to propose a zero-shot visual decoding and reconstruction framework that can be proven effective on EEG datasets. Our manuscript provides more empirical guidance on practical BCI applications. We hope that the BCI and neuroscience communities pay more attention to the implementation of similar technologies and real data rather than overfitting only on existing fMRI datasets.

**Q2. What is the innovation of this paper? (How is it different from previous work?)**

**Our work is by no means a pile-up or improvement of existing work.** We propose a novel and feasible EEG-based zero-shot image decoding and reconstruction framework. Although it utilizes existing machine learning techniques , we demonstrate for the first time that EEG-based zero-shot visual decoding and reconstruction can be competitive with fMRI.
Previous published work either focused on the EEG image decoding [7], or reconstructed images of known categories on small-scale and controversial datasets [4]; or focused on the fMRI based image decoding and reconstruction, to achieve advanced performance on the fMRI datasets [3].
We believe that zero-shot visual reconstruction of EEG is essential, which is an important prerequisite for decoding imaginary images. The work in Song et al. [9] is that they only use Graph-attention or Self-attention modules, without any learnable token embedding strategies and Feed Forward Layer, and even position encoding schemes. In addition, our method provides a plug-and-play spatial-temporal convolution module, and we prove that even if EEGNetV4 is used as a convolution module, the performance is still robust. Our reconstruction scheme is also different from Bencherit Y el al. [8], which only used the method in MindEye [1] in their paper for MEG data, and the performance of our two-stage image reconstruction framework is far behind.

**Q3. The conclusions of this article seem to violate neuroscientific priors?**

The results of this paper are exactly mutually confirmed to the conclusions of previous papers and are consistent with neuroscience priors [7][8]. According to our results in **Appendix H.1 Accuracy for time windows**, **Figure 29** shows that for all embeddings, a clear peak can be observed for windows ending around 200-250 ms after image onset. Comparing **Figure 28** and **Figure 29**, we can see that, unlike [8], our time window results on THINGS-EEG dataset do not have a second peak, which may be mainly affected by the experimental paradigm. We can see that in the first 50ms-200ms, the image reconstructed at 50 ms is completely messy and has no semantics; the semantics of the reconstructed image after 250 ms is basically correct and gradually stabilizes, and after 500ms, due to the lack of additional visual response, the content of the image reconstruction is more stable. Similar results are also shown in **Figure 4 A of [8]**: their method can also reconstruct high-quality images at 100ms. **This just shows that our reconstruction results are in line with the neuroscience prior.**
However, this does not mean that the EEG data after the absence of visual response (200ms) loses its contribution to decoding, because the processing of high-level visual features (corresponding to the visual features of CLIP) may be involved over time. 


**Reference**

[1]Scotti P, Banerjee A, Goode J, et al. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2]Xia W, de Charette R, Öztireli C, et al. UMBRAE: Unified Multimodal Decoding of Brain Signals[J]. arXiv preprint arXiv:2404.07202, 2024.

[3]Allen, E. J. et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci. 25, 116–126 (2022).

[4]Kavasidis I, Palazzo S, Spampinato C, et al. Brain2image: Converting brain signals into images[C]//Proceedings of the 25th ACM international conference on Multimedia. 2017: 1809-1817.

[5]Bai, Yunpeng, et al. "Dreamdiffusion: Generating high-quality images from brain eeg signals." arXiv preprint arXiv:2306.16934 (2023).

[6]Lan, Yu-Ting, et al. "Seeing through the brain: image reconstruction of visual perception from human brain signals." arXiv preprint arXiv:2308.02510 (2023).

[7]Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

[8]Benchetrit Y, Banville H, King J R. Brain decoding: toward real-time reconstruction of visual perception[C]//The Twelfth International Conference on Learning Representations.

Meta-review: This paper demonstrates visual decoding and reconstruction of viewed
images from recorded EEG using the THINGS EEG(2) dataset.  They
demonstrate their algorithm on image retrieval, category
classification, and image reconstruction tasks.  Results are
impressive and exciting.  The results (timing, most relevant
electrodes) are consistent with the Neuroscience literature, but the
ability to do this with EEG is surprising.

The biggest issue with this paper, as raised by one reviewer is
measuring performance as maximal on test data.  While the author responded
that it should not matter as all methods are tested similarly, this is
not true as methods with higher variance are more benefitted by this
approach.  It is well known that parameters must be fit on data
separate from the test data.  If this paper is accepted, this will
have to be changed in the camera ready.	 From the plots, it appears that
this will only change the numbers slightly, and	so I recommend acceptance.

Minor:
It appears there might be a typo in the	supplemental text.  On line
432, it is mentioned that "we averaged across the 4 EEG trials from
the same image in the test set" but the test set has 80 repeats and
later on line 513, they say they used an "averaging technique on 80 repeated instances
within the test set".

Finally, there is contemporaneous work for reconstruction from EEG
here https://arxiv.org/abs/2404.01250
Decision: Accept (poster)