Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion

Authors: dongyang  li
chen  wei
shiying  li
jiachen  zou
quanying  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 09:06:37

Abstract: How to decode human vision through neural signals has attracted a long-standing interest in neuroscience and machine learning. Modern contrastive learning and generative models improved the performance of visual decoding and reconstruction based on functional Magnetic Resonance Imaging (fMRI). However, the high cost and low temporal resolution of fMRI limit their applications in brain-computer interfaces (BCIs), prompting a high need for visual decoding based on electroencephalography (EEG). In this study, we present an end-to-end EEG-based visual reconstruction zero-shot framework, consisting of a tailored brain encoder, called the Adaptive Thinking Mapper (ATM), which projects neural signals from different sources into the shared subspace as the clip embedding, and a two-stage multi-pipe EEG-to-image generation strategy. In stage one, EEG is embedded to align the high-level clip embedding, and then the prior diffusion model refines EEG embedding into image priors. A blurry image also decoded from EEG for maintaining the low-level feature. In stage two, we input both the high-level clip embedding, the blurry image and  caption from EEG latent to a pre-trained diffusion model. Furthermore, we analyzed the impacts of different time windows and brain regions on decoding and reconstruction. The versatility of our framework is demonstrated in the magnetoencephalogram (MEG) data modality. The experimental results indicate that our EEG-based visual zero-shot framework achieves SOTA performance in classification, retrieval and reconstruction, highlighting the portability, low cost, and high temporal resolution of EEG, enabling a wide range of BCI applications. Our code is available at https://github.com/ncclab-sustech/EEG_Image_decode.

Main paper: 
 introduction: 
 A key technical challenge in BCIs is to decode/reconstruct the visual world seen by humans through non-invasive brain recordings, such as fMRI, MEG or EEG. These highly dynamic brain activities reflect human perception of the visual world, which is influenced by properties of the external visual stimulus, our internal states, emotions and even personal experiences. Thus, visual decoding and reconstruction based on neural signals can uncover how the human brain processes and interprets natural visual stimulus, as well as promote non-invasive BCI applications.Contrastive learning and generative models have greatly advanced fMRI-based visual decoding in both decoding tasks (e.g., image classification and retrieval) and generative tasks (e.g., image reconstruction). By combining pre-trained visual models, existing fMRI decoding models can learn highly-refined feature embeddings in limited data [1, 2] . Using these embedded fMRI features, generative models such as diffusion models can reconstruct the image one is seeing [2, 3] . However, despite many advances in fMRI-based visual decoding, fMRI equipment is unportable, expensive, and difficult to operate, largely limiting its application in BCIs. Alternatively, EEG is portable, cheap, and universal, facilitating a wide range of BCI applications. EEG has higher temporal resolution and can effectively capture rapid changes in brain activity when processing complex, dynamic visual stimulus.EEG has long been considered incomparable to fMRI in natural image decoding/reconstruction tasks, as EEG suffers from low signal-to-noise ratio, low spatial resolution, and large inter-subject variability. Recent advances in multimodal alignment have made MEG/EEG visual decoding possible, although the performance is still inferior to fMRI [4, 5, 6] . Yohann Benchetrit et al. used the CLIP model to extract the latent representation of the image and trained the MEG encoder to align it with the image representation extracted by CLIP. It achieved excellent retrieval and reconstruction performance on MEG and fMRI datasets, demonstrating the potential for real-time visual decoding and reconstruction using EEG/MEG signals. Recently, Song et al. [5] used an EEG encoder based on ShallowNet [7] and performed representation alignment through contrastive learning, achieving excellent decoding performance on the THING-EEG dataset [8] . These two studies provide preliminary evidence of the potential of EEG/MEG-based visual decoding. However, there is a significant gap in their performance compared to the fMRI-level performance. This gap is largely caused by the fact that the framework of EEG visual decoding and reconstruction have not yet been thoroughly explored.To fill this gap, we have developed a visual decoding framework based on EEG/MEG, including a novel EEG encoder and a two-stage image generation strategy. Our work has three main contributions:1. We present brain decoding framework, which is the first work allows zero-shot image classification, retrieval, and reconstruction via EEG data. Experimental results demonstrate that our framework is applicable to various common EEG encoder architectures. 2. By extensively studying the existing EEG encoder modules, we construct a tailored EEG encoder ATM, which achieves state-of-the-art performance in three downstream visual decoding tasks. 3. We report a two-stage EEG-to-image generation strategy, which separately extracts highlevel and low-level visual features from EEG and refining these features with an additional lightweight prior diffusion model, enabling reliable reconstruction of images using less than 500ms EEG. 
method: 
 To learn high-quality latent representations of EEG data, it is crucial to consider the spatial position of EEG channels and the Temporal-Spatial properties of EEG signals. Let T represent the length of the time window of the data, C the number of EEG channels, and N the total number of data samples. Our objective is to derive EEG embeddings Z E = f (E) ∈ R N ×F from the brain activity data E ∈ R N ×C×T , where f is the EEG encoder and F is the projection dimension of the embeddings. Concurrently, we use the CLIP model to extract image embeddings Z I ∈ R N ×F from images I. Our goal is to effectively align the EEG representation with the image representation, as illustrated in Fig. 2 . In the training phase, the EEG encoder is trained with EEG and image pairs using a contrastive learning framework. In the inference phase, the EEG embeddings from the trained EEG projector can be used for a variety of zero-shot tasks, including EEG-based image classification, retrieval, and image reconstruction.Inspired by advanced time series models [9, 10] , we develop an EEG encoder called ATM, for aligning the original EEG signals to its feature representation space (Fig. 3 ). ATM is based on the channel-wise Transformer encoder, Temporal-Spatial convolution and multilayer perceptron (MLP) architecture.In contrast to other conventional practices, the original EEG does not need to be segmented, and each sequence acts as a patch. After sinusoidal position embedding, these patches are processed through a channel-attention module to integrate the information of different series. Subsequently, through the Temporal-Spatial aggregation, we project the output with a MLP to get rational shape representations. The Temporal-Spatial convolution module is an effective way to represent EEG data with a small number of parameters [5] , prevent overfitting in training. The difference is our components is plug-and-play and can be flexibly replaced with different types of Temporal-Spatial convolution components as needed to adapt to various EEG/MEG datasets. Finally, MLP projectior consists of M simple residual components and fully connected layers, with LayerNorm applied in the output to ensure the stability of training. In addition to entering the original series, we provide an identification input for a known subject and can specifically use this token for downstream tasks. For unknown subjects, we use shared tokens or average all tokens equally directly into the MLP projector.Many previous studies have explored various training strategies to train deep neural networks for image embedding, such as VGG-19 and ResNet trained with supervised learning, CLIP, DINO trained with contrastive learning, and VAEs with self-supervised learning [11, 12, 5, 6] . They have reported that DINO and CLIP models pre-trained using the Vision Transformer (ViT) architecture perform better in a range of downstream tasks, including image decoding and reconstruction, compared to models trained using supervised learning methods (such as VGG, ResNet) and self-supervised VAE frameworks. Thus, in this study, we use CLIP for image embedding, denoted asZ I ∈ R N ×1024 , instead of Z I ∈ R N ×257×768, with the EEG embeddings. Before formal training, all images undergo the standard preprocessing procedure [13] . In this study, we present a two-stage pipeline for generating images that serve as visual stimulus for EEG recordings, as shown in the bottom right of Fig. 2 . In the left of Fig. 3 we have obtained the EEG embeddings z E for each image by the EEG encoder ATM. Now our goal is to use these EEG embeddings to generate the corresponding images. The joint distribution of images, EEG embeddings, and image embeddings can be expressed as p(I, z E , z I ) = p(z I |z E )p(I|z I ), corresponding to the prior diffusion and CLIP-guided generation, respectively. In Stage I, we first focus on the prior diffusion stage. Inspired by DALL-E 2 [14] and Mind's Eyes [2] , we train a diffusion model conditioned on the EEG embeddings ẐE to learn the distribution of CLIP embeddings p(z I |z E ). In this stage, we construct a lightweight U-Net: ϵ prior (z t I , t, z E ), where z t I represents the noisy CLIP embedding at diffusion time step t. We train the prior diffusion model using EEG and CLIP embeddings. Through this diffusion model, we can generate corresponding CLIP embeddings z I from EEG embeddings as a prior for stage II. In Stage II, we employ the pre-trained SDXL [15] and IP-Adapter [16] models to model the generator p(I|z I ), thereby sampling image I according to z I . In addition, we introduce the low-level features here using img2img [17] . Further details are provided in Appendix C.Following the methodology outlined by Benchetrit et al. [12] , we adopt a dual approach to loss functions, serving distinct objectives. For the classification and retrieval tasks, we only utilize the CLIP loss, which is inspired by the contrastive learning approach described in Radford et al. [13] . This loss function aids in aligning the EEG data E with corresponding image data I, thereby facilitating the identification of EEG-image pairs and maximizing the boundaries of EEG representations. For the generation tasks, besides the CLIP loss, we add a Mean Squared Error (MSE) loss to facilitate consistency learning in regression. Thus the overall loss function for our model is a combination of these two distinct loss types, expressed as:Loss = λ • L CLIP + (1 -λ) • L M SEHere, λ is a hyperparameter that balances the contribution of each loss type. 3 Experiments 
training and computational considerations: 
 We conducted our experiments on the THINGS-EEG dataset's training set [8, 6] . To verify the versatility of ATM for embedding electrophysiological data, we tested it on MEG data modality using the THINGS-MEG dataset [18] . All experiments can be completed in a single NVIDIA RTX 4090 GPU. We used the Adam optimizer [19] to train the across-subject model on a set of approximately 496,200 samples, and the within-subject model on a set of about 66,160 samples, with an initial learning rate of 3 × 10 -4 and batch sizes of 16 and 1024. Our initial temperature parameter was set to 0.07. We splited the last batch of the original training set as the validation set and selected the best model based on the minimum validation loss over 40 epochs. For fairness, all models' hyperparameters were kept consistent. In our study, we compared the performance of different encoders on the within-subject test set and cross-subject (leave-one-subject-out) test set (see Appendix H).Our method obtains the EEG embedding for the classification task. We output the category of EEG with the highest cosine similarity with text embeddings(Fig. 5a ). Fig. 5c presents the average accuracy across different methods in the subjects, and shows that our method outperforms others. More details of the EEG-based image classification are in Appendix B.In Fig. 5 , we test the effectiveness of EEG embeddings in the image retrieval task. We calculate the cosine similarity between the EEG embeddings and the CLIP embeddings instead of text embeddings in the image dataset (with 200 images). Fig. 5d shows the average results for all subjects. We take the highest test accuracy in the evaluation process as the statistical result. Fig. 5b shows the Top-5 retrieved images corresponding to the real visual stimulus seen by subjects. Compared with the previous models, the Top-1 accuracy of our model is significantly improved, and the Top-5 images all maintain a high degree of similarity with the original images. See the Tab. 8 in Appendix for more detailed averages of test accuracy in subjects.Ablation Study on ATM We systematically deconstructed and analyzed each layer of our EEG projector. We conducted an ablation study for each component in ATM (i.e., the MLP projector, the Temporal-Spatial convolution module and the channel-wise attention block). We specified two different convolution architectures, ShallowNet (ATM-S) and EEGNetV4 (ATM-E), as our convolution backbone. Appendix B.3 showed the results obtained under different experimental configurations. Fig. 6a shows the process of generating images under the guidance of EEG embedding and evaluating the quality of the generated images. To evaluate the generation performance, we conducted an image retrieval task. Specifically, we extract the CLIP embedding of the generated images and compare the similarity between the CLIP embeddings of all images to retrieve the generated image.Fig. 6b shows the similarity of distribution. Fig. 6c shows the generated samples. The generated images have high semantic similarity with the seen images and have good diversity in low-level visual features, which can be manipulated by the guidance scale hyperparameter (Fig. 6d ). We also report the decoding and reconstruction performance for EEG, MEG, and fMRI across various metrics from different datasets in the Tab. 1. To investigate the effects of different EEG time window on visual decoding, we calculated the average top-1 classification accuracy for sliding and growing time windows: [0, t], including the entire period from the onset of visual stimulus to time point t, and [t-100, t], only including the data 100ms before time point t. We compared the accuracy with a randomly selected baseline (0.5% chance level) to test predictive performance (Fig. 7 ). Our results show that within 500ms after visual stimulus, the accuracy reaches an upper limit of about 30%, after which the accuracy no longer improves (Fig. 7a ). The MEG decoding shows a similar profile as the time window expands (Fig. 7b ). We exhibit the generated images under different EEG time windows, [0, t] in Fig. 7c . The similarity is low when the time window is less than 150ms, and this similarity gradually increase as the time window expands. After 500 milliseconds, EEG-guided image generation can reliably reveal the semantics of the images seen. Interestingly, we find differences in the optimal reconstruction time windows for different categories of images, for example, jelly beans (200ms) are faster than aircraft carrier (500ms), implying that the human brain may process different visual objects at different speeds. This finding highlights the advantage of EEG's high temporal resolution in studying fast visual processing compared with the lower temporal resolution of fMRI.To investigate the contribution of different brain regions to visual decoding, we divided the EEG electrodes from the THING-EEG data into five distinct brain regions (i.e., Frontal, Temporal, Center, Parietal, Occipital regions in Fig. 8a ), and then conducted ablation experiments on retrieval task (Fig. 8b ) and the reconstruction task (Fig. 8c ). The results showed that using information from all brain regions is optimal, for both retrieval and generation tasks. The occipital had the highest retrieval accuracy and reconstruction performance compared to other regions. Parietal and temporal regions contain some semantic information, whereas frontal and central regions contribute the least useful information to the visual decoding. 
related works: 
 Visual decoding using neural signals: Decoding visual information from our brain has been a long-standing pursuit in neuroscience and computer science [22, 23] . Some progress has been made in decoding steady-state visual stimulus. However, accurately and rapidly decoding semantic information in natural images remains a challenge [24] . fMRI has been widely used to estimate semantic and shape information in visual processing within the brain [25, 26] . However, the demand for high-speed and practical applications in brain-computer interfaces calls for alternative approaches. EEG, due to its high temporal resolution and portability, emerges as a promising option [27] . Yet, the overall performance across different subjects and biological plausibility remains unresolved [28] . Furthermore, previous approaches often relied on supervised learning methods with limited image categories, overlooking the intrinsic relationship between image stimulus and brain responses [1, 29, 30] .Neural decoding using EEG/MEG data: Previous studies have shown the efficacy of Temporal-Spatial modules in representing neural data [7, 31] . For example, lightweight convolutional neural networks such as EEGNet and ShallowNet [7] have achieved considerable performance in small EEG and MEG datasets. Using contrastive learning, it has been shown that merely using convolutional neural networks and projection layers can yield satisfactory results on neural datasets [32] . More recently, Benchetrit et al proposed a method towards real-time MEG-based reconstruction of visual perception [12] . Song et al. presented an EEG encoder using ShallowNet Temporal-Spatial convolution module with a large convolution kernel with a few parameters for EEG embedding, resulting in favorable performance on EEG-based visual decoding [5] .Limitations of previous studies: Previous EEG studies are primarily oriented toward understanding visual perception in the human brain rather than maximizing EEG decoding performance. Thus the visual decoding performance is far from optimal. Specifically, previous studies have trained linear models to (1) classify a small set of images from brain activity [33, 34] , (2) to predict brain activity from the latent representations of images [4] , or (3) to quantify the similarity analysis between these two patterns with representational similarity [4, 33, 8, 35] . While these studies also utilize image embeddings, their linear decoders are limited to classifying a small group of object categories or distinguishing image pairs. Moreover, several deep neural networks have been applied to maximize classification of speech [36] , cognitive load [37] , and images [38, 39, 40] in EEG recordings. [38] proposed a deep convolutional neural network for classifying natural images using EEG signals.Unfortunately, the experiment presented all images of the same category in a single block, probably misleading the decoder to rely on autocorrelated noise rather than the hidden informative patterns of brain activity [30] . Also, these EEG studies only classify a relatively small number of image categories. 
discussion and conclusion: 
 In this study, we proposed a novel and feasible EEG-based zero-shot image reconstruction framework.Although it utilizes existing machine learning techniques , we demonstrate for the first time that EEG-based zero-shot visual decoding and reconstruction can be competitive with MEG and fMRI.Technical Impact: Our technical contributions are mainly on the EEG encoder and the two-stage zero-shot visual reconstruction framework (Fig. 2 ). First, we developed the ATM, an EEG encoder which can efficiently represent EEG/MEG features for three tasks. Our comprehensive experiments of the EEG encoder (Fig. 3 ), compared to various architectures and training methods, achieves SOTA performance across various metrics and tasks (Figs. 4b, 5 ). Second, our two-stage EEG guidance image reconstruction framework achieves performance close to fMRI using only EEG data (Figs. 6, 14, Tab. 1, 6), and this method is compatible with MEG data (Figs. 4c, 7b ).Neuroscience Insights: Our results offer insights into the relationship between brain activity and visual perception. We analyzed EEG-based visual decoding within different time windows to examine when visual information is perceived in the brain (Fig. 7 ). Our results revealed that visual information in EEG data is predominantly contained within the 200-400ms range (Fig. 7a ), consistent with previous EEG studies [11, 6, 5] . Interestingly, the visual information in MEG data last up to 800ms, much longer than EEG (Fig. 7b ), in line with the results reported by a previous MEG study [12, 5] . We also found that EEG performs better than MEG in visual tasks (See Appendix D for Tab. 6), which is different from other fields, such as speech decoding [36] . In addition, through ablation experiments of spatial information, we found that visual information is mainly encoded in the occipital and parietal areas (Fig. 8 ).Interesting Phenomena and Future Directions: First, there are non-negligible performance differences between cross-subject and within-subject settings. This performance gap arises from inter-subject differences in EEG signals [41, 42] , likely attribute to heterogeneity in individual brain, differences in visual perception between individuals, and even shifts in noise distribution during EEG recording. So it calls for more efforts on EEG encoder, such as more flexible neural network architectures or better weight initialization of pre-trained models [43, 44] . Transfer learning and meta-learning are also future directions worth exploring [45, 46, 47] . Moreover, how to unify various electrode montages of different EEG datasets when pre-training large EEG models is a challenge. EEG source localization, which converts senor-level EEG signals into the standard brain source space [48, 49] , might be a potential solution.Retrieval accuracy Similar to the classification task, in the retrieval task, the objective is to retrieve the Top-K images most related to a given stimulus image via its corresponding EEG signal. This implies that by changing the text embeddings of image labels to image embeddings, we can transition the task from classification to image retrieval. Given that contrastive learning is known to be sensitive to batch size, we also compared the performance improvement of different methods under varying batch sizes (batch size=16, 1024) (Appendix H). 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: In the Section 3.1 of this article, we dedicate a section to describe in detail the environment configuration for running our experiments. We ensure that important results can be reproduced intact. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We fully adhere to the NeurIPS Code of Ethics throughout the entire process. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts • The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We open up the code and model weights of our method for reproducible results. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: We used publicly available datasets containing EEG and MEG from human subjects studies. We strictly adhere to the relevant requirements of neuroscience ethics. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.  
 
 Appendix: The generation task presents more challenges than the other tasks. For each image condition in the test set, we generate 10 different images from 10 subjects based on the corresponding EEG. Subsequently, image retrieval is performed for each generated image. The Top-1 and Top-5 accuracies are calculated. It helps in evaluating the semantic alignment between the generated images and their original counterparts.In the upstream EEG encoder part, we compared various methods. For the B.D. method [12] , we replicated the network structure as described in the original work, with the difference being in the shape of the input data due to the original study's focus on MEG. It is worth mentioning that we used the leave one for subject method in the testing process so we modify its subject-wise layer as an linear layer for modeling the time dimension. To ensure fairness, we did not use the same hyperparameters as in the original paper. Instead, we chose settings yielded excellent results upon reproduction. Across all methods, we used identical hyperparameters, apart from the network structures. These included batch size, optimizer, initial learning rate, and temperature parameters. To validate the efficacy of our EEG encoder, we experimented with a variety of empirical setups aimed at optimizing the model's efficiency. we leverage joint subject training to adapt to new subjects. Once a model is trained, it can be used for both reasoning about known subjects (subject-specific tokens) and reasoning about unknown subjects (shared tokens). In the context of channel-wise attention layer, we explored four distinct approaches for enhancing downstream retrieval capabilities: leveraging subject-specific tokens for retrieval, averaging all tokens for a more generalized retrieval, flattening the entire token set for direct retrieval, and preserving token dimensions to feed into the Temporal-Spatial convolution module for feature integration. Notably, the strategy of preserving dimensions and using the Temporal-Spatial convolution module emerged as the most effective.In our quest to optimize token embeddings, we experimented with a variety of approaches, including using 1 × 1 convolution and linear layers. Under our framework, using linear layers performs better than convolutions. Moving beyond, we delved into the efficacy of diverse Feed Forward Networks (FFNs) within the Transformer encoder layer. Our findings indicated that an FFN tailored to the temporal dimension emerged as the superior option. We also conducted a comparative analysis of various positional encoding techniques. Interestingly, for Temporal-Spatial convolution utilized in retrieval tasks, the significance of positional encoding diminished.In a further exploration, we examined the impact of deploying convolutions at various stages and even contemplated the complete removal of the convolution module. It was discovered that situating convolutions post the Transformer encoder layer yielded the most favorable outcomes. Conversely, a shift to a MLP or the removal of the convolution module led to a notable degradation in performance.Our assumption is though the convolution's inherent translational invariance is compromised when the context of the time dimension is disrupted, the efficiency of parameters it possesses may confer a resistance to overfitting, thereby maintaining its effectiveness. Here, we provide a concise overview of the conditional diffusion model framework used in EEGguided image generation, following the presentation of continuous-time diffusion models in [51, 52] .Diffusion models Diffusion Models (DMs) engage in a generative process by transforming highvariance Gaussian noise into structured data representations. This transformation is achieved by gradually reducing noise levels across a sequence of steps. Specifically, we begin with a high-variance Gaussian noise x M ∼ N (0, σ 2 max ) and systematically denoise it through a series of steps to obtain x t ∼ p(x t ; t), where σ t < σ t+1 and σ M = σ max . For a well-calibrated DM, and with σ 0 = 0, the final x 0 aligns with the original data distribution.The sampling in DMs is implemented by numerically simulating a Probability Flow ordinary differential equation (ODE) or a stochastic differential equation (SDE). The ODE is represented as: dx = -σ(t)σ(t)∇ x log p(x; t)dt, (1) where ∇ x log p(x; t) is the score function, and σ(t) is a pre-defined schedule with its time derivative σ(t). The SDE variant includes a Langevin diffusion component and is expressed as:EQUATIONwhere dω t is the standard Wiener process.The core of DM training is to learn a model for the score function. This is typically achieved through denoising score matching (DSM), where ϵ θ is a learnable denoiser. The training process can be formulated as:EQUATIONwhere ϵ is Gaussian noise with variance σ 2 t , and c represents a condition.The initiation of the EEG-conditioned diffusion phase is paramount in our EEG-based image generation framework, leveraging the classifier-free guidance strategy alongside data pairs of CLIP embeddings and EEG embeddings (z I , z E ). where w ≥ 0 represents the guidance scale. This mechanism facilitates concurrent training of the conditional and unconditional models within a singular network framework, periodically substituting the EEG embedding z E with a null vector to promote training variability, i.e. 10% of the time. The primary objective of this method is to enhance the sample quality produced by DMs while maintaining output diversity.In Fig. 10 , we compare the effects of one-stage and two-stage EEG-guided image generation. We show images generated using EEG embeddings directly (One-stage) and images generated using image embeddings obtained via prior diffusion (Two-stage). It can be seen that the two-stage EEGguided image generation can more accurately reconstruct the semantic and low-level visual features of the original image, and the style is more realistic. (a) We present the images that subjects seen (Seen), our reconstructed images directly using EEG embeddings (One-stage), and the reconstructed images from low level and high level image embeddings obtained by the prior diffusion (Two-stage). These results indicate that the strategy of our two-stage generation can better reconstruct the seen visual stimulus. (b) We employed ATM-S to compare the generated images with the original images in a retrieval task. Our result indicates that the images generated in two stages significantly enhance the performance of the original model on the retrieval task.In the second stage of our EEG-based image generation approach, the CLIP embedding z I derived from the EEG-conditioned diffusion acts as the precursor for synthesizing visual objects I based on z I . This is achieved by harnessing the synergies of advanced pre-trained models, namely SDXL and IP-Adapter [15, 16] , facilitating the creation of high-caliber images.The cornerstone of our synthesis process is the SDXL framework, acclaimed for its proficiency in textto-image conversion. The integration of the IP-Adapter introduces dual cross-attention mechanisms, allowing the CLIP embedding z I to serve as a directive input and guide the denoising trajectory within the U-Net structure. The synthesis model is denoted as ϵ SD (z t , t, z I ), where z t denotes the SDXL Variational Autoencoder's (VAE) disturbed latents.To augment the efficiency of our framework, we additionally explore the SDXL-Turbo [53] , a refined iteration of SDXL optimized for swift image synthesis. This variant proves especially beneficial in scenarios demanding quick generation of high-fidelity visuals.The IP-Adapter, with its compact design, has proven to be effective in enhancing image prompt adaptability within pre-trained text-to-image models. Its compatibility with text prompts for multimodal image generation extends the versatility of our EEG-based image synthesis approach.Compared with pure vision pre-training models such as (ViT, ResNet, DINO, etc.), the CLIP model lacks low-level visual features. Therefore, in order to make up for this shortcoming, our framework introduces a low-level visual reconstruction pipeline. We hope to restore basic such as contour, posture, orientation and other pixel-level information from EEG by aligning with the latent of VAE.Past work [25] has found that in the denoising stage of the early diffusion model, z signals (corresponding to the VAE latent in our framework) dominated prediction of fMRI signals. And during the middle step of the denoising process, z c predicted activity within higher visual cortex much better than z. However, note that this is only an analysis based on decoding accuracy. These analyses do not have a strong neuroscience causal relationship. We still cannot conclude that the low-level features of neural data are modeled by VAE. We trained the low-level pipe for 200 epochs, trying a latent mean squared error (MSE) loss, along with a contrastive learning loss, and a variational autoencoder (VAE) image reconstruction loss to align the 4 × 64 × 64 EEG latents obtained from a projection layer and an upsampled CNN with the VAE latents. Nevertheless, reconstruction loss or contrastive learning loss performs worse than only applying the loss in latent space and also requires significantly more GPU memory. In addition, we found that using a low-level visual model for distillation learning in the low-level pipeline is not only unhelpful for VAE latent training, but also leads to overfitting. Similar conclusions were reached in MindEye [2] . Our results suggested that the low-level zeroshot reconstruction in EEG is not stable enough and may mislead the model results. When using the low level pipeline, we usually set the inference steps of SDXL to 10 (or SDXL-turbo to 4) and the image-to-image denoising strength to 0.5. We give several reconstruction examples in Fig. 11 to compare the impact before and after using low level pipeline. Moreover, low level alignment was validated through retrieval performance tests, as depicted in Tab. 5. Our findings indicate that retrieval using EEG latents can also achieve excellent performance. This further substantiates the feasibility of aligning the low-level consistency through VAE latents.In addition to using EEG latent and low-level pipelines during reconstruction, we also add a corresponding semantic level pipeline guided by text captions during the image reconstruction. We input the 1 × 1024 EEG features output by prior Diffusion into the trained image projector to obtain 256 × 1024 image features. Using the GIT model [54] , we can directly generate a caption from the latent features of the image. IP-Adapter accepts such a caption as a text prompt to guide the semantic level reconstruction of the image. It should be noted that due to the difficulty of the zeroshot task itself and the low dimensionality of the EEG features, the caption generated from the latent may be unstable, thereby interfering with the original correct EEG semantics. Considering that the image features extracted by the CLIP model itself are already high-level visual features and do not require the introduction of more semantic information, this framework retains the entry of the text prompt, and the reconstructed image presented does not force the use of the semantic level pipeline. Comparison metrics Our study uses various metrics to evaluate how well we can recreate visual stimulus from brain data (EEG, MEG, fMRI) (Tab. 1 and Tab. 7). These metrics include PixCorr (pixelwise correlation, between ground truth and reconstructions), SSIM (structural similarity index metric) [55] , SwAV (SwAV-ResNet50, refer to average correlation distance) [56] , and two-way identification using neural networks (AlexNet(2/5), Inception, CLIP. Here AlexNet(2/5) the 2nd and 5th feature layers of AlexNet) for both low-level and high-level image features. Here two-way identification can be seem as a two-way retrieval task described in [20] . In Tab. 1, our results showed that on the THINGS dataset, we could achieve performance over MEG on EEG reconstruction using ATM. Tab. 6 shows the decoding performance of different data sets (fMRI, MEG, EEG) on visual stimulus tasks, and we even achieved the same or better performance than fMRI and MEG. Our results suggest that a suitable neural representation plays a decisive role in the downstream task. As depicted in Fig. 12 , we showcase the representational similarity matrix and visualization in the latent space. To investigate the relationship between the representations obtained from EEG and those of images, we conducted a representational similarity matrix. We focused on Subject 8, who exhibited the highest retrieval accuracy. By applying a clustering algorithm to the image embeddings corresponding to 200 images in the test set, we observed distinct within-category clustering. We generated similarity matrices based on both image and text embeddings, which were then compared with EEG representations. As shown in Fig. 12 , clear within-category clustering is observable in the representational similarity matrix with image, whereas this phenomenon is not present in the representational similarity matrix with text.We have adopted the concept embedding encoder proposed by Wei et al. [62] , which encodes the clip embedding of the original image into a 42-dimensional vector, with each dimension representing a distinct concept. Utilizing the ATM for direct projection on the EEG data of 200 categories from the test set, we obtained EEG embeddings that were then fed into the concept encoder with frozen weights, yielding 200 concept embeddings, as depicted in Fig. 13 . An analysis of representational similarity at the concept level indicates that the EEG embeddings derived from our EEG projector effectively align with the conceptual space. This ensures semantic consistency at a high level of alignment, providing compelling evidence for the reconstruction of images from EEG data. We visualize the best, medium and worst generated images in Fig. 14 . We randomly selected the EEG data of a subject viewing 100 images, and extracted EEG embeddings to guide image generation. By calculating the cosine similarity of the CLIP embedding between the generated image and the original image, we found 12 images each with the best, medium and worst generation effects. It can be seen that in the best group, the generated image is not only highly consistent with the semantics of the original image, but also well retains the low-level visual features. in the medium group, the generated image maintains the semantic features of the original image, and the low-level visual features are well preserved. Visual features were altered. in the worst group, both semantic features and low-level visual features were altered. Generated Seen Generated SeenAccording to our results in 30 for time windows, which shows that for all embeddings, a clear peak can be observed for windows ending around 200-250 ms after image onset. Comparing 29 and 30, we can see that, unlike [12] , our time window results on THINGS-EEG dataset do not have a second peak, which may be mainly affected by the experimental paradigm. In 7, we can see that in the first 50ms-200ms, the image reconstructed at 50 ms is completely messy and has no semantics; the semantics of the reconstructed image after 250 ms is basically correct and gradually stabilizes, and after 500ms, due to the lack of additional visual response, the content of the image reconstruction is more stable. Similar results are also shown in Figure 4 A of [12] : their method can also reconstruct high-quality images at 100ms. This just shows that our reconstruction results are in line with the neuroscience prior. However, this does not mean that the EEG data after the absence of visual response (200ms) loses its contribution to decoding, because the processing of high-level visual features (corresponding to the visual features of CLIP) may be involved over time. Table 8 : Overall performance of zero-shot Retrieval on THINGS-EEG dataset. We divided the last batch from the original training set as the validation set, selected the best model according to the minimum validation loss in 40 epochs, and finally evaluated the performance on the test set. We showed in-subject and cross-subject retrieval task performance (Ave ± Std.%) under the condition of batch size=1024. We compared the 2-way, 4-way, 10-way, the Top-1 and Top-5 accuracy of 200-way from different EEG embedding methods. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: In the box plots and line charts in this article, we provide accurate error bars to describe the performance differences between different settings and different subjects. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 


Decision: This paper demonstrates visual decoding and reconstruction of viewed
images from recorded EEG using the THINGS EEG(2) dataset.  They
demonstrate their algorithm on image retrieval, category
classification, and image reconstruction tasks.  Results are
impressive and exciting.  The results (timing, most relevant
electrodes) are consistent with the Neuroscience literature, but the
ability to do this with EEG is surprising.

The biggest issue with this paper, as raised by one reviewer is
measuring performance as maximal on test data.  While the author responded
that it should not matter as all methods are tested similarly, this is
not true as methods with higher variance are more benefitted by this
approach.  It is well known that parameters must be fit on data
separate from the test data.  If this paper is accepted, this will
have to be changed in the camera ready.	 From the plots, it appears that
this will only change the numbers slightly, and	so I recommend acceptance.

Minor:
It appears there might be a typo in the	supplemental text.  On line
432, it is mentioned that "we averaged across the 4 EEG trials from
the same image in the test set" but the test set has 80 repeats and
later on line 513, they say they used an "averaging technique on 80 repeated instances
within the test set".

Finally, there is contemporaneous work for reconstruction from EEG
here https://arxiv.org/abs/2404.01250

Reviews: review: paper_summary: paper_summary: This paper proposes a learning framework to decode images from EEG signals. It introduces a tailored brain encoder, the Adaptive Thinking Mapper, which projects neural signals to the clip embedding space. Subsequently, a two-stage image generation strategy is applied to produce images, progressing from blurry to high-quality reconstructions. This is an interesting and innovative work.
strength_weakness: strength_weakness: * Some experimental settings were not very clearly illustrated. For instance, in Section 3.1, it is mentioned that the experiments were trained on an EEG dataset and tested on an MEG dataset. How did you align the channel heterogeneity? Is it a zero-shot approach?

* Table 1 is a bit confusing to me. It would be helpful to clarify and explain the experimental settings of these additional datasets.

* Although EEG has a fast response, the design of the proposed framework does not seem to leverage motivations related to EEG data. It would be better to demonstrate the effectiveness of the proposed framework compared to existing fMRI-image decoding methodologies. For instance, you could replace the EEG encoder with an fMRI encoder and test whether the framework can outperform existing methodologies.
questions: questions: * Is the spider plot in Figure 4 showing normalized performance? Please clarify.

* Please address my queries in the Weakness section:
limitations: limitations: I agree with the limitation regarding the cross-subject performance drop. Future work could consider incorporating existing cross-subject generalization efforts from the EEG domain to address this issue.
paper_summary: paper_summary: The study proposes an end-to-end EEG\MEG-to-image reconstruction framework, consisting of a tailored brain encoder ATM to project neural signals into the shared subspace as the clip embedding, and a two-stage image generation block. The model achieves successful cross-subject EEG\MEG decoding and SOTA performance in classification, retrieval, and reconstruction.
strength_weakness: strength_weakness: 1. I think some parts of the model and implementation have not been clarified very well:
* I think it hasn’t been clarified in the main text whether the main results (starting from 3.2 to 3.5) show within-subject or cross-subject performance, unless I missed something. Adding some notices in figure\table captions or some summarizing sentences at the beginning of each section might be helpful.
* What model is inside the frozen “VAE image encoder” for low-level image generation in Figure 2? Also, what diffusion model is this study conditioning on? Is it built and retained from a pre-trained model (like stable diffusion), or was it trained from scratch by the authors?
* I think the authors didn’t introduce the “image2text” component (BLIP2), unless I missed it.
2. There are two recent EEG-to-image reconstruction works that this study has not discussed or compared. While it is understandable that the authors did not compare them, as they are still preprints and use different datasets, it might be beneficial to discuss them in the related work section given the limited literature in this field:
* Bai, Yunpeng, et al. "Dreamdiffusion: Generating high-quality images from brain eeg signals." arXiv preprint arXiv:2306.16934 (2023).
* Lan, Yu-Ting, et al. "Seeing through the brain: image reconstruction of visual perception from human brain signals." arXiv preprint arXiv:2308.02510 (2023).

For other minor comments please see Questions.
questions: questions: 1. For low-level metrics, in Mindeye, Ozcelik et al.’s, and other fMRI-to-image reconstruction works, they usually provide PixCorr values. I wonder why this metric wasn't included here.
2. For Figure 4, the plots are impressive, but it would be helpful to include numerical indications of the downstream task performances, such as 0.7 or 0.8. This would provide a clearer understanding of how well the model is performing.
3. Could the author elaborate more on the functioning of the shared token and subject tokens?
limitations: limitations: The authors have adequately discussed the limitations.
paper_summary: paper_summary: The paper presents a end-to-end EEG-based visual reconstruction zero-shot framework, featuring the Adaptive Thinking Mapper (ATM) and a two-stage EEG-to-image generation strategy. This method achievies state-of-the-art performance in classification, retrieval, and reconstruction tasks, and significantly advancing the field of brain-computer interfaces.
strength_weakness: strength_weakness: 1.The manuscript has several significant deficiencies. First, its motivation is based on the signal differences between EEG and fMRI, concluding that EEG's performance limitations are due to constraints in decoding and reconstruction frameworks. However, the proposed EEG encoder merely adds a Channel Attention layer compared to NICE [1], with no detailed explanation provided. Additionally, the loss function in Section 2.4 is taken directly from [2], demonstrating a lack of originality. Furthermore, the visual reconstruction framework shows no substantial difference from existing fMRI methods [3,4], as it also utilizes pre-trained stable diffusion models and their variants, with minor differences, such as the incorporation of Sdedit [5], being tricks to enhance generation quality. Thus, the manuscript fails to substantiate its claims and contributions convincingly.
2.The processing of visual information in the brain involves multiple stages and typically takes around 100-150 milliseconds to reach higher visual areas where complex processing occurs. EEG signals at 50 milliseconds are likely still within the retina and optic nerve stages. Thus, the generation of images from 50ms EEG signals, as shown in Figure 7, contradicts established neuroscience principles. This suggests that the visual stimulus reconstruction framework heavily relies on the image generation model, which may have limited significance for the field of neuroscience.

[1] Song, Yonghao, et al. "Decoding Natural Images from EEG for Object Recognition." arXiv preprint arXiv:2308.13234 (2023).
[2] Benchetrit, Yohann, Hubert Banville, and Jean-Rémi King. "Brain decoding: toward real-time reconstruction of visual perception." arXiv preprint arXiv:2310.19812 (2023).
[3] Chen, Zijiao, et al. "Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[4] Lu, Yizhuo, et al. "Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion." Proceedings of the 31st ACM International Conference on Multimedia. 2023.
[5] Meng, Chenlin, et al. "Sdedit: Guided image synthesis and editing with stochastic differential equations." arXiv preprint arXiv:2108.01073 (2021).
questions: questions: I have no quentions, please see the weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This paper presents an end-to-end EEG-based visual decoding framework that includes two stages: a brain encoder for EEG feature extraction and a "generator" for producing reconstructed images. The experiments demonstrate effective results in retrieval, classification, and reconstruction tasks across two datasets, suggesting potential applications for real-time brain-computer interfaces.
strength_weakness: strength_weakness: The paper's novelty in the context of brain-image decoding methodologies is not distinctly clear.

1. The pipeline, which includes a contrastive learning-driven encoder and a diffusion-based decoder, is already well-established in the field.

2. The ATM encoder, which incorporates channel-wise attention, spatial convolution, and temporal convolution, does not significantly differ from previous studies, such as mentioned in the introduction, Benchetrit et al. and Song et al.

3. The method for disentangling low- and high-level features within the framework is unclear, particularly how the VAE is supposed to provide low-level image features.

The evaluation section contains many conclusions that conflict with scientific validity, which may cause serious misleading.

1. How can we get clear reconstruction results with only 50 ms signals after the onset as in Figure 7c? The visual stimuli haven’t reached V1 in such a short time. 

2. We know the temporal cortex is strongly related to object recognition. But in Figure 8b, the temporal channels contribute very small. Based on that, should we think the retrieval task was based on some low-level information? 

3. The visual response is very quick after the onset, finished before 200 ms. It cannot be thought of as visual responses at 500 ms after the onset on Page 7 Line 182, and contained within 200-400 ms on Page 9 Line 246. 

4. On page 9 Line 251, It can’t be concluded that EEG is better than MEG in visual tasks where the paradigms used for these two datasets were different.
questions: questions: 1. The paper mentions selecting the highest test accuracy during the training process as the statistical result. It would be more rigorous to test the model only once after it is fully trained.

2. In the framework illustrated in Figure 2, which input to the diffusion process is most critical for image generation—the reconstructed output after VAE, the image embedding transferred from EEG, or the generated caption?

3. How does the model ensure that it captures low-level features after the VAE encoder?

4. What is the impact of the pre-trained language and generative models on the final performance?

5. Could you provide more details about the cross-subject settings described in P3L91? Specifically, what roles do the subject token and shared token in Figure 3 play, and do they enhance cross-subject capability?

6. Figure 11a seems to show no significant correlation between text and EEG features. Could you comment on this observation?
limitations: limitations: Yes. The authors adequately addressed the limitations and future works.

Comments: title: Response to Reviewer gPm4
please respond to the extensive rebuttal
Response to Reviewer 4xL4
Thank you for your further suggestive comments
Response to Reviewer gPm4
Response to authors
Sincere Request for Feedback on Response Clarifications and Explanations
Response to Reviewer zVcP
Thank you for the rebuttal
comment: Sorry to bother you. We are about to run out of time to respond.

We have made complements and explanations for this work with the help of all the reviews. We would be grateful if you could confirm whether the rebuttal meets your expectations and if there is any other suggestion.

Thank you once again for your time and insightful comments!
Reviewer 4xL4 please respond to the extensive rebuttal provided by the authors.
Thank you for your comments and feedback!

**Q1:  ”Its motivation is based on the signal differences between EEG and fMRI, concluding that EEG's performance limitations are due to constraints in decoding and reconstruction frameworks. ”**

Although fMRI plays an important role in neuroimaging research, fMRI has low temporal resolution, bulky and non-portable equipment, high cost, and is non-invasive but limited by the magnetic field environment. So fMRI-based work is almost impossible to apply in practice, and this have hindered the development of the brain-computer interface (BCI). This motivates us to propose a zero-shot visual decoding and reconstruction framework that can be proven effective on Image-EEG datasets. Our manuscript provides an empirical guidance on practical BCI applications. We hope that the BCI and neuroscience communities pay more attention to the implementation of similar technologies and real data rather than overfitting only on existing fMRI datasets.

**Q2: “However, the proposed EEG encoder merely adds a Channel Attention layer compared to NICE [1], with no detailed explanation provided. Additionally, the loss function in Section 2.4 is taken directly from [2], demonstrating a lack of originality. Furthermore, the visual reconstruction framework shows no substantial difference from existing fMRI methods [3,4], as it also utilizes pre-trained stable diffusion models and their variants, with minor differences, such as the incorporation of Sdedit [5], being tricks to enhance generation quality. Thus, the manuscript fails to substantiate its claims and contributions convincingly.”**

Neural network model structures, including image reconstruction strategies, are often inductive biased. On the one hand, researchers in the BCI community have been using spatial-temporal convolutional models to process EEG signals since EEGNet. On the other hand, the open source of StableDiffusion allows us to leverage models like CLIP that were trained with massive datasets as a teacher to guide the training of our brain models where we have a relative scarcity of data. 

Although it utilizes existing machine learning techniques , we demonstrate for the first time that EEG-based zero-shot visual decoding and reconstruction can be competitive with fMRI. Previous published work either focused on the EEG image decoding, or reconstructed images of known categories on small-scale and controversial datasets; or focused on the fMRI based image decoding and reconstruction, to achieve advanced performance on the fMRI datasets. 

Our work has introduced joint subject training for cross-subject evaluation, which is expected to solve the problem of decreased decoding performance due to subject differences when the amount of training data is sufficient. To the best of our knowledge, this is the first work to simultaneously achieve state-of-the-art performance on downstream zero-shot retrieval, classification, and reconstruction tasks on a dataset of the size of THINGS-EEG.


**Q3: “The processing of visual information in the brain involves multiple stages and typically takes around 100-150 milliseconds to reach higher visual areas where complex processing occurs. EEG signals at 50 milliseconds are likely still within the retina and optic nerve stages. Thus, the generation of images from 50ms EEG signals, as shown in Figure 7, contradicts established neuroscience principles. This suggests that the visual stimulus reconstruction framework heavily relies on the image generation model, which may have limited significance for the field of neuroscience.”**

We have updated codes in the anonymous code link (https://anonymous.4open.science/r/Visual_Reconstruction-AC56). And we provided examples of reconstructing images with different random seeds for growing windows across time scales (README.md). From the two example stimulus images provided, the reconstructed images show uncertainty between 0 to 50 ms and 0 to 250 ms because the stimulus has just arrived at the primary visual cortex and it takes time for the brain to react and be captured, the reconstructed image during this period shows uncertainty (probably due to noise unrelated to the stimulus). Here, since the prior of natural images is added to the two-stage framework, high quality images can be reconstructed even from 0 to 50 ms and 0 to 250 ms. Then, with the accumulation of visual stimulus information, the semantics of the reconstructed images gradually become clear. After 500 ms, the images decoded by EEG tend to be stable, which means that there may be no new information added. 

Combined with Figure 7, the random seed image reconstruction example figure linked to the anonymous code we provided, and Figure 29 in Appendix H.1, we can see that these results just confirm the previous research and are consistent with the priors of neuroscience.
We further explain the question (1) you have raised. We added the performance of different EEG encoders on the test set with batch sizes of 16 and 1024 in the anonymous code (https://anonymous.4open.science/r/Visual_Reconstruction-AC56). It can be seen that all methods gradually converge with the increase of training epochs and present a performance value with very small variance on the test set. 

Therefore, **Figure 5c** and **Figure 5d** of this article use the epoch with the highest test set accuracy in the same total epochs, under the statistics of multiple random seeds. The results in **Table 7** are the average test set accuracy of the last 10 epochs after training convergence. In the **one-page pdf** file we uploaded, there is a more comprehensive performance of each subject's data for reference.
We appreciate the reviewer's thoughtful feedback and recognize the significance of the highlighted concerns. 

(1) Thank you for your suggestion. We will add the results on the validation set in the official version of the paper. In fact, we initially considered that if we adopt the same approach as in [1], that is, to divide a small number of trials in the training set as the validation set for evaluating the model, this may violate the zero-shot task setting. Different from those classification and retrieval tasks with known categories, due to our unique processing form (in a dataset with a total of 1864 categories, we use 1654 categories of samples as the training set and the remaining 200 categories as the test set), so we hope to always evaluate the performance of the model in a zero-shot form.

Our original approach was to train the models of different encoders for a sufficient number of rounds (30 epochs in the experiment) to ensure model convergence, and test them on the test set in the last 10 epochs. Since all models use the same evaluation method, and we strictly control the random seed and ensure that the data is not leaked, the final evaluation results are also unbiased.

(2) This is another interesting question. First of all, our statement has some typo. What we want to express is "The low-level and high-level here refer to the representation that obtained from the pixel-level after VAE alignment with or the semantic-level after CLIP alignment with".
Past work [2] has found that in the denoising stage of the early diffusion model, z signals (corresponding to the VAE latent in our framework) dominated prediction of fMRI signals. And during the middle step of the denoising process, zc predicted activity within higher visual cortex much better than z. However, please note that this is only an analysis based on decoding accuracy. These analyses do not have a strong neuroscience causal relationship. We still cannot conclude that the low-level features of fMRI are modeled by VAE. 

From our experimental results, the more VAE latent is used in the denoising process, the more certain the overall reconstructed image is and the less details it has. Conversely, the more details it has. Therefore, the contribution of VAE latent and clip latent to reconstruction tends to be a balance relationship. Our future work should focus on achieving more brain-like decoding, paying attention to both well low-level and high-level reconstruction, rather than maintaining an either-or relationship between the two.


**Reference**

[1] Song Y, Liu B, Li X, et al. Decoding Natural Images from EEG for Object Recognition[C]//The Twelfth International Conference on Learning Representations.

[2] Takagi Y, Nishimoto S. High-resolution image reconstruction with latent diffusion models from human brain activity[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14453-14463
Thanks for your kind reply. I still have concerns: 1) Q7, Why not use a validation set? It's not convincing to use test sets in training processing. 2) Q9, what are the gains of low-level features from VAE for the overall framework, if they are pixel-level representations after CLIP alignment, instead of low-level features defined in vision?
Dear Chairs and Reviewers,

Thanks for the insightful questions and suggestions to our manuscript. 

Could you please provide feedback on our responses to your questions? We are sincerely eager to know if our answers have addressed your concerns. Additionally, are there any further comments or insights you might have regarding our work?

Your input is crucial for us to enhance the quality of our submission, and we greatly appreciate your guidance and time.

Best wishes,

Paper Authors
Thank you to the reviewer for the impressive questions and suggestions.

We are very grateful to the reviewer for pointing out the technical details that were not clarified in our article. The content presented in this article is quite comprehensive and sufficient, but we sincerely hope the reviewer to pay further attention to our innovation and unique contributions:

1. Technically, we consider the positional relationship between channels and the modeling of the time dimension, so we introduce a channel-wise patch embedding and feedforward neural network in the time dimension (Figure 3). EEG data is different from fMRI, which requires sufficiently effective feature extraction and neuroscience-specific inductive paranoia to maximize the performance expectations of decoding - we have achieved state-of-the-art on different tasks (Figure 4). Finally, in the uploaded pdf file, we provide comprehensive evaluation results.

2. In terms of scientific insights, based on strong performance, we further provide analysis results of different dimensions such as time distribution (Figure 7), spatial distribution (Figure 8), representation distribution (Figure 11) and concept distribution (Figure 12), which strongly proves the effectiveness, scalability, interpretability and causality of our framework for EEG decoding and reconstruction.

3. In terms of potential impact, our work enhances the interpretability of the model on existing neural decoding tasks as much as possible, including decoding of cognitive concept foundations (Figure 12) and revealing neural mechanisms (Figure 8). Furthermore, we focus on the study of causal relationships. In order to understand the reasons for the high decoding accuracy, we conducted a number of spatial-temporal ablation experiments, compared and demonstrated in detail the corresponding viewpoints in EEG decoding and neuroscience. In addition, in order to prove the effectiveness of the results, future work will focus on the verification of real-world applications - we are collecting enough EEG data from different subjects for full parameter or efficient parameter fine tuning to further verify the effectiveness of our framework.

4. We have updated codes in the anonymous code link (https://anonymous.4open.science/r/Visual_Reconstruction-AC56), and all codes provided in the review area will be refined and be public after the anonymous phase. And we provided examples of reconstructing images with different random seeds for growing windows across time scales (README.md). From the two example stimulus images provided, the reconstructed images show uncertainty between 0 to 50 ms and 0 to 250 ms because the stimulus has just arrived at the primary visual cortex and it takes time for the brain to react and be captured, the reconstructed image during this period shows uncertainty (probably due to noise unrelated to the stimulus). Here, since the prior of natural images is added to the two-stage framework, high quality images can be reconstructed even from 0 to 50 ms and 0 to 250 ms. Then, with the accumulation of visual stimulus information, the semantics of the reconstructed images gradually become clear. After 500 ms, the images decoded by EEG tend to be stable, which means that there may be no new information added.
Thank you for the detailed explanations! I will maintain my current score.