EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals
Authors: Xuanhao Liu
Yan-Kai Liu
Yansen Wang
Kan Ren
Hanwen Shi
Zilong Wang
Dongsheng Li
Bao-liang Lu
Wei-Long Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: EEG
video generation
diffusion model
brain-computer interface
Primary area: neuroscience_and_cognitive_science
TL;DR: We build an EEG-video dataset and propose a framework to generate videos from EEG signals, taking an important step towards decoding dynamic visual perception from EEG.

Abstract: Our visual experience in daily life are dominated by dynamic change. Decoding such dynamic information from brain activity can enhance the understanding of the brain’s visual processing system. However, previous studies predominately focus on reconstructing static visual stimuli. In this paper, we explore to decode dynamic visual perception from electroencephalography (EEG), a neuroimaging technique able to record brain activity with high temporal resolution (1000 Hz) for capturing rapid changes in brains. Our contributions are threefold: Firstly, we develop a large dataset recording signals from 20 subjects while they were watching 1400 dynamic video clips of 40 concepts. This dataset fills the gap in the lack of EEG-video pairs. Secondly, we annotate each video clips to investigate the potential for decoding some specific meta information (e.g., color, dynamic, human or not) from EEG. Thirdly, we propose a novel baseline EEG2Video for video reconstruction from EEG signals that better aligns dynamic movements with high temporal resolution brain signals by Seq2Seq architecture. EEG2Video achieves a 2-way accuracy of 79.8% in semantic classification tasks and 0.256 in structural similarity index (SSIM). Overall, our works takes an important step towards decoding dynamic visual perception from EEG signals. Our dataset and code will be released soon.

Video: Towards Decoding Dynamic Visual Perception from EEG Signals
Xuan-Hao Liu Yan-Kai Liu 1x Yansen Wang"1, Kan Ren Hanwen Shi',
Zilong Wang", Dongsheng Li?, Bao-Liang Lu', Wei-Long Zheng' Shanghai Jiao Tong University "Micrssoft Research Asia "ShanghaiTech University
https://bcmi. http//mmi.jj.uu... edu. an/home/ /enne/eeg
Abstract
Our visual experience in daily life are dominated by dynamic change. Decoding such dynamic information from brain activity can enhance the understanding of the brain's visual processing system. However, previous studies predominately focus on reconstructing static visual stimuli. In this paper, we explore to decode dynamic visual perception from elecrreencephaloraahyy (EEG), a neuroimaging technique able record brain activity with high temporal resolution (1000 Hz) for capturing rapid changes in brains. Our contributions are threefold: Firstly, we develop a large dataset recording signals from 20 subjects while they were watching 1400 dynamic video clips of 40 concepts. This dataset fills the gap in the lack of EEG-video pairs. Secondly, we annotate each video clip to investigate the potential for decoding some specific meta information (e.g., color, dynamic, human not) from EEG. Thirdly, we propose a novel baseline REG2Vided for video reconstruction from EEG signals that better aligns dynamic movements with high temporal resolution brain signals by Seq2Seq architecture. EEG2Vidco achieves 2-way accuracy of 79.8% in semantic classification tasks and 0.256 in structural similarity index (SSIM), Overall, our works takes an important step towards decoding dynamic visual perception from EEG signals.
1 Introduction
Our visual experience are composed of continuously evolving scenes caused by the movement of objects and viewing perspective [ The intricate and complex visual system our brains which enables us to explore the wonderful and ever-changing visual world has been appealing interests from philosophers and scientists for centuries (266.. To investigate the mechanism of visual system, various neuroimaging techniques have been used to analyze brain activities, especially non-invasive methods like functional Magnetic Resonance Imaging (fMRI)
g]
magneneencephhaogrppyMMMMM IT. and elettrenncehhaloggaplyEEECC 12 13
Compared fMRI and MEG which need be recorded by large and expensive medical devices, EEG is relatively low-cost and portable and thus has been applied across many human visual studies 14-18 For instance, a recent work achieved an accuracy of 15.6% in 200-way zero-shot tasks on an EEG-image dataset [19], demonstrating the rich visual information in BEG signals. However, these studies adopt either artificial strobe (SSVEP) [44 [5] or static image [66 as stimulation, which is far different from the dynamic visual world and not suitable for studying brain activities in naturalistic paradigm. As far as we know, there is currently no research on decoding video from EEG
"uaan-Har Liu and Kan-Kai Liu contribute equally.
'The work was conducted during Xuan-Hao Liu's internship Microsoft Research Asia. Corresponding authors: yansenwang @ microsoft.com renkan @ whanghaitech.edu.crr weilong @ sjtu.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

signals. Consequently, people have limited knowledge about 1) whether can we decode video from EEG signals? 2) yes, what kind of visual information can we decode?
In order to fill the gap, we develop a large EEG dataset, called STTU EEG Dataset for Dynamic Vision (SEED-DV) dataset, collected from 20 subjects while they were watching series of natural videos belonging to 40 different concepts. Also, we annotate some meta information for each video clip to comprehensively explore the boundary of which visual information can be decoded from EEG signals, offering benchmark containing various visual decoding tasks across object recognition, color/motion perception, and human/face detection.
Besides the fundamental classification tasks, reconstructing visual perceptions from corresponding brain signals helps to advance the understanding of our visual neural system. With the development of the representation learning and artificial intelligence generated content (AIGC), numerous works have reconstructed vivid images from brain activities 200 which utilize the text-to-image generation models pretrained on large amounts of visual-language pairs by aligning the brain signals with corresponding text embeddings. Recently, some works reconstruct high-quality two-sccond videos from a single IMRI data frame 32). However, limited by the low temporal resolution of fMRI, these video generation frameworks lack the ability of capturing high dynamic changes.
To this end, we propose a novel baseline, VEG2Video, for video reconstruction from EEG signals based on Seq2Seq architectures which extract continuous low-level dynamic visual perception such as color and position from the brain signals of high temporal resolution. Afterwards, a dynamic- aware noise-adding (DANA) method is adopted for the diffusion process according the dynamic information decoded from EEG. At last, we adopt the inflated diffusion model [33 fine-tuned on our dataset for video generation using the semantic information predicted from EEG. Our method densely extract visual information from high temporal resolution brain signals thus can better recover fast changes. Overall, our video reconstruction results take important step towards decoding dynamic visual perception from BEG.
In conclusion, contribution are as follows:
For the first time, we develop a large EEG dataset named SEED-DV dataset collected from 20 subjects, offering 1400 EEG-video pairs from 40 concepts for studying dynamic visual information in EEG signals.
We annotate the meta information of each video clip for comprehensively analyzing the visual information in EEG, presenting the EEG-VP benchmark.
We evaluate various EEG models the EEG-VP benchmark to determine the decoding ability of different visual information in raw EEG signals and -ettaneettracted features.
We propose novel framework named EEG2Video for video reconstruction from EEG signals based on Seq2Seq architecture to densely utilize the highly dynamic information.
The ablation study showcases the effectiveness of Seq2Seq and DANA modules in Video, which are designed based on the decoding results of different visual information on the EPG-VP benchmark.
2 Related Work
2.1 Decoding Static Visual Perception from Brain Activities
Researchers have been trying decode low-level static visual perception (e.g., shape, color, and position) from brain activities for decades (34)7)) revealing the abundant visual information hidden in brain signals. Early approaches tried generate hazy silhouette or higher-quality images with Deconvolutional Neural Networks (DeCNNs) ) 39], Generative Adversarial Networks (4O
44. and Variational Autoencoders (VAEs) 44 Recent studies have obtained impressive results on
decoding static visual stimuli from various brain activities like fMRI (7) MEG (8 and EEG
[9]30] with the Latent Diffusion Models (LDMs), also named Stable Diffusion (SD) 45 47]. as
image generation module, which is pretrained on large-scale image dataset to generate vivid images based on text prompts. Specifically, text prompts are embedded into a text-image sharing latent space by CLIP 48 multi-modal encoder to align visual and language representations. By encoding brain signals into the same sharing space, these methods can decode statie visual perception with high diversity and fidelity using LDMs.

---

Compared to the high spatial resolutions of brain signals recorded by large devices like fMRI ( 100,000 voxels) and MEG (=z 300 sensors), the limited spatial resolution of EEG (+e 60 sensors) brings difficulties to decode accurate visual perception from EEG in both semantic and pixel levels. Although previous studies claimed achieve over 60% of semantic decoding accuracy on an EEG dataset with 40 classes (29 (30, the dataset[49 was blamed due to the block design [50]. The follow-
up rigorous experiment [51] by randomly arranging all images showed at most 7.0% classification accuracy of the EEG siganls (chance level is 2.5%), exposing the fact that it is still challenging and insufficiently supported by appropriate datasets to decode visual perception from EEG.
2.2 Decoding Dynamic Visual Perception from Brain Activities
Observing the great success decoding static visual perception, many endeavours have been devoted to decoding dynamic visual perception from fMRI (52154 32] These works were conducted on an IMRI-video dataset collected from 3 female subjects while watching series of videos, including animals, humans, and natural scenery [52] Due to the data sparsity. the DeCNN, VAE and GAN- based methods can only decode hazy perception of dynamic videos from fMRI 52
54 To generate
high-quality videos, MinD- Video [3] and NeuroCine (32 utilized an inflated SD model (33 for video generation, which incorporates network temporal inflation by adding temporal attention modules in the original SD to ensure the consistency between frames. The video SD is firstly fine-tuned using the text-video pairs the training dataset (the text prompts were generated by an automatic image caption model called BLIP (55)). then the pre-trained fMRI encoder were co-trained with the video SD to enhance the fMRI guidance.
However, IMRI intrinsically lacks the ability of capturing dynamic visual perception due to the low temporal resolution, which is limited by the time scale of blood flow and results in a single fMRI frame every two seconds. MinD-Videof3 Video[31] and NeuroCine [32 all decoded two-second videos from only one fMRI frame. Thus, they are unable to decode changes faster than 0.5 Hz. Instead, other neuroimaging techniques such as EEG with high temporal resolution up to thousands Hz can offer more appropriate alternatives. To the best of our knowledge, there is no such dataset studying decoding video stimuli from such signals, and we are the first to support this research direction with the dataset, benchmarks, and the decoding framework.
3 EEG Dynamic Vision Dataset and Benchmarks
In this section, we first describe how we construct the SJTU EEG Dataset for Dynamic Vision (SEED- DV). Then we introduce two benchmarks built on the SEED-DV dataset: EEG visual perception classification benchmark and video reconstruction benchmark. The purpose of building this new dataset is to answer the following research questions:
RQ1- Whether can we decode dynamic visual information from EEG signals?
RQ2: If yes, which visual information can be decoded?
RQ3 To what extent can we reconstruct video from EEG signals?
Hence, we carefully selected video clips suitable for studying dynamic vision and annotated their meta information.
3.1 Participants
Twenty healthy students from Shanghai Jiao Tong University (SJTU) participated (mean age: 21.75 STD: 2.05; 10 females, 10 males), all having normal or corrected-to-normal vision. All subjects were informed of the experimental process and signed informed consent forms before the experiment, then received monetary reimbursement after finishing. This study was approved by the ethical committee of SJTU Institutional Review Board for Human Research Protections.
3.2 Visual Stimuli Selection
We elaborately selected concepts of videos in our experiment to study. The number of concepts follows previous research on EEG-image pairs [5] 49]. It is worth noting that EEG-Things [
3

---

A
B
Video Stiulli
2
Eye
EEG Amplifier
EEG Signals
7 video blocks in total
min 40
230
8 min 40
230 s
min 40
video block
reat phase
8 min s
video clips: 2s
Figure The meta information of video clips of 40 concepts, and experiment protocol. (A) Visualizations of the meta information for all video clips of 40 concepts, plot the average of each meta information for each concept. (B) The data collecting environment. (C) Demonstration of whole data collecting session. A session will contain 7 video blocks to be watched, and there are rest phases at least 30 seconds each between blocks. (D) Demonstration of a video block, there is a 3-second hint before different video clips of the same concept.
dataset (with 1854 object concepts) employed rapid serial visual presentation (RSVP) paradigm 55 57, which cannot be used for EEG-video experiment as video continuous stimulation. We further cluster the concepts into 9 coarser classes, as is demonstrated in Figure [A]:: (land animal, water animal, plant, exercise, human, natural scene, food, musical instrument, transportation).
All video clips are collected from two online video websites, Bilibil and YouTube We selected 35 different two-second video clips for each concept, which are divided randomly into 7 groups, each group has video clips. Afterwards, 40 groups of different concepts are arranged sequentially form single block, each block containing 40 blocks in our experiment, where the order of 40 categories of videos within each block is random and different from each other mitigate the temporal bias EEG signals.
200 video clips. Consequently, there are
3.3 Experiment Protocol
To ensure the quality of the acquired EEG data, the experiments were conducted in a controlled laboratory environment to minimize noise and other environmental disturbances. 62-channel EEG signals were collected by an active AgCI electrode cap with an international 10-10 system. The EEG signals were acquired using the ESI NeuroScan System at a sampling rate of 1000 Hz. Besides EEG data, EOG and ECG signals were recorded simultaneously during the experiment. We also adopt Tobii Pro Fusion eye tracker to collect eye movements a sampling rate of 250Hz.
During the experiment, all subjects were instructed to watch a series of color video clips presented with the resolution of 1980 1080 (16:9) in full-screen mode 25-inch display. In each block, the 5 video clips with the same category were displayed continuously, and before playing these same-class videos, there is hint the screen to inform the subjects what class they will see next, which will last for three seconds. Consequently, there are 40 x (3 + x 2) = 520 seconds, i.c., 8 min 40 s for each block.
There are blocks of videos to watch in our experiment for a subject. After finishing watching a block (40 classes), the subject required rest for at least 30 seconds to mitigate fatigue and assesses his/her own attention level (ranging from 1 to 5. means sleepy and 5 means very concentrating)
hinp::///www..bbbiii.. com
https://www http://wwwww.uubbe.eee// ywwttbe.uccoom.

---

No Human
Human
Human
Face
No Face
Blue
305
Histogram Optical Flow Score
65.6%
Red
Number
28.6%
11. 2%
71.4%
62.3%
21.6%
Grey
8.1%
19.0%
Human
No Face
Yallow
White
Optical flow Score
Figure 2: Statistics of each meta information: (A) the fraction of human appearance. (B) the fraction of face appearance (only count the videos with humans). (C) the distribution of different object numbers. (D) the distribution of different object colors. (E) the histogram of OFS.
before starting the next block. As result, the average attention level across all the subjects and blocks is 4.01 + 0.83, ensuring high-quality EEG signals with acceptable concentrations.
3.4 Meta Information of Video Clips
Besides the 40 concepts and 9 coarser classes, we also labeled some other information for each video clip.
Color: The main color of the main object. There 7 color categories: (Red, Yellow, Blue, Green, White, Grey, Colorful). Colorful indicates the color is too complex for identifying an accurate color from single video clip.
Optical Flow Score:
The optical flow score (OFS) of each 24 FPS two-second video clip obtained by averaging the length of the optical flow vectors, ranging from 0.008 (almost static) to 6.252 (rapidly changing). Further, based on the OFS, we divide all the video clips into 2 categories: (Fast, Slow). We choose the median OFS of 1.799 as the threshold to make sure the label is balanced.
The number of the main objects. There are 3 categories: (One, Two, Many]. Many indicates the number of the main objects is equal to or more than three.
Object Number:
Human: If there are any humans appearing in the video, the label is 1, otherwise is 0.
Human Face: If there are any human faces appearing in the video, the label is otherwise is O.
We depict the average meta information of each concept in Figure A):
3.5
EEG Visual Perception Classification Benchmark
To better understand the visual information in EEG signals,
low-level and high-level visual perception tasks are investigated in our dataset, presenting an EEG Viusal Perception (EEG-VP benchmark. There are EEG classification tasks based the video label and meta information detailed in Section 3.4 whose statistics are presented in Figure 2
The 40-class classification of the fine fine-grained concept of the video clip.
The 9-class classification of the course concept of the video clip.
The 6-class classification of which color of the main object the video clip. The data while watching Colorful videos are discarded in this task.
The binary classification of whether the video is fast or slow based on the OFS.
The 3-class classification the number of the main objects the video clip.
The binary classification of whether human appears the video or not.
The binary classification of whether any human face appears in the video or not. The data while watching videos without human appearance are discarded in this task.
3.6 Video Reconstruction Benchmark
This task is to reconstruct two-second video clips from the corresponding EEG segments. Following
the previous video reconstruction from fMRI studies 31 32 52 54), we utilize the same metrics to
5

---

A
EEG Signals
gogg
2
Generated Video
x, Ag
Seq2Seq Model
Video Diffusion Model
B
Decoder
Eong
Embodding
Leyer
Semantic Predictor
2n
Static Noise
Diverse Noise 24
Concal
B
By
By
Dynamic Predictor
Egtabal
Exocul
E
Video
Zo
ZT
Zg
Generated Video
VAE Encoder
Diffusion Process
Inflated Diffusion UNet
VAE Decoder
200000.00 0000000
Semantic Guidance
5000055
yellow cal playtig with a ball of yam
Clip Text Encoder
Freeze
Fine-Tune
Figure 3: (A-B) GLMNet Encoders. (C-E) Overview of our EEG2Video framework. (A) The visual cortex, basically in the occipital lobe. (B) GLMNet architecture which combines the global and local embedding. (CyThe framework guidance é, with Seq2Seq model and predictor, respectively. A video diffusion model is then be adopted for generating videos using 20 and é (D) Dynamic-aware noise-adding process based on the decoded dynamic information B. (E) Using large amounts of video-text pairs to fine-tune the inflated diffusion UNet for video generation. The texts are obtained by BLIP.
EEG2Vider which predicts the latent variabels &o and semantic
evaluate the quality of generated videos, roughly classified as frame- basedd metrics and video-based metrics. Definition of these metries can be found Appendix(A]
Methodology
4.1 EEG Encoder:
Mixture of Global and Local Features.
EEG classifiers take into an EEG segment æ E X' C CxT where C denotes EEG channels and T denotes time samples, and decode the target information VV, which can defined as X' - y
Previous studies on EEG classification always treat all channels equally 49
58
59
: f(x)
Eglobat( where Eyiobal is global encoder for decoding EEG. However, research neuroscience indicates that human visual cortex are basically in the occipital lobe, as shown in Figure 3 A), treating all channels equally is unable highlight the visual cortex features. To this end, propose simple yet effective network to combine the global features decoded from whole channels and local features decoded from the visual associated channels called Global Local Mixture Network (GLMNet). Depicted in Figure B). GLMNet utilizes local encoder Etocal to extract the vision- associated features, which can be denoted as f(x) emb (ooo(CCoca(@ee (Eeooaa(Egloba()))) Etocat (z)))
4.2 EEG22idee: High Temporal Resolution Brain Decoding Framework
Compared the fMRI-video reconstruction framework [I which are constrained by the limited temporal resolution that results in decoding two-second dynamic videos from one fMRI data frame, EEG signals have higher temporal resolution for better capturing dynamic visual perception to facilitate video reconstruction. As demonstrated in Figure C). we introduce EEG2Video, novel framework which utilize a Seq2Seq model to densely reconstruct low-level visual perception from continuous EEG embeddings extracted by an overlapping sliding window, and decode the semantic
CC)

---

and dynamic information with two other modules for guiding an inflated diffusion model to recreate videos.
EEG Embeddings Extraction For a two-second EEG segment æ E RCXT we apply an overlapping sliding window for slicing it into shorter segments (x1, T2, ,......,....... Wr),
where & is the total number. EEG
embeddings Eeeg = {ee: e ................................ , ) are then extracted from these shorter segments with an EEG encoder & that (zx, = ext i =
Seq2Seq Model In contrast to video reconstruction from fMRI (generating several frames from single data frame), it essential to align the high temporal resolution brain signals with videos in video reconstruction from EEG (generating several frames from several EEG segments) for capturing rapid changes. The Seq2Seq models are naturally introduced for extracting the continuous visual information from high temporal resolution brain signals. We employ the Transformer architecture as the Seq2Seq model in our framework, which can be formulated as a stack of several blocks, each block containing multi-head attention (MHA) layer and feed-forward network (FFN) layer[60]. Denoting the input of the i-th Transformer block as iin, the calculation of the output z is given by:
a'id SMHHA (LA(x! + 2i Auu = fFEN (LA(((LAmmma))) + midd
where ÍMHA is the MHA layer. SEEN is the FFN layer, and LA is layer normalization. In our framework, the input of Transformer is the addition of EEG embeddings and position embeddings (PE): = = eeeg + PE, and the output is the latent variables &o of the corresponding video frames. As depicted in Figure E), the Ground Truth (GT) video frames are fed into the frozen VAE encoder to obatin the GT latent variables 2u- We apply mean squared error (MSE) loss MSE (20 20) for training the Seq2Seq model that densely predicts the continuous visual information of frames.
Semantic Predictor In order to utilize the pre-trained diffusion models for generating high-quality videos, we first generate the corresponding text description of each video by feeding the medium frame to a caption model called BLIP (55. then align the EEG signals with the text embeddings e, E 767766 which are acquired by the frozen CLIP text encoder as shown in Figure E). An MLP is adopted as the semantic predictor to project EEG data into the same dimension to obtain é. Finally, the MSE loss MSE(E E(e,, ex) is employed for aligning EEG and text embeddings.
Dynamic-Aware Noise Adding Process The frames in a video with high OFS are more diverse from each other than those in videos with low OFS. Based on the dynamic decoding results, can roughly classify whether the video is high dynamic or not from EÉG signals. Hence, we introduce the static noise e, and diverse noise ed into the diffusion process and balance them by the decoded dynamic information & The diverse noise e, = {+),
ea) consists of n different noises, each e2 ~ (0, 1). The static noise e, has the same noise e ~ NO0, 1) replicated n times. The ratio of the static noise is smaller when the èd indicates the video is more dynamic. The diffusion process can be formulated as follows to acquire the noise 27 at time steps T:
27 v aT x E0 + vI (V3x BI-Bx xxee) ea),
(2)
where dr is the coefficient for adding noise directly to the noise at time steps T, B is depended by the decoded dynamic information. Here, We set 3 = 0.2 when the video is high dynamic and B = 0.3 when the video is low dynamic.
Video Difffusion Model For reconstructing vivid videos from EEG signals, we utilize the Tune- A- Video technique which fine-tunes an inflated text-to-image diffusion model (33]. The network inflation trick adding sparse temporal attention layer in the image generation model ensure the consistency between frames, in which each frame is calculated with the the first frame and the frame the attention formulated as:
before it. Using the same notations 3
Q = W Zo, K = W [B V WV [Z0
(3)
where !:] denotes concatenation operation, and žn, the i-th frame. To fine-tune the video generation model in our framework, the corresponding text description of each video by feeding the medium frame to a caption model called BLIIP (55). Afterwards, all video-text pairs in the training set are used for fine-tuning the Stable Diffusion Model V1-4 [45).
7

---

Experiment on SEED-DV Dataset
In this section, we evaluate the performance of our methods and other baselines on the proposed two benchmarks: the EEG-VP benchmark and the video reconstruction benchmark. The details of data pre-processing, model implementation and training can be found in Appendix B
5.1
EEG-VP Benchmark
5.1.1 Quantitative Results
We present the overall accuracy of different EEG classifiers in Table 1 Besides raw EEG Signals, we also run experiments on the PSD features and the DE features [6] frequency bands.
From the result, we can see that (1) our GLMNet outperformed the baselines consistently across all the classification tasks, which indicates the importance of extracting iision-associated features for visual perception tasks, (2) Different types of EEG features yield similar results. (3) Different tasks have different difficulties. Some meta information are distinguishable via EEG signals, e.g.. color, dynamics, while the statistical significance indicates that the number of main object and whether human/face appear are difficult or even impossible to be classified. The difference may be attributed to different processing mechanism by our brains which can inspire research in neuroscience.
To conclude, we can answer RQ1 and RQ2: Yes, some of visual information be decoded from EEG signals, including category, color, dynamics. However, the overall visual perception benchmark is rather challenging and calls for more advanced algorithms. Refer to Appendix Cor the confusion matrix and analysis across subjects.
Table 1: Average classification accuracy (%) and std across all subjects with different EEG classifiers on different tasks. Chance level is the percentage of the largest class. The star symbol (*) represents the result is above chance level with statistical significance (two-sample t-test: p < 0.05).
Methods
40cc top-1
401- top-5
9.e top-1
9.2 top-3
Color
Fast/Slow low
Numbers
Human Face
Human
Chance
71.43
Raw LEG Signals
NeillowNei
5.59/2.27
16934.66
21.401.96
49.6272.5
21102209
56.62/1.77
66.15/0.89
64.87/1 54
75.2171
42
4.56/1.522
14.30/1
26.37/1995
55.4280.59
24
ERGNNe
4.640086*
47.04/1
41/1311
51,99/2.00
72.38/099
Conformer
4.93/1,57"
14
20.920098"
49.25/1 ,99
37.
55.02/0.8"
65.7340.26
64.96/1.14 14
73.000 85
TSCom [9
4.92/0.99"
15.05/2.31"
20.00/1.01"
47.76/1.51"
26.89/1.83'
55.32X0.99
65.39/0.41
64.39/1477
72.68/0.67
GLMNot (Our)
17.75/4.24
50.01/2.52'
27.33/1445
57.35/1.98
73.34/1.31
PSDTectuurs Fentures
SVM
3,19/2.81
19:0273.77
21.377
53.361.11
61.15/122
S8391221
70.9171.8
6.20/5.02*
21.59/3.00
61,480092
63.94/113
776
GLMNer (Oums
6.23/2.91
18.98/5.62
21.69/3.20
50.03/4100
26.40/2999'
55.42/1.32
64.68/0922
64.22/1.43
72.27/1.57
DE Features
SVM
4.33/2800
19.0513.9
21,07/288
$3.3421.25
63.62/773
S7.873
70.25/1
MLP
6.12/3.08"
19.02/5.71
21.17/3.24
19.404.94
25/91/3.27"
54.76/1.25
64.100.70
63.41/1.57
71.74/176
GLMNN: (Ours)
6.16/3.18'
19.12/6.07
21.34/3.34'
49.55/4.57
26.15/3.24
55.06/1.200
64.25/0.74
63.63/1.80
72.27/1.58
5.1.2 Analysis of Brain Areas
To find electrodes or brain areas most associated with dynamic visual perception, we conduct one-channel classification task to test the classification quality of each electrode. Due to the reason that only one chan- nel is used, we simplify the task to binary classifica- tion: Human/Animal and Fast/Slow tasks, which related to object recognition and dynamic perception, respectively. can be observed from Figure 4 A) that the electrodes in the occipital area have higher accuracy on Human/Animal tasks, demonstrating the object recognition are related to the occipital area where the visual cortex located, presenting a sim-
Human Animal
8x2
6.00
Perate Occatal Brain region (eblated)
Figure 4: Spatial Analysis. (A-B). Topogra- phies of each electrode's accuracy for Hu- man/Animal and Fast/Slow tasks. (C). Ablate electrodes of different brain regions.
ilar result as previous works [64
19 30 However, not all dynamic visual perception are the occipital region. Figure 4 B) reveals that the brain area associ-

---

ated to movements are around the temporal region where the sensory and motor cortex lies, consistent with the previous neuroscience study [65]
To verify the findings, we conduct the ablation study by removing electrodes from different brain regions and show the 40-c top-1 accuracy in Figure CC). Removing occipital region significantly damages the performance (p < 0.01). The performance also declines without the temporal region.
5.2
Video Reconstruction
5.2.1
Quantitative Results
In this section, evaluate our framework the SEED-DV dataset and three subsets which contains less categories to answer Q3. The quantitative results under four cases are reported in Table 2 With the number of classes increases, the reconstruction performances decrease. Our framework achieves 34.0% of 40-way semantic- accuracy when dealing with the subset contains 10 classes and 15.9% when facing the whole 40 classes. Meanwhile, the structural similarity index measure (SSIM) [66]. which reflects the pixel-level similarity between reconstructed samples and ground truth samples, is up to 0.300 when facing the smallest subset, and drops to 0.256 when facing whole dataset. While the scores cannot be directly compared, our reconstructed videos have very similar and even higher SSIM to that reconstructed by IMRI reported [32], intuitively demonstrating the capability of EEG signals to reconstruct dynamic visual perceptions. And it is also worth noting that with this more sophisticated model, the generated videos have much higher 40-way classification accuracy than that reported in EEG-VI VP benchmark, highlighting the potential to advance this research direction.
Based the findings from the results of the EEG-V VP benchmark (the statistic significance proves that Category, Color, and Dynamics can be decoded from EEG signals), we design the DANA module for injecting the Fast/Slow into diffusion process, the semantic predictor inject Category information, and the general Seq2Seq for decoding low-level visual information like Color. We further conduct the ablation study by removing the Seq2Seq module and the DANA process respectively, and can see huge performance drop without either module. This indicates that capturing the dynamics of both EEG and video is crucial for successful video reconstruction.
Table 2: Quantitative results of each methods on different size of subsets. Standard deviation is calculated across random seeds.
# Classes
Video-based
Frame-basee based
Metrics
Semantic-level
Semantic-level
Pixel-level
Models
2-way
40-way
2-way
40-way
SSIM
Full Model
0.852+20.02
0.340-10.01
0.798:10.00
0.232-20.02
0.300+0.05
10
w/o Seq2Seq
0.772++0.02
0.117:00.1
0.696:t0.02
0.155=00.03
0.187=+0.0
w/o DANA
0.803++000
0.1833+0.01
0.679:+0.02
0.092=0000
0.292++0.05
Full Model
0.813=0.02
0.273
0.785
0.184
0.242++0:03
20
w/o Seq2Seq
0.800
0.119:10.01
0.685
0.09
0.187
w/o DANA
0.801
0.240
0.784.10.01
0.164
0.201
Full Model
0.794+0.02
0.209-00.05
0.785++0.04
0,1 00.002
0.228+0004
30
w/o Seq2Seq
0.770=0.02
0. 0.122
0.741+0.02
0.173
0,207=+0.0
w/o DANA
0.789=0.02
0.160++0.02
0.769
0.139
0.198
Model
03
0.159-10.01
0.774.1:002
00
0.256:20.5
40
w/o Seq2Seq
0.786=0003
0.113++0.01
0.734-1:0.02
0. 12=20.11
0.189:0.05
w/o DANA
0.770=0.02
0.1
0.732++0.03
0.109=00.3
0.217+0.02
5.2.2
Reconstructed Examples
We present some visual examples Figure can be observed that various videos are reconstructed, and as long as the semantic and low-level visual information is correctly predicted by the model, the downstream diffusion process can generate vivid and high-quality videos. More examples including failure cases with full 6 frames can be found in Appendix D

---

Ours
Figure 5: Reconstruction Presentations. Various video clips with low dynamics (e.g., Mountain, Beach, Face) and high dynamics (e.g.. Skiing, Fireworks, Dancing) across animals, scenes, persons, and activities can be correctly recovered.
6 Conclusion
In this paper, we developed the large dataset, SEED-DV, to reconstruct videos from EEG signals, upon which we built the EEG Visual Perception Classification benchmark and the Video Reconstruction benchmark to support evaluating the advances of EEG-based video reconstruction. Moreover, proposed a novel baseline EEG2 Video for video reconstruction from EEG signals that can align visual dynamics with EEG based on the Seq2Seq architecture, and we presented vivid generated examples by training our framework on SEED-DV.
As the first attempt, we open new possibility for BCI researchers to decode dynamic visual percep- tion from EEG signals. Although the overall performance on SEED-DV are still in a preliminary stage, we hold the strong belief that the game-changing results for the BCI area can soon be discovered.
Broader Impacts
Reconstructing dynamic visual perception from brain activities helps to advance the understanding of our visual system in brains. EEG is physiological signal widely used in clinical practice and brain-computer interfaces. Compared to non-portable and expensive neuroimaging techniques like fMRI and MEG, work provides convenient and cheap solution for decoding visual information from brain activities. This technique can used for visualize our mind, offering a novel approach for listening the inner world of people patients with mental illnesses like autistic and depression.
However, every coin has two sides. Personal privacy may leak through our brain activities and be abused by malicious attackers for reading one's 's mind from EEG signals without aeknowledgment More strict regulations are supposed be made for protecting the privacy of people's biological data by government and medical institutions.
Limitations
Our SEED-DV dataset currently records the EEG signals of each subject with one session. leading to the requirement for collecting more EEG-Video pairs with more sessions, which is significant for studying the stable neural patterns over time. Our framework evaluated under the subject-dependent settings, the cross-subject ability remains unexplored due to individual variations. Future work be focused on the transferability and stability the video reconstruction framework by exploiting generalizable EEG encoders and Seq2Seq model.
10

---

Acknowledgments
This work was supported in part by grants from National Natural Science Foundation of China (Grant No. 62376158). STI 2030-Major Projecus+2022Z20088500 Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZD ZX), Shanghai Pujiang Program (Grant No. 22PJ1408600), Medical-Engineering Interdisciplinary Research Foundation of Shanghai Jiao Tong University "Jiao Tong Star" Program (YG2023ZD25, YG2024ZD25 and YG2024QNA03), Shanghai Pilot Program for Basic Research Shanghai Jiao Tong University (No. 2ITQ1400203) and GuangCi Professorship Program of RuiJin Hospital Shanghai Jiao Tong University School of Medicine.
References
[1] R. Hendee and P. N. Wells, The perception of visual information. Springer Science & Business Media, 1997.
[2] K. Grill-Spector and R. Malach, "The human visual cortex," Annu. Rev. Neurosci,, vol. 27, pp. 649-677, 2004.
[3] F. Tong, "Primary visual cortex and visual awareness," Nature Reviews Neuroscience, vol. 4, no. 3, pp. 219-229, 2003.
[4] D. H. Hubel and T. N. Wiesel, "Receptive fields of single neurones in the cat's 's striate cortex." The Journal of Physiology, vol. 148, no. 3, 574, 1959.
[5] H. Jang and F. Tong, "Improved modeling of human vision by incorporating robustness to blur in convolu- tional neural networks," Nature Communications, vol. 15, no. 1. p. 1989, 2024.
[6] S. Lees, N. Dayan, Cecotti, P. McCullagh, Maguire, Lotte, and D. Coyle, "A review of rapid serial visual presentation-bast brain-computer interfaces." " Journal of neural engineering, vol. 15, no. 2, p. 021001, 2018.
[7] S. A. Engel, D. E. Rumelhart, B. A. Wandell, A. T. Lee, G. H. Glover, E.-J Chichilnisky, M. N. Shadlen er al., "fmri of human visual cortex," Nature, vol. 369, no. 6481, pp. 525-525, 1994.
[8] K. Han. H. Wen, Shi, K.-H. Lu, Zhang. D. Fu, and Z. Liu, "Variational autoencoder: An unsupervised model for encoding and decoding fmri activity in visual cortex," Neurolmage, vol. 198, pp. 125-136, 2019.
[9] F. Moradi, L. Liu, K. Cheng, R. A. Waggoner, K. Tanaka, and A. A. loannides, "Consistent and precise localization of brain activity in human primary visual cortex by meg and fmri," Neuroimage, vol. 18, no. pp. 595-609,966666 2003.
[10] P. Adjamian, A. Hadjipapas, G. R. Barnes, A. Hillebrand, and I. E. Holliday, "Induced gamma activity in primary visual cortex is related to luminance and not color contrast: An meg study," " Journal of Vision, vol. 8, no. 7, pp. 4-4, 4, 2008.
[11] P. Ramkumar, B. C. Hansen, S. Pannasch, and C. Loschky, "Visual information representation and rapid-scene categorization are simultaneous across cortex: An meg study," Neuroimage, vol. 134, pp. 295-304, 2016.
[12] N. A. Busch, J. Dubois, and VanRullen, "The phase of ongoing eeg oscillations predicts visual perception," " Journal of neumscience, vol. 29, no. 24, pp. 7869-7876, 2009.
[13] A. Т. Gifford, K. Dwivedi, G. Roig, and R. M. Cichy, "A large and rich EEG dataset for modeling human visual object recognition," Neurolmage, vol. 264, p. 119754, 2022.
[14] D. Regan, "Some characteristics of average steady-state and transient responses evoked by modulated light." Eleetroenccehhalogrupyy and clinical neurophysiology. vol. 20, no. 3, pp. 238-248, 1966.
[15] N. K. N. Aznan, S. Bonner, J. Connolly, N. Al Moubayed, and T. Breckon, "On the classification of ssvep-based dry-eeg signals via convolutional neural networks." in 2018 IEEE international conference on systems, man, and cybermetics (SMC). IEEE, 2018 pp. 3726-3731.
[16] S. Bagchi and D. R. Bathula, "Feg-coontranssoommr for single-trial cog-hased visual stimulus classification,"" " Pattern Recognition, vol. 129, p. 108757, 2022.
[17] D. Veniero, J. Gross, S. Morand, F. Duecker, A. T. Sack, and G. Thut, "Top-down control of visual cortex by the frontal eye fields through oscillatory realignment," Nature communications, vol. 12, no. 1, 1757, 2021.
11

---

[18] T. Grootswagers, I. Zhou, A. K. Robinson, M. N. Hebart, and T. A. Carlson, "Human EEG recordings for 1,854 concepts presented in rapid serial visual presentation streams," Scientific Data, vol. 9, no. 1. p. 2022.
[19] Y. Song, B. Liu, I., Shi, Y. Wang. and X. Gao, "Decoding natural images from EEG for object recognition," " The Twelfth International Conference on Learning Representations, 2024, [Online]. Available: https://openceeee..ee//foummm [
[20] Y. Takagi and S. Nishimoto, feigutresolution image reconstruction with latent diffusion models from human brain activity," in Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), 2023. pp. 14453- 14463.
[21] 7. Chen, J. Qing, T. Xiang, W. I., Yue, and J. H. Zhou, "Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2023, pp. 22 22710-22720 720.
[22] Sun. M. Li, Chen, Y. Zhang, S. Wang, and M.-F. Moens, "Contrast, attend and diffuse to decode high- resolution images from brain activities," Advances in Neural Information Processing Systems (NeurIPS) (NeurlPS), vol. 36, 2024.
[23] B. Zeng. S. Li. X. Liu, S. Gao, X. Jiang, X. Tang, Y. Hu, J. Liu, and B. Zhang, "Controllable mind visual diffusion model," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 7, 2024, pp. 6935-6943.
[24] P. Scottii, A. Banerjee, J. Goode, Shabalin, A. Nguyen, A. Dempster, N. Verlinde, F. Yundler, D. Weisherg. K. Norman et al., "Reconstructing the mind's eye: fmrii-t--mmaee with contrastive learning and diffusion priors." Advances in Neural Information Processing Systems (NeurIPS). vol. 36, 2024.
[25] Xia, de Charette, C. Oztireli, and J.-H. Xue, "Dream: Visual decoding from reversing human visual system," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024. pp. 8226-8235.
[26] F. Ozcelik and R. VanRullen, "Natural scene reconstruction from fmri signals using generative latent diffusionn" Scientific Reports, vol. 13, no. 1. 15666, 2023.
[27] P. S. Scotti, M. Tripathy, C. K. T. Villanueva, R. Kneeland, T. Chen, A. Narang, C. Santhirasegaran, J. Xu, T. Naselaris, K. A. Norman et al., "Mindeye2: Shared-subjer models enable firri-to-maan image with hour of data," arXiv preprint arXiv:2403.I /207, 2024.
[28] Y. Benchetrit, H. Banville, and J.RR. King, "Brain decoding: toward real-time reconstruction of visual perception." " The Twelfth International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://peerrviiowneef/ooo wwt.://wwwww/ww/n//...m 3d=3y KóbuO8c
[29] Y. Bai, X. Wang, Y.pp Cao, Y. Ge, C. Yuan, and Y. Shan, "Dreamdiffusion: Generating high-quality images from hrain EEG signals," arXiv preprint ar ar3iv:23006. 16934, 2023.
[30] Y.TT. Lan, K. Ren, Y. Wang, W.-L. Zheng, D. Li, B.-L. Lu, and L. Qiu, "Seeing through the brain: image reconstruction of visual perception from human brain signals," arXiv preprint arXiv:2308. 025100 2023.
[31] 7. Chen, J. Qing, and J. H. Zhou, "Cinematic mindscapes: High-quality video reconstruction from brain activity," Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2024.
[32] J. Sun, M. Li. Z. Chen, and M.-F Moens, "Neurocine: Decoding vivid video sequences from human brain activties." arXiv preprint or arxi:22402.01599 015000 2024.
[33] J. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. W. Hsu, Y. Shan, X. Qie, and M. 7. Shou, "Tune-a-video: One-shot tuning of image diffusion models for text-to-viden generation," in Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023, pp. 7623-7633.
[34] Y. Miyawaki, Uchida, O. Yamashita, M.a. Sato, Morito, H. Tanabe, N. Sadato, and Kamitani, "Visual image reconstruction from human brain activity using a combination of multiscale local image decoders," Newron, vol. 60, no. 5, pp. 915-929 2008
[35] T. Naselaris, R. J. Prenger, K. N. Kay, M. Oliver, and J. L. Gallant, "Bayesian reconstruction of natural images from human brain activity," Neuron, vol. 63, no. 6, pp. 902-915, 2009.
[36] G. J. Brouwer and D. J. Heeger. "Decoding and reconstructing color from responses in human visual cortex," Journal of Neuroscience, vol. 29, no. 44, pp. 13 992-992-49 143, 2009.
12

---

[37] S. Schoenmakers. M. Barth. T. Heskes, and M. Van Gerven, "Linear reconstruction of perceived images from human brain activity." Neurolmage, vol. 83, pp. 951-961, 2013.
[38] R. Beliy; G. Gaziv, A. Hoogi, F. Strappini, T. Golan, and M. Trani, "From voxels to pixels and back: Self-supervision in natural natural-image reconstruction from fmri,"" Advances in Neural Information Processing Systems (NeurlPS), vol. 32, 2019.
[39] G. Gaziv, R. Beliy, N. Granot, A. Hoogi. F, Strappini, T. Golan, and M. Trani, "Self-supervised natural image reconstruction and large-scale semantic classification from brain activity," Neurolmage, vol. 254, p. 119121,0222
[40] S. Lin, T. Sprague, and A. K. Singh, "Mind reader: Reconstructing complex images from brain activities," Advances in Neural Information Processing Systems (NeurlPS), vol. 35, pp. 29 624 29 636, 2022.
[41] F. Ozcelik, B. Choksi, M. Mozafari, L. Reddy, and R. VanRullen, "Reconstruction of perceived images from firr patterns and semantic brain exploration using instance-conditionn gans," in 2022 International Joint Conference on Neural Networks (JCNN). IEEE, 2022, 1-8.
[42] L. Meng and Yang. "Semantics- guided hierarchical feature encoding generative adversarial network for visual image reconstruction from brain activity," JEEE Transactions on Neurul Systems and Rehabilitation Engineering, vol. 32, pp. 1267-1283, 2024.
[43] S. Palazzo, C. Spampinato, Kavasidis, D. Giordano, and M. Shah, "Generative adversarial networks conditioned by brain signals." in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. pp. 3410-3418.
[44] Kavasidis. S. Palazzo. C. Spampinato, Giordano, and Shah, "Brain2image: Converting brain signals into images," in Proceedings of the 25th ACM International Conference on Multimedia, 2017. pp. 1809-1817.
[45] R. Rombach, Blattmann, D. Lorenz, Esser, and Ommer, "High- renghuresotttio image synthesis with latent diffusion models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition (CVPR), 2022. pp. 10 69.
[46] G. Kim, T. Kwon, and C. Ye, "Diffusionclip: Text-guided diffusion models for robust image manipula- tion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp.
[47] Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 6840-6851, 2020.
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. Clark et al., "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning (ICML). PMLR, 2021, pp. 8748-8763.
[49] C. Spampinato, S. Palazzo, I. Kavasidis, D. Giordano, N. Souly, and M. Shah, "Deep learning human mind for automated visual classification." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6809-6817.
[50] R. Li, J. S. Johansen, H. Ahmed, T. V. Ilyevsky, R. B. Wilbur, H. M. Bharadwaj. and J. M. Siskind, "The perils and pitfalls of block design for EEG classification experiments," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 316-333, 2020.
[51] H. Ahmed, R. B. Wilbur, H. M. Bharadwaj. and J. M. Siskind, "Object classitication from randomized EEG trials." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3845 3854.
[52] H. Wen, J. Shi, Y. Zhang, K.-H. Lu, J. Cao, and Z. Liu, "Neural encoding and decoding with deep learning for dynamic natural vision." Cerebral Cortex, vol. 28, no. 12, pp. 4136-4160, 2018.
[53] G. Kupershmidt, R. Beliy, G. Gaziv, and M. Irani, "A penny for your (visual) thoughts: Self Self-supervised reconstruction of natural movies from brain activity." arXiv preprint arXiv:2206.03544 2022.
[54] C. Wang. H. Yan, W. Huang. J. Li, Y. Wang. Y.S.. Fan, W. Sheng, T. Liu, R. Li, and H. Chen, "Recon- structing rapid natural vision with immi-condition video generative adversarial network." " Cerebral Cortex. vol. 32, no. 20, pp. 4502-4511, 2022.
[55] J. Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-iman -mauua pre- ore-training for unified vision- language understanding and generation," in International Conference on Machine Learning (ICML). PMLR, 2022, pp. 12 1888-129000
13

---

[56] H. Intraub. "Rapid conceptual identification of sequentially presented pictures." Journal of Experimental Psychology: Human Perception and Performance, vol. 7, no. 3. p. 604, 1981.
[57] K. N. Kay. T. Naselaris, R. J. Prenger, and J. L. Gallant, "Identifying natural images from human brain activity." Nature, vol. 452, no. 7185, pp. 352-355, 2008.
[58] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, "EEGnet: compact convolutional neural network for EEG-based brain-computer interfaces." Journal of neural engineering, vol. 15, no. 5, p. 056013, 2018.
[59] Y. Song. Q. Zheng. B. Liu, and X. Gao, "EEG conformer: Convolutional transformer for EEG decoding and visualization," IEEE Trunsactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 710-719, 2022.
[60] A. Vaswani. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[61] A. Hyvärinen, "New approximations differential entropy for independent component analysis and projection pursuit," Advances in Neural Information Processing Systems (NeurIPS), vol. 10, 1997.
[62] R. T. Schirrmeister, Springenberg. L. J. Fiederer, Glasstetter, K. Eggensperger, Tangermann. F. Hutter, W. Burgard, and T. Ball, "Deep learning with convolutional neural networks for EEG decoding and visualization." Human brain mapping, vol. 38, no. 11, pp. 5391-5420, 2017.
[63] C. Cortes and V. Vapnik, "Suppprt-vectoor networks," Machine learning, vol. 20, pp. 273-297, 1995.
[64] P. Bao, She. M. McGill, and Y. Tsao, "A map of object space primate inferotemporal cortex." Nature, vol. 583, no. 7814, pp. 103-108, 2020.
[65] H. Maunsell and D. C. Van Essen, "Functional properties of neurons in middle temporal visual area of the macaque monkey. ii binocular interactions and sensitivity to binocular disparity," Journal of neurophysiology. vol. 49, no. 5. pp. 1148-1167, 1983.
[66] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE transactions on image processing, vol. 13, no. 4, pp. 600-612, 2004.
[67] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenct: A large-scale hicrarchical image database," in 2009 IEEE conference computer vision and patlern recognition. leee, 2009, pp. 248-255.
[68] Z. Tong, Y. Song, J. Wang, and L.. Wang, "Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training," Advonces in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 11007-10099) 2022.
[69] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijaynnaasiimhan F. Viola, T. Green, T. Back, P. Natsev al., "The kinetics human action video dataset," arXiv preprint 1785v:1705.06550. 2017.
[70] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," arXiv preprint arXiv: 20X00:0010002502, 2020.
[71] E. Orchard-Mills, D. Alais, and E. Van der Burg, "Cross-modal associations between vision, touch, and audition influence visual search through top-down attention, not bottom-up capture," Attention, Perception, & Psychophysics, vol. 75, pp. 1892-1905, 2013.
[72] J. Gu, B. Liu, X. Li, P. Wang, and B. Wang. "Cross-modal representations in early visual and auditory cortices revealed by multi-voxel pattern analysis." " Brain Imaging and Behavior, vol. 14, no. 5, pp. 1908- 1920, 2020.
[73] V. Marian, S. Hayakawa, and S. R. Schroeder, "Cross-modal interaction between auditory and visual input impacts memory retrieval," Frontiers in Neuroscience, vol. 15, p. 661477, 2021.
14

---

A Evaluation Metrics for Video Reconstruction Benchmark
In this section, detail the metries use for the video reconstruction benchmark. The metrics to evaluate the quality of generated videos can be roughly classified as frame-based metrics and video-based metrics.
Frame-based Metries Two levels of metrics are considered to judge the quality of generated frames: the pixel-level and the semantics-level metrics. For the pixel level, we calculate the average structural similarity index measure (SSIM) [66] of each frame between the ground-truth video and the reconstructed video. For the semantic level, CLIP-based classifier [48] trained on ImageNet [67] is adopted to compute the N- N-way top-K accuracy of predicted frames. If the ground-truth class is within the top-K probability of the predicted frames classification results from N arbitrary classes (including ground-truth class), the semantic-leve. reconstruction is regarded successful.
Video-based Metric As the ImageNet classifier is unable to well understand videos, VideoMAE- based [68] video classifier trained on Kinetics-40K dataset [69], which can understand 400 dynamic concepts (e.g.. changes, human motions), is applied to compute the video semantic-level accuracy,
B Experiment Setup and Implementation Details
B.1 EEG Signals Preprocessing
The raw data was recorded with the 62-channel EEG cap with a sample rate of 1000 Hz and stored in the continuous EEG data file format (.ent), single file for the experiment of each subject. We applied the 0.1-100 Hz band pass filter to filter out the DC interference and very high-frequency interference and down-sampled the EEG data to 200Hz accelerate computations. A one-second sliding window with 500 ms overlapping is used for EEG segmentation and frequency feature extraction. Specifically, power spectral density (PSD) and differential entropy (DE) [6] of five frequency bands (6: 1-4 Hz, 0: 4-8 Hz, a: 8-14 Hz. 9: 14-31 Hz, and y; 31-99 Hz) are extracted.
B.2 Classification Experiment Details
We perform 7-fold cross- validatiin and report the average accuracy. Specifically, we select each single video block as the testing set one by one, the block before testing set as the validation set, and the remaining 5 blocks compose the training set. Several EEG classifiers, including ShallowNet, DeepNet (62.. EEGNet [58], Conformer [59], and TSConv [19, are adopted as baselines for processing raw EEG. For frequency features, we compare the performance of support vector machine (SVM) [63] and multilayer perceptron (MLP). In our experiment, Our GLMNet uses a concise global encoder, which has the same architecture of ShallowNet on raw signals, and MLP on frequency featrues. All models are implemented with PyTorch and evaluated on an Nvidia A 100 GPU. Adam optimizer used with the learning rate r = 0.001. Batch size is set to 256 for all methods, and the number of training epochs is 100.
B.3 Video Reconstruction Experiment Details
We reconstruct a two-second video clip from the corresponding two-second EEG segment. For efficient training and testing, down-sampled the 24 FPS 1080p original videos to small video of resolution of 512 x 288 (16:9) with 3 FPS, resulting in 6 frames for each video. A 500 ms sliding window with 250 ms overlapping is used for EEG segmentation, forming total of 7 segments in an EEG embeddings Geng E T xx where is the embedding dimension and we set = 512.
We use the first 6 blocks from all the sessions as the training set and the last blocks the testing set in our experiment. The Transformer used as the Seq2Seq model has encoder layers and decoder layers. Semantic predictor has 4 layers with ReLU activation. Dynamic predictor is GLMNet. An Adam optimizer with learning rate of 0.0005 and cosine scheduler was adopted for training the above models with 200 epochs. The inflated video generation model is fine- tinettund using the same setting in B3] on the training set with learning rate of 0.00003 and cosine scheduler for 200 epochs, which takes about hours. The inference is performed with 100 DDIM 770 steps.
Limited to the low signal-to-noise ratio (SNR) and spatial resolution, decoding videos from EEG is somehow difficult. Thus, besides applying the full dataset of 40 concepts, select (Cat, Shark,
15

---

Flower, Dancing, Face, Buildings, Road. Pizza. Cuitar, Airplane) to form 10-class subset, and the first [1-20] categories and the first [1-30) categories form other subsets whose sizes 20 and 30.
C More Results on EEG-VP benchmark
C.1 40-class Classification Task
We plot the confusion matrices of GLMNet the 40-class task. It can be seen that though the accuracy is not high, there is a faint diagonal lines in both confusion matrices. Moreover, small square in the right bottom corner is being observed, of which categories are (Drum, Guitar, and Piano) (32 34 class). This discovery may caused by the reason that the musical instruments stimulate the auditory cortex in our brains with these visual cues, well-studied phenomenon named cross-modal perception 7713 which then has similar reflections EEG signals.
A
GLMNet on DE Features
GLMNet on raw EEG signals
+0.175
0.200
10
10
True Label
2.050
0.000
35
2.025
35-
0.025
2,000
20
0.000
Predict Label
Predict Label
Figure 6: Confusion Matrices of GLMNet (A). The performance using DE features. (B). The performance using raw EEG signals.
C.2 Performance of Each Subject
EEG signal is notorious for its large variations among different subjects. In order to verify whether the dynamic visual perception decoding generalize across different people, we plot the results on the EEG-V VP benchmark of all the subjects separately in Figure 7 Generally speaking, model trained on all the individual subject can achieve reasonable results across different tasks, demonstrating that the phenomenon of visual perception information being contained in the FEG is common across different people. Meanwhile, we can see the differences visual specificity contained in the EEG signals among different subjects. For example, EEG from Subject 3 predicts well on the class of the objects, EEG from Subject 4 and are more sensitive color information, and EEG from Subject is better at capturing motor information.
D More Reconstructed Samples
We display more reconstruction results for demonstrating the effectiveness of our EEG2Video framework. It can be observed that various videos across animals, plants, people, and activities. Please kindly refer to the supplementary files to find GIFs of these reconstructed videos.
Some failure samples are displayed in Figure 13 These failures are typically caused by the inability of the model to infer either semantic information or the low-level visual information correctly, can still see from these failed examples
resulting the irrelevantly generated videos. However, that the model reconstructs some features of the real video, from shapes, movements, to the scene dynamics of the vidco. For example, the man practicing boxing is reconstructed as a panda practicing
16

---

51
51
51
110
516
318
512
51
518
50
54
510
19
10
510
715
515
57
515
113
112
510
512
510
320
317
510
40-class Classification
9-class Classification
Color
Fast/Slow
520
536
516
218
50
54
97
510
337
25
56
516
56
516
715
515
57
314
11
213
517
513
512
$20
51:
510
Number
Human Face
Human
Figure 7: Performance of each subject on different tasks on EEG-VP benchmark.
boxing, the ship sailing on the sea is reconstructed shark, and the fast-moving car reconstructed fast-moving person.
E Algorithm of EEG2Video
In this section, we write the algorithm of HEG2Video framework in Algorithm
Algorithm Training Stage of ViGov Framework
Input: (1) training set Dtrain { ), where is EEG segments, v; is video clips, d; is the fast/slow label (2) stable diffusion model T2I, whose VAE encoder Evee (3) image caption
model B
Output: (1) video diffusion model T2V, (2) Seq2Seq model Seq2Seg, (3) semantic predictor P..
(4) dynamic predictor Pa
1: Initialize text prompts training dataset == {4:}
2: for each (w:) E Dirain do
3: 04 fe...........
4: t, + BBF(
5: end for
6: Initialize latent vectors of all frames L {2;}
7: for each (w:) E Dtrain do
8:
f2. fn)
9: 2; = (4,42.....
for each (fz) E Uj do
11:
1, + VAF(J))
12: end for
13:
end for
Fine-tune the 72/ with {v,, t;} obtain video diffusion model 72V
15: Train the Seq2Seq model Seg2Seq with all {;} using MSE loss
Train the semantic predictor P. with all {æ, f.) using MSE loss
17: Train the dynamic predictor Pa with all {æ, d,} using Cross Entropy loss
return T2V, Seq2Seq, P., Pa
17

---

GT
Ours
GT
Ours
GT
GT
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 8: Various videos reconstruction samples.
18

---

GT
Ours
GT
Ours
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 9: Various videos reconstruction samples.
19

---

Ours
GT
Ours
Ours
Ours
GT
Ours
Ours
Ours
GT
Ours
Figure 10: Various videos reconstruction samples.
20

---

GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 11: Various videos reconstruction samples.
21

---

GT
Ours
GT
Ours
GT
Ours
Ours
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 12: Various videos reconstruction samples.
22

---

GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 13: Some failure samples reconstructed in ours.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We made clear claims of our contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of our research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Our paper dose not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we demonstrate all the information needed to reproduce the main experi- mental results. Morcover, will public our code for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: Yes, we have provide the code for implementing our experiments and part of the datasets (one data and labels) in the supplementary. However, due the size limitation and the principle of anonymity, we are unable upload whole dataset. We will soon publish our dataset and code on our official website of the institute.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https /CodeSubmeei /nipss.c//pullic/guu Socpe///ggiddeeeeCeuSuuubbibbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We demonstrated our method and the experiment settings clearly in Section 5.1. Also, we will public our code of training and testing process.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we present the error bar and statistical significances of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we demonstrated in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss the broader impacts of the EEG2Video technique.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will check the purpose of people who applied our dataset. Our models and data do not have high risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all original papers of EEG models used in experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, our novel dataset is well documented. And the documentation will be provided alongside the dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Yes, we provide the paper include the full text of instructions and sufficient monetary reimbursement subjects, which is above the minimum wage.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: Our experiment has obtained the IRB approvals from our local institution.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "M34tWOHV8b", "reviewer": "Reviewer_nkHY", "summary": "This paper presents a novel framework named EEG2Video for video reconstruction from EEG signals based on Seq2Seq architecture to densely utilize the highly dynamic information in brain signals. It also developed a large EEG dataset named EEG-DV dataset collected from 20 subjects, offering 1400 EEG-video pairs from 40 concepts for studying dynamic visual information in EEG signals.", "questions": "It is suggested to highlight the originality of the proposed method in Section 4. It is also summarize the steps constituting the proposed method as an algorithm.", "limitations": "Yes. The limitations are discussed in Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A large dataset, called EEG-DV, to reconstruct videos from EEG signals, upon which two benchmarks were generated (i.e., EEG Visual Perception Classification benchmark and the Video Reconstruction benchmark) to support evaluating the advances of EEG-based video reconstruction. \n- A novel baseline, called EEG2Video, for video reconstruction from EEG signals that can align visual dynamics with EEG based on the Seq2Seq architecture.", "weaknesses": "- The different steps constituting the proposed method in Section 4 are not well highlighted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below we have addressed your questions and concerns point-by-point.\n\n**Weaknesses**\n\n> The different steps constituting the proposed method in Section 4 are not well highlighted.\n\nThanks for your constructive suggestion, we will add an algorithm to demonstrate the process in our paper. To address your question here, we detail the steps of our proposed EEG2Video below:\n\nTraining: \n\n1. Using video-text pairs in training set to implement an inflated diffusion model T2V, which generates videos using text embeddings $e_t$ and frame latent vectors $z_0$;\n2. Training a Seq2Seq model for mapping EEG embeddings $e_{eeg}$ to frame latent vectors $z_0$, where $z_0$ is obatined by feeding the original frames into the VAE encoder of the stable diffusion;\n3. Training a semantic predictor for mapping $e_{eeg}$ to the corresponding text embedding $e_t$;\n4. Training a dynamic predictor which is a binary classifier for predicting *Fast* or *Slow* from $e_{eeg}$.\n\nInference:\n\n1. Using the Seq2Seq model to get the predicted frame latent vectors $\\hat{z}_0$;\n2. Using the dynamic predictor to predict the *Fast* or *Slow* from $e_{eeg}$ and adopting the Dynamic-Aware Noise-Adding Process for adding noise to $\\hat{z}_0$ to obtain the noise $z_T$ at time steps $T$;\n3. Using the semantic predictor to predict the text embedding $\\hat{e}_t$ from $e_{eeg}$;\n4. Using the T2V model to generate the videos with predicted $\\hat{z}_0$ (after DANA process is $z_T$) and $\\hat{e}_t$.\n\nWe acknowledge that we didn't allocate more space to highlight our proposed method EEG2Video in our main content. However, it is worth mentioning that our contribution is not limited to the proposed algorithm. The goal of our work is to explore the potential of using EEG signals for brain decoding. As a pioneering work, we build the dataset to support the area, we conduct the EEG-VP benchmark to figure out what kind of visual information we can decode from EEG signals. Based on the empirical findings that it is possible to decode *Color* and *Fast/Slow* from EEG, we develop two modules in EEG2Video: Seq2Seq for *Color* and DANA for *Fast/Slow*. \n\nWe appreciate that you can acknowledge these contributions when summarizing the strengths of our paper. These contributions are logically coherent with each other, and hence, we'd argue that the dataset, classification tasks, recontruction tasks, as well as the method itself, are equally important as our contributions. It is fair for them to having equal presenting space in the main content. But we truly value your suggestion and will add more details to the paper, probably in the appendix in the final version.\n\n**Questions**\n\n> It is suggested to highlight the originality of the proposed method in Section 4. It is also summarize the steps constituting the proposed method as an algorithm.\n\nThanks for your constructive suggestion. We will add an algorithm in the appendix to demonstrate the different steps constitued our proposed EEG2Video as stated above. Please kindly refer to the PDF file in the general response, where we write the algorithm of EEG2Video.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nwDdGz0J9B", "reviewer": "Reviewer_9iBh", "summary": "The authors provide an EEG-video paired dataset, addressing the lack of data for decoding dynamic visual perception tasks from EEG signals. They also propose a dynamic noise-adding perception video reconstruction method for this dataset.", "questions": "In the dataset creation (Fig.1), five different video sequences from the same concept are viewed consecutively, followed by a 3-second hint. Could this approach lead to interference between brain signals from different video sequences? Would having a hint between each video be better? Please explain the rationale for this setup.", "limitations": "See the Weaknesses and Questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.The authors introduce a new EEG-video paired dataset, providing valuable data support for studying dynamic perception using EEG signals.\n2.The dataset includes various classification labels, facilitating the analysis of EEG responses to different shapes, colors, frequencies, and other stimuli.\n3.The authors propose an adaptive noise-adding method for image generation, tailored to different OFS.", "weaknesses": "1.Method innovation: The video generation method primarily comprises modules from previous methods, limiting its innovation.\n2.Comparison methods: The authors should compare their method with other video generation approaches, such as those mentioned in the paper that use fMRI to generate dynamic videos (references [31, 32]), simply replacing fMRI features with EEG features.\n3.The article's focus is scattered: According to the title and abstract, the article should primarily focus on EEG-to-video generation. However, the experimental section offers limited analysis of this task, concentrating more on classification performance. While classification performance and analysis can reflect the dataset's quality, presenting more analysis of video generation results in the main text, rather than in the appendix, might be more appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments, and we'd like to express our appreciation that our contributions of the dataset and the benchmarks are well recognized. Below we address your questions and concerns point-by-point.\n\n**Weaknesses**\n> Method innovation: ...\n\nWe would like to emphasize that our contribution and novelty for the EEG2Video framework, while we're the first to propose the DANA module, are definitely not in implementing these techniques per se. Instead, it lies in applying these techniques in a new and challenging domain: reconstructing visual stimuli from the dynamic brain activity.\n\nOur work is at the intersection of neuroscience and CV, where the focus is not solely on inventing new tricks or models. The main aim is to design a novel framework to tackle the unique challenges of what visual perceptions and how we can decode from EEG with adaptation of state-of-the-art generative models to our specific task. \n\nOn the one hand, the EEG-to-video method is more than a simple adaptation of the fMRI-to-video method because of the significantly lower spatial resolution and higher temporal resolution, thus we naturally become the first to apply the Seq2Seq framework upon the newly proposed neuro-signal decoding task. \n\nOn the other hand, the signal-to-noise ratio (SNR) of EEG is less than fMRI, and we probably need some intermediate visual information to reconstruct high-quality and semantically-correct videos. Hence, based on the findings from the results of the EEG-VP benchmark, we design the dynamic predictor and DANA for injecting the *Fast/Slow* into video generation process, the semantic predictor to inject *class information*, and the general Seq2Seq for decoding low-level visual information like *Color*.\n\nOur contribution is not constrained within EEG2Video framework, even though, we argue that our EEG2Video framework is novel as the Seq2Seq and DANA modules are novel and designed by a prior experimental basis rather than randomly combining existing modules together.\n\n> Comparison methods: ...\n\nActually, we have compared these fMRI-to-video methods [2,3] in our work. The ablation variants *w/o Seq2Seq* is the simple adaptation of [2,3] on EEG-to-video task.\nWe denote the video diffusion model as T2V, fMRI-to-video methods [2,3] all can be decoupled as an fMRI encoder $E_{fmri}$ and T2V, where $E_{fmri}$ maps fMRI data to text embeddings $e_t$, and the reconstructed video V = T2V(e_t).\n\nOur EEG2Video has two more modules besides T2V and $E_{eeg}$: the Seq2Seq model that predicts the frame latent vectors and dynamic predictor for DANA. As the pre-training methods for fMRI data cannot be applied on EEG data, the ablation variants *w/o Seq2Seq* is actually a simple adaptation of [2,3]. The difference between *w/o Seq2Seq* and [2,3] is that $E_{eeg}$ is trained without any pre-training methods.\n\nIt can be seen from Table 2 that Seq2Seq and DANA all enhance the video generation performance on all metrices. In other word, our EEG2Video outperforms previous SOTA fMRI-to-video methods [2,3] on EEG-to-video task.\n\n> The article's focus is scattered: ... .\n\nThanks for your careful thinking of our paper, however, we beg to differ that the focuses of the paper are scattered. Instead, all our contributions detailed in the main content are meticulously crafted around the title, and are logically-inseparable from each other. \n\nThe goal of our work is exploring the possibility of using EEG signals for reconstructing visual perceptions. Due to the low SNR, it's almost impossible to reconstruct video pixel by pixel directly. Indirect method involves decoding intermediate visual information, however, nobody knows the boundary of the decoding ability of EEG at the time the dataset is newly built. \n\nTo this end, we conducted comprehensive experiments on the EEG-VP benchmark to figure out the distinguishable ones, e.g. *Class*, *Color*, *Fast/Slow* compared to indistinguishable ones, e.g. *number of objects*. Until then, we can finally design the modules (DANA for *Fast/Slow*, Semantic predictor for *Class*, etc.) in the EEG2Video framework. Hence, we argue that the dataset, classification tasks, and recontruction tasks are equally important as our contribution. It's fair for them to having equal presenting space.\n\nWe carefully choose the expression \"EEG2Video: **Towards** Decoding ...\" in our title, because as a pioneering work, rather than providing a standalone model, we believe it's more valuable to share our thoughts thoughout the whole research process with the neuroscience community to facilitate the future brain decoding research.\n\n**Questions**\n> In the dataset creation (Fig.1), ... the rationale for this setup.\n\nWe appreciate your careful reading. Actually, we spent lots of time discussing the experimental protocol. including what you're concerning about.\n\nThe interference between brain signals from different video sequences, especially from different class of videos, should definitely be mitigated to the minimum. Therefore, compared to the datasets [4] used by fMRI-to-video works, we decide to add intervals between different scenes. Nevertheless, given the selected number of classes (40) and video clips (1400), if adding intervals after every clip, let's say only 1-second interval, the total length will be 4200s = 1h10min without any break and relaxation. No one can tolerant such an experiment and the fatigue and the distraction will harm the data quality. As a result, we made a compromise to only add an 3-second interval every 5 videos, and add rest phase between blocks for relaxing.\n\n\n[1] J. Wu, et al. “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation”\n\n[2] Z. Chen, et al. “Cinematic mindscapes: High-quality video reconstruction from brain activity”\n\n[3] J. Sun, et al. “Neurocine: Decoding vivid video sequences from human brain activties”\n\n[4] H. Wen, et al. “Neural encoding and decoding with deep learning for dynamic natural vision”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "75PbXQiNW9", "reviewer": "Reviewer_ACPC", "summary": "The authors present a novel annotated dataset of EEG-video pairs and an approach to reconstruct videos from EEG brain activity data. The dataset contains brain responses of 20 subjects watching 2-s videos from 40 general concepts. A total of 7 classification tasks are built based on the metadata available in the dataset (e.g. finegrained concept, coarse concept, color, etc.) and a video diffusion pipeline is trained to reconstruct videos. Classification results on the different tasks are presented, along with generated video frames and image quality metrics.", "questions": "1. What is the architecture and hyperparameters for the EEG encoder (Section 4.2)?\n2. In Section 4.1: What is meant by “treating all channels equally”? Most deep learning encoders trained on EEG data have some kind of spatial processing layer, e.g. a convolutional layer, that learns to reweigh different channels end-to-end [1].\n3. The 40-class classification performance appears very low, however generations are qualitatively very good. Can you describe the process for selecting the generations shown in the paper?\n4. In Table 2, what does 40-way classification refer to when fewer than 40 classes are used?\n\n[1] Schirrmeister, Robin Tibor, et al. \"Deep learning with convolutional neural networks for EEG decoding and visualization.\" Human brain mapping 38.11 (2017): 5391-5420.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Originality: the study of video decoding from EEG has not been the focus of much attention yet, and the presentation of a new dataset is both original and useful to the community.\n* Clarity: the manuscript is overall clearly written and the general approach is well motivated.\n* Quality: interesting analysis of what information can be decoded (color, optical flow, object number, human face, human) through the different classification tasks described in Section 3.5.\n* Significance: the presented dataset and analysis set the stage for more generalizable results in visual decoding from EEG.", "weaknesses": "1. The dataset contains videos spanning 40 concepts which are seen in both training and test sets as described in Section B.2, i.e. there is \"categorical leakage\" between the two sets. This makes it very likely that the model learns to mostly (or solely) predict a concept, rather than predict the finer grained visual information contained in a video. Following this hypothesis, the EEG encoder, the seq2seq model and the semantic predictor could all be replaced by a single classifier that outputs the label of one of 40 concepts, followed by a lookup table that returns the corresponding concept-specific conditioning vector to be fed to the video diffusion model, and generation performance would remain similar. To test whether that is indeed the case, it would be interesting to train the model on e.g. 25 concepts, and test it on the remaining left out 5 concepts (also taking into account the next point about finetuning the video model).\n2. Moreover, as described in Section 4.2, line 244: “[...] all video-text pairs are used for fine-tuning the Stable Diffusion Model [...]”. If that is indeed the case, this means that the video diffusion model has already seen the specific videos it tries to predict later on, which makes the generation task significantly easier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments with expertise. Below we have addressed your questions and concerns point-by-point.\n\n> W1.1 ... \"categorical leakage\" between the two sets.\n\nWe may be misunderstanding your concern, but we argue that \"Categorical leakage\" is a pseudo-concept and should not be considered a problem in machine learning. Conversely, all learnings relies on such shared distributions to achieve **in-domain generalization**. For instance, a diffusion model must be trained on cat images to generate cat images.\n\n**Leakage**, on the other hand, describes the situation where the information that is inaccessible or only accessible during test is used to construct the model in training. Based on your description, the most similar concept may be the label leakage, where the inaccessible label information is added to training steps. However, in our EEG2Video framework, the only input is the EEG data collected when the subject is shown the video. And the \"categorical\" information is inferred by the model implicitly with the semantic predictor, thus won't cause any leakage problem. \n\n> W1.2 This ... mostly (or solely) predict a concept ...\n\nThe pixel-level and higher-level decoding recover visual stimuli from two different perspectives, where the trade-off between fidelity and meaningfulness needs to be considered. \n\nDecoding from EEG is challenging due to several reasons. The classification results shown in Table 1 even struggles to decode information like *Numbers* and *Face*, not to mention finer-grained visual information at current stage. Hence, in this work, we prioritize the video recovery of visual stimuli via intermediate semantics in EEG, which is also crucial for understanding the complex mechanism of human perception. We recognize the contribution of categorical information to generate semantically closer results. Nonetheless, there are more the model can facilitate such as the *color* and *fast/slow* information. All in all, our aim is to establish a foundation that integrates both pixel-level features and visual semantics in this particular task.\n\n> W1.3 Following ... would remain similar.\n\nEssentially, our framework is based on the alignment in both semantic and visual features, and the pre-trained diffusion priors. The predicted dynamic information and other latent clues are also introduced in the diffusion process for enhancing decoding performance. It is definitely different from a simple combination of classifier + look-up dictionary. \n\nThe best classification result of 40 classes is 6.23% in Table 1, which is the upper-bound semantic-level accuracy of a simple combination. However, our framework achieved a semantic-level accuracy of 15.9%, more than twice of that.\n\n> W1.4 To test ... , train the model on e.g. 25 concepts, and test it ...\n\nWe argue that it is an impossible task for the current neuroscience and AI community. Training on N classes and testing on other unseen M classes is called zero-shot transfer, studied by large pre-training model, e.g., [1] pre-trained on $4*10^8$ image-text pairs. Even so, they can only generate images of seen concepts and their combinations.\n\nAs a pioneering work, we focus on providing the basic logistic and the first batch of data with only 40 concepts and 1400 videos. This is far not enough to do zero-shot learning.\n\n> W2 Moreover, ..., the generation task significantly easier.\n\nThanks for your careful reading! The right expression is \"all video-text pairs **from the training set** are used for fine-tuning ...\". We promise that the whole framework including the diffusion model has never seen any videos it tries to predict later during any training stages. We apologize for the confusion and will revise it.\n\n> Q1 ... the architecture and hyperparameters ...?\n\nWe adopt our GLMnet as EEG encoder due to the outstanding visual decoding ability. The hyperparameters are detailed in the appendix B. \n\n> Q2 ... “treating all channels equally”? ...\n\n\"Treating all channels equally\" means adding no inductive bias upon different channels. While deep models learns to reweigh input, correct inductive bias can prevent model from learning from spurious features and improve the generalization ability [2]. In fact, all the modifications on model structure can be treated as some kind of inductive bias.\n\nIn our case, GLMnet adds another data stream to focus on the visual-related channels, which motivates networks to focus on the visual cortex. As a result, it performs better on all the tasks in the EEG-VP benchmark and thus selected as the EEG encoder.\n\n> Q3 The 40-class classification performance appears very low, ...\n\nTo clarify, we assure you that we have presented a representative range of qualitative results of the generated videos. More importantly, we also evaluated the quantitative performance upon all EEG-video pairs in the testing set.\n\nFrom Table 2, thus, we ran our EEG2Video on the testing set and selected the most representative qualitative results for demonstrating the effectiveness. We also selected typical failure samples in Fig. 13 in the appendix including messing up between categories, wrong color, wrong main object, etc. Please kindly refer to Appendix F for failure cases.\n\n> Q4 In Table 2, what does 40-way ...\n\nThe metric is to verify the generated images/videos class with a pre-trained images/videos classifier. For all cases, we adopt the same 40-class classifier to fairly compare the reconstruction performances, though our generative model may be trained on a fewer number of classes. We follow [3] and used the same code for calculating this metric, which has been submitted in Supplementary.\n\n\n[1] R. Alec, et al. \"Learning transferable visual models from natural language supervision.\" in ICML, 2021.\n\n[2] Bo Li, et al., “Sparse Mixture-of-Experts are Domain Generalizable Learners” ICLR 2023, Oral \n\n[3] Z. Chen, et. al. “Cinematic mindscapes: High-quality video reconstruction from brain activity,” in NeurIPS, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sUONTdicIM", "reviewer": "Reviewer_Kswp", "summary": "The manuscript proposes an EEG decoding model that aims to reconstruct the video stimuli presented to participants. To this end, a large dataset corresponding to twenty participants is collected. The dataset is annotated with respect to different features such as the dominant colour of the video or the presence of a human in the video frames. Lastly, they also developed an eeg2video that can be considered as the baseline for this dataset in future research.", "questions": "Why SSIM is not reported for video-based evaluation in Table 2?\n\nThe choices for the nine classes (land animal, water animal, plant, exercise, human, natural scene, food, musical instrument, transportation) seem a bit arbitrary. It would be nice to read the rationale behind these categories.\n\nIt would be helpful to include experimental results to support the benefit of global and local data streams in the proposed model. If we perform an ablation study where the global branch is lesioned, how does the performance change?", "limitations": "The limitations of the work are sufficiently discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem this manuscript addresses is gaining more importance in recent years due to numerous potential applications of brain-computer interfaces. Subsequently, the dataset developed as part of this work could add value to the research community.\n\nThe source code is provided in the supplementary materials, which makes the reproduction of reported results easier.\n\nThe recontacted videos are visually appealing which helps to showcase the potential of EEG decoding.", "weaknesses": "The decoding power that recorded EEG signals offer is questionable with respect to some of the annotations. For instance, the chance level for the \"Human\" task is 71.43 and the best method (the proposed model) only reaches 73.43. Similar observations can be made for almost all other tasks. In some scenarios, the reported accuracies are even below the chance level, for instance using DE features in the \"Numbers\" task, the best-performing model reaches 64.2 and the chance level is at 65.64.\n\nFollowing the point above, it's unclear whether even those cases that are above chance level have any statistical significance as no Student t-test or Wilcoxon Signed-Rank test is conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below we have addressed your questions and concerns point-by-point.\n\n> W1. The decoding power ...\n\nWe appreciate your careful reading. We can't agree more and believe that the decoding power of the recorded EEG signals w.r.t. some specific tasks is *more than* questionable. Actually, in Line 258, we claim them as \"difficult or even **impossible to classify**\". However, we deliberately added experiments on these tasks for the following reason and we beg to differ regarding this point as a weakness. \n\nAs a pioneering work, our goal is exploring the potential of using EEG signals for reconstructing visual perceptions. At the time the dataset is created, nobody knows the boundary of the decoding ability of EEG. We'd like to answer what kind of visual information we can decode from EEG and use it as intermediate clues to further enhance the reconstruction ability of the EEG2Video framework. \n\nTo this end, we not only studied distinguishing among the 40 concepts, but also investigated other decoding tasks, across both low levels (*Color*, *Fast/Slow*, *Number*) and the high levels (*Human*, *Face*). We conducted comprehensive experiments to validate the decoding performance of each task with 7 machine learning methods on raw EEG data and two other human-extracted EEG features.\n\nAs a result, we reach a quick conclusion that *Number*, *Human*, *Face* are difficult and probably impossible tasks. On the other hand, we figure out that it is possible to decode visual information like *Color* and *Fast* from EEG, which guided us to develop some useful modules in our EEG2Video framework for incorporating such information, e.g. semantic predictor for class information, and DANA for *slow/fast* information.\n\nOf course, we can definitely omit the results of indistinguishable tasks in the paper for better presentation. However, we insist to add them in Table 1 and believe they can offer some helpful empirical findings to the neuroscience community and facilitate the future brain decoding research by focusing on only promising semantics.\n\n> W2. Following the point above, ...\n\nThanks for your constructive advice. We conduct the Student t-test and calculate the *p*-value on the performance of all classification tasks using raw EEG of our GLMnet model compared to the chance level. The results are as follows and we will add them in our paper:\n\n||40c t1|40c t5|9c t1|9c t3|Color|Fast|Numbers|Face|Human|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|*p*-value|0.00|0.00|0.00|0.00|0.00|0.00|0.28|0.07| 0.17|\n\nAccording to the statistical significance analysis, it can be concluded that the classification results of the first 6 columns are significantly above the chance level ($p$ <0.005), while there are no significant gap for the last 3 columns ($p$ >0.05). We will take your advice and add the results to strengthen our claims and intuitions.\n\n> Q1. Why SSIM is not reported for video-based evaluation in Table 2?\n\nSSIM is a metric that reflects the structural similarity between two images. To acquire the SSIM between two videos, what the algorithm do is using each frame pair in these two video and computing the SSIM between each frame pair.\n\nIn our paper, we calculate the SSIM of each frame between the ground-truth video and the corresponding reconstructed video. There is no need to add a video-based SSIM since the meanings of frame-based SSIM and the video-based SSIM are the same, all reflecting the pixel-level similarity.\n\n> Q2. The choices for the nine classes seem a bit arbitrary ...\n\nThat is a very interesting question. In fact, we spent lots of time deciding the class to use and before we finalized the list, we had already considered the EEG-VP tasks. Then, the general idea is to use natural videos instead of artificial ones (like anime) and balance different types of videos for EEG-VP tasks as much as we could. Specifically, we referred to several related works including [1][2][3][4] and obtained the list of classes according to the following guidelines:\n1. We remove some static classes that is not suitable to be presented as videos, e.g., golf balls, keyboards, etc.\n2. We would like to involve roughly 1/3 classes with human beings, 1/3 classes with animals and plants, and 1/3 of non-living scenes or objects.\n3. We would like to have roughly 1/2 videos with rapidly changing scenes, and the other half with relatively static objects.\n4. We would like to balance the numbers of the main colors. \n\nAs a result, we obtained classes as described in Figure 1, and present the statistics in Figure 2. However, it's very hard to accurately control the numbers so we call for future work to design a probably more reasonable sets of videos as visual stimuli.\n\n> Q3. It would be helpful ...\n\nThanks for your constructive advice. Actually, the GLMnet's global and local encoder are both simple CNN or MLP. We use the ShallowNet (for raw EEG) or MLP (for EEG features) as our GLMnet's global encoder. ShallowNet and MLP (equal to the ablated GLMnet) all had been compared as baseline models in Table 1.\nIt can be seen that a local encoder can improve the performances for brain decoding tasks. The reason why local data stream works is that it introduce inductive bias in neural networks, which motivates netowrks to focus on the visual cortex.\n\n[1] C. Spampinato, et al. “Deep learning human mind for automated visual classification”\n\n[2] H. Ahmed, et al. “Object classification from randomized EEG trials”\n\n[3] H. Wen, et al. “Neural encoding and decoding with deep learning for dynamic natural vision,”\n\n[4] Allen, et al. \"A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for your proficient and valuable comments and suggestions. We are cheerful to find that all the reviewers have reached the consensus that our datasets are novel and valuable to the research community. Moreover, we're also glad to see that our contributions in building the baselines (nkHY, 9iBh), interesting analysis (ACPC), visually appealing results (Kswp) are well recognized, and the paper is thought to be clearly presented (9iBh, ACPC).

**Here, we sincerely invite all the reviewers to read this general response before diving into the detailed responses to your individual concerns.** To help readers to logically understand our research workflow, we introduce our deep thought on **what the actual problem is**, **how we reach there step by step**, and **what we are contributing**, and most concerns about the scattering contributions, the indirect prediction methodology, the framework, and the allocation of the 9 pages can be naturally resolved. 

1. **The roadmap for EEG2Video**. Ultimately, we, standing between neuroscience and AI, are seeking solutions to reconstruct the dynamic visual stimuli from the EEG signals due to its significantly higher temporal resolution and low latency compared to other brain signals like fMRI. We could easily figure out two paths to achieve it. a) **directly** decoding the pixel-level information of the video from EEG. b) **indirectly** decoding the video via intermediate semantics. As the first attempt, we dispose the direct way almost immediately. From neuroscience perspective, only the primary visual cortex(V1) is related to this very low-level perception, and then there will be only two channels (O1, O2) useful for the decoding. However, the full visual pathways span almost everywhere in our brain to process the information into high-level concepts like colors, motions, recognition, etc., and would be very useful for indirect decoding. From AI perspective, it's even impossible to generate arbitrary pixel combinations with contemporary generative models which involve OOD generalization. As a result, **we want to try our best to facilitate intermediate information to help the video reconstruction in our work.**
2. **The EEG-VP benchmark and the classification result.** Setting the indirect decoding mechanism as the main approach, the next question we need to answer is **what should be the intermediate information to use**. Nobody can answer this question before we build the EEG-DV dataset simply because of the omission of data resources for analysis (By the way, this adds to the value of our dataset and we're glad to see that most reviewers can recognize it). In fact, we had selected the classification tasks of our interests based on our knowledge as early as the stage when we're choosing the videos and we tried to balance different types of videos as much as possible. As a result, findings from the classification task demonstrate that some semantics are possibly helpful such as *Slow/Fast*, *Class* and some are almost impossible to decode such as *number of objects*. These experiments greatly guides our design of the EEG2Video framework and possibly future attempts, thus we insist to include indistinguishable tasks in the paper and the benchmark. In this sense, **the EEG-VP benchmark and the classification results are not only the icing on the cake, but actually a logically indispensable part of our contributions**.
3. **The design of EEG2Video.** Now we finally reach the point where people may be mostly interested in. Our contributions in the framework are two-folded. **The first part comes from the task's property**. For the first time, we can model the stimuli reconstruction from brain signal task into a Sequence-to-Sequence task given the modality of input and output. Also, as for decoding visual stimuli, we emphasize the visual cortex by adding inductive bias to the model and construct the GLMNet. **The second part comes from the findings of the classification results**. We tried to incorporate the distinguishable information by implementing the semantic predictor, dynamic-aware noise-adding process, etc.. We admit that we're not actually inventing new neural network blocks, but selecting the effective inductive bias to the model should definitely be considered as our contribution.

Now we have stated the coherent logic among our contributions in collecting the dataset, building the two benchmarks, and designing the EEG2Video framework. They are equally important to achieve the final goal, so we're not going to reallocate pages for them. Meanwhile, we still value all the suggestions from the reviewers which definitely help we improve the paper quality. Therefore, we will make the following modifications to the paper accordingly:
1. We will add more details about the model architecture and hyperparameter selection in the appendix.
2. We will add the workflow in a format of an algorithm as is shown in the attached .pdf file.
3. We will make some expression more accurate and clear. 
4. We will add the above discussion in the publicly available version, or a link to this page to help the readers better understand it regardless of the final decision of this paper. 

Finally, we would express our greatest appreciation and excitement again that all the reviewers recognize potential of our work to the whole neuroscience and AI, especially the BCI community. The word "*Towards*" in the title expresses our recognition of its position: as a pioneering work, this marks the beginning rather than the end of a new frontier. While the methods and the results are still in a very preliminary stage compared to the final desired goal, we hold the deepest belief that with our dataset, our benchmarks, and our framework, the publication of EEG2Video would open immeasurable possibilities to push the area moving forward.

Meta-review: This paper develops a dataset and first	algorithm for decoding videos
from EEG.  They show that certain features of video (color, motion
speed) can be decoded, but there were other aspects (number of
objects, presence of human face) they were not able to decode with
their network and dataset.

There was a lot of discussion and the authors answered a lot of the
questions.  (One of the reviewers kept their score at 4 as they
believed the changes would be a lot, but agreed that the work is a
valuable contribution - I feel like the changes do not change what has
been done, but just reflect clarifying some of the presentation).
This area is of great interest and appears to be the first decoding
video from EEG.  There are interesting findings and algorithmic design
choices that others in the field can build on.  It is also clearly written. 
I think this work is worthy of NeurIPS presentation and will receive a 
lot of interest.
Decision: Accept (poster)