Decoding Natural Images from EEG for Object Recognition
Authors: Yonghao Song
Bingchuan Liu
Xiang Li
Nanlin Shi
Yijun Wang
Xiaorong Gao
Venue: ICLR 2024 poster (2024)
Keywords: EEG
object recognition
contrastive learning
brain-computer interface
Primary area: applications to neuroscience & cognitive science
TL;DR: This study propose a self-supervised framework to decode natural images from EEG for object recognition, achieving remarkable results in zero-shot tasks with rich biological evidence.

Abstract: Electroencephalography (EEG) signals, known for convenient non-invasive acquisition but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised framework to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Contrastive learning aligns these two modalities by constraining their similarity. Our approach achieves state-of-the-art results on a comprehensive EEG-image dataset, with a top-1 accuracy of 15.6% and a top-5 accuracy of 42.8% in 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plausibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-world scenarios. Code available at https://github.com/eeyhsong/NICE-EEG.

Conference paper at ICLR 2024
DECODING NATURAL IMAGES FROM EEG FOR OBJECT RECOGNITION
Yonghao Song'. Bingchuan Liu', Xiang Li', Nanlin Shi', Yijun Wang", Xiaorong Gaol*
1 Department of Biomedical Engineering, Tsinghua University 2 Institute of Semiconductors, CAS gxr-dea ts
ts inghua edu.cn
ABSTRACT
Eleetroenceephalograpyy (EEG) signals, known for convenient non-invasive acquisi- tion but low signal-to-noise ratio, have recently gained substantial attention due to the potential to decode natural images. This paper presents a self-supervised frame- work to demonstrate the feasibility of learning image representations from EEG signals, particularly for object recognition. The framework utilizes image and EEG encoders to extract features from paired image stimuli and EEG responses. Con- trastive learning aligns these two modalities by constraining their similarity. Our approach achieves state-of-the-ar results on a comprehensive EEG-image dataset, with a top-1 I accuracy of 15.6% and top-5 accuracy of 42.8% in 200-way zero-shot tasks. Moreover, we perform extensive experiments to explore the biological plau- sibility by resolving the temporal, spatial, spectral, and semantic aspects of EEG signals. Besides, we introduce attention modules to capture spatial correlations, providing implicit evidence of the brain activity perceived from EEG data. These findings yield valuable insights for neural decoding and brain-computer interfaces in real-wor world scenarios. Code available
INTRODUCTION
Our daily life relies on accurately and rapidly identifying objects in complex visual environments (DiCarlo & Cox, 2007). Researchers have pursued decoding natural images from brain activity. aiming to deepen our understanding of the brain and create user-friendly brain-computer interfaces (BCIs) (Kamitani & Tong 2005 Kay al., 2008 Gao al. 202T]. Functional magnetic resonance imaging (IMRI), which records blood oxygen level-dependent signals, has been popular choice for categorizing objects observed by humans (Du et al., 2023) Horikawa & Kamitani] [017 Allen al., 2022). Despite its high spatial resolution, fMRI typically requires several seconds to provide stable response to single stimulus, limiting its real-time applicability in daily interactions (Lin et al.,
2022). Similarly, Magnetoenceephalogran (MEG), with high time resolution, has been employed for
this purpose, but is hindered by cost and large devices (Cichy et al. 2014:) Hebart et al. 2023).
Eleetroenccepallogaam (EEG) has emerged evoked brain activities (Spampinato et al., 2017). EEG has high time resolution, low cost, and good portability, but the low to-noi-to-noise ratio takes problems (Pan et al. 2022 Kobler al.) 2022). Some efforts have yielded impressive classification results and captured saliency maps using EEG signals (Palazzo et al. 2021). However, its flawed block-design experiments group images of the same class within one block, resulting in classification that relies on block-level temporal correlation rather than stimulus-relatee activity (Li et al.) 202T). Some studies have achieved above-chance performance but were restricted to much data with one subject data (Ahmed et al.) (02T). Moreover, available datasets often feature limited number of image categories, posing challenges for real- wolld image decoding tasks. The latest study has significantly expanded the situation by amassing large and diverse EEG dataset comprising 16,740 image stimuli spanning 1,854 concepts, employing rapid serial visual presentation (Gifford et al. 2022). While demonstrating the feasibility of image-to- EEG encoding and category separability, this work primarily focused on analyzing electrodes in the occipital and parietal regions, overlooking the inferior temporal cortex plays necessary role in object recognition (Dapello et al. 2023). In addition, the work focuses on the peak of pairwise decoding about 10 after stimuli onset. Generally, this latency involves visual conduction and primary
valuable tool for decoding images based visual-

---

Conference paper at ICLR 2024
visual processing by large margin, insufficient for semantic understanding Xu al. (2023) In short, the issues of multi-class object recognition and biological plausibility warrant further exploration.
Pattern recognition plays a pivotal role in EEG analysis. Existing work has predominantly relied on supervised learning, often dealing with limited data from few categories (Cheng et al. (022). Ap- plying machine learning methods directly, such as support vector machines and deep neural networks, has proven challenging in obtaining intrinsic representations (Fu et al. 2022 Liu et al. 2021). People have performed self-supervised learning and leveraged information from other modalities computer vision research (Wang et al. 2022). Text replaces the traditional hard labels to enable self-supervised learnigg to produce superior image representations, often leading to impressive zero-shot results on downstream tasks (Radford et al. (202)). Researchers have also used contrastive learning to obtain consistent embedding of neural recordings across multiple animals (Schneider al. 2023). Building upon these insights, we explore the intriguing possibility of recognizing visual objects from EEG signals a self-supervised manner. Another issue pertains to the limitations of commonly employed EEG feature extractors, typically structured around convolutional layers applied separately along temporal and spatial dimensions of raw EEG signals (Schirrmeister et al. 2017) Lawhern et al.. 2018 Song et al., 2023). This organization disrupts the correlations between electrode channels, hindering our perceiving spatial properties of brain activity (Ding et al.) 2023).
To address the above limitations, introduce self-supervised framework to decode image repre- sentations from EEG signals, focusing on object recognition. Our approach uses contrastive learning the bridge connecting image stimuli and EEG responses. We feed image-EEG pairs the model and process them by an image encoder and an EEG encoder separately. These modalities are aligned by constraining their cosine similarity, resulting in the ability perform zero-shot EEG decoding after training, achieving remarkable classification accuracy for previously unseen image categories. With the framework, we provide an overall resolving of the visual object recognition processing in our brain from spatial, temporal, spectral, and semantic aspects with comparative experiments. The results are consistent with established neuroscientific knowledge, further demonstrating the feasibility of BEG-based image decoding. Furthermore, we present two spatial modules with self-attention and graph attention that integrate with the EEG encoder (Dosovitskiy et al. 2021 Veličković et al. 20188. helping to emphasize key brain regions relevant object recognition.
Our main contributions can be summarized as follows:
We propose self- supervupee framework for EEG-based object recognition with contrastive learning. Remarkable zero-shot performance has been achieved on large and rich datasets.
We demonstrate the feasibility of investigating natural image information from EEG signals. Extensive experiments affirm the biological plausibility, which brings a resolving of human object recognition from temporal, spatial, spectral, and semantic aspects.
We apply two and-play modules to capture spatial correlations among EEG channels, offering evidence that the model discerns the spatial dynamics of object recognition.
2
RELATED WORKS
Decoding visual information from our brain has been a long-standing pursuit in neuroscience and computer science (Miyawaki et al. 2008 Tia et al.) [021). While some progress has been made in decoding steady-state visual stimuli, the accurate and rapid decoding of semantic information in natural images has remained challenge (Shi et al. 2023). fMRI has been widely used estimate semantic and shape information from visual processing in the brain (Ho et al. 2023 Takagi & Nishimoto, 2023). However, the demand for high-speed and practical applications in compucr interaction necessitates alternative approaches. EEG has emerged as a promising option due to its high temporal resolution and portability (Willett et al.) (021). But general performance on different subjects and biological plausibility remain unsolved (Ahmed et al., 2021). Additionally, prior approaches have often relied on supervised learning methods with limited image categories, overlooking the intrinsic relationship between image stimuli and brain responses (Shen et al. 2019
Li al. 2021 Liu et al.] 2023). These limitations hinder their practicality in real-world scenarios with generalization to new categories. Motivated by these challenges, we present an EEG-based image decoding framework that employs self-supervised learning. enabling the model to achieve zero-shot generalization in object recognition tasks, further demonstrating the feasibility.
2

---

Conference paper at ICLR 2024
A
B
Pre-trained CLIP, VIT
SelfiGraph Att
Image Encoder
Train
Temporal Conv
Contrastive
Avg Pooling
Spatial Conv
EEG Encoder
Test
Linear
Match
Figure 1: (A) Overall framework for EEG-based object recognition. During training, image-EEG pairs are processed by an image encoder (pre-trained) and an EEG encoder. The objective is to increase the similarity between matched pairs while decreasing for unmatched pairs. During testing, few unseen images of target concepts (classes) are processed in advance into templates. Then, the model obtains results by matching test data templates. (B) Architecture of the EEG encoder. Temporal-spatial convolution is used with spatial modules, made with self and graph attention, to reveal spatial features brain activity. The linear layer used to project the feature dimension.
3 METHODS
3.1
OVERVIEW
We propose self-supervised framework, Natural Image Contrast EEG (NICE), to decode images from EEG signals. The overall framework depicted Fig. AA).
During training, stimulus-respon response pairs comprising images and EEG signals are fed into the frame- work. The image and EEG encoder extract features from their respective modalities. Contrastive learning is employed to align these features, optimizing the similarity between matched pairs and reducing it for unmatched pairs. This self-supervised approach enables the model to learn shared information between the modalities instead of traditional supervised learning with predefined labels. Before the test process, we use few images that belong to the image concepts (classes) to be classified, and have not appeared in the EEG collection experiments. These images are processed and averaged to obtain one template for each concept. Next, the EEG encoder processes the test signals, and their similarity with all templates calculated for matching results.
For the EEG encoder, we adapt semporal-spatioe convolution (TSConv), which is widely used in EEG analysis, as shown in Fig. BB). In addition, we integrate two plug-and-play spatial modules with self-atrention and graph attention. These modules are instrumental in preserving the spatial characteristics EEG channels, reflecting intrinsic patterns brain activity. Note that the image encoder has been pre-trained other image datasets. The leading studies in computer vision help us with larger sample space, when we cannot casily collect numbers brain responses.
3.2
NETWORK ARCHITECTURE
3.2.
EEG ENCODER
Researchers usually arrange the raw EEG trials into two dimensions C T, in which C denotes electrode channels, and T denotes time samples. Convolution along the two dimensions is widely used in deep learning-basse EEG analysis models. In the same way, we present a very concise network architecture as shown in Table One-dimension convolution is first applied to capture the temporal features with kernels of size (1, m1) and stride of (1, 1). An average pooling layer with kemne size of (1, m2) and stride of (1, introduced to alleviate overfitting and smooth the temporal features. Another one- dieddmmensoon convolution is then used for spatial features keeping kernels of size (ch, 1) and stride of (1, 1), where ch usually equals the number of electrodes. After
3

---

Conference paper at ICLR 2024
Table 1: Architecture of EEG encoder with SA GA followed by TSCon
Layer
In
Out Kernel
Stride
Dimension
SA GA
[1 C,T) + C,,, batch
Temporal Conv
1
k
(1.m))
(1.
(b, C.T mi
Avg Pooling
k
&
(1.ma)
(1,6) (b,k, (T' my ma + 1)/s +
Spatial Conv
&
(C,1) (1,1) (6,
ma + 1)/a m2 1)/e+ + 1)-+ shape of image feature]
Flatten&
[t * ((T my
each convolutional layer, batch normalization and exponential linear units (ELU) activation functions used for better training and nonlinearity (Clevert al. 2016). Finally, linear layer is added as projector to transform the features to the same size as the output of the image encoder.
3.2.2 IMAGE ENCODER
The image features are extracted with the pairing EEG features for contrastive learning. There have been some excellent computer vision models that can obtain image features with semantic discrimination. We prefer directly using models pre-trained on other image datasets get large sample space. The feature extraction parts Contrastive Language- Image Pre-training (CLIP) (Radford al. [2021), Vision Transfomer (ViT) (Dosovitskiy al.) (201),, and Residual Neural Networks (ResNet) (He et al., 2016) are tried separately our framework for demonstration. All the images for stimuli can be processed in advance, thus speeding up model training.
3.2.3 CONTRASTIVE LEARNING
The framework is constructed with contrastive learning, shown in Algorithm I The outputs of the image and EEG encoder are normalized separately. Then, the dot product is used to evaluate the similarity of all image-EEG feature pairs. A scaled temperature parameter used adjust distribution probability. InfoNCE loss (Radford et al., [22T]] van den Oord et al.) (2019) is employed as objective function to increase the similarities between matched pairs and decrease those between unmatched pairs. After adequate training, the EEG encoder can extract representations similar to corresponding images. The self-supervised strategy allows us to learn inherent patterns from EEG signals without labels, rather than directly separate different classes with supervised learning.
Algorithm Natural Image Contrast EEG framework
1: Input: (Image, EEG) stinulus & response
Model: Enc img CLIP Enc_eeg _eee- TSConv
3: # E (batch, channel, electrode, sample)
b batch of input EEG batch of input images
4: # 1- (batch, channel, height, width)
5: learned temperature parameter
6: # extract normalized representations from the raw image and EEG
7: E f Norm(Linear (enearrrEnnc,a cog(E)))
8: 1 I Norm(Ene img(1))
be obtained before training
9: # scaled pairwise cosine similarity
10: logits = dot(E f, f.t) exp(r)
11: # symmetric loss function
12: labels arange(batch)
13: loss e cross entropy loss(logits, labels, axis- 0))
14: loss i cross entropy loss( (logits, labels, axis-1)
15: loss (loss + loss
4

---

Conference paper at ICLR 2024
3.3 PLUG-AND-PLAY MODULE
3.3.1 SELF-ATTENTION
We utilize two approaches, self-attention (SA) Vaswani et al. (2017) and graph attention (GA) Veličković et al. (2018), to supplement the EEG encoder as "spatial filters" encapsulate electrode correlations and reflect the spatial dynamics of brain activity. We use SA on the electrode channels to evaluate the spatial correlations of EEG data. The input Tin E RCxx is linearly transformed into equal-sized with weight metries Wg. Wk, Wo:
WgTiin (WETTi)) WoTin
int Softmax
vd
where z'n E RCXT denotes the output after the SA module, d, the time length of EEG data, is scaled factor accommodate the Softmax function.
3.3.2 GRAPHI ATTENTION
We employ the GA module to update each electrode with all the others, using the implementation from Brody et al. (2022). We treat each electrode as node n; 1IXxT I,..., ch, which has edges to all the other electrodes N₁. The process to update one electrode is as follows:
n ai an
Σ
OG. Wn,
JEN.
where ni, denotes each node after processing, 04 is attention coefficients indicating importance of node j features to node i, and W is the weight of linear transformation. Calculate oz. as:
exp(a" (ea [ng ng)))
Oii = EkENGU exp(a" LeakyRel.U(WWW
[xx]))
where E 2T denotes the weight of feedforward layer for attention calculation, () is the transposition operator, and || is the concatenation operator. LeakyReLU with a slope of 0.2 used for nonlincarity in calculation. Residual connection is employed by integrating the input and output of the SA and GA modules separately, contributing to stable training (He et al. 2016).
EXPERIMENTS AND RESULTS
4.1 DATASETS AND PREPROCESSING
The dataset Gifford et al. 2022 contains EEG data from ten participants with a time-efficient rapid serial visual presentation (RSVP) paradigm. The training set includes 1654 concepts x 10 images x repetitions. The test set includes 200 conceptsx| I image x 80 repetitions. Images for training and testing appear in a pseudo-randomizzd order, and target image is used reduce eye blinks and other artifacts. Each image displays 100 ms, followed by a 100 ms blank screen. Raw EEG data filtered [0.1, 100] Hz has 63 channels and a sample rate of 1000 Hz.
For preprocessing, we epoched EEG data into trials ranging from O to 1000 ms after stimuli onset. Baseline correction was performed with the mean of 200 ms pre-stimulus data. All electrodes were preserved for analysis with down-sampling to 250 Hz, and multivariate noise normalization was performed with training data (Guggenmos al. 2018). We averaged all EEG repetitions of one image to ensure the signal-to-noise ratio and compared the impact of repetitions. Images were resized to 224 x 224 and normalized before being processed by the image encoder.
4.2 EXPERIMENT DETAILS
Our method is implemented with PyTorch on a Geforce 4090 GPU. We randomly select 740 trials from training data as the validation set in each run of the code. Best models are saved when the validation loss reaches a minimum of 200 epochs in the training process. We compute the results of the test set once after training. It takes about minutes per subject to train with a batch size 1000, since get the image features in advance. The in TSConv is set to 40, m, 25, m2 to 51, and to by pre-experiments. Adam optimizer used with the learning rate, Bn and 32 of 0,0002, 0.5, and 0.999, respectively. Wilcoxon Signed-Rank Test employed to analyze the statistical significance.
5

---

Conference paper at ICLR 2024
Table 2: Overall accuracy (%) of 200-way zero-shot classification: top-1 and top-5
Subyet
Subjoct 2
Subject
4
Subject 5 Subjeet Subjoct 7
Subject
Subject
Subhjct 10
Ave
Method
Top-1
1op-5
top-1
top-5
top-1
top-5
top-1 top-5 top-1 top-5 top-1 top-5 top-1 top-5
top-1
top-5
top-1
top-5
top-1 top-5
top-1
top-5
Subjestdependentt
dependent train and test on one subject
BraVL
6.1
17.
4.9
14.9
5.6
174
5.0
15.1
4.0
13.4
6.0
18.2
6.5
20.4
8.8
23.7
4.
14.0
7.0
19.7
5.8
17.5
NICK
12.3
366
10.4
1199
13.1
19.0
164
47.0
8.0
26.9
14.1
4066
15.2
42.1
20.0
49.9
13.3
37.1
14.9
41.9
1388
39.5
NICE-SA
13.3
40.
12.1
15.3
39.6
9.8
34,4
142
42.4
17.9
43.6
18.2
50.2
14.4
38.7
16.0
42.8
14.7
41.7
NICE-GA
15.2
40.1
13.9
40.1
14.7
42.7
17.6
48.9
9.8
29.7
44.4
14.9
43.1
20.3
52.1
14.1
39.7
19.6
46.7
15.6
42.8
Subject
indepeneen
subject
oUt
lest
BaVI.
2.3
8.0
1.5
6.3
1.4
5.9
1.7
6.7
1.5
5.6
1.8
7.2
2.1
8.1
2.2
7.6
1.6
644
2.3
8.5
1.8
7.0
NICE
21.4
NICE-SA
7.0
22.6
6.6
23.2
7,5
23.7
5.4
21.4
6.4
22.2
7.5
22.5
3.8
19.1
8.5
24.4
7.4
22.3
9.8
29.6
7.0
23.1
NICE- GA
5.9
21.4
6.4
22.7
5.5
20.1
6.1
21.0
4.7
19.5
6.2
22.5
59
19.1
7.3
25.3
4.8
18.3
6.2
26.3
5.9
21.6
Table 3: Classification accuracy (%) with different EEG encoder and Image encoder: top-1 (top-5)
Methoui
Subject
Subject
Subjoct Subject 4
Subject
6
Subject? ?
Subject 8
Suijjcct 9
Subject 10
Ave
EBG encoder
ShullowNet 6.(16..))
3.5 (21.5) (28.) 160(41.0) (16.5 (3.00 (30(31))) (24.5 2.00 (2..5) 7.9 (24.2)
(9.3)
NeepNet (3.0
7.5 (27.5) 10.5
4.5 (36.5)
(330) (30.0) (410)) (31.)) (33.))
(31.7) 26c5.5)
Conformer
(3.0
8,5 (28.0) (3.5
6.0 (36.
100
(4.0 (35,) (3.5 (3.5
(33.)) (6.5
BEGNet
(38.0)
(3555 (37.5 (4.5
13.0
(1.5 (4.0
(42.5
(3.0 14.5
13.1
TSComv
(36.6)
(33.9)
(29.0) (6.1
(269)))))
14.1 (1.2) (1.9) (3.3 (19 138 (39.5) (6.5)
Image
ResNet
9,5.2000)
(115) 7.5((6.)) 10.0 (1500 80((4.)) (0.0 14.0 6.0((..) (.0
(.7.)) 2.2
ResNer (..)))
60((1.55 (1140)
(.0.9.5)
(52)) (10.5) (14.5) 6..22) (13.)
(11.)
VIT
(18.)
(15.0 10.5 (20.5) (20.)) 6.0
(19.) (205)
(21. (1700) (7,
(18.) (.8
V
(2.0
(28.5) 13.0 (29.0) (32.5) (21.5) 12.5 (29.0) (3.0 (38.) 65 (24.)) (1.0 12.1 (294) (4.))
CLIP
(3666)
(33.9) 13.1 (39.0) (4.4)) (26.9) (40.)) 15.2
20.0 (69.9) (3..) (4.9) 13,8 (39.5) (3.))
pro-trined ResNes-50 and VIT-B/16 models.
4.3 OVERALL PERFORMANCE
The decoding results are outlined in Table with sanity check in Appendix A.1 We evaluated the base framework (NICE) alongside its vanants enhanced with self-attention (NICE-SA) and graph attention (NICE-GA). The evaluation was on the latest and most extensive image-EEG dataset, with only one state-of-thh-an BraVL. (Du et al., 2023) for comparison. This dataset posed 200-way zero-shot task, comprising 200 untrained image concepts, with a chance level of 0.5%. In subject- dependent experiments, NICE achieved top-1 accuracy of 13.8% and top-5 accuracy of 39.5%, surpassing BL by 8.0% and 22.0%, respectively. The introduction of SA and GA improved the top-1 by 0.9% (p < 0.05) and 1.8% (p 0.01), separately. Besides, in subject-independent experiments, our method achieved top-1 of 6.2% and top-5 of 21.4%. NICE-SA improved the top-1 by 0.8% (p 0.05), while NICE-GA only improved for several subjects, with the average top-1 decreasing by 0.3%. We employed base NÍCE for fair comparison in the following experiments.
4.4 ENCODER COMPARISON
The comparison of EEG encoders and image encoders is present in Table We carefully selected several representative methods. including ShallowNet, DeepNet (Schinmeister et al. 2017), Con- former (Song et al. 2023). and EEGNet (Lawhern al. 2018). Our custom-designed TSConv, which utilizes temporal and spatial convolution, outperformed these methods. The average top-1 accuracy of TSConv is 5.9% higher than ShallowNet (p 0.001). 3.9% higher than DeepNet (p < 0.001). 2.7% higher than Conformer (p < 0.001), and 0.7% higher than EEGNet (p > 0.05). The robustness of TSConv reflected by standard deviation was not superior.
We utilized image encoders, including ResNet-50 pre-trained on ImageNet-1k, k, VIT-B/16 pre-trained on ImageNet-21k and fine-tuned on ImageNet- 1k, and CIT--VVT-L/14 pre-trained on 400 million image-text pairs. CLIP pre-train tre-trained on huge amounts of data helped us achieve the highest results. Interestingly, VIT also worked well with an average top-1 accuracy 1.7% lower than CLIP (p > 0.05). The results of ResNet were 6.9% lower (p 0.001). See Appendix A.2 for the latest update.
We also showed the impact of pre-traine pre-training compared with non-pre- non-pre-trained ResNet-50 and ViT-B/16. Pre- trairaindd ViT outperformed the non-pre-traiee trennr by 4.2% (p 0.01). Surprisingly, ResNet pre- trained with smaller ImageNet-Ie decreased by 1.9% (p < 0.05). Note that non-pre-traime models performed well but incurred substantially higher computational costs, with details in Appendix A.3

---

Conference paper at ICLR 2024
A
0-100
100-200
200-300
300-400
400-500
500-600
600-700
700-800
800-900
900-1000
10
B
10.3
lewe
15.0
20.0
Accuracy (%)
Accuracy (%)
10.0
15.0
0.0
100
200
300
400
500
600
700
800
960
1000
5.0
Frontal Central Perietal TemppraaoOccppiia
(ms)
Brain region (ablated)
D
103
Occipital
Temporal
Parietal
Suuraay (%)
12
Frequency
600
200
500
800
200
400
820
dett
thata
alpha
bata
gamma
(ms)
t (ms)
t (ms)
Frequency band
Figure 2: Temporal, spatial, spectral analysis. (A) Topographies of each 100 ms by averaging all training trials. The temporal lobe has a clear response between 100-600 ms. (B) Averaged accuracy of all subjects with different time lengths. The region of interest is 100-600 ms, after hysteresis in visual systems. (C) Ablate electrodes of different brain regions. The occipital, temporal, and parietal lobes contribute significantly to image decoding.
(D) Time- frequffeyyency maps of the occipital, temporal, and parictal lobes data from one subject. The main components are below 30 Hz, and high-frrquenne components be observed on the temporal lobe. (E) Averaged accuracy of different rhythms. Theta (~>4 Hz) and beta (~~14-8 Hz) bands show effective performance.
4.5 TEMPORAL, SPATIAL, AND SPECTRAL DYNAMICS
Beyond the self-supervised framework, we try to demonstrate the biological plausibility by resolving the visual processing of EEG signals. We conducted a detailed analysis from the temporal, spatial, and spectral dynamics perspective in Fig. The topographies were first plotted with each 100 ms in Fig. 2 A) by averaging all training trials from the first subject. Visual masking (Keysers & Perrett collected in many rapid series sequences. A clear response could be observed on the temporal cortex 100-600 ms after the onset, although the 200 ms stimulus onset asynchrony (SOA) still caused periodic responses on the occipital cortex. Besides, the parietal cortex also had response after 100 ms. The phenomenon is consistent with the bottom-up hierarchy of visual system (DiCarlo Cox 2007), that the visual stimulus processed sequentially by the V1, V2, V4 on the occipital cortex, and inferotemporal (IT) on the temporal cortex along the ventral stream for object recognition (Bao et al. 2020).
2002) would be alleviated because the data used
We explored the active time range in three ways: incrementing forward, decrementing backward, and segmentation, as Fig. 2 B).. could be seen that the region of interest was 100 600 ms. The average top-1 accuracy of [0, 1000] ms was 0.1% lower than that of [100, 1000] ms without significance (p > 0.05). These findings suggest that the initial 100 ms following stimulus onset contained limited information, possibly due the hysteresis effect in the visual pathway (Sayal et al.) 2020). Besides, the signal after 600 ms brought negative effect, probably due to the noise from other stimuli and cognitive processing, emerging response increasing the frontal lobe in Fig. AA)
We conducted an ablation study involving electrodes from different cortices as in Fig. CC). The average accuracy sharply declined 3.8% (p 0.01) without occipital electrodes. Ablation of temporal,
7

---

Conference paper at ICLR 2024
A
Similarity
B
Ground
Top
snimal
1.0
rood
EEG features vehicle
0.6
golf
bike
tool
0.4
handbrake
punchz
vise
tongs
tool
handbrake
athers
COOF
tube top
sult
COaC
hoodle
bonnet
0.0
image features
pear
radish
garli:
bok,
pear
Figure 3: Semantic similarity analysis and visualization. (A) Cosine similarity of feature pairs of 200 concepts in the test set. The results calculated by the trained models of 10 subjects were averaged, and all the concepts were rearranged into five categories: animal, food, vehicle, tool, and others. (B) Classification results visualized with ground truth (first column) and the top-5 predicted.
parietal electrodes and the central area near motor cortex decreased the results by 1.9% (p 0.05), 1.6% (p 0.05), and 0.5% (p 0.05) separately. The accuracy improved by 0.2% (p 0.05) to discard frontal electrodes. The results are still reasonable with the low spatial resolution of EEG.
We plotted time-firquee frequency maps in Fig. D) by averaging all training trials from the first subject. The main components were below 30 Hz from the occipital, temporal, and parictal region data. High-frequency responses could be observed from electrodes the temporal cortex. Interestingly, the frequency responses had an upward trend along with visual processing.
We further conducted classification tests with different frequency bands in Fig. [). Theta and delta bands near 4 Hz. performed well, where theta was more stable for different subjects. The beta and alpha bands also demonstrated performance above the chance level, coarsely aligning with previous findings (Bastos et al. 2015 Michalareas et al.) 2016). However, the gamma band could hardly help us in decoding, for which we have two speculations. Capturing pure gamma oscillations from EEG can be challenging due to the susceptibility of high-frequency artifacts (Fries et al. 2008 YureeeGree et al., 2008). Besides, the amplitude of the gamma band is quite low easily modulated by other cognitive processing, such as attention and memory (Herrmann & Demiralp 2005). This may explain why some studies with MEG have also disregarded the analysis of gamma band (Hebart et al. (002).. Inspired by the latest work Benchetrit et al. (2023), we provided preliminary analysis on MEG data for reference in Appendix A.6
4.6 SEMANTIC SIMILARITY
We show the semantic similarity and classification visualization in Fig. 3 One concern that the discrinination we obtain from EEG is not due to semantic information but only the basic visual components, such as color, brightness, contrast, etc. We performed representational similarity analysis (RSA) (Cichy& & Oliva, 2020), and got the similarity matrices by averaging all subjects, and categorizing the 200 concepts in the test set into animal, food, vehicle, tool, and others. As shown in Fig. 3 AA, could observe distinct intra-category aggregation. This meant the obtained EEG representations were closer to the image representations within the corresponding semantic category.
The images in the test set were used for visualization in Fig. B)) We randomly chose several top-5 decoding results from the first subject. It could be noticed that the predicted results were semantically similar the ground truth, such as the bike predicted into several vehicles with wheels.
8

---

Conference paper at ICLR 2024
A
All regetitins
All conditons
20.0
SubS
Sub10
Input
Accuracy (%)
15.0
Accuracy (%)
15.0
Graph.Att Solf
0.0
25
50
100
0.0
1
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
Amount of training data (%)
Repeat
Times
Figure 4: Effect of data size and repetition in training and test set, and visualization of SA and GA. (A) Accuracy with quarters of conditions and all repetitions, with quarters of repetitions and all conditions of training images. Adding more conditions can potentially further improve the performance. (B) Accuracy with different repetitions of the test images. Average ten times to achieve an accuracy of 9.9 10.. (C) -rads of SA and GA show activation on temporal and occipital regions.
4.7 DATA SIZE AND REPETITION
We compared the impact of the data size and repetition of the training and test sets, which may be crucial for the performance, in Fig. 4 There were 1654 concepts 10 image conditions repeated four times the training set. We averaged the four repetitions of each condition for training. Two cases were compared in Fig. A), adding conditions by 25% intervals with all repetitions, and adding repetitions by 25% with all conditions. Increasing conditions, i.e., data size, contributed significantly to decoding accuracy from 25% to 50% (p 0.01). from 50% to 75% (p 0.05), and from 75% to 100% (p < 0.05). There were also significant contributions by increasing repetitions from 25% to 50% (p < 0.01), from 50% to 75% (p < 0.05), except from 75% to 100% (p 0.05). The results suggest that increasing repetitions is of limited help, while more data promises further improvements.
From another view, repetitions of test data determine the practice efficiency. We compared the number of repetitions for averaging, from 5 to 80, with an interval of 5 in Fig. BB). Ten repetitions achieved a 9.9% (30.1 1)) accuracy, which tended to stabilize above 13.0% (37.0%) after twenty-five repetitions.
4.8 EFFECT OF SPATIAL MODULES
We tried provide additional evidence of brain responses facilitated by spatial modules. Gradient- weighted class activation mapping (Grad-CAM) (Selvaraju et al., 2017) was used to visualize the interest region of SA and GA in Fig. CC) and Appendix A.4 The original data was largely affected by the response on the frontal lobe. Surprisingly, self and graph attention focuses on the temporal and occipital lobes, providing implicit evidence that object recognition from EEG a feasible endeavor.
5 DISCUSSION AND CONCLUSION
There have been some studies achieving commendable results in image reconstruction with IMRL. We turn more convenient and fast EEG signals and focus on the object recognition tasks, in which the semantic information is the significant gain by natural image decoding compared to visual decoding of contrast, color, etc. We have tried demonstrate the feasibility and plausibility of EEG-based image decoding from three folds, zero-shot classification performance, detailed resolving of the brain activity, and model interpreting. There are also some limitations of this paper. Firstly, we observed that a stable response required multiple repetitions. This could be attributed to the brief 100 ms stimulus duration, which may make it susceptible to being missed or disrupted by stimuli before and after. We will aim identify more optimal window length for stimulus presentation. Secondly, have yet to capture useful information from the gamma band, which deserves further investigation.
In conclusion, propose self supervied framework to decode natural images from EEG for object recognition. Our framework has achieved remarkable results in zero-shot tasks with rich biological evidence. The results provide new inspiration for practical brain-computer interfaces.

---

Conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant U2241208, and the Key Research and Development Program of Ningxia under Grant 2023BEG02063,
REFERENCES
Hamad Ahmed, Ronnie B. Wilbur, Hari M. Bharadwaj, and Jeffrey Mark Siskind. Object classification from randomized EEG trials. In 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3844-3853, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544- 509-2. doi: AAL109/CVPR44447.2221000000
Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 941593-021-00092-x
January 2022. ISSN 1546-1726. doi: 10. 10.1038/
Pinglei Bao, Liang She, Mason McGill, and Doris Y. Tsao. A map of object space in primate July 2020. ISSN 1476-4687. doi: 10.1038/
inferotemporal cortex. Nature, 341386-020-2350-55
André Moraes Bastos, Julien Vezoli, Conrado Arturo Bosman, Jan-Mathijs Schoffelen, Robert Oostenveld, Jarrod Robert Dowdall, Peter De Weerd, Henry Kennedy, and Pascal Fries. Visual Areas Exert Feedforward and Feedback Influences through Distinct Frequency Channels. Neuron, 85(2):390-44) January 2015. ISSN 0896-6273. doi: 2..1016/..eurrn.2011 12.0018.
Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: Toward real-time reconstruction of visual perception, October 2023.
Shaked Brody, Uri Alon, and Eran Yahav. How Attentive are Graph Attention Networks?, January 2022.
Xinyu Cheng, Wei Wei, Changde Du, Shuang Qiu, Sanli Tian, Xiaojun Ma, and Huiguang He. VigilanceNet: Decouple Intra- and Inter-Modality Learning for Multimodal Vigilance Estimation in RSVP-Based BCI. In Proceedings of the 30th ACM International Conference Multimedia, pp. 209-217, Lisboa Portugal, October 2022. ACM. ISBN 978-1-4503-9203-2 doi: 10. 145/ 3503161.354836
Radoslaw M. Cichy and Aude Oliva. A M/EEG-IMRI Fusion Primer: Resolving Human Brain Responses in Space and Time. Neuron, 107(5):77--71, September 2020. ISSN 0896-6277 6273. doi: 10.10166/j meuron.200..07..000
Radoslaw Martin Cichy, Dimitrios Pantazis, and Aude Oliva. Resolving human object recognition in space and time. Nature Neuroscience, 17(3):455 462, March 2014. ISSN 1546-172 1726. doi: 10.1038/nn.3635
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learn- ing by exponential linear units (elus). In International Conference on Learning Representations, 2016.
Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Baldwin Geary, Michael Ferguson, David Daniel Cox, and James J. DiCarlo. Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-HHe -uman Behavioral Alignment and Adversarial Robustness. In The Eleventh International Conference Learning Representations, February 2023.
James J. DiCarlo and David D. Cox. Untangling invariant object recognition. Trends in Cognitive Sciences, (8):33333333))) -411, August 2007. ISSN 1364-6613, doi: htt01///..ics200071. 00.0010.
Yi Ding, Neethu Robinson, Chengxuan Tong, Qiuhao Zeng, and Cuntai Guan. LGGNet: Learning From Local-Global Representations for Brain-Commiin Computer Interface. IEEE Transactions Neural Nenworks and Learning Systems, pp. 1-14, 2023. ISSN 2162-2366 168-2388, 2023.3236635.
doi: 10. 10./TNSS.
10

---

Conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, April 2021.
Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Lan Visual-L Linguusii Features. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-17, 2023. ISSN 1939-3539.
doi: 10.1109/TPAM..2233332422
Pascal Fries, René Scheeringa, and Robert Oostenveld. Finding Gamma. Neuron, 58(3):303-305, May 2008, ISSN 0896-6273. doi: 10.1016/j 10.10166j.neuron..o www.neuu.nnn000m 00000.04.022...
Kaicheng Fu, Changde Du, Shengpei Wang, and Huiguang He. Multi-View Multi-Label Fine-Grained Emotion Decoding From Human Brain Activity. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2022. ISSN 2162-2388. doi: 10.1109/TNNLS
Xiaorong Gao, Yijun Wang. Xiaogang Chen, and Shangkai Gao. Interface, interaction, and intelli- gence in generalized brain-computer interfaces. Trends in Cognitive Sciences, 25(8):671-6844 August 2021. ISSN 1364-6613. doi: w0.1016fj.icc..021...
Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, and Radoslaw M. Cichy. A large and rich EEG dataset for modeling human visual object recognition. Neurolmage, 264:1 19754, December 2022. ISSN 1053-8119. doi: 10.10166j. 1e.1016j.ccuroimmee.20000. 199744.
Matthias Guggenmos, Philipp Sterzer, and Radoslaw Martin Cichy. Multivariate pattern analysis for MEG: A comparison of dissimilarity measures. Neurolmage, 173:434-447, June 2018. ISSN 1053-8119, doi: 10.1016/j www.oof.e.rrmmm..20....... 201.000044.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978- 778-1-4673-88511 doi: 10. 10.1109/C 2016.90.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Parhki,, and Chris I Baker. THINGS- data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior, eLife, 12:e82580, February 2023. ISSN 2050-084X. doi: 10.7554/eLife.82500
C. Herrmann and T. Demiralp. Human BEG gamma oscillations in neuropsychiatric disorders. Clinical Neurophysiology, 116(12):219--733) December 2005. ISSN 1388-2457. doi: 10.1016/j. clinph.2005.07. 07.007
Jun Kai Ho, Tomoyasu Horikawa, Kei Majima, Fan Cheng, and Yukiyasu Kamitani. Inter-individual deep image reconstruction via hierarchical neural code conversion. Neurolmage, 271: May 2023. ISSN 1053-8119. doi: hww.:...cerrimae.20.000.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8(1):15037, May 2017. ISSN 2041-1723. doi: 10. 10..00////ommms 15007.
Xiaoxuan Jia, Ha Hong, and James J DiCarlo. Unsupervised changes core object recognition behavior predicted by neural plasticity in inferior temporal cortex. eLife, 10:e60830, June 2021. ISSN 2050-084X. doi: 10.7554/cLife.600 60033.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents the human brain. Nature Neuroscience, (03x679-685, May 2005. ISSN 1546-1726. doi:
1444.
Kendrick N. Kay, Thomas Nasclaris, Ryan J. Prenger, and Jack L. Gallant. Identifying natural images from human brain activity. Nature, 452(7185))352-32 March 2008. ISSN 1476-4687. doi: 10.1038/aauue0061333 13.
Christian Keysers and David I Perrett. Visual masking and RSVP reveal neural competition. Trends in Cognitive Sciences, 01852-0.
March 2002. ISSN 1364-6613, doi: 10.1016/S13344-611300
11

---

Conference paper at ICLR 2024
Reinmar Kobler, Jun-ichiro Hirayama, Qibin Zhao, and Motoaki Kawanabe. SPD domain-specine batch normalization to crack interpretable unsupervised domain adaptation in EEG, Advances in Neural Information Processing Systems, 35:6219 6235, December 2022.
Vernon J. Lawhern, Amelia Solon, Nicholas Waytowich. Stephen M. Gordon, Chou Hung, and Brent J. Lance. EEGNet: A compact convolutional neural network for EEG-based coain-computer interfaces. Journal of Neural Engineering, 15(5):056013, July 2018. ISSN 1741-2552. doi:
(0.1088/1741-25522aacee
Ren Li, Jared S. Johansen, Hamad Ahmed, Thomas V. Ilyevsky, Ronnie B. Wilbur, Hari M. Bharad- waj. and Jeffrey Mark Siskind. The Perils and Pitfalls of Block Design for EEG Classification Experiments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4361):316-333, January 2021. ISSN 1939-3539. doi: 10.1 109/TPAML.2020.2933155
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind Reader: Reconstructing complex images from brain activities. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A, Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 29624-29636. Curran Associates, Inc., 2022.
Bingchuan Liu, Xiaogang Chen, Nanlin Shi, Yijun Wang. Shangkai Gao, and Xiaorong Gao. Improv- ing the Performance of Individually Calibrated SSVEP-BCI by Task- Discriminant Component Analysis. IEEE Transactions Neural Systems and Rehabilitation Engineering, 29:1998-2000 2021. ISSN 1558-0210. doi: 10. 1 1091
Dongjun Liu, Weichen Dai, Hangkui Zhang, Xuanyu Jin, Jianting Cao, and Wanzeng Kong, Brain- Machine Coupled Learning Method for Facial Emotion Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-15, 2023. ISSN 1939-3539. doi: 10. 10.1109/TPMM1.2023
3257846.
Georgios Michalareas, Julien Vezoli, Stan van Pelt, Jan- hun-Mathijs Schoffelen, Henry Kennedy, and Pascal Fries. Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas. Neuron, 89(2):384-39) January 2016. ISSN 0896-6273. doi: me..16/..euroon2015522222
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, and Yukiyasu Kamitani. Visual Image Reconstruction from Human Brain Activity using Combination of Multiscale Local Image Decoders. Neuron, 60(5):915-929, December 2008. ISSN 0896-6273. doi: 10.1016/ ww..10666.neurrn..o newr.nnerooo.0000 2008. 1.004.
Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, Joseph Schmidt, and Mubarak Shah. Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11): 3833-3849, November 2021. ISSN 0162-8828. doi: 10. T..100//PAM...200..0O 2995999..
Yue-Ting Pan, Jing-Lun Chou, and Chun-Shu Wei. MAtt: Manifold Attention Network for EEG Decoding, Advances in Neural Information Processing Systems, 35:311 16-31129,,, December 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Alexandre Sayal, Teresa Sousa, João V. Duarte, Gabriel N. Costa, Ricardo Martins, and Miguel Castelo-Branco. Identification of competing neural mechanisms underlying positive and negative perceptual hysteresis in the human visual system. Neurolmage, 221:1 17153, November 2020.
ISSN 1053-8119. doi: wwt.0//j.nurriiaae..ooo 2020.1115.2..
Robin Tibor Schirrmesster, Jost Tobias Springenberg, Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping, 38(1) (8(11))539113 1):5911-5420, 2017. ISSN 1097-0193. doi: 10. 1002000m..37..500
12

---

Conference paper at ICLR 2024
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960)3360-33)) 368, May 2023. ISSN 1476-4687. doi: (580088/4156-002-000000
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626,
October 2017. doi: 10.1 10.110//CC222211...2.
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLOS Computational Biology, 15(()=e1006633, 2019. ISSN 1553-7358. doi: 10.137 10.1371/joummal.poo 106663....333
Nanlin Shi, Xiang Li, Bingchuan Liu, Chen Yang. Yijun Wang, and Xiaorong Gao. Representative- Based Cold Start for Adaptive SSVEP-BCI. IEEE Transactions on Neural Systems and Rehabilita- fion Engineering, 31:1521-15311 2023. ISSN 1558-0210. doi: 10. 1099TNSSE.
Yonghao Song, Qingqing Zheng. Bingehuan Liu, and Xiaorong Gao. EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:710-719, 2023. ISSN 1558-0210. doi: 10. 10.1009/TNS. 3230250.
C. Spampinato, S. Palazzo, I. Kavasidis, D. Giordano, N. Souly, and M. Shah. Deep Learning Human Mind for Automated Visual Classification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4503-4511, July 2017. doi: 10.1 00.100//VPR.2211.4733
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved Training Techniques for CLIP at Scale, March 2023.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. 2023 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, January 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9886) 2605, 2008. ISSN 1533-7928.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks, In International Conference on Learning Representations, April 2018
Junke Wang. Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. OmniVL: One foundation model for image-language and video-language tasks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 5696-5710. Curran Associates, Inc., 2022.
Francis R. Willett, Donald T. Avansino, Leigh R. Hochberg, Jaimie M. Henderson, and Krishna V. Shenoy. bighhperformance brain-to-text communication via handwriting. Nature, 593(7858)) 249-254, May 2021. ISSN 1476-4687. doi: 1410088/41586--00103336
Rui Xu, Wenjing Zhou, Xin Xu, Zhipei Ling, Hesheng Liu, and Bo Hong. temporal hierarchy of object processing in human visual cortex, September 2023.
Shlomit
Orr Tomer, Alon S. Keren, Israel Nelken, and Leon Y. Deouell. Transient Induced Gamma- Band Response in EEG as a Manifestation Miniature Saccades. Neuron, 58(3): 429-441, May 2008. ISSN 0896-6273, doi: http://jnnrro.20000.0
13

---

Conference paper at ICLR 2024
A APPENDIX
A.1 1 SANITY CHECK
In this work, we perform 200-way zero-shot tasks. Here, we provide an experimental chance by shuffling pairs compare with the ideal chance of 0.5% in Table The experimental chance is close to the ideal chance, and significantly lower than the performance of the NICE model (p < 0.001).
Table 4: Chance level of the experiments
Condition
Top-1 (%)
Top-5 (%)
ideal chance
0.5
2.5
experimental chance
2.3
NICE
13.8
39.5
A.2 BETTER ENCODERS
The adaptability of the NICE framework extends to any other well-designed EEG or image encoders, as shown in 3 We further present results utilizing EVA-CLIP Sun et al. (2023) pre-trained with LAION-2B as the image encoder in Table | The enhancements in both overall top-1 (p < 0.001) and top-5 (p < 0.01) accuracy are surprising. The results further show the potential of this task and hint at the consistency of large pre-trained models and brain responses. Beyond that, we recommend not solely focusing on overall performance, but the specific insight gained.
Table 5: Overall accuracy (%) with EVA-CLIP image encoder: top-1 and top-5
Subject
Subject 2
Subject
Subject
Subject
Subject
Subject 7
Subject
Subject 9
Subject 10
Ave
Method
Iop-1
top-5
top-l top-5
top-1 1op-5
top
top
top
top
top
top
top
top
top-1 top:5
top tep
top top
top top-5
NICE-SA
17.1
454
15.3
44.
174
49.
22.
520
15.6
42.4
21.1
505
234
52.5
26.3
58.8
18.3
49.1
22.1
53.5
19.9
49.7
NICE-CG GA
19.8
46.7
19.6
44.7
198
51.5
25.7
54.5
13.2
39.2
234
54.1
22.
529
2688
58.9
178
47.1
23.6
53.9
21.2
50.4
A.3 COMPARISON OF PRE-TRAINING
We employed pre-trained models the image encoder for two reasons: 1) Pre-trained models like CLIP and VIT possess extensive feature extraction capabilities gained from large-scale training. We sought to leverage these models to enhance the EEG model, trained smaller dataset. 2) Training the image encoder demands substantial computational resources. In our current pre-trained setup, one GPU with 8GB memory yields well-trained EEG encoder for one subject in 5 minutes.
We conducted additional experiments without pre- pre-trained image encoders Table 66 training the randomly initialed image encoder and the EEG encoder simultaneously. VIT-B/16 and ResNet-50 were used as examples. CLIP also relies on ViT-L, but with more layers than ViT-B (24 vs. 12), impeding completion. In the scenario with non-pre-trained, we observed that although training loss reduced rapidly, validation loss demonstrated weaker decline, which is sign of limited generalization. During testing, pre-trained ViT outperformed (p 0.01) non-pre-trained models. Pre-trained ResNet decreased by 1.9% (p < 0.05) instead, hinting at avenues of refining the image encoder to enhance overall performance. But still, training the image encoder imposes significant computational load. We should mention that the decrease in computational cost actually comes from freezing of the pre-trained image encoder and getting image features beforehand.
A.4 SPATIAL MODULE INTERPRETATION
We provide Frad-CAMs and attention weights of all subjects Fig. 5] These visualizations show the alleviation of prefrontal region influence- often tied to cognitive behavior like workload and memory- by both SA and GA. Intriguingly. GA focuses on the occipital region, integral to visual processing, across multiple subjects. SA appears adept at capturing temporal region activities
14

---

Conference paper at ICLR 2024
Table 6: Comparison of pre-trained and non-pre-trained image encoders
Image Enc
Top-1 (%)
Top-5 (%)
Params.
Memory
Time subject
VIT
7.9
18.5
88.0 M
155 G
130 min
V (frozen)
12.1
29.4
1.8 M
~8 G
~ min
ResNet
8.8
17.3
25.3 M
112
~60 min
ResNet" (frozen)
11.8
1.8
G
~ min
pre-trained ResNet-50 and ViT-B/16. The image features were obtained before training.
SUD1
Sub2
SUB3
SUD4
SuB5
Sub6
Sub7
SuB8
Sube
Sub10
Imput
Graph-Att Selk-Att
Gragh-Atz Self-Ait W
Sub1
Sub2
Suba
Sub4
Sub5
Subéi
Sub7
Sub8
Suba
Sub10
Figure 5: (A) Grad-CAM for self-attention (SA) and graph attention (GA) modules of individual subjects and the average across subjects. (B) Attention weights of SA and GA with individual subjects and the average across subjects. The visualizations highlight the regions of interest, focusing on the temporal and occipital brain areas, which are known to be associated with visual processing.
associated with high-level visual processing. Our model interpretation demonstrates that EEG responses can capture and reveal brain activity in the occipital and temporal areas, which are strongly associated with object recognition processes. This provides implicit evidence for the effectiveness of EEG-based object recognition, and further validates the plausibility of our approach.
A.5 PARAMETER SENSITIVITY
The temperature parameter (7) contrastive learning plays crucial role in shaping the similarity distribution. In our implementation, we optimized this temperature directly log-parameterized multiplicative scalar during training, similar to the CLIP model. explore the impact of temperature settings, conducted series of comparisons, and the results presented in Table 7 These comparisons demonstrate the significant influence of temperature adjustments on the results. Besides, pre-experiments help determine conventional parameters such as batch size, decay, etc.
Table 7: Comparison of the temperature parameter in contrastive learning
Temperature
learnable
0.001
0.005
0.01
0.03
0.05
0.1
Top-1 (%)
13.8
12.4
13.45
14.7
signif. (p)
>.0
>.05
<.01
<.01
<.05
<001
<.001
15

---

Conference paper at ICLR 2024
A.6 MEG EXPERIMENTS
We provided analysis on an additional MEG dataset Hebart et al. 2023) to corroborate the results on EEG data. The MEG dataset of four participants has 271 channels and stable responses with a long stimulus duration of 500 ms followed by a blank screen of 1000--200) ms. There are 1854 concepts 12 imagesx I repetitions in the training stage and 200 concepts x imagex x repetitions in the test stage. We discarded 200 test concepts from the training set to construct the zero-shot task. The MEG data were epoched into trials from
1000 ms after the stimuli onset. We used a band-pass filter of [0.1, , 100] Hz and baseline correction after down- sampling to 200 Hz for preprocessing. We averaged all MEG repetitions one image to ensure the signal-to-non ratio. Note that the statistical analysis was not performed on the MEG dataset due to the few participants.
Table 8: Overall accuracy (%) of 200-way zero-shot classification and retrieval on MEG data
Sub
Sub 2
Sub 3
Sub
Ave
Task
Method
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
NICE
6.9
20.5
15.3
37.1
12.3
35.0
5.8
21.1
10.1
28.4
classification
w/ SA
7.3
22.6
13.1
37.1
10.2
30.2
8.3
24.5
9.7
28.6
w/ GA
6.4
23.4
16.2
40.8
14.0
38.9
6.4
25.2
10.8
32.1
9.6
27.8
18.5
14.2
41.6
9.0
26.6
12.8
retrieval
9.8
28.1
18.6
38.4
11.7
27.2
12.7
8.7
30.5
21.8
56.6
16.5
49.7
10.3
32.3
14.3
42.3
0-100 ms
100-200 ms
200 300 ms
300 ms
400-300 ms
500-600 ms
600-700 ms
700-800 ms
800 900 ms
900-1000 ms
3.0
60
[0.1
It
1000)
1200
12.0
Accuracy (%)
Accuracy (%)
8.0
4.-
4.0
3.0
200
300
600
700
900
1000
0.0
Frontal Central Rorietal Temporal Deciptbal
1 (ms)
Brain region
D
Occipital
Temporal
Parietal
Frequency (Hz)
Accuracy
8.0
4.0
200
200
400
600 (ms)
400 200
600
0.0 della
theta
alphe
beta
ganima
(ms)
(ms)
Frequency band
Figure 6: Temporal, spatial, spectral analysis of MEG data. (A) Topographies of each 100 ms by averaging training trials. (B) Averaged accuracy with different time lengths. (C) Averaged accuracy of different brain regions (not ablation here for clarity). (D) Time-frequency maps of the occipital, temporal, and parictal data from one subject. (E) Averaged accuracy of different rhythms.
Classification and retrieval tasks with the MEG dataset have been conducted with the NICE framework in Table A similar 200-way zero-shot classification showed above-chance performance on MEG data with average top-1 accuracy of 10.1% and top-5 accuracy of 28.4%. Besides, we provided the results on a retrieval task, which meant directly using the stimulus images for matching templates instead of other images belonging the concepts. The spatial module GA could still help with the
16

---

Conference paper at ICLR 2024
results, but SA varied across individuals. It is worth noting that more suitable encoder for extracting MEG features should help with the performance Benchetrit et al. (2023). Moreover, we roughly conducted the temporal, spatial, and spectral analysis of MEG data in Fig. The results were similar to Fig. 2 while the temporal responses changed because of the longer stimuli duration.
A.7 EEG ENCODER COMPARISON
The residual connection He al. 2016) used with SA and GA modules stabilize the training procedure. We compared the effect of the residual connection in Table It can be seen that significantly helped GA in top-1 accuracy (p<0.01) but significance for SA (p > 0.05).
The EEG encoder TSConv in our framework has concise architecture consisting of basic temporal and spatial convolutional layers. We compared the details Table because the design was similar to ShallowNet Schirreeister et al. (2017). The comparison encompassed several issues, including the kernel size, pooling order, and activation function. All three cases caused significant effect on the top-1 accuracy compared to TSConv (p < 0.01). Besides, we provided the number of parameters and the output dimension within different EEG encoders Table []] We believe that other well-designed EEG feature extractors can help us improve the performance in this framework.
Table 9: Residual connection comparison for spatial modules
Module
Res
Top-1 (std)
Top-5 (std)
w/ res
14.7 (2.3)
41.7 (4.6)
w/o
13.8 (2.8)
39.6 (5.7)
w/ res
15.6 15.6 (2.9)
42.8 (5.5)
GA
w/o
10.1 (3.2)
29.9 (5.3)
Table 10: Comparison of the architecture of TSConv
Condition
Top-1 (std)
Top-5 (std)
original ShallowNet
7.9 (4.3)
24.2 (9.3)
kernel size as ShallowNet pooling order BLU->ReLU
9.9 (2.4)
33.5 (5.4)
10.9 (3.0)
34.3 (7.2)
11.0 (2.5)
33.3 (6.6)
TSConv
13.8 (3.3)
39.5 (6.5)
A.8 FEATURES DISTRIBUTION
We plotted t-SNE van der Maaten Hinton 2008 in Fig. to show the distribution variance between training and test sets with visual and EEG features separately, be supplement of the RSA in Fig.
A
Visual features
B
EEG features
an mal
training test
animal
training test
food
vehicle tool
vehicle tool
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 7: t-SNE visualization of four categories: animal, food, vehicle, and tool. (A) Visual feature distribution training and test sets. (B) EEG feature distribution of training and test sets.
17

---

Conference paper at ICLR 2024
Table 11: Details of different EEG encoders
EEG encoder
Params
Out dim
ShallowNet
102.0 k
440
DeepNet
303.3 k
1400
Conformer
161.2 k
EEGNet
12.8
1248
TSConv 102.0
1440
Four categories, animal, food, vehicle, and tool, were selected for the visualization, which revealed why we could achieve zero-shot performance after training with rich data.
18


Reviews:
[{"review_id": "H66CzSuMNB", "reviewer": "Reviewer_ixpa", "summary": "The manuscript aims to match images to the EEG signals they evoke using a contrastive loss on the encodings of an EEG model and a vision model. The models are evaluated on zero-shot-discriminating 200 images, that were shown 80 times each, and the corresponding 80 EEG signals are averaged for each image.  The manuscript introduces an EEG model specific for this task and results show it to reach improved decoding performance over existing commonly used EEG models. Further, the manuscript finds that semantically similar images of the same category (like animal, food) have more similar learned EEG representations compared to images of different categories. Temporal analysis shows 100 to 600ms to yield the highest decoding accuracy. Frequency analysis show delta theta and beta bands contain information for reasonable decoding accuracies. The manuscript also performs a variety of spatial analyses.", "questions": "I am not sure I understood the graph attention method details, I found the notation a bit confusing, does aT mean the result fo applying the feedforward layer is transposed? Also it would be nice to mention if this is a unique way of doing graph attention or a well-established way, then would be good to provide references for it. \n\nFor the GradCam Analysis, I am not sure I understood exactly which gradients of which layer do you use and with regard to what? These details may be helpful to write more clearly.\n\nA very recent work tackling similar problems is https://arxiv.org/abs/2310.19812. Due to its recency, I do not expect any comparison, but a brief mention of this work as concurrent work would be great.\n\nFont in Figure3 b could be bigger\nFigure 4 Colored stars could be bigger\n\n\nTypo p. 8: anaylasis", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The manuscript is well-written \n* method is straightforward to understand\n* a lot of detailed analysis of the results on different levels\n* helpful figures", "weaknesses": "Some details I found not easy to understand, see below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Z75jpBYiZ", "reviewer": "Reviewer_JYif", "summary": "The authors present a deep learning pipeline to retrieve the images seen by participants whose brain activity was collected using EEG. The pipeline combines a pretrained frozen image encoder (CLIP, ViT or ResNet) and an EEG encoder (a parameter efficient ConvNet optionally prepended with a spatial attention layer) trained with a contrastive loss and evaluated in a retrieval scenario. Results favorably compare to an existing baseline and to existing neural network architectures. An analysis of the decoding performance and of the biological plausibility of the results highlights spatial, temporal and spectral patterns underlying the image decoding task.", "questions": "Q1. In Section 3.1, the authors write “These images are processed and averaged to obtain one template for each concept.” It is not clear to me how these averaged images are used in the experiments - was this in the semantic similarity analysis? Moreover, were the images averaged in pixel space or in latent space?\n\nQ2. In Figure 2A: What is plotted on the topomaps? I assume it is the average EEG amplitude of each channel across the 100 ms window. If so, how was the data normalized (since the colorbar is from -1 to 1)? Also, in Figure 2B: What do the shaded areas represent? Is it variability across subjects?\n\nQ3. In Section 4.5: “High-frequency responses could be observed from electrodes on the temporal cortex.” How is that measured? I find it hard to conclude from visual analysis of Figure 2D. Along with the chance-level performance of the gamma band (Figure 2E) I don’t see compelling evidence for the following claim: “These results align with established principles, coarsely indicating that the bottom-up feedforward is carried by the synchronization of theta and gamma bands.” \n\nQ4. In Figure 4C and in Suppl. Figure 1: If I understand the topomaps correctly it looks like graph attention actually mostly ignored occipital electrodes for most subjects (as opposed to self-attention for which the occipitally-located pattern is very clear). It would be interesting to look at the average attention weights predicted by both attention modules instead of using GradCAM for this kind of analysis. More generally, given the two formulations (self-attention and graph attention) end up predicting C weights, directly comparing their respective attention weights might give more insight into what drives their different performance. Finally, a central difference between the two approaches appears to be the use of a residual connection after the attention module - it would be interesting to evaluate whether that explains (part of) their differences.\n\nQ5. Existing image decoding studies, e.g. in fMRI (Ozcelik & VanRullen, 2023) but also in EEG (Palazzo et al., 2020), have extended a similar approach up to image generation by feeding predicted image latents to a generative image model. Have the authors attempted this step?\n\nOzcelik, Furkan, and Rufin VanRullen. \"Natural scene reconstruction from fMRI signals using generative latent diffusion.\" Scientific Reports 13.1 (2023): 15666.\nPalazzo, Simone, et al. \"Decoding brain representations by multimodal learning of neural activity and visual features.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 43.11 (2020): 3833-3849.\n\nQ6. The description of the analysis of Section A2 is a bit confusing. The comparison appears to be made between (1) using pretrained frozen image encoders and (2) using randomly initialized image encoders which are trained along the EEG encoders. The description doesn’t make it clear that the difference in computational resources between both approaches comes from the frozen/unfrozen dimension, not from the pretrained/non-pretrained dimension.\n\nQ7. Algorithm 1 describes the CLIP loss (Radford et al., 2021). I think this should be introduced and cited as such, rather than through the lens of InfoNCE.\n\nRadford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The proposed approach combining a frozen image encoder and a trained brain encoder has previously been used on fMRI, however it appears its application to EEG data is novel.\n\nQuality: The manuscript is overall of good quality. The experiments adequately answer the core research question of the paper and provide supporting evidence for the physiological plausibility of the decoded patterns (i.e. through the complementary analyses of the decoding performance, spatial/temporal/spectral patterns and impact of number of stimulus repetitions).\n\nClarity: The manuscript is clearly written, and most figures and results are presented clearly.\n\nSignificance: This study goes beyond the existing image decoding results from EEG in which a limited number of classes were considered. It also is an important building block toward better image decoding from EEG, following in the footsteps of previous studies in fMRI.", "weaknesses": "- Some of the secondary claims do not appear to be adequately supported by the presented results. For instance, the claims on gamma-based decoding (see Q3 below) and on the use of occipital information in the graph attention layer (Q4). \n\n- Based on previous image decoding work in fMRI and EEG, a logical continuation of the proposed contrastive learning pipeline is to then pass predicted latents to a generative image model (see Q5). I believe the presented image decoding performance results (along with the analyses of Sections 4.5-4.8) are interesting in themselves, however as a result the scope of the paper is narrower than existing studies on the topic. I believe the authors should discuss this in their manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SrnG0qHTSz", "reviewer": "Reviewer_9bdz", "summary": "The work presents a method for the task of EEG-based object recognition, using a recent dataset collected with the Rapid Serial Visual Presentation (RSVP) paradigm. The authors propose a self-supervised framework where an image and an EEG encoder network are used to extract visual and EEG features. During training, the method maximizes the similarity between positive pairs of samples (i.e. when the signals of the EEG sample of a pair are obtained from the image sample of the pair), and minimizes the similarity for negative pairs. After training, the authors perform zero-shot EEG decoding, by matching EEG features to visual features from image categories that were not present during training. Experiments are performed in one dataset, using various network architectures for EEG and image feature extraction, presenting results that are superior to the previous state-of-the-art and demonstrating the importance of EEG electrode areas, EEG frequency bands, and the temporal window timings of the input EEG signals.\n\nThe contributions are:\n1) an architecture for EEG feature extraction, named \"TSConv\", along with two plug-and-play attention modules that can be incorporated in the architecture\n2) the training methodology of the proposed method, called \"NICE\" (Natural Image Contrast EEG)\n3) the investigation of natural image information from EEG signals", "questions": "Important information about the experiments is missing from the manuscript. The number of trainable parameters for each EEG encoding architecture is not reported. Moreover, the dimensionality of the EEG features extracted from each EEG encoding architecture, exactly before being mapped to the dimensionality of the image features, should also be reported. This would allow the reader to obtain a better understanding on the details behind each architecture's performance. For example, it could be the case (among other reasons) that some EEG encoding architectures have better performance because of a larger EEG feature dimensionality (hence an EEG-to-image feature mapping layer of larger learning capacity).\n\nThe results presented in Figure 3 are not sufficiently explained. It is unclear to what do rows and columns correspond, i.e. which dimension corresponds to EEG features and which corresponds to image features. It would also be helpful to get an idea of the similarities between the visual features from (some of the) concepts of the training set and the visual features from concepts of the test set.\n\nIn Figure 2, sub-figure (D) is not sufficiently explained, i.e. the exact method that was used for obtaining the time-frequency map is not mentioned, the quantity that is depicted is not stated and it is unclear to what does the range [-1, 0] correspond.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There is some originality behind the work presented in the manuscript. The authors work on a recent RSVP dataset of EEG & visual data (Gifford et al., 2022), being among the first works that conduct experiments on the task of EEG-based object recognition using this dataset and presenting superior results compared to the previous state-of-the-art (Du et al., 2023). The contrastive loss employed during training, is different from contrastive losses that have been used in other works for EEG-based object recognition (e.g. in (Palazzo et al., 2021)).\n\nGifford et al., \"A large and rich EEG dataset for modeling human visual object recognition\", NeuroImage 2022\nDu et al., \"Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features\", TPAMI 2023\nPalazzo et al., \"Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features\", TPAMI 2021\n\nThe authors are aware of the challenges on the studied task, presenting a clear summary of previous works relevant to EEG-based object recognition. The presentation of the results provides clear insights for several factors that are involved (e.g. intra-subject versus cross-subject testing in Table 2, network architectures in Table 3, ablation studies on the attention modules in Table 2, electrode areas, EEG timings and EEG frequency bands in Figure 2).", "weaknesses": "The proposed architecture for EEG feature extraction, TSConv (described in Table 1 of the manuscript), is highly similar to that of the ShallowNet (as described in (Schirrmeister et al., 2017)) architecture, which the authors include as a baseline method. Specifically, the ShallowNet architecture originally consists of a temporal and a spatial convolutional layer, followed by average pooling and a fully-connected classification layer. Leaving aside the differences on the employed activation functions between TSConv and ShallowNet, one could say that the difference lies only on the order of placement for the average pooling layer. Given that the results reported in Table 3 are quite different for these two architectures (i.e. ShallowNet is the worst performing architecture and TSConv is the best performing architecture), what could be the interpretation?\nSchirrmeister et al., \"Deep learning with convolutional neural networks for EEG decoding and visualization\", Human Brain Mapping 2017\n\nRegarding the two plug-and-play spatial modules, the authors state that they are \"instrumental in preserving the spatial characteristics of EEG channels\" (page 3, Section 3.1). Considering the self-attention (SA) module and looking at Equation (1), one can notice the spatial filtering nature of the proposed self-attention. However the authors do not provide any insight on the spatial characteristics of the signals obtained through Equation (1), i.e. are the signals more or less correlated after this operation? The authors also mention that, following this transformation of the SA module, \"Residual connection is employed by integrating the input and output of the SA module\", but it is not clear how much does this integrating procedure affect the spatial correlations between the EEG channels.\n\nThe graph attention mechanism proposed by the authors (page 5, Section 3.3.2) is similar to that of (Li et al., 2023).It would be interesting if the authors could explain the differences.\nLi et al., \"STGATE: Spatial-temporal graph attention network with a transformer encoder for EEG-based emotion recognition\", Frontiers in Human Neuroscience, 2023\n\nRegarding the intuition behind the proposed training methodology, the authors state that \"The self-supervised strategy allows us to learn inherent patterns from EEG signals without labels, rather than directly separate different classes with supervised learning.\" (page 4, Section 3.2.3). This argument is not well-supported, as image embeddings extracted from pretrained models like ResNet, CLIP and ViT may in fact contain class-related information (Van Gansbeke et al., 2020).\n\"SCAN: Learning to classify images without labels\", Van Gansbeke et al., ECCV 2020\n\nOther comments:\n\nThere are some typos, grammar and syntax issues throughout the manuscript, e.g.:\n- \"We device\" (page 2)\n- \"has remained a challenging.\" (page 2)\n- \"as the widely use way\" (page 3\n- \"representation similarity anaylasis\" (page 8)\n- \"categoring\" (page 8)\n- In Table 1, while s_1 is not presented in any layer's description, it is then used in the descriptions of output dimensions. I think that this is a mistake and that s_2 should be used instead.\n\nThis particular phrase is a bit vague and informal, the process could be stated more clearly: \"we look for a few images that belong to the image concepts\" (page 3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwFq7dVqbd", "reviewer": "Reviewer_pm39", "summary": "The authors propose a self-supervised framework: Natural Image Contrast EEG (NICE) to decode image from EEG", "questions": "Q1: In the second paragraph of the Introduction, the author mentioned four problems with existing work. Are these problems the motivation of this paper? If so, why are some issues not mentioned or solved in the follow-up; if not, why are the issues that have not been addressed and paid attention to mentioned in the paper?\nQ2: If so, for P3: “overlooking the inferior temporal cortex plays a necessary role in object recognition (fifth line)”, the inferior temporal cortex is located at the bottom of the frontal lobe, and EEG mainly records signals from the surface layer, so how does NICE pay attention to it from EEG and how the signals from the inferior temporal cortex can help NICE perform Vision Decoding?\nQ3: For P4: \"focuses on the peak of pairwise decoding about 110 ms after stimulus onset. (fourth line from the bottom)\", the paper does not mention whether NICE's focus on the temporal dimension of the EEG signal is biologically interpretable, but P4 points out that the problem with the current model is that the focus of specific EEG time intervals is unreasonable", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Clarity: \nDescribes the proposed NICE model with very clear logic and diagrams", "weaknesses": "Motivation: \nThe authors' main MOTIVATION consists in solving the following two problems (the problems mentioned in the second paragraph of the introduction are not sure whether they are motivations, so they are placed in the Question section): 1) Existing work has predominantly relied on supervised learning, often dealing with limited data from a few categories (introduction, first line of the third paragraph); 2) Limitations of the EEG feature extractor: convolution in the temporal and spatial dimensions respectively (the second sentence from the bottom of the third paragraph of introduction)\nW1:Indeed most of the existing EEG tasks (emotion recognition, motor imagery) are supervised learning, but Decoding tasks (such as EEG2speech, EEG2Image) are different from traditional EEG tasks. The essence of the task is cross-modal conversion rather than Classification, self-supervised learning is a very common method in decoding tasks. Comparisons with supervised learning for traditional tasks are therefore unreasonable.\nW2: Many existing EEG feature extraction methods focus on the spatial-temporal relationship of EEG, such as methods based on spatial-temporal graph, or spatial-temporal attention, or spatial-temporal convolution that is very similar to TSconv in NICE (Such as TSception). However, these very common methods are not mentioned in this paper, so this motivation is unreasonable.\n\nExperiment：\nW1: The main experiment only included one baseline, but there has been a lot of work on EEG vision decoding, such as Mind-Vis in CVPR. It is difficult to demonstrate the superiority of NICE by only comparing it with one baseline on a not commonly used dataset.\nW2: There is no analysis of why NICE achieves better results than BraVL.\nW3: In the Encoder comparison experiment, the baseline compared with TSConv did not include methods used for EEG feature extraction in the past five years.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)