Human-Aligned Image Models Improve Visual Decoding from the Brain
Authors: Nona Rajabi
Antonio H. Ribeiro
Miguel Vasco
Farzaneh Taleb
Mårten Björkman
Danica Kragic
Venue: ICML 2025 poster (2025)
Keywords: Visual Decoding
Brain-Computer Interface
EEG
Contrastive Learning
Human-Alignment
Primary area: applications->neuroscience_cognitive_science
TL;DR: This paper proposes leveraging human-aligned image encoders to map brain signals to images using self-supervised learning, achieving up to 21% higher accuracy compared to state of the art.

Abstract: Decoding visual images from brain activity has significant potential for advancing brain-computer interaction and enhancing the understanding of human perception. Recent approaches align the representation spaces of images and brain activity to enable visual decoding. In this paper, we introduce the use of human-aligned image encoders to map brain signals to images. We hypothesize that these models more effectively capture perceptual attributes associated with the rapid visual stimuli presentations commonly used in visual brain data recording experiments. Our empirical results support this hypothesis, demonstrating that this simple modification improves image retrieval accuracy by up to 21\% compared to state-of-the-art methods. Comprehensive experiments confirm consistent performance improvements across diverse EEG architectures, image encoders, alignment methods, participants, and brain imaging modalities.

Human-Aligned
Image Models Improve Visual Decoding from the Brain
Nona Rajabi' Antônio H. Ribeiro Miguel Vasco
Farzaneh Taleb
Marten Björkman
Danica Kragic'
Abstract
Topl Retrieval Performance
Human-Aligned
Decoding visual images from brain activity has significant potential for advancing brain-computer interaction and enhancing the understanding of human perception. Recent approaches align the representation spaces of images and brain activity enable visual decoding. In this paper, intro- duce the use of nlmann-lligme image encoders to map brain signals to images. We hypothesize that these models more effectively capture perceptual attributes associated with the rapid visual stim- uli presentations commonly used in visual brain data recording experiments. Our empirical results support this hypothesis, demonstrating that this simple modification improves image retrieval ac- curacy by up to 21% compared to state-of-the methods. Comprehensive experiments confirm consistent performance improvements across di- verse EEG architectures, image encoders, align- ment methods, participants, and brain imaging modalities!.
0.6
Original
0.4
0.2
0.0
CUP
DINON2
DINO
ENSEMBLE E
OPENCUR
SYNCLA
Query
Figure I. Performance of image retrieval from EEG data. We show how the use of human-aligned versions of standard image encoders consistently leads to an increase performance across image retrieval tasks from brain activity. The bars represent the top-1 retrieval performance averaged over subjects.
1. Introduction
shared representation space.
Decoding visual information from brain activity has long been a key goal neuroscience and computer vision, with the potential to revolutionize brain-computer interfaces, deepen our understanding of neural processes, and enable advanced assistive technologies. Current state-of-the methods for visual decoding rely on three core components: (i) a brain-signal encoder that processes high-ding digh-iimensional, multivariate time-series data from the brain; (íi) a pre-trained image encoder capable of generating semantically rich im- age embeddings; and (iii) self-super superviedd loss function that aligns brain signals and image embeddings within
Despite progress, visual decoding, especially from low hignal-to-nole ratio data like electroencephalogran (EEG), remains a significant challenge. Previous approaches have explored various brain signal encoders (Benchetrit et al., 2023; Song et al., 2024: Li et al., 2024), additional encoding models (Scotti et al., 2023; Li al., 2024), and auxiliary loss functions (Ferrante et al., 2024). Yet, overall performance remains limited; for example, state-of-the-art BEG-based models achieve only 28% top-1 retrieval accuracy from a database of 200 unseen test images. Here, we take another approach and focus on the nature the image encoder and how it affects the performance of visual decoding methods.
'Equal contribution 'Division of Robotics, Perception, and Learning, KTH Royal Institute of Technology. Stockholm, Sweden *Department of Information Technology, Uppsala University, Upp- sala, Sweden. Correspondence to: Nona Rajabi < @ kt..
We investigate the use of human-aligned image represen- tation models (Fel et al., 2022; Muttenthaler et al., 2023b; Sundaram et al., 2024) for visual decoding from brain sig- nals. These models are specifically designed to ensure that images perceived as similar by humans are closer in the representation space, effectively aligning the embeddings with human perception. This alignment particularly rele-
Proceedings of the nnnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 com/
by the author(s).
All codes available NonaRjb/Al A1 wnw.iii...giii.......

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Alignment
(optiona)
Trainable Parameters
Parameters
Human User
Reference
Image encoder 9a
Fine-tunning
Training
Visual Stimuli
Brain encoder
Self supervisddd
Image encoddrr
Target Image
Decoding
Visual Stimuli
Brain encoder Jn
Latent Space
Top-k Image Retrieval
Figure 2. Learning to map brain activity to images. Image embeddings E IRd are extracted from frozen image encoders fo(-), (-), while IRd computed using mopality-specifie encoders g-(-). A contrastive loss then aligns brain-signal representations to the image space. We consider both original pre-trained image encoders and human-aligned versions fine-tuned on human image similarity judgments. Images for the similarity judgment dataset and retrieval results are borrowed from NIGHTS (Fu al., 2023) and THINGS (Hebart et al., 2019) datasets.
brain signal embeddings
vant for decoding visual information from brain signals, human judgments emerge from underlying brain processes.
2. Brain-to-Image Decoding via Latent Space Alignment
We demonstrate that incorporating pre-trained, human- aligned image encoders significantly improves visual decod- ing performance from brain signals. Through an extensive empirical study spanning multiple brain modalities, brain- signal encoders, and image foundation models, show consistent and substantial improvements achieved through human alignment. Furthermore, we analyze the differences between models trained with human-aligned and baseline image encoders, revealing that human-aligned models fo- cus on signal components known to be relevant to visual processing. By leveraging human-aligned image encoders, improve the accuracy of state-of oftte-of-thearr models by up to 11, establishing new direction for visual decoding from the brain (Figure 1).
We frame the problem of visual decoding from the brain as retrieving the corresponding viewed image from a given image database based on brain activity. We exploit an archi- tecture composed of train-signal encoder and an image encoder. After training the brain encoder, the test brain activity is embedded, and its distances to all images in the test database are computed. The images are then ranked by their distance to the brain embedding, and the top-k images are retrieved. The procedure and overall architecture the method are summarized in Figure 2.
Formally, let x denote an image from a subset of an image dataset, and let b represent the brain-activity measurement recorded from a subject while observing We define two neural
functions, so and Pø- that map images and brain signals, respectively, to a shared latent vector space Z E Rd:
Our key contributions are as follows:
We propose and demonstrate that human-aligned mod- els consistently and significantly improve visual re- trieval from the brain.
x fo(x) =v, b + (b) = V,W Rd (I)
We conduct an extensive empirical study of various human-alignment methods, image and brain encoders, and neuroimaging modalities.
Our goal train the brain encoder gø such that the embed- dings (v, w) for each pair (x, b) are aligned in the shared latent space Z E Rd To achieve this, encoders fo derived from pre-trained image image-text models, which provide rich, generalizable representations without requiring fine-tuning. Aligning brain embeddings w with the image embeddings v enables the retrieval of the corresponding viewed image x from the brain signal.
frozen image
We hypothesize that human similarity judgmenl datasets capture early human impressions of images and are more closely aligned with the brain signals. support this hypothesis, we present experiments that provide further evidence.
2

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
This alignment is enforced using the multimodal InfoNCE loss (Radford et al., 2021):
complex images from brain activity. Research has explored natural image and video decoding using functional mag- netic resonance imaging (fMRI (Scotti et al., 2023; 2024), magnetoencephalogram (MEG) (Benchetrit et al., 2023), and EEG (Song et al., 2024; Li et al., 2024). While fMRI achieves higher decoding accuracy. MEG and EEG offer better temporal resolution and accessibility. A common ap- proach in these studies maps brain signals into pre-trained image representation spaces like CLIP (Radford et al., 2021) and DINO (Caron et al., 2021) using self-supervised loss, such as InfoNCE (Fquation 2). These mappings enable image retrieval and can serve as conditioning signals for image generative models like Stable Diffusion (Rombach et al., 2022) to reconstruct viewed images. Unlike prior work focusing on brain encoders and auxiliary modalities like text, we investigate the role of image encoders. Specifi- cally, we examine how aligning image encoders with human perception impacts visual decoding from brain signals.
Lo = 1
N
exp(sim Wi, V+)/T)
log
Mn exp(sim( w v2)/T)
(2)
exp(sim(vs, w.)/r)
+ log N ==1 exp(sim(vs, w3)/T)
where is the temperature parameter and sim(a, b) is distance metric between the two embeddings: in our case, we employ cosine similarity. The same subscripts denote matching pairs. We investigate the impact of image encoders on visual decoding from brain activity, specifically assessing whether human perceptual biases in image representation spaces influence performance. Our hypothesis is:
Hypothesis: Replacing the image encoder fx) with its perceptually aligned version significantly improves visual decoding performance from brain signals.
3.2. Perceptual Alignment of Vision Models
We based our hypothesis on the intuition that human sim- ilarity judgments originate from brain processes and thus reflect the similarities between brain signals correspond- ing to image stimuli. Additionally, we propose that this improvement is linked to the experimental design of the mmage-similarity dataset. Specifically, since brain signals are elicited by a rapid visual stimulus (100 ms viewing time), they primarily capture early impressions of images, such as shape, orientation, and color. These features are better represented in image-similarity datasets focused on early perceptual attributes. We corroborate this hypothesis through the experiments provided in later sections.
Early computer vision models were inspired by human per- ception (Fukushima, 1980; Serre et al.. 2007; Krizhevsky et al., 2012), but modern models often diverge from it (Golan et al., 2020; Lindsay, 2021; Kumar et al., 2022: Fel et al., 2022; Muttenthaler et al., 2023a). Recent research has sought to bridge this gap by aligning machine vision models with human perception. Fel et al. (2022) proposed a neural harmonizer that co-trains deep networks to align with human visual strategies while maintaining task accuracy. Their ap- proach leverages datasets with human judgments on salient image features from ImageNet (Russakovsky et al., 2015). Similarly, Muttenthaler et al. (2023b) introduced gLocal, a linear transformation that aligns the global structure of neu- ral representations with human similarity judgments while preserving local structure. Unlike naive transformations that degrade downstream task performance, gLocal maintains or slightly improves performance while enhancing alignment with human perception. More recently, Fu et al. (2023) in- troduced Dreamsim, metric for fine-tuning vision models on human image similarity dataset. Sundaram et al. (2024) demonstrated that Dreamsim-aligner models outperform un- aligned ones on several downstream vision tasks, including object counting, instance retrieval, and image segmentation.
3. Related Work
Decoding images from brain activity and aligning vision models with human perception are two independently re- searched directions. However, perceptually aligned models have yet to be explored for visual decoding from the brain. This section reviews prior work in both areas, highlighting the gap our study addresses.
3.1. Decoding Images from Brain Activity
Understanding how humans perceive images and decode this information from brain signals has been a long-standing question in neuroscience and, more recently, in computer vision (Hubel & Wiesel, 1962; Kamitani & Tong, 2005: Scotti et al., 2024). Early studies focused on decoding simple visual features like stimulus orientation (Kamitani & Tong, 2005) and basic geometric shapes (Du et al., 2017). Advances in computer vision (Radford et al., 2021; Dosovitskiy, 2020; Rombach et al., 2022) and large-scale datasets (Gifford et al., 2022; Allen et al., 2022; Hebart et al., 2023) have enabled the retrieval and reconstruction of
4. Materials and Methods
Here we describe the datasets, pre-trained image encoders, and EEG encoder architectures employed in our study.
4.1. Datasets
We used the Things EEG2 dataset (Gifford et al., 2022) to train and evaluate our framework. Things EEG2 is, to our knowledge, the only dataset with a sufficient number of participants and trials per participant support more
3

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Retrieval Performance
Top1 Retrieval Performance
Top1 Retrieval Performance
DREANSIP Or arginal
BVN Orig ma
OLOCAL
0.8
Originl
5.6-
0.2
0.0
0.0
Top5 Retrieval Performance
Top5 Retrieval
Top5 Retrieval Performance
0.0
5.6
0.2
0.0
OPENEL wea
STROI JIBEE
0.0
CuP Meds
LEVT
DavG
(b)
(C)
Figure 200-way image retrieval performance from EEG signals. For this task. we employ the NICE EEG encoder (Song et al., 2024) and evaluate different human- -ligmmmtt methods and image encoders. Each vertical pair of panels demonstrates the results for one human-alignment method. The results highlight a consistent improvement image decoding performance when considering human-aligned image encoders. Stars represent significant difference confirmed by paired T-test (p < 0.05). Best viewed with zoom.
robust and generalizable findings. It also employs random- order design, avoiding biases associated with block-order designs (Li et al., 2020).
vision models aligned to human perceptual judgments pro- posed by Sundaram et al. (2024) (which we refer to as Dreamsim) and compare them with their original unaligned version. Specifically, we use vision transformer (ViT) (Doso- vitskiy, 2020) backbones including CLIP (Radford et al., 2021), OpenCIIP (Cherti et al., 2023), DINO (Caron et al., 2021), DINOv2 (Oquab et al., 2024), and SynCLR (Tian et al., 2024). We also include results obtained by concate- nated features from DINO, CLJP, and OpenCLIP referred to the Ensemble model (Sundaram et al., 2024).
EEG data were recorded from 10 participants exposed to visual stimuli via a rapid serial visual presentation (RSVP) paradigm (100 ms per image). The training set includes 1,654 unique concepts, each with 10 images shown in ran- dom order and repeated 4 times, totaling 1654 x 10 samples per participant. The test set contains 200 distinct concepts, each with I image shown 80 times, yielding 200 x x 80 samples per participant. Training and test concepts are disjoint. To improve the signal-to-noise ratio, repetitions were averaged, reducing training and test sam- ples to 1654 x 10 and 200 1 per participant. Models were trained with 90/10% split and evaluated on the test set.
4.3. EEG Architectures and Training
We used the NICE BEG encoder proposed by Song al. (2024) to embed EEG signals in the experiments described in Sections 5.1, 5.3, and 5.4. NICE is recent convolutional neural network (CNN) that applies temporal and spatial convolutions to multivariate time-series input data. Addi- tionally, in Section 5.2, we evaluated our method with other EEG encoders by training three recent architectures: EEG- Netv4 (Lawhern al., 2018), a widely used CNN for EEG tasks; EEGConformer (Song al., 2022), a transformer- based model designed for EEG signals; and ATM-S (Li et al., 2024), which combines transformer with a CNN.
For the results in Section 5.4, used preprocessed MEG data from (Hebart et al., 2023), recorded from 4 participants in response to 1,854 unique concepts (500 ms per image). The training set contains 1854 x 12 x 1 samples, while the test set includes 200 x x 12 samples per participant. We excluded the 200 test concepts from training and averaged the 12 repetitions per test concept, resulting in 1654 x 12 training samples and 200 1 test samples per participant. For both EEG and MEG, models were trained signals from 0 to 1000 ms relative to stimulus onset. Further dataset details are in Appendix A.
To speed up the training, we pre-computed the image embed- dings using the frozen image encoders. Dreamsim models were used as human -alnan- image encoders throughout the paper unless explicitly mentioned otherwise. We trained all models using the InfoNCE loss (Equation 2) with five different seeds (which also resulted in five different train/val splits). All results presented were averaged over all seeds. More training details are presented in the Appendix B.
4.2. Pretrained Vision Models
We train the whole EEG encoder by aligning its final layer representations pretrained vision model's representa- tions. For this purpose, we use image embeddings from
4

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Table 200-way top-1 image retrieval performance from EEG signals. We consider different EEG and image encoders during the training of the models. The results show that using human-aligned aligned (HA) image encoders consistently improves the performance of the overall model, regardless the specific image and EEG encoder employed. Best viewed with zoom.
Eavemblle
S
CUP
OpvaCUiP
DINO??
HA
Base
BA
Base
HA
Basc
HA
Base
BA
Base
HA
Base
LEGHeeM
39.51
1.42
31.81
1.50
38.98
1.27
31.22
1.04
28.03
1.40
20.35 a U36
31:31
1.1 1.74
24
37.46
1.42
3134 1:33
27.45
1.12
15:33 = 1.07
ERGConlimer
20.78
1.25
1.32
26.25
1.01
18.6
1.37
19.02
+ 1.01
21.06
0.96
23.92
+: 1.30
1948
18.14
+ 1.04
0.26 D.65
NIC
54.66
2.57
17.30
2.20
56.44
2:20
61.19
2,00
12.82
2.17
18. 78.59+1.58 1.32
19.59
2 2.40
36.20
54.21
2 2:19
45.90 192
41.87
2: 1.87
2001 + 1.55
A
56.36
2.14
1.99
02.53
1.59
46.15
46.06
à: 1.62
2
53.25
1 1,93
10.1
56.30
1.0.1.1000 1.02
4804
44.73
2: 1.01
2:40
Table 2. Cross-subject 200-way top-1 image retrieval performance from EEG signals. We encode EEG using the NICE encoder and consider different image encoders during the training of the models. HA refers to the human- -linnnd models. Best viewed with zoom. CIJP
Fasemblle
SynCUR
OpenCIIP
DINO
DINOv2
HA
Base
Rase
Base
S1
11.4
+ 1.71
12.2 1 1.36
9.1 1 1,16
10.0 1 0.84
10.3 1: 1.96
6.0 2: 1.05
11.5 2: 0.55
9.0 4: 0.37
10.6 1: 1.11
10.0 4 0.71
9.8 1 1.21
9.5 1: 1.00
S2
15.0
1667
15
13.2 163
12.8 1.12
0.95
2.84
15.1
1.25
13.5 1.73
13.5
1.38
S3
8.1
1.99 0:98
9.5 1.0
12.8 1.91
8.7 +: 1.50
11.9 119::1.0 1.98
7.5 1.00
1.15
7.1 0.73
9.1 1.31
9.4 1.24
8.4 1.88
7.8 1 0.75
1.58
12.0 1.05
10.2:1.500
1.36
9.21
0.40
10.0
6.1::.24
9.2
1.32
1:21
85
7.3
2.42
0.86
7.6 0.58
866 + 1.28
7.4+ 1.36
4.4 1.39
6.1 1.88
6.2 0 1:29
1.17
8.1 1.68
7.1 0.97
6.4+1. 1.16
S6
21.8
1:00
18.9 1.24
17.0
1.36
16.7 1.47
0.40
2..0
1.39
2.55
15.5 1.97
11.87 + 1.78
11.3
S7
10.8
1: 1.72
10.5
4 1.45
20.7
8.8 L.33
8.5 1.52
5.9
=.6=1.71
7.0 1: 1.38
9.5 1.30
10.2 1.82
9.5 1.30
5.5 0.45
S8
14.5
1: 1.45
13.6::0.00
15.9
12.7
1244 1.56
1.00
13.6
0.55
14.2
13.2
13.8
1: 1.35
89
13944444
13.3+2.0
13.8+0..5 0.51
1.65
10.0+1.44
0.6
11.4+1.46
9.7 1.91
14:2+00.0
11.8+11.12
10.0
9.8 1.81
810
15.8
+ 2.06
0.73
1.33
1.00
14.1 1,20
10.2
12.3 1.50
1.2+100 1.94
12.6
13.2 1.03
10.1 1.65
11.9 1.32
Arg.
12.8 1.37
12.9 1.04
12.4 0.92
0.00
11.4
0.87
0.75
10.8 0.74
+ 0.58
11.5 0.82
11.4
10.2 0.72
0.67
4.4. Evaluation
methods. For consistency, we employ the NICE EEG en- coder (Song et al., 2024) across all evaluations. We present the main results in Figure 3.
We evaluated model performance using top-1 and top-5 image retrieval accuracy. Specifically, during testing, we computed the similarity of each test brain embedding and the image embeddings of all 200 unseen images in the test set, then ranked them accordingly. The top-k accuracy measures whether the correct image appears among the k most similar image embeddings to the given brain embedding.
EEG encoders trained with human-aligned embeddings us- ing Dreamsim consistently and significantly outperform those trained with unaligned embeddings, regardless of the type of image encoder used (Fig. 3a). Notably, these human- aligned models achieve up to 21% higher top-1 and 29% higher top-5 accuracy in 200-way retrieval tasks. A paired T-test further confirms the significance of these improve- ments (p < 0.001 for all except DINO and Ensemble, where p < 0.01 and p 0.05, respectively, for the top-1 perfor- mance).
5. Results
We conduct an extensive empirical study comparing the performance of human-aligned and unaligned image en- coders in visual decoding from brain activity. Specifically, we assess their impact across various image encoder ar- chitectures and alignment methods (Section 5.1), as well as various EEG encoders (Section 5.2). We also assess whether human-alignee models generalize better to unseen subjects (Section 5.3) and examine their effectiveness in other neuroimaging modalities, such as MEG (Section 5.4).
Both gLocal and Harmonization perform comparably to original unaligned models and sometimes even surpass them (Fig. 3b, Fig. 3c). However, they still underperform relative to Dreamsim. We attribute this to differences in the datasets used to align these models with human perception. Dream- sim is trained image similarity judgment datasets that capture mid-level similarities, which are more likely to be reflected in brain data collected via RSVP. In contrast, gLo- cal and Harmonization emphasize similarities that require more attention and longer viewing times. We further discuss these differences in Section 6.
Finally, we perform an interpretability analysis (Section 5.5)
to highlight differences between EEG models trained with human-aligned aligned and original image embeddings. Overall, our results highlight the consistent improvement of using human-aligned models for visual decoding from the brain.
5.1. Image Encoders and Alignment Methods
5.2. EEG Architecture
We start by evaluating how different image encoders and alignment methods affect visual decoding performance from EEG activity. In particular, we consider the Dreamsim (Fu al., 2023; Sundaram et al., 2024), gLocal (Muttenthaler al., 2023a) and Harmonization (Fel et al., 2022) alignment
We now assess the impact of different EEG architectures on the overall performance of the method. We evaluate the EEGNerV4 (Lawhern et al., 2018), EEGConformer (Song et al., 2022), NICE (Song et al., 2024), and ATM-S (Li et al., 2024) EEG encoders in our evaluation. As before, base-
5

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Table 200-way top-1 image retrieval performance from MEG signals. encode MEG using the NICE encoder and consider different image encoders during the training of the models. The results show that using human-aligned aligned (HA) image encoders consistently improves the performance of the overall system, regardless of the specific image encoder employed. Best viewed with zoom.
Enaemble
SycCRR
CLIP
OpenCIJ
DINO
DINO12
IIA
Base
IIA
Base
11A
Base
11A
Base
1IA
Base
[IA
Base
S1
13.4 1 182
12.1 1 188
15.3 1 1.14
13.0 1 1.30
12.6 1 0.96
0.49
11.4 1.11
8.7 1. 0.88
10.7 1 1.03
11.8 1 1.75
11.7 2.02
1.5 1 1.10
52
36.7 1: 2.56
368 0.97
24. 1.52
26.3 1: 2.75
1384:181
29.6:122:11
20.6
33.6 1: 1.56
24.75
4..75122.61
90.6 1.28
12171:1.17
22.7 1: 1.08
20:6 L02
2..6
16.9
22.2
1.3
1.34
21.4
1.39
21.1 1.77
19.12
17.5
0.9
$4
14.8 + 1.96
12.2 + 1.72
14.7 0
12.7
+ 2.46
14.3 + 1.33
8.1 + 0.73
11.1 + 1.68
10.9 + 2.08
10.2 + 1.17
10.2 + 1.48
10.8 + 1.75
5.8 + 1.78
Avg.
21.9 + 5.34
18.6 6.16
23.2 +: 5.18
16,6 1..8.1.2.75 2.75
19.3 + 3.64
10.6 1.6.1.1. 1.75
18.4 4.44
13,9 2,66
1.59:::5.51
1.557
17.7 + 4.58
8.3 1.40
line embeddings are compared with human- embed- dings from multiple image encoders. We consider Dream- sim as the alignment method for this evaluation. The results are summarized in Table 1.
scribed in Section 4.1, and employed a similar architecture to the NICE encoder. We present the top-1 retrieval per- formances, discriminated by subject, in Table 3, and top-5 performance in Appendix D.
Once again, we observe consistent and significant improve- ment when we consider the human-aligned version of the image embeddings during training. In particular, human- alignment allows us to achieve a 62% top-1 accuracy, significant improvement over the 46% accuracy achieved by non-aligned versions. As expected from prior literature, certain architectures (NICE, ATM-S) perform better overall than others (EEGNeIV4, V4, EEGConformer). Nonetheless, en- coders trained with human-aligned embeddings consistently outperformed those trained with unaligned embeddings, re- gardless of the underlying EEG architecture.
Results consistently show improved retrieval performance across all participants using human-aligned embeddings with Dreamsim over original image encoders. While statis- tical tests were not conducted due to the limited number of participants, the findings highlight the potential of human- aligned image encoders to enhance visual decoding across brain activity modalities.
Finally, we also evaluated the use of human- -linand models for image retrieval from IMRI signals (Allen al., 2022), with results presented in Appendix E.
5.5. Biological Interpretation
5.3. Transferring to New Subjects
To investigate whether the improved performance with per- ceptually aligned embeddings is linked attributes of vi- sual processing in the brain, we analyzed the gradients of the EEG encoder from spatial, temporal, and spectral per- spectives. Specifically, we applied gradient-weighted class activation mapping (Grad-CAM) (Selvaraju et al., 2017) to identify the regions in the signals that contributed most to the model's predictions. Figure 4a presents the histogram of gradient values exceeding the 99th percentile over time for representative participant. Notably, EEG encoders trained with human -alaane image embeddings exhibit a higher peak at carlier time points (0- - 200 ms relative to stimu- lus onset). This period overlaps with the image stimulus presentation time and associated with low-level visual pro- cessing, such as orientation and color (Lamme & Roelfsema, 2000; Hillyard & Anllo-Vento. 1998). Figure further sup- ports this, demonstrating that human-alignn models, com- pared to original ones, more accurately retrieve attributes like texture, pattern, color, and shape, which are consis- tently shared among the top-5 retrieved images. Additional examples are provided in Appendix F.
Generalization of trained models to unseen participants is critical yet challenging task in brain data studies, as the subjective nature of brain signals often leads to poor per- formance on new participants' data. In this experiment, we compared the naive generalization capability of the NICE EEG encoder trained with human-aligned embeddings to its unaligned counterpart. Specifically, the NICE encoder was trained using data from 9 participants and tested on the held-out participant's 'ss test data. The results for each test participant and image encoder are presented in Table 2.
The results highlight the challenge of transferring brain sig- nals to unseen participants. A paired T-test showed human- aligned models significantly outperformed original models for CLIP (p- < 0.001) and OpenCLIP (p- < 0.05), while no significant differences were found for other architectures. Though improving transferability is beyond the scope of this work, we plan to explore strategies for enhancing gen- cralization in future research.
5.4. Validation on MEG Signals
To evaluate whether the performance improvement of using human-aligned image encoders extends to other neuroimag- ing modalities, we replicate the experimental procedure of Section 5.1, this time using MEG data. Specifically, we utilized the MEG dataset from Hebart et al. (2023), as de-
EEG signals are typically decomposed into five frequency bands linked to cognitive processes: Delta (0- 4) H2), Theta ([4 - 8] Hz), Alpha ([S - 12] Hz), Beta (12 - 30] Hz), and Gamma (> 30 Hz)). To assess each band's 's con- tribution, we computed the power spectral density of the
6

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Histogram of Gradients Across Time
PSD of Gradients
0.20
Human-Aligned
Human-Aligned Original
0.5
Original
Probabiiity (x104)
0.15
Normalized Energy
0.4
0.3
0.10
0.2
0.05
0.1
0.03
0.00
0.0
200
600
800
1000
Delta
Theta
Alpha
Beta
Gamma
Time (ms)
Frequency Band
(a)
(b)
Human-Aligned
ms
100 ms
200 ms
300 ms
400 ms
500 ms
600 ms
700 ms
800 ms
900 ms
Original
ms
100 ms
200 ms
300 ms
400 ms
500 ms
600 ms
700 ms
800 ms
900 ms
LVV
(c)
Figure 4. Analyzing gradient maps for NICE EEG Encoder. We computed the gradients for trained EEG encoders with human-aligned and original unaligned image embeddings and analyzed them from (a) temporal, (b) spectral, and (c) spatial perspectives. The results are presented for a representative participant (S4) and image backbone (SynCLR).
gradients and normalized them by total power. Fig. 4b com- pares the energy of each band across human-aligned and original models, with statistical significance assessed via an unpaired T-test (p < 0.005 for Delta, Alpha, and Beta; p 0..05 for Gamma). Models trained with original embed- dings focused more on Delta, while human-aligned models emphasized Alpha, Beta, and Gamma, aligning with prior research linking these bands to visual processing (Bastos et al., 2015; Michalareas et al., 2016).
6. Discussion
In this paper, we investigate the use of human-aligned im- age models for decoding visual information from brain signals. We demonstrate that brain encoders trained with human-aligned image embeddings consistently outperform those trained with unaligned embeddings in image retrieval tasks. This finding is validated through extensive evalua- tions across various image and brain encoders, alignment methods, and brain modalities. Finally, we provide some in- sights on spatial, temporal, and spectral differences between the models by analyzing the gradients of the brain encoder.
We also examined which EEG electrodes contribute more to the predictions of the model. Both EEG encoders, whether trained with human-aligned relied on information from electrodes positioned over the oc- cipital and parietal brain regions, which are known to be in- volved in visual processing (Grill-Spector & Malach, 2004; Wandell et al., 2007) (Figure 4c). From this perspective, no significant differences were observed between the models. More results and details are provided in Appendix G.
original image embeddings,
Perceptual Alignment and Brain Decoding. Human sim- ilarity judgments stem from brain processes and reflect the similarities between brain signals elicited by image stimuli, To evaluate the effect of using alignment, in Section 5.5 we analyze gradient maps produced by the brain encoder across three domains: spatial, temporal, and spectral. The spatial analysis showed that both human-ale aligned and non-human-

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
HA 20
HL.022
BA-0224
HA
Ouery Image
Query
8..0.35
H1.
RL-027
01.-0.28
N62-0.26
86:-8.25
HA-0.23
Query Image
Figure 5. Top-5 retrieved images of unseen classes from EEG signals using human-aligned (top) and base (bottom) models. We use Dreamsim for the aligned model with CLIP backbone for both. The cosine similarity between EEG and image embeddings is shown above the images. Images retrieved from human-aligned models emphasize attributes like texture, pattern, color, and shape.
aligned models focus on similar brain regions, particularly the visual cortex. However, differences emerged in the tem- poral and spectral domains, with the human-aligne model placing greater emphasis on early stimulus responses and frequency components in Alpha, Beta, and Gamma bands.
low-level perturbations (e.g.. blurring, noise) or high-level categorical distinctions, NIGHTS prioritizes features that are also relevant to rapid visual processing, aligning more closely with brain-computer interaction tasks in this study, Finally, Harmonization leverages the ClickMe (Linsley et al., 2019) and Clicktionary (Linsley et al., 2017) datasets, which use
We attribute this behavior to the experimental design of the mmage-simiiarity dataset. Brain signals, elicited by rapid visual stimuli (100 ms for EEG and 500 ms for MEG), pri- marily capture early impressions of images, such as shape, orientation, and color. These features are better represented in image-similarity datasets like NIGHTS, which emphasize mid-level similarities, compared to those focusing on fine- grained categorical differences. As a result, image encoders fine-tuned with such datasets are more aligned to differences between images that are also reflected in the recorded brain signals. We visualized some of the retrieved images from EEG signals in Figure 5, showing that the retrieved images for the noman-alignee models indeed share some attributes such as color, shape, and orientation.
saliency maps to train image encoders aligned with human perception. However, tasks in ClickMe and Clicktionary require careful inspection and attention to longes--oonditonss that do not reflect the rapid visual stimulus used in brain data collection experiments. As a result, Harmonization models are less likely to capture the same features as the brain datasets used in this study.
Future Work. Our work opens several new possibilities for future work. We mainly looked at this problem from the perspective of image retrieval to provide a more direct and eosier-to- scenario that allows us to isolate the effect of human alignment better. Another important down- stream task is reconstructing viewed images from brain activity using image generative models like Stable Diffu- sion (Rombach et al., 2022). This venue of exploration could be achieved by fine-tuning the image generation pro- cess on a auman-aligned conditioning representation space. Moreover, methods for quantifying the alignment (Sucholut- sky al., 2023) can provide significant insight and could also be explored. Finally, to generalize ture work should validate the results on datasets featuring brain responses to images from other databases. We plan to extend this study to fMRI, which offers higher spatial resolution and longer stimulus times, to test the consistency of our findings.
Why Dreamsim outperforms other alignment methods. We also evaluated embeddings generated by numan-aligned image encoders trained with carlier methods. While these models performed similarly to those using unaligned em- beddings, they did not achieve the substantial improvements observed with Dreamsim models. This difference likely arises from variations in datasets and training methodolo- gies. For example, gLocal relies on THINGS odd-one-out triplets (Hebart et al., 2019), which emphasize high-level cat- egory differences, whereas Dreamsim uses the NIGHTS (Fu et al., 2023) similarity judgment dataset, capturing mid-level attributes such as object appearance, viewing angles, cam- era poses, and layout. Unlike earlier datasets focused on
findings, fu-
Conclusion. In this work, we proposed that human- aligned models consistently and significantly improve vi-
8

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
sual decoding from the brain. Our extensive empirical study highlighted the pivotal role of image representations and experimental design in visual decoding from brain data. We hope our study introduces a new perspective on learning brain-to-image mappings and lays the groundwork for future advances in visual decoding from brain activity.
Bastos, A. M., Vezoli, J., Bosman, C. A., Schoffelen, J.-M., Oostenveld, R., Dowdall, J. R., De Weerd, P., Kennedy, H., and Fries, P. Visual areas exert feedforward and feedback influences through distinet frequency channels. Neuron, 85(2):390-401, 2015.
Benchetrit, Y., Banville, H., and King, J.-R. Brain decoding: toward real-time reconstruction of visual perception. In The Twelfth International Conference on Learning Repre- sentations, 2023.
Acknowledgments
This work supported by the Knut and Alice Wallenberg Foundation, the Swedish Research Council, European Re- search Council (ERC) D2Smell 101118977 and ERC BIRD 88480 grants. The computations and data handling were enabled mainly by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Su- percomputer Centre and partly by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), par- tially funded by the Swedish Research Council through grant agreement no. 2022-06725.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650- 2021.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for con- trastive language-iman image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
We would also like to thank Mohammed Al-Jaff for his valuable insights and constructive discussions.
Impact Statement
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint MXXY::0010.112299 2020.
Decoding images from brain signals has seen significant progress in recent years. However, this technology remains limited to research environments and is not yet suitable for applications outside controlled laboratory settings. These models remain the research phase, and no consequential decisions should be based on their outputs. The technology is not yet reliable enough for any form of critical appli- cation, and its results should be interpreted with caution. An application of this research could be in interaction, particularly for assistive technologies aiding individuals with communication disabilities. While these advancements could enhance accessibility, current models remain untested on patient data. Further research is needed to validate their use in medical settings.
Du, C., Du, C., and He, H. Sharing deep generative repre- sentation for perceived image reconstruction from human brain activity. In 2017 international joint conference on neural networks (ijenn), pp. 1049-1055 1056. IEEE, 2017.
Fel, T., Rodriguez Rodriguez, I. F., Linsley, D., and Serre, T. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in neural infor- mation processing systems, 35:9432-9446, 2022.
Ferrante, M., Boccato, T., Bargione, S., and Toschi, N. Decoding visual brain representations from electroen- cephalography through knowledge distillation and latent diffusion models. Computers in Biology and Medicine, pp. 108701, 2024.
Given that human data is used in this research, ethical ap- proval is a necessary prerequisite for data collection. In our work, we exclusively utilized publicly available datasets that were ethically reviewed and approved by official orga- nizations. Furthermore, all data used in this study has been anonymized to ensure the privacy and confidentiality of participants, aligning with best practices in ethical research.
Fu, S., Tamir, N., Sundaram, S., Chai, L.,, Zhang, R., Dekel, T., and Isola, P. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. Advances in Neural Information Processing Systems, 36, 2023.
Fukushima, K. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36 14):1932 1980.
References
Allen, E. J., St- ves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F,, Charest, I., et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 26 2022.
Gifford, A. T., Dwivedi, K., Roig, G., and Cichy, R. M. A large and rich eeg dataset for modeling human visual object recognition. Neurolmage, 264:1 19754, 2022.

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Golan, T., Raju, P. C., and Kriegeskorte, N. Controversial stimuli: Pitting neural networks against each other models of human cognition. Proceedings of the National 2020.
Li, D., Wei, C., Li, S., Zou, J., Qin, H., and Liu, Q. Vi- sual decoding and reconstruction via eeg embeddings with guided diffusion. Advances in Neural Information Processing Systems, 2024.
Academy of Sciences. 117(47)29330-299))
Grill-Spector, K. and Malach, R. The human visual cortex. Annu. Rev. Neurosci., 27(1):649-677, 2004.
Li, R.,, Johansen, J. S., Ahmed, H.,, Ilyevsky, T. V., Wilbur, R. B.,, Bharadwaj. H. M., and Siskind, J. M. The per- ils and pitfalls of block design for eeg classification ex- periments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):316-333 2020.
Guggenmos, M., Sterzer, P., and Cichy, R. M. Multivariate pattern analysis for meg: A comparison of dissimilarity measures, Neuroimage, 173:434 447, 2018.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer vision-ECCCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. 740- 755. Springer, 2014.
Hebart, M. N., Dickter. A. H., Kidder, A., Kwok, W. Y., Corriveau, A., Van Wicklin, C., and Baker, C. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10)=0223392, 2019.
Hebart, M. N., Contier, O., Teichmann, L., Rockter, A. H., Zheng, C. Y., Kidder, A., Corriveau, A., Vaziri-Pashkam, M., and Baker, C. I. Things-data, a multimodal collection of large-scale datasets for investigating object represen- tations human brain and behavior. Elife, 12:e82580, 2023.
Lindsay, G. W. Convolutional neural networks as a model of the visual system: Past, present, and future. Journal of cognitive neuroscience, 33(10):201722031 2021.
Linsley, D., Eberhardt, S., Sharma, T., Gupta, P., and Serre, T. What are the visual features underlying human versus machine vision? In Proceedings of the IEEE Interna- tional Conference Computer Vision Workshops, pp. 2706 2706-2714, 2017.
Hillyard, S. A. and Anllo-Vent Vento, L. Event-related brain potentials in the study of visual selective attention. Pro- ceedings of the National Academy of Sciences, 95(3): 781-787, 1998.
Linsley, D., Shiebler, D., Eberhardt, S., and Serre, T. Learn- ing what and where attend. In International Conference Learning Representations, 2019.
Hubel, D. H. and Wiesel, T. N. Receptive fields, binocular interaction and functional architecture in the cat's s visual cortex. The Journal of Physiology, 160(1):106, 1962.
Michalareas, G., Vezoli, J., Van Pelt, S., Schoffelen, J.- M., Kennedy, H., and Fries, P. Alpha-beta and gamma rhythms subserve feedback and feedforward influences among human visual cortical areas. Neuron, 89(2):384- 397, 2016
Kamitani, Y. and Tong, F. Decoding the visual and subjec- tive contents of the human brain. Nature neuroscience, 8 6355679-685, 2005.
Muttenthaler, L., Dippel, J., Linhardt, L.,, Vandermeulen, R. A., and Kornblith, S. Human alignment of neural network representations. In The Eleventh International Conference on Learning Representations, 2023a.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Muttenthaler, L.,, Linhardt, L.,, Dippel, J., Vandermeulen, R. A., Hermann, K., Lampinen, A., and Kornblith, S. Improving neural network representations using human similarity judgments. Advances in Neural Information Processing Systems, 36, 2023b.
Kumar, M., Houlsby, N., Kalchbrenner, N., and Cubuk, E. D. Do better imagenet classifiers assess perceptual similarity better? Transactions on Machine Learning Research, 2022.
Lamme, V. A. and Roelfsema, R. The distinct modes of vision offered by feedforward and recurrent processing. Trends in neurosciences, 23(11):571-577 2000.
Oquab, M., Darcet, T. Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. P., and Lance, B. J. Eegnet: a com- pact convolutional neural network for eeg-based brain- computer interfaces. Journal of neural engineering, 15 (5):056013, 2018
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural
10

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
language supervision. In International conference on machine learning, pp. 8748-8766 PMLR, 2021.
Takagi, Y. and Nishimoto, S. High-resolution image re- construction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 14453-14463, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10684-10695, 2022.
Tian, Y., Fan, I.., Chen, K., Katabi, D., Krishnan, D., and Isola, P. Learning vision from models rivals learning vision from data. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 15887-15898, 2024.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:
Wandell, B. A., Dumoulin, S. O., and Brewer, A. A. Visual field maps in human cortex. Neuron, 56(2):366-383, 2007.
211-252, 2015.
Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., Verlinde, N., Yundler, E., Weisberg, D., Norman, K., et al. Reconstructing the mind's eye: fmri- to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2023.
Scotti, P. S., Tripathy, M., Torrico, C., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K. A., al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. In Forty-first International Conference Machine Learning, 2024.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via maadient-based localization. In Proceedings of the IEEE international conference computer vision, pp. 618 626, 2017.
Serre, T., Oliva, A., and Poggio, T. A feedforward architec- ture accounts for rapid categorization. Proceedings of the national academy of sciences, 104(15):6424-447) 6429, 2007.
Song, Y., Zheng, Q., Liu, B., and Gao, X. Beg conformer: Convolutional transformer for ceg decoding and visu- alization. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 31:710-719, 2022.
Song. Y., Liu, B., Li, X., Shi, N., Wang, Y., and Gao, X. Decoding natural images from eeg for object recognition. In The Twelfth International Conference on Learning Representations, 2024.
Sucholutsky, I., Muttenthaler, L., Weller, A., Peng, A., Bobu, A., Kim, B., Love, B. C., Grant, E., Groen, I., Achterberg, J., et al. Getting aligned on representational alignment. arXiv preprint arXiv:2310. 13018, 2023.
Sundaram, S., Fu, S., Muttenthaler, I.., Tamir, N. Y., Chai, L., Kornblith, S., Darrell, T., and Isola, P. When does per- ceptual alignment benefit vision representations? In The eiiht Annual Conference on Neural Information Processing Systems, 2024.

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
A. Datasets Details
EEG Data We used EEG data from Gifford al. (2022) and preprocessed in the same way as Song et al. (2024); Li al. (2024). Particularly, first, a multivariate noise normalization (Guggenmos al., 2018) was applied to the data from each session before epoching the signals. Then, signals from 63 EEG electrodes were filtered the range [0.1-100] Hz and downsampled to 250 Hz. EEG data epoched from 0 1000 ms relative to the stimulus onset, with baseline correction applied using the 200 ms pre-stimulus interval.
MEG Data For the MEG experiments, we used the preprocessed MEG data from Hebart al. (2023). The dataset includes recordings from four participants with 271 channels. Signals were band- passed between 0. and 40 Hz and baseline corrected by subtracting the mean and dividing by the standard deviation of the 100 ms period before stimulus onset. While the original data was epoched from 110 to 1300 ms relative to stimulus onset, we used a subset from to 1000 ms. The data was initially sampled at 1200 Hz but was downsampled 200 Hz during preprocessing.
B. Implementation Details
Aligned Image Encoders We obtained human-aligned image encoders directly from the Dreamsim, gl.ocll, and Harmo- nization repositories provided by the authors, ensuring they function exactly as reported in their respective papers without any retraining. The code for generating embeddings for images in the SHINGS-EEGE and THINGS-MEG2 datasets is available in the project repository.
Unaligned Image Encoders For the original unaligned encoders, we used publicly available pretrained models using Hugging Face transformers (Table 4) timm (Table 5) libraries.
Model Name
Hugging Face Repository Path
CLIP_VIT-B32
openal//llip-uttt-baaeppaattttt
CLIP-ViT-B16
opena//cliip-V11--basspptttthd
CLIP ViT-L14
pgenal//llp--itt-laageetttthhll
DINO.ViT-B16
facebook/diino-iitbbb6
DINO.ViT-B8
facebook//din-vvitbb
DINOv2_ViT-B14
faceboot//dhnvv2-baee
DINOV2.ViT-114
facabook/dihov2-large
DeenCLIR.viT-LL11.lste
OpenCIIe.WiT-LL1laaionn00
UpenCIIP-WII--S3_Laaloo000
http:://mw...bbpppppppaaaaaaaaa2 hf ht-rau::t.mmmreeeettto..ppp.pppp
OpenCLIP.RN50
Table Model names and their corresponding Hugging Face repository paths.
Model Type
Timm Model Name
ViT-B16
vit.base.patcch16.. 6-224
50
resnet50
VGG16
vgg16
ConvNEXT
convnext.tiny
LeViT
levit.conv.
Table 5. Mapping of model types to their corresponding timm model names.
Training We performed a grid search on sample subject to determine the optimal learning rate, batch size, and temperature parameter, selecting the best values based on minimum validation loss. For per-participant EEG experiments, NICE encoders were trained for up to 50 epochs with batch size of 128, learning rate of 0.0002, and a temperature of 0.04. The same hyperparameters were used for A7M-S, except for the number of epochs, which was set 80. EEGNet and
12

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
EEGConformer both trained for 200 epochs. EEGConformer used learning of 0.0002, a batch size of 128, and temperature of 0.07, while EEGNer used 0.01, 512, and 0.1, respectively. For cross-participant training, NICE was trained for up to 150 epochs with a batch size of 512 and a learning rate of 0.0001. For MEG experiments, we used a learning rate of 0.00005, batch size of 256, and a temperature of 0.1, training for up to 50 epochs. Training was halted in all models if validation loss did not improve for 25 consecutive epochs.
C. Cross-Subject EEG Top-5 Retrieval Performances
Table 6 presents the top-5 performance of the experiment presented in Section 5.3.
Table Cross-subject 200-way top-5 image retricval performance of NICE encoder on EEG signals using human-aligned or original image embeddings.
Fisemble
SynCLR
CLIIP
OpenCT.IP IIP
DINO
DINO
HA
Base
HA
Base
HA
Base
HA
Base
Base
HA
Base
SI
33.6 2.0
3L.5
1.0
32.6 2.7
27.0 + 1.7
33.7 +
25.0
1,9
33.9 + 3.2
25.400.
32.3 + 2.9
28.2 + 1.3
29.3
29. +
S2
38.1
+ 1.6
38.4 1.4
34.1 3.4
32.8
32.0 +
25.2
30.1
4-3.
27.8 + 2.3
35.3 + 1.3
33.3 + 1.7
34.2
27.6 1.6
S3
28.3 1.4
31.9
33.5
11.9
30.7 3.3
23.9
33.2 4.4
24.61:033
30.8 1.4
29.9
2.4
96.6118
25.0
$4
3.9
33.6 1.9
29
30.1 336
28.6
21.8 1.5
26.7
21.2+
27.9 1.7
27.3
27.6 28
S5
23.5
0.8
1.1
2.
21.6+
22.1
16.5
1.3
23.3+337
20.5 20.5+336
21.6+ 1.9
22.2 1.8
1..3+344
16.9
S6
46.9
1..4
4.
36.7 3.7
38.2 1.88
32.9 1.4
36.5
12.3
36.2 1.9
1.3
34.7
26.3
1 1.7
57
35.2226
3.1
29.8
29.
28.8
21.5 1.6
2.3
27.124
32.0
31.2
25.9
20.0
35.6299
35.1
31.5
89
2.
33.6
2.0
3.0
27.1 3.5
2..1
26.1
26.1
32.1
312#11
26.1
2.3
S10
40.8+4..
40.2
1.9
3..4+22.6
32.7 + 1.3
33.1 + 1.5
28.4 2.7
36.2 + 1.6
32.2 2.3
32.1 4.4
28.2 2.3
31.9
Avg.
316424
21
30.7.2211 32
30.6 + 2.5
25.3
30.9 2.6
3166 27
30.5
28.7
25.6
D. MEG Top-5 Retrieval Performances
Table 7 presents the top-5 performance of the experiment presented in Section 5.4.
Table 200-way zero-shot 5 performance of the NICE encoder on MEG data using human -algned or original image embeddings.
Ensemble
SynCLR
CLIP
OpeaCLIP
DINO
DINOv2
HA
Base
HA
Base
HA
HA
Basc
HA
Base
HA
Basc
S1
10.1 1 3.7
40.1 1
39.6 1 1.2
35.6 1
35.7 3.3.
26.6 1:
35.9
2.4
29.0 1: 1.1
35.7 1.1
34.1 1 2.3
33.4 1.4
21.2 1
$2
72.7 + 1.4
62.9 + 1.3
75.2 2 1.8
62.1 2.0
62.5 22.5+23
44.7
64.5
1.2
53.1 + 1.5
66.5 1.9
58.1 +
60.7 3.6
34.1
49.6
47.7
36.0
42.9
23.1
S4
36.4+22
35.1 + 1.4
38.6 0.8
31.9+3377
36.7 1.8
26.4 2.1
32.9 1.3
33.5 0334330
37.3-40 40
30.6 +
32.4++1.0
19.4 1.1
Avg.
51.9 1. 2.3
46.9 4: 1.6
52.1 1.5
44.3 2.5
45.9 1. 2.3
33.4 1: 1.6
46.6 1...1.2 2.0
39.3 1: 2.0
47.9 1 2.2
41.4 2.4
43.3 1: 2.5
24.4
E. Evaluation on MMRI Data
To further validate the benefits of using numan--lignen models, we extended our experiments to the NSD dataset (Allen et al., 2022). This dataset contains 7-Tesla fMRI responses of human participants to natural scenes from the MS-COCO image database (Lin et al., 2014). We followed the same preprocessing and data splitting steps as Takagi & Nishimoto (2023); Scotti et al. (2023; 2024) and refer the reader to those works for further details. We trained individual-subject models for the four participants who completed all scanning sessions (participants images that were presented to all participants. This resulted in dataset comprising 24,980 training samples and 2,770 test samples. For the test set, we averaged the brain responses across the three repetitions of each image, reducing the test set to 982 unique samples, while the training set remained unaveraged.
5, and 7), using a test set based on the 1,000
We trained the MLP fMRI encoder with residual connections proposed by Scotti et al. (2023) for 50 epochs with learning rate of 0.0001 and batch size of 128. Top-1 and top-5 1000-way way retrieval accuracies are presented in Tables 8 and 9, respectively.
13

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Table 1000-way zero-shot top-1 performance of the MLP encoder fMRT data using human-aligned or original image embeddings.
Ensemble
SynCLR
CLIP
OpenCLIP
DINO
DINOv2
HA
Buse
HA
Buse
HA
Base
HA
Base
HA
Base
HA
Base
60.4
0.9
52.9 1
61.5 1.1
52.0.007
49.2
27.7 1: 1.7
53.6 0.6
40.2 0.8
53.7
49.9
1.
24.8
82
62.9
+ 0.8
44.3 + 0.9
67.5+0.
43.9 1.0
4.0
0.8
48.6
31.5+00:
47.9 0.9
43.0 0.5
20,2+00. 4..6+.11
85
62.
1.2
56.5+13
022202
67.8
3..7
0.
5..9
43.2
56.5=12
53.7
14
272213 0.72007
57
43.1
+ 1.3
38.8
45.6++1.0
1.3
33.8
18.0
1.7
27.0 1.2
39.3 0.9
37.0 1.1
36.5
17.9
Arg.
54.6
+ 4.3
18.1 1.0
58.2 + 4.6
18.2 1.3
15.0 4.3
2.4
19.5 = 1.3
36.1
49.1 + 3.7
4.79
45.3 + 3.6
2..
Table 1000-way zero-shot S performance of the MLP encoder IMRI data using human-aligned or original image embeddings.
Fisemblle
SynCIR R
CLIPP
OpenCTIIP IIP
DINO
DINOv2
HA
Base
HA
Base
HA
Basc
HA
Base
HA
Basc
HA
Basc
S1
47.8
0.4
83.3 + 0.6
80.4 +: 0.9
8 0.9
79.9 +: 0.6
0.9
83.2 +:
72.4 + 0.7
83.6 4: 1.0
50.6 + 0.5
79.4
51.7 +: 1.5
$2
82.5 + 1.1
77.84 + 1.0
80.7 0.7
7.
76.0 1.3
50.8 0.3
78.0 0.5
(4.7700.8
79.1 0.8
73.6
+ 0.3
74.7+
45.0 +
S5
89.9
86.5+
9.9
0.7
8.
0.
0.
7.88
0.5 0.7
56.9 0.0
S7
73.5 1.0
70.8 1.6
7..
7.
6.44
1.
08.4
0..4
6..6
6..2
65.9 0.5
0.6
Avg.
83.4 3.7
3.1
86.7 = 3.8
7.1
76.6 + 4.2
79.2 = 4.1
67.8
79.7
75.5 3.6
3.5
49.1 3.2
F. More Examples of Retrieved Images
In addition to the original Things EEG2 test set of 200 samples, we explored retrieved images from a larger dataset. Specifically, using EEG data from the test set, we computed the distances between EEG embeddings and all image embeddings from both the training and test sets, expanding the search space to 16,740 images with a chance performance of 0.006%. This broader search provides deeper insights into the neighboring images of an EEG sample and highlights differences between the representation spaces learned with human-aligned and unaligned image embeddings.
Figure 6 presents examples of the top-5 retrieved images from this larger dataset. The results suggest that buman-aligme (Dreamsim) models are more sensitive to both color and pattern. As seen in Figures 6e, 6f, 6g, and 6i, while both the top (human-aligned) and bottom (original) rows contain images with similar patterns to the ground truth (query) image, those retrieved by human-aligned models also share similar colors with the ground truth. Additionally, while the original models sometimes better estimate object categories, human- aulnnne models appear to be more effective in retrieving the correct orientation and overall shape of the object (see Figures 6a, 6b, 6c, 6d, 6h, 6j). Table 10 show the retrieval performance when using the large image set for representative subject. The results show that for human-aligned models the top-1 performance can achieve up to 12.5% which is significantly higher than the chance performance (0.006%).
We also visualized some of the retrieved images using human-aligned models with gLocal as the alignment method, as shown in Figure 7. Interestingly, images retrieved from gLocal model embeddings exhibit greater consistency in object category compared to both the original unaligned versions and Dreamsim models. However, unlike Dreamsim, images retrieved from gLocal embeddings do not necessarily match the ground truth image in terms of orientation and color (see Figures 7b, 7c, 7d, and 7f).
This highlights an important point: each human-aligned model prioritizes attributes emphasized in the dataset used for alignment with human perception, which directly influences their learned representations (as discussed in Section 5.5). Additionally, the way participants experience image stimuli during visual brain data collection affects their perception of the images. Currently available datasets have been collected using the RSVP paradigm, which primarily captures low-level and mid-level features. As result, they align more closely with Dreamsim's 's approach. Future work should explore alternative experimental designs for brain data collection and investigate their relationship with different human- alignmmet methods.
Table 10. 400-wayPae zero-shot performance of the NICE encoder on EEG data of representative subject using a larger image database for retrieval. numbers show top-1 accuracies and top-5 in paranthesis.
Ensemble
SynCLR
CLIP
OpenCIIPP
DINO
DINOv2
HA
Base
HA
Base
HA
Base
HA
Base
HA
Base
HA
Base
$10
9.5 (22.0)
3.0 (12.0) 12.5 (28.0) 3.0 (12.0) 4.5 (14.5)
3.5 (8.5)
6.0 (19.5)
4.5 (11.5)
7.5 (24.0)
3.5 (12.5)
3.0(10.0)
0.5(2.0)
14

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
G. Grad-CAM Analysis Per Participant
G.I. Temporal Analysis
In Section 5.5, we plotted a histogram of large gradient values over time for a representative participant and image encoder. We extracted gradients from the projection layer of the NICE model's convolutional encoder (encoder. 0 www.ppoeettonoooo 0) for models trained with five different seeds on one participant's data. Only positive gradient values were kept, highlighting features that positively contributed to predictions. These gradients were then normalized between O and and interpolated to match the EEG input size. For each model (with specific seed), we determined the 99th percentile of gradient amplitudes across all 200 test samples and used this as the amplitude threshold for that model, resulting in five different thresholds. Finally, we computed a histogram showing the number of gradients exceeding their respective thresholds across time for all images and seeds (1,000 samples in total).
We extended this analysis to all participants and image encoders, with the results shown in Figures 8 to 10. The findings consistently indicate that human-aligned models focus more on earlier time windows, associated with low-level visual processing, compared their unaligned counterparts, further corroborating the main paper's 's results.
Figure 11 illustrates the same procedure applied to two sample models (the top performers in the retrieval task) trained with gLocal image embeddings. The results indicate that there is no consistent difference in the time windows attended to by gLocal models compared to the original models.
Finally, we computed the gradients for models trained with Harmonization embeddings. Figure 12 and Figure 13 show histograms of large gradient values over time for four different architectures. Compared their unaligned counterparts, the Harmonized models tend to exhibit larger gradients at later time points. Interestingly, this shift is more pronounced in the ConvNext and LEVIT architectures (Figure 12), where human alignment was shown to degrade the model performance (Figure 3c).
G.2. Spectral Analysis
To analyze the frequency contributions to the model's predictions, we computed the energy of the Fourier trans- form of the gradient maps. Following the same procedure as in the previous section, we extracted gradients from encoder. 0. projection. c, retained only positive values, normalized them between 0 and I, and interpolated them to match the EEG input size. The five frequency bands were defined as described in Section 5.5. We measured the power spectral density (PSD) using the periodogram method. The periodogram of a signal obtained by squaring Fourier transform values, as shown in Equation (3).
Sax (fx)
|x
k= 1=., N-1,
(3)
N
where X is the discrete Fourier transform of the signal and is computed as:
X
e 12nkn/N
(4)
The normalized energy of each frequency band was calculated by summing the power values within that band and dividing by the total signal energy (sum of power values across all frequencies). This process was performed separately for each seed, and the bars in Figure 4b and Figures 14-16 represent the mean and standard error across all seeds.
We repeated the procedure for different subjects and image encoders. As shown in Figures 14-16, models trained with human-aligned embeddings consistently exhibited greater focus on the Alpha and Beta bands, while original models emphasized the Delta band. This trend was observed across nearly all participants and image encoders. Although the Gamma band contributed more to predictions in some cases for human- allinedd models, its effect was less consistent than the differences observed in other frequency bands.
Following the same approach, we repeated the analysis for the two best-performing gLocal models. The results, visualized in Figure 17, show that, in most cases, the original models attend more to Alpha and Beta frequencies compared to the gLocal models.
Similar to the previous section, we computed the normalized energy of the gradients each EEG frequency band for the Harmonized models and compared them to their unaligned counterparts. Interestingly, for the LEVIT and ConvNext
15

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
architectures, where the unaligned models outperform their human-aligned versions-the original models show higher energy in the alpha and beta bands (Figure 18). In contrast, for ResNet-50 and VGG16, where the human-aligned models achieve slightly better performance, the aligned models exhibit greater attention to the alpha and beta bands (Figure 19).
As discussed in the paper, we believe the difference between gLocal and Dreamsim stems from the datasets used to train them. The superior performance of Dreamsim in visual decoding from brain activity may be due to its focus on information that is emphasized by the experimental design used to collect brain data. In contrast, other models are optimized to highlight features such categorical differences, which may be well-captured in brain data collected using RSVP experiments.
16

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Ourry Imuge
Qurry Inugo
Original
Ostinal
Query
tougs
Original 6:32
Origiaal 6.12
Original 6,32
Oriital
Orieteal
Oetitnl
Original
Original 0.29
Original
(c)
(d)
HAL 6:83
0.13
0.47
Original 0.27
Original 6.26
Oriinal 6.26
Origtal 0.23
Origteal 023
Original 0.33
Original
Ortginal
Original 0..33
Oriyinal 0.31
Hi 0..9
HL.
H1 0.48
IKL 0.60
Ouery Imsage
Query Image
0.35
Qpery Image
Query Image
Original
Original
Figure 6. Top-5 retrieved images from larger image database (Dreamsim). Query EEG signals were selected from the test set but the search for the corresponding image happened the combined test and train image sets. Top rows show aligned (HA) and bottom rows show original models. The numbers cosine similarity between the EEG and image embeddings. Dreamsim used as the alignment method.
17

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
0.33
0.33
Oser Imugy
Ourn Imary
(b)
Onery Iauae
Oenn baugy
Oriiiaal
Original
Oriiial 8.31
Origial
Original
Original Ostginal
Ortginal Original
0L.34
Origiaal 8:36
Original 6:36
Original 8.315
Original -835
Original 8:14
Orlginal 0.31
Original 8.33
Original -0.32
Ohiginal 8.33
Original- 8
HLL 0.38
ILL
ILL 0.37
8.1-0.37
IBL 0.37
HA-833
H&X 6L36
EIA 013-66166
HL-6.35
Origiaal- 8:34
8.34
-.31
Original -8.1
Origial 8:11
Deigihal-8..1
Original -.33
Ortyinal
Odiyial--8311
Original 8.31
(
(h)
Figure 7. Top-5 retrieved images from a larger image database (gLocal). Query EEG signals were selected from the test set but the search for the corresponding image happened the combined test and train image sets. Top show human-aligned (HA) and bottom rows show original models. The numbers are cosine similarity between the EEG and image embeddings.
18

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
51
S2
S3
SS
HA
0.025
Original
0.020
Probability
0.015
0.010
0.005
0.000
56
57
58
59
510
0.025
0.020
Probability
0.015
0.010
0.000
200
400
800
1000
400
600
800
1000
600
800
l000
200
400
600
800
1000
400
600
800
1000
(ms)
SynCLR
51
S2
S3
S4
55
0.030
HA
Original
0.025
0.020
0.015
0.010
0.005
0.000
57
S10
0.025
0.020
Probability
0.015
0.010
0.005
0.000
600
800
1000
200
400
600
800
1000
800
1000
200
400
600
890
1080
200
600
800
1000
Time (ms)
(b) ENSEMBLE
Figure 8. Comparing gradient maps for EEG models- temporal analysis (Part 1). EBG encoders trained with original unaligned and human-aligned image embeddings using Dreamsim as the alignment method.
19

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S2
S3
SS
0.030
HA Original
0.025
Probability
0.020
0.015
0.010
0.005
0.000
56
57
58
59
510
Probability
0.015
0.010
0.005
0.000
200
400
800
1000
400
600
800
1000
800
l000
200
400
600
800
1000
400
600
800
1000
(ms)
CLIP
51
$2
S3
S4
55
0.030
HA
Original
0.025
0.020
0.000
S7
S10
0.020
Probability
0.015
0.010
0.000
600
800
1000
200
400
600
800
1000
800
1000
200
400
600
800
1000
200
600
800
1000
Time (ms)
(b) OpenCLIP
Figure 9. Comparing gradient maps for EEG models-tempores temporal analysis (Part 2).EEG encoders trained with original unaligned and human-aligned image embeddings using Dreamsim as the alignment method.
20

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S2
S3
0.030
0.025
Original
Probability
0.020
0.015
0.010
0.005
0.000
56
57
58
59
510
0.020
Probability
0.015
0.010
0.005
0.000
200
400
800
1000
400
600
800
1000
600
800
1000
200
400
600
800
1000
400
600
800
1000
(ms)
(a) DINO
51
S2
S3
S4
55
0.025
HA
Original
Probabilty
0.015
0.010
0.005
0.000
57
510
0.020
Probability
0.010
0.005
0.000
600
800
1000
200
400
600
800
1000
800
1000
200
400
600
800
1080
200
400
600
800
1000
Time (ms)
(b) DINOv2
Figure 10. Comparing gradient maps for EEG models-temporal analysis (Part 3). EEG encoders trained with original unaligned and human-aligned image embeddings using Dreamsim as the alignment method.
21

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S2
S3
SS
0.025
HA
Original
0.020
Probability
0.015
0.010
0.005
0.000
56
S7
58
59
510
0.025
0.020
Probability
0.015
0.010
0.005
0.000
200
400
600
800
1000
200
400
600
800
1000
200
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
(ms)
CLIP-RN50
51
S2
S3
S4
55
0.025
HA
Original
0.020
Probability
0.010
0.005
0.000
S7
S8
S10
0.025
Probability
0.015
0.010
0.005
0.000
600
800
1000
200
400
600
800
1000
800
1000
200
400
600
800
1080
200
400
600
800
1000
Time (ms)
(b) DINO
Figure I.. Comparing gradient maps for EEG
analysis (gLocal), EEG encoders trained with original unaligned
and human-aligned image embeddings using gLocal as the alignment method.
22

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S2
S3
SS
0.025
Original
Probability
0.020
0.010
0.005
0.000
56
57
58
59
510
0.030
0.025
Probability
0.020
0.015
0.010
0.005
0.000
200
400
800
1000
200
400
600
800
1000
200
600
800
1000
200
400
600
800
1000
400
600
800
1000
(ms)
COnY
51
S2
S3
S4
55
0.035
HA
0.030
Original
0.025
0.015
0.005
0.000
57
510
0.035
0.030
0.025
Probability
0.020
0.010
0.005
0.000
600
800
1000
200
400
600
800
1000
800
1000
200
400
600
800
1000
200
400
600
800
1000
Time (ms)
(b) LEVIT
Figure 12. Comparing Gradient Maps for EEG Models-Temporrs Analysis (HarmonizationnPaal 1), EEG encoders trained with original unaligned and human-aligned image embeddings using Harmonization as the alignment method.
23

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S2
S3
Original
0.020
Probability
0.015
0.010
0.005
0.000
56
57
58
59
510
0.025
0.020
Probability
0.015
0.010
0.005
0.000
200
400
800
1000
200
400
600
800
1000
200
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
(ms)
(a) ResNet-50
51
S2
S3
S4
55
0.025
HA
Original
0.020
Probability
0.010
0.000
57
510
0.025
Probability
0.015
0.010
0.005
0.000
600
800
1000
200
400
600
800
1000
800
1000
200
400
600
800
1080
200
600
800
1000
Time (ms)
(b) VGG16
Figure 13. Comparing gradient maps for EEG models-temporal analysis (Harmonization-Pan 2). EEG encoders trained with original unaligned and human-aligned image embeddings using Harmonization as the alignment method.
24

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S5
Human Aligned Original
0.5
Nomalized Energy
0.3
0.1
0.0
56
57
S8
59
510
0.5
Nomalized Energy
0.4
0.3
0.2
0.1
0.0
Dett
Thera
Alpha
Beta
Gamma
Detta
Theta
Alptia
Beta
Gamma
Datta
Theta
Alpna
Beta
Gammaa
Deta
Thete
Alpha
Beta
Gamma
Detaa
Thete
Alpna
Beta
Gamma
Prequency Band
(a) SynCLR
0.5
Huuman-Aligned Original
Noralized Energy
0.3
0.1
56
S7
S8
59
S10
0.5
Normalized Energy
0.4
0.3
0.2
0.0
Datta
Thota
Alpha
Bata
Gamma
Delt
Thea
Alpha
Beta
Gamma
Detta
Mhra
Alpha
Bera
Gamma
Deta
Thata
Alpha
Beta
Gamma
Delta
Theta Alpta
Beta
Gamma
Frequency Band
(b) ENSEMBLE
Figure Comparing gradient maps for EEG models-spectral analysis (Part 1). EBG encoders are trained with original unaligned and human-aligned image embeddings using Dreamsim as the alignment method. Stars over bars represent a significant difference (greater) with p < 0.05.
25

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S5
0.5
Human Huimen-ligneed Original
Normalzzdd Energy
0.4
0.2
0.1
0.0
56
57
S8
59
510
0.5
Nomalized Energy
0.4
0.3
0.2
0.0
Dett
Thera
Alpha
Beta
Gamma
Detta
Theta
Alptia
Beta
Gamma
Datta
Theta
Alpna
Beta
Gamma
Deta
Thete
Alpha
Beta
Gamma
Detaa
Thete
Alpna
Beta
Gamma
Frequency Band
CLIP
Huuman-AAigned Original
Noralized Energy
0.4
0.3
0.1
56
S7
S8
59
S10
0.5
Normalized Energy
0.4
0.3
0.2
0.0
Datta
Thota
Alpha
Bata
Gamma
Delta
Thea
Alpha
Beta
Gamma
Detta
Thera
Alpha
Bera
Gamma
Dela
Thata
Alpha
Beta
Gamma
Delaa Theta Alpta
Beta
Gamma
Frequency Band
(b) OpenCLIP
Figure Comparing gradient maps for EEG models-spectral analysis (Part EBG encoders are trained with original unaligned and human-aligned image embeddings using Dreamsim as the alignment method. Stars over bars represent a significant difference (greater) with p < 0.05.
26

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S5
Human Aligned Original
Noralizedd Energy
0.4
0.3
0.2
0.0
56
57
58
59
510
Nomalized Energy
0.4-
0.3
0.1
0.0
Dett
Thera
Alpha
Beta
Gamma
Detta
Theta
Alptia
Beta
Gamma
Datta
Thera
Alpna
Beta
Gammia
Deta
Thete
Alpha
Beta
Gamma
Detaa
Theta
Alpna
Beta
Gamma
Frequency Band
(a) DINO
Huuman-Aligned Original
0.5
Nomalzzed Energy
0.4
0.2
0.1
56
57
S8
59
S10
0.5
Normalized Energy
0.4
0.1
0.0
Datt
Thota
Alpha
Bata
Gamma
Delta
Thea
Alpha
Beta
Gamma
Detta
Mera
Alpha
Bera
Gamma
Deta
Thata
Alpha
Beta
Gamma
Detaa
Theta Alpta
Beta
Gamma
Frequency Band
(b) DINOv2
Figure Comparing gradient maps for EEG models-spectral analysis (Part 3), EBG encoders are trained with original unaligned and human-aligned image embeddings using Dreamsim as the alignment method. Stars over bars represent a significant difference (greater) with p 0.05.
27

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Human Aligned Original
Nomalized Energy
0.4
0.3
0.2
0.1
0.0
56
57
58
59
510
0.6
Normalized Energy
0.5
0.4
0.3
0.2
0.1
0.0
Delta
Theta
Alpna
Beta
Gamma
Deta
Theta
Alpta
Beta
Gamma
Dataa
Theta
Alpraa
Beta
Gamma
Deta
Theta
Alpna
Beta
Gamma
Deta
Theta
Alpna
Beta
Ganma
Frequency Band
CLIP-RNSO
0.5
Humann-Aligner Original
Nomalized Energy
0.3
0.1
56
S7
SB
59
S10
0.5
Normalized Energy
0.4
0.3
0.2
0.0
Dalta
Theta
Alpha
Bata
Gamma
Dett
Thea
Alpha
Beta
Gania
Delta
Meta
Alpha
Beta
Gamma
Delta
Theta
Alpha
Beta
Gamma
Detaa
Theta Alpta
Beta Gamma
Frequency Band
(b) DINO-VITI V
Figure 17. Comparing gradient maps for EEG moeets-spectral analysis (gLocal). EEG encoders are trained with original unaligned and human human-aligned image embeddings using gLocal as the alignment method. Stars over bars represent significant difference (greater) with p < 0.05.
28

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
Human Hligen-lignned Original
Normalized Energy
0.3
0.0
57
S8
59
510
0.6
Nomalized Energy
0.5
0.2
0.1
0.0
Dett
Thera
Alpha
Beta
Gamma
Deta
Theta
Alpria
Beta
Gamma
Dataa
Thera
Alpna
Beta
Gamma
Deta
There
Alpha
Beta
Gamma
Detaa
Theta
Alpnaa
Bata
Gamma
Prequency Band
ConyNEXT
Hluman-AAigned Original
0.5
Normaized Energy
0.4
0.3
0.1
56
57
S8
59
S10
0.5
Normalized Energy
0.4
0.3
0.2
0.1
0.0
Datt
Thota
Alpha
Bata
Gamma
Delta
Thta
Alpha
Beta
Gamma
Detta
Mhra
Alpha
Beda
Gamma
Deta
Thata
Alpha
Beta
Gamma
Detaa
Theta
Alpta
Beta
Gamma
Frequency Band
(b) LEVIT
Figure Comparing gradient maps for EEG models-spectral analysis (Harmonization-Pat 1). EEG encoders are trained with original unaligned and human-aligned image embeddings using Harmonization the alignment method. Stars over bars represent significant difference (greater) with p < 0.05.
29

---

Human-Aligned Image Models Improve Visual Decoding from the Brain
S5
Human Huimen-ligneed Original
Nomalized Energy
0.4
0.3
0.2
0.1
0.0
56
57
58
59
510
0.5
Noralized Energy
0.4
0.3
0.0
Dett
Thera
Alpha
Beta
Gamma
Deta
Theta
Alptia
Beta
Gamma
Dataa
Thera
Alpna
Beta
Gamma
Deta
There
Alpha
Beta
Gamma
Detaa
Thete
Alpna
Beta
Gamma
Prequency Band
(a) ResNet-50
Huuman-AAignedd Original
Noralized Energy
0.4
0.3
0.2
0.1
56
S7
S8
59
S10
Normallzed Energy
0.4
0.3
0.1
0.0
Datt
Thota
Alpha
Bata
Gamma
Delt
Thia
Alpha
Beta
Gamma
Detta
Mhra
Alpha
Bera
Gamma
Deta
Thata
Alpha
Beta
Gamma
Delta
Theta
Alpta
Beta
Gamma
Frequency Band
(b) VGG16
Figure Comparing gradient maps for EEG models-spectral analysis (Harmonization-Part 2). EEG encoders are trained with original unaligned and human-aligned image embeddings using Harmonization the alignment method. Stars over bars represent significant difference (greater) with p < 0.05.
30


Reviews:
[{"review_id": "mevLBslHyR", "reviewer": "Reviewer_ewMg", "summary": "The paper explores the application of human-aligned image encoders to enhance the decoding of visual information from brain signals, specifically EEG and MEG data. The authors propose that image encoders fine-tuned to align with human perceptual similarity judgments improve the mapping of brain activity to visual stimuli compared to standard pre-trained encoders. The main findings indicate that integrating human-aligned image encoders, such as Dreamsim, into a brain-to-image decoding framework increases image retrieval accuracy. The study employs a contrastive learning approach, aligning brain signal embeddings with image embeddings in a shared latent space using the InfoNCE loss. Comprehensive experiments across various EEG architectures, image encoders, and brain imaging modalities demonstrate consistent performance improvements. The authors also provide biological insights, showing that human-aligned models focus on early visual processing features and frequency bands linked to visual perception.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ewMG for their thorough review and constructive feedback. Below, we provide our response to your questions as well as some clarifications on the points you raised.\n\n**Regarding the image reconstruction task:**\n\n Thank you for this thoughtful comment. We agree that visual decoding goes beyond retrieval, and incorporating reconstruction would strengthen our claims. We will revise the framing to more clearly reflect our focus on evaluating whether human-aligned image embeddings improve alignment with brain activity. Since retrieval directly reflects this alignment, it was our primary evaluation metric.\n \nState-of-the-art reconstruction methods typically use pretrained diffusion models based on latents from the CLIP representation space and project brain activity to that using an alignment loss such as contrastive or MSE loss (Scotti et al. 2024, Li et al. 2024, Benchetrit et al. 2024). However, a challenge is that current generative models are trained on original CLIP embeddings, so using human-aligned embeddings would require retraining or fine-tuning the generative model. This task is broader than brain-image alignment and primarily tests the general utility of human-aligned embeddings as a latent space for generative diffusion models. We thought our setup provided a more direct and easier-to-evaluate scenario that allows us to isolate the effect of human alignment better, and that is why it was our focus. \n\nAn alternative, inspired by MindEye, decouples retrieval and reconstruction. In our case, we can have two separate representation spaces: In one, brain signals are aligned with human-aligned embeddings for retrieval and with the original CLIP embeddings for reconstruction. We are also experimenting with a two-stage pipeline: brain signals are first mapped to the human-aligned space (for retrieval) and then further projected into the original CLIP space for reconstruction. This framework is implemented and under active evaluation. Preliminary results are promising, but further experimentation is needed to draw firm conclusions. \n\nWe hope our ongoing work and future directions help address your concern regarding the broader impact of the study.\n\n**Regarding the use of other datasets:** \n\nThank you for this question. Our primary focus is on EEG, as decoding performance in this modality remains relatively low and presents clear opportunities for improvement. We included MEG due to its similar input structure, allowing us to use the same architectures and demonstrate consistent gains from aligning neural signals with human-aligned image embeddings. \n\nWe agree that including other modalities, such as fMRI, could further strengthen our claims. In response to your suggestion, we extended our experiments to the NSD dataset (Allen et al., 2022) and observed similar improvements in retrieval performance using human-aligned embeddings. These results, will be included in the revision and reinforce the generalizability of our findings across neural recording modalities.\n\n|       | CLIP     |          | DINOv2   |          | DINO     |          | ENSEMBLE |          | OpenCLIP |          | SynCLR   |          |\n|-------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n|       | HA       | Base     | HA       | Base     | HA       | Base     | HA       | Base     | HA       | Base     | HA       | Base     |\n| top-1 | **45.0±4.3** | 23.8±2.4 | **45.3±3.6** | 22.5±2.1 | **49.1±3.7** | 45.9±3.7 | **54.6±4.3** | 48.1±4.0 | **49.5±4.3** | 35.4±3.8 | **58.2±4.6** | 48.2±4.3 |\n| top-5 | **76.6±4.2** | 53.8±4.1 | **75.5±3.5** | 49.1±3.2 | **79.7±3.7** | 76.8±3.6 | **83.4±3.7** | 79.6±3.4 | **79.2±4.1** | 67.8±4.3 | **86.7±3.6** | 78.7±3.1 |", "claims_and_evidence": "The primary claim—that human-aligned image encoders improve visual decoding from brain signals—is supported by extensive empirical evidence. The authors present quantitative results, such as top-1 and top-5 retrieval accuracies, across multiple participants, image encoders, and EEG architectures, showing consistent improvements with human-aligned models. The 21% improvement over state-of-the-art (from 28% to 62% top-1 accuracy with NICE and Ensemble models) is convincingly demonstrated in Figure 1 and Table 1. Additional evidence from gradient analyses (Figures 4 and 5) supports the claim that these models capture perceptually relevant signal components, aligning with biological expectations of visual processing.\n\nHowever, the broader claim that these models enhance \"visual decoding\" in general is less substantiated, as the evaluation focuses solely on image retrieval tasks without exploring other decoding paradigms, such as image reconstruction. This limitation weakens the generalizability of the claim beyond retrieval. The evidence is clear and convincing within the scope of retrieval, but the paper overstates its implications for brain decoding as a whole without broader task validation.", "methods_and_evaluation_criteria": "The methods—using pre-trained human-aligned image encoders (e.g., Dreamsim) and existing EEG/MEG encoders (e.g., NICE)—are appropriate for the retrieval task, leveraging established contrastive learning frameworks (InfoNCE loss) to align brain and image embeddings. The evaluation criteria, primarily top-1 and top-5 retrieval accuracy on the Things EEG2 and MEG datasets, are standard for retrieval tasks and align with prior work (e.g., Song et al., 2024). However, the reliance on a single EEG dataset (Things EEG2) and one MEG dataset limits the robustness of the findings across diverse brain signal recordings. While the datasets are well-suited for the task, the lack of variety in datasets and tasks (e.g., reconstruction) restricts the method's applicability.", "theoretical_claims": "The paper does not present formal theoretical proofs but proposes a hypothesis: human-aligned encoders improve decoding by better capturing perceptual attributes reflected in brain signals. This is supported empirically.", "experimental_designs_or_analyses": "I reviewed the experimental designs in Sections 4 and 5, focusing on the retrieval tasks. The design is sound: training on averaged EEG/MEG repetitions, using a 90/10 train/validation split, and testing on unseen images with multiple seeds ensures reproducibility and reduces overfitting. The use of paired T-tests is statistically valid, with significance levels (p<0.05) appropriately reported. The usage of architectures and main training loss for alignment is mostly well-reported and ensures reproducibility. However, the method for creating a human-aligned model, though referenced, were not explained clearly in the paper.", "relation_to_broader_scientific_literature": "The paper mainly builds upon the Dreamsim work (Fu, 2023) to create its main results, proving its effectiveness over classical vision models like CLIP (Radford, 2021). In terms of decoding the paper is related to Decoding natural images from eeg for object recognition (Song 2024), which takes a step further by utilizing human-aligned image encoders for contrastive learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: The paper’s strength lies in its thorough experimentation—spanning multiple encoders, modalities, and participants—demonstrating robustness within the retrieval task. The idea of leveraging human-aligned models is conceptually sound, addressing the semantic collapse in text-paired contrastive learning (e.g., CLIP), and the biological insights (Section 5.5) add depth. The clarity of presentation, with figures (e.g., Figure 5) and tables, enhances accessibility.\n\nWeaknesses: Originality is limited, as the core methods (Dreamsim, NICE) are borrowed, and the contribution is an application rather than a novel algorithmic advance. The significance is tempered by the narrow focus on retrieval, omitting tasks like reconstruction, which are critical for general brain decoding (e.g., Scotti et al., 2023, used Stable Diffusion for reconstruction). The reliance on a single EEG dataset (Things EEG2) further limits generalizability. While the direction is promising, the work feels accumulative, reinforcing the utility of human-aligned models rather than breaking new ground.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Why was image reconstruction not explored alongside retrieval to support the broader claim of improved visual decoding? A response demonstrating feasibility or plans for reconstruction could strengthen the paper’s significance, potentially shifting my recommendation to weak accept. Without this, the claim feels overstated.\n\nCan you justify the exclusive use of the Things EEG2 dataset and MEG dataset, given other image-paired neural datasets (e.g., Allen et al., 2022) could test robustness?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qg8tU8sDZn", "reviewer": "Reviewer_9vzB", "summary": "This paper compares the image identification (decoding) performance of human-aligned image embedding models and their unaligned counterparts. The authors find that human-aligned models generally performed better than the unaligned models in EEG and MEG, after evaluating several different base image encoders and brain signal encoders. The authors also claim that the gradients of the EEG encoder more closely match human perception when using human-aligned image embeddings yields.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 9vzB for their careful review and constructive suggestions. We appreciate your thoughtful engagement with our work.\n\nYour concern about the image encoders differing across alignment methods is valid. However, we would like to clarify that our comparisons are always made within the same image encoder, specifically, between its original embeddings and its human-aligned embeddings. Our goal is to assess whether human alignment improves brain-image alignment for a given encoder, not to compare across encoder architectures. The inclusion of multiple image encoders was intended to test the consistency of this effect across different visual representation models. Thank you also for suggesting the inclusion of Nishimoto et al., 2011. We agree it is a relevant and influential study, and we will incorporate it into the related work section of the revised paper. Below, we address your specific questions in detail.\n\n**Question 1:**\n\nThank you for this important question. Based on Fu et al. (2023), the THINGS categories used in the THINGS-EEG and THINGS-MEG datasets are not directly included in the NIGHTS dataset. The Harmonization component of NIGHTS is based on ImageNet-derived datasets. As for gLocal, it uses odd-one-out triplets from the same THINGS database, so there is likely some concept overlap between its training data and the EEG/MEG test sets.\n\nHowever, we would like to emphasize two points. First, despite potential concept overlap, gLocal-aligned models do not outperform unaligned models on EEG/MEG retrieval, suggesting that overlap alone does not explain performance differences. Second, these concepts are already likely present in the pretraining datasets of the original image encoders, which are large-scale vision models trained on broad and diverse datasets. Therefore, any overlap applies equally to both human-aligned and unaligned models and does not undermine the claims of the paper.\n\n**Question 2:**\n\nIn the main results, the encoder only has access to the 200 test concepts and their associated images during retrieval. We will clarify this more explicitly in the revised text. For completeness, we also evaluated retrieval on a larger image database—including both training and test concepts—and reported those results in Appendix E. Notably, models trained with human-aligned embeddings continue to outperform the original models in this more challenging setting.\n\n**Question 3:**\n\nOne possible explanation is that DINOv2 embeddings may capture different visual features or emphasize different inductive biases compared to DINO, leading to a less optimal alignment with EEG representations in the unaligned setting. After human alignment, however, both models are optimized toward human perceptual similarity, which reduces the gap by bringing their representations closer to behaviorally relevant dimensions. This suggests that human alignment can partially mitigate architectural differences by enforcing a shared similarity structure.\n\n**Question 4:**\n\nCross-subject decoding introduces additional variability due to individual differences in neural representations, which may overshadow the benefits of human alignment. In this setting, the EEG encoder must generalize across subjects, making it more difficult to exploit the finer-grained structure introduced by human alignment.\n\n**Question 5:**\n\nThank you for catching this. The reported value is a typo—the correct performance for OpenCLIP with HA on S1 is **11.5 ± 0.55**. As noted in the paper, all results are averaged over five random seeds to reduce sensitivity to a specific seed. We will correct this in the revision.\n\n**Question 6:**\n\nYou are right; we only included temporal gradient maps for Dreamsim and gLocal in the original submission. Based on your suggestion, we computed the gradient distributions over time for models trained with Harmonization-aligned embeddings and will include them in Appendix F1 of the revised paper. As anticipated, we do not observe a consistent shift toward earlier time points compared to the unaligned models. This suggests that Harmonization-aligned features do not rely more heavily on early, low-level EEG responses, and the alignment may not strongly alter temporal emphasis in this context.", "claims_and_evidence": "1. The authors claim that Dreamsim performs better than the two alignment alternatives (gLocal and Harmonization) and go on to speculate the cause. But, the image encoders used to measure improvement are not the same for each alignment method, so their results are not directly comparable. In Figure 3, it is also unclear which image encoders _do_ have direct comparisons between alignment methods (i.e., which encoders are both in 3a and 3b).", "methods_and_evaluation_criteria": "The authors use retrieval to evaluate the quality of their decoder.\n1. I believe some of the specifics about the retrieval process (like which images are made available to the decoder) are not stated.\n2. Further, I believe a task like open domain reconstruction (rather than selecting from a known set of candidates) would be more interesting, albeit significantly more computationally expensive. I think this because the test set, unlike the training set, seems to contain 200 concepts each with a single unique image. This means that the model might not need to learn to (for example) differentiate between viewpoints of a single object -- simply differentiating something like texture or color may be sufficient.", "theoretical_claims": "No theoretical claims were made in this paper.", "experimental_designs_or_analyses": "The analyses seem sound -- in particular, the authors are explicit about which concepts are distinct to which split of the EEG and MEG datasets. I do have a concern about an overlap in concepts in the human alignment datasets and the brain datasets. (I ask about this in the Questions section as well.)", "relation_to_broader_scientific_literature": "This work shows that human-alignment efforts in machine learning may have deeper impacts in modeling (decoding) brain processes.", "essential_references_not_discussed": "Unless the authors are specifically discussing previous efforts to decode still images in Lines 154-161 (which I believe they should not), discussion of Nishimoto et al. 2011 paper belongs in Section 3.1: https://www.sciencedirect.com/science/article/pii/S0960982211009377", "other_strengths_and_weaknesses": "Other strengths:\n1. The authors are thorough and trained a wide combination of EEG encoders and image encoders.", "other_comments_or_suggestions": "1. In Figure 3a, consider reordering the bars to match the order of the models in Tables 1-3.", "questions_for_authors": "1. Were any of the concepts in the EEG and MEG datasets also present in the human alignment datasets (e.g. NIGHTS)? In particular were they part of the test split of the brain datasets?\n2. At test time, which concepts/images is the encoder given access to? Only the 200 images associated with the 200 test concepts, or also the images/concepts from the training set? (If I missed this in the paper, please point me to where it was discussed.)\n2. In Table 1, the base performance of DINOv2 is substantially worse than DINO, though the gap narrows after human alignment. What could be the cause of this?\n3. In Table 2, the Ensemble and DINO models show no improvement after HA. What makes cross-subject decoding so different from within-subject decoding to cause this?\n4. In Table 2, S1 shows a catastrophic drop in performance for OpenCLIP after HA. What is going on there? It doesn't seem to just be a bad seed.\n5. Around Line 408 in column 2, the authors speculate that the datasets for Harmonization emphasize features that \"require careful inspection and attention\", while the RSVP setup of the experiments emphasize the recognition of more low-/mid-level features. This would be borne out by looking at the temporal gradients -- is a shift toward later timepoints actually seen in the data? (In the appendix I saw some gradient maps for gLocal but not Harmonization.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T2WWZTBe8S", "reviewer": "Reviewer_q6J2", "summary": "This paper explores the problem of decoding visual images from the brain by replacing the visual encoder with a human-aligned visual encoder. The experiments demonstrate that the use of human-aligned visual encoder effectively improves the brain-image retrieval performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer q6J2 for their thoughtful review. Please find our responses to your concerns and suggestions below.\n\n**Regarding the image reconstruction task:**\n\nThank you for highlighting this important point. We agree that image reconstruction is a key aspect of brain decoding and would be an important addition to our retrieval results. Our current focus is on retrieval as it provides a direct measure of alignment between brain activity and image embeddings—our primary objective in this work. That said, we recognize the significance of reconstruction and are exploring it in our current work.\n\nState-of-the-art reconstruction methods typically use pretrained diffusion models based on latents from the CLIP representation space and project brain activity to that using an alignment loss such as contrastive or MSE loss (Scotti et al. 2024, Li et al. 2024, Benchetrit et al. 2024). However, a challenge is that current generative models are trained on original CLIP embeddings, so using human-aligned embeddings would require retraining or fine-tuning the generative model. This task is broader than brain-image alignment and primarily tests the general utility of human-aligned embeddings as a latent space for generative diffusion models. We thought our setup provided a more direct and easier-to-evaluate scenario that allows us to isolate the effect of human alignment better, and that is why it was our focus.\n\nAn alternative, inspired by MindEye, decouples retrieval and reconstruction. In our case, we can have two separate representation spaces: In one, brain signals are aligned with human-aligned embeddings for retrieval and with the original CLIP embeddings for reconstruction. We are also experimenting with a two-stage pipeline: brain signals are first mapped to the human-aligned space (for retrieval) and then further projected into the original CLIP space for reconstruction. This framework is implemented and under active evaluation. Preliminary results are promising, but further experimentation is needed to draw firm conclusions.\n\nWe hope our ongoing work and future directions help address your concern regarding the broader impact of the study.\n\n**Regarding the novelty:**\n\nThank you for the feedback. While the model design is intentionally simple, we believe the contribution lies in the insight that brain signals are more aligned with image embeddings that have been aligned with human similarity judgments. This alignment raises the interesting question of whether the representations of currently used models are suitable for brain signals. Also, it opens new doors to further investigate the connection between human-alignment and brain signal decoding. To our knowledge, this is the first work to systematically evaluate the impact of human-aligned representations on brain-based image retrieval across multiple modalities, models, and datasets. We hope this finding encourages future work to explore cognitively grounded embeddings in brain decoding.\n\n**Regarding the biological interpretations:**\n\nThank you for this important observation. We agree that the visualizations should be interpreted with caution. As noted in the paper, we performed statistical tests to assess the significance of the differences and only reported effects that met the significance threshold. However, we acknowledge that these differences—while statistically supported—are subtle and should be viewed as suggestive evidence rather than definitive explanations.\n\nOur intent was to provide preliminary insight into how human alignment might influence model attention across spectral and spatial domains. We agree that more extensive visualizations across additional models and datasets would help strengthen these interpretations. While additional visualizations for other models and participants are included in Appendix F2, we will revise the text to better highlight the limitations of these findings.", "claims_and_evidence": "See Other Strengths And Weaknesses.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "See Other Strengths And Weaknesses.", "experimental_designs_or_analyses": "See Other Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "See Other Strengths And Weaknesses.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n\n- This paper is well-written and easy to follow.\n- This paper conducts extensive experiments across various EEG and image encoders, demonstrating the effectiveness of the use of human-aligned image encoders.\n- This paper reveals an interesting finding that human-aligned encoders demonstrate greater compatibility with brain visual decoding tasks.\n\nWeaknesses\n\n- This paper focuses only on image retrieval task. However, brain decoding has another important task, reconstructing images from brain signals, which is more challenging and important. Extending the experiments to include the reconstruction task would provide a more convincing demonstration of the method’s effectiveness.\n- The novelty is limited. Although effective, the model’s design and alignment strategy is quite simple, i.e., using the human-aligned models to align with EEG features through contrastive loss. \n- The visualization results in Fig. 4 cannot fully support the biological interpretation in Section 5.5. According to Fig. 4 (b), the PSD difference of human-aligned models and original models is trivial，making it difficult to conclude that “Models trained with original embeddings focused more on Delta, while human-aligned models emphasized Alpha, Beta, and Gamma”. Additionally, in Fig. 4(c), EEG features aligned using different methods appear to focus on nearly identical brain regions, which does not sufficiently explain why the human-aligned models perform better. Further visualization across more models may be necessary to provide stronger evidence for these claims.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses.", "questions_for_authors": "See Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "arc60BEznk", "reviewer": "Reviewer_KJCA", "summary": "The authors tackle the task of image decoding from brain activity (EEG and MEG). On this task, most of the existing papers map brain data to pretrained vision encoders such as CLIP and DINO. The authors propose to analyze the role of these vision encoders and show that aligning them with human perception boosts performance on image retrieval tasks. They conduct experiments by analyzing different types of neural architectures for mapping brain data to vision embeddings and provide a biological analysis showing the representation learned by these human-aligned encoders have more meaningful spectral and temporal response compared to non-aligned ones.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer KJCA for their constructive review and insightful suggestions. We agree that evaluating the impact of human-aligned embeddings across a broader range of vision tasks would be highly valuable. In this work, we focused on image retrieval as a direct and interpretable proxy for brain-image alignment. Nonetheless, we see strong potential for extending this framework to other tasks that rely on shared embedding spaces. As part of our ongoing work, we are exploring image reconstruction and plan to investigate whether human-aligned embeddings can also enhance brain-to-text mappings.\n\nRegarding your suggestion on incorporating fMRI data, we fully agree. In response, we have conducted additional experiments using the Natural Scenes Dataset (NSD) which has been commonly used in similar studies. The results—provided below—show a consistent improvement in retrieval performance when using human-aligned embeddings over unaligned ones, further supporting the generalizability of our findings.\n\n|       | CLIP     |          | DINOv2   |          | DINO     |          | ENSEMBLE |          | OpenCLIP |          | SynCLR   |          |\n|-------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n|       | HA       | Base     | HA       | Base     | HA       | Base     | HA       | Base     | HA       | Base     | HA       | Base     |\n| top-1 | **45.0±4.3** | 23.8±2.4 | **45.3±3.6** | 22.5±2.1 | **49.1±3.7** | 45.9±3.7 | **54.6±4.3** | 48.1±4.0 | **49.5±4.3** | 35.4±3.8 | **58.2±4.6** | 48.2±4.3 |\n| top-5 | **76.6±4.2** | 53.8±4.1 | **75.5±3.5** | 49.1±3.2 | **79.7±3.7** | 76.8±3.6 | **83.4±3.7** | 79.6±3.4 | **79.2±4.1** | 67.8±4.3 | **86.7±3.6** | 78.7±3.1 |\n\n**Regarding your question:**\n\nThank you for raising this question. Interestingly, while the original (unaligned) performance of models like DINO is comparable to that of human-aligned CLIP and OpenCLIP in the cross-subject setting, human alignment improves performance only for the latter. One possible explanation is that CLIP and OpenCLIP embeddings, due to their language supervision, have more semantically structured representations that benefit from additional alignment with human similarity judgments. In contrast, DINO and SynCLR—trained with self-supervised visual objectives—may already be well-matched to the structure of EEG signals in a way that does not benefit further from human alignment in the cross-subject setting. This suggests that the effectiveness of human alignment in promoting generalization may depend on the underlying inductive biases of the image encoder.", "claims_and_evidence": "The claims are clear.", "methods_and_evaluation_criteria": "The method makes sense and is well justified.", "theoretical_claims": "No proofs", "experimental_designs_or_analyses": "The experimental part is well designed but would benefit from additional experimental results to validate the claim stating that human-aligned models are better at decoding. \n- Evaluation on more tasks. The authors only evaluate models on image retrieval. it would be interesting to see if it generalizes to other vision tasks such as image generation, captioning.\n- Evaluation on other brain modalities. The authors choose to focus on MEG and EEG data using THINGS datasets. Since the fMRI modality is also available in THINGS dataset and is commonly used for image decoding, it would be valuable to examine the impact of human-aligned encoders on fMRI data to determine if the findings are also applicable to this modality too.", "relation_to_broader_scientific_literature": "The paper focuses on image decoding from brain activity, a field that has seen significant advancements in recent years due to the emergence of large pretrained vision models. Most existing studies involve aligning brain representations with image encoders like CLIP which was shown to not align well with human perception. This paper demonstrates that using an image encoder better aligned with human similarity judgments yields a better decoding performance on retrieval tasks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is well written and easy to follow. It presents interesting findings by showing that human-aligned image encoders enhance the performance of image retrieval from brain signals, opening up new avenues for future research.", "other_comments_or_suggestions": "No", "questions_for_authors": "The results related to the generalization to other subjects in Section 5.3 are not very clear. Indeed, could you please elaborate on why the generalization would work for CLIP and openCLIP and not DINO network ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an application human-aligned image encoders to enhance the decoding of visual information from brain signals, alongside a series of evaluations on EEG and MEG data. Reviewers reported felt that the  claim—that human-aligned image encoders improve visual decoding from brain signals in the form of image retrieval—is well supported, but there were some concerns that the claim could be strengthened by inlcuding additional modalities, such as fMRI data, and that the claim was perhaps too broadly articulated in the paper, given that there were no image reconstruction tasks reported. More specifically, reviewers reported the following strengths:
* The paper is well-written and easy to follow.
* Extensive experiments across various EEG and image encoders, demonstrate the effectiveness of the use of human-aligned image encoders.
* The proposed method consistently improved measured alignment across multiple models and subjects, in some cases by a wide margin.
* The work included interesting additional findings, including those related to spectral and temporal responses.

Specific limitations and concerns discussed by reviewers include:
* The paper focuses on image retrieval task, while image reconstruction from brain signals, is another relevant, and more challenging, task. Experiments on the reconstruction task would provide a more convincing demonstration of the method’s effectiveness. Nonetheless, the authors robustly defend their focus on retrieval as it provides a direct measure of alignment between brain activity and image embeddings. 
* The novelty is limited. Although effective, the model’s design and alignment strategy draws heavily on prior work.

The authors are encouraged to revise their paper along the lines they discuss in reviewer responses, including re-framing the motivation to more clearly reflect a focus on image retrieval, and adding the additional results on fMRI data from (Allen et al., 2022). 

While some reviewers increased their scores on rebuttal, no one reviewer definitively championed the paper (all recommending "weak accept"). I have reflected this in my recommendation.
Decision: Accept (poster)