SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding
Authors: Juhyeon Park
Peter Yongho Kim
Jiook Cha
Shinjae Yoo
Taesup Moon
Venue: ICLR 2026 Poster (2026)
Keywords: Brain Decoding
fMRI
Evaluation Metric
Primary area: applications to neuroscience & cognitive science

Abstract: We present SEED ($\textbf{Se}$mantic $\textbf{E}$valuation for Visual Brain $\textbf{D}$ecoding), a novel metric for evaluating the semantic decoding performance of visual brain decoding models. It integrates three complementary metrics, each capturing a different aspect of semantic similarity between images inspired by neuroscientific findings. Using carefully crowd-sourced human evaluation data, we demonstrate that SEED achieves the highest alignment with human evaluation, outperforming other widely used metrics.
Through the evaluation of existing visual brain decoding models with SEED, we further reveal that crucial information is often lost in translation, even in the state-of-the-art models that achieve near-perfect scores on existing metrics. This finding highlights the limitations of current evaluation practices and provides guidance for future improvements in decoding models.
Finally, to facilitate further research, we open-source the human evaluation data, encouraging the development of more advanced evaluation methods for brain decoding. Our code and the human evaluation data are available at https://github.com/Concarne2/SEED.

Published as a conference paper at ICLR 2026
SEED: TOWARDS MORE ACCURATE SEMANTIC EVAL- UATION FOR VISUAL BRAIN DECODING
Juhyeon Park', Peter Yongho Kim Jiook Cha Shal,, You*, Taesup Moon 1,2,5 IPAL Scoul National University, "ECE, Scoul National University. spsychology, Seoul National University, 'Brookhaven National Lab, 5 "ASRI/ INMC/ AIIS, Scoul National University (parkjh9229, peterkim98, (sncon)@snu.ae
ABSTRACT
We present SEED (Semantic Evaluation for Visual Brain Decoding), a novel met- ric for evaluating the semantic decoding performance of visual brain decoding models. It integrates three complementary metries, each capturing a different aspect of semantic similarity between images inspired by neuroscientific find- ings. Using carefully crowd-sourced human evaluation data, we demonstrate that SEED achieves the highest alignment with human evaluation, outperform- ing other widely used metrics. Through the evaluation of existing visual brain decoding models with SEED, we further reveal that crucial information is often lost in translation, even in the sfate-of-the-art models that achieve near-perfect scores on existing metrics. This finding highlights the limitations of current evaluation practices and provides guidance for future improvements in decoding models. Finally, to facilitate further research, we open-source the human evalu- ation data, encouraging the development of more advanced evaluation methods for brain decoding. Our code and the human evaluation data are available at ptttps://gittub..com///
INTRODUCTION
Visual brain decoding focuses on reconstruct- ing visual stimuli from brain signals, such as functional magnetic resonance imaging (IMRI), thereby bridging the fields of neuro- science and computer vision. This field of re- search pivotal for developing
GT
Recon
Decoding
Model
compuer interface (BCI) systems (Mai et al. 2024 Du al.] 2022, Saha et al. (021) and provides key insights into the working mechanisms of complex human perceptual systems (Mai et al.
Evaluation
The semantic similarity rank 846h 1,000!
Metric
Similarty
CLIP
97.7%
Disparity
2024). Reflecting its importance, numerous studies have been dedicated to advancing this
Incepsion
Current Evaluation
Human Evaluation
domain (Scotti et al., 2023 2024 Wang al.
2024a Huo et al. 2024 Xia et al. 2024a Wang al. 2024b Tian al. 2025).
Figure Current evaluation metrics assess the semantic similarity between ground-truth and constructions in a way that significantly differs from human evaluation, often giving relatively high scores to reconstructions that are semanti- cally misaligned.
With the recent advent of diffusion-based de- coding models (Scotti et al. 2023 2024 Wang
et al. 2024a b) Huo et al. 2024 Tian et al.
2025 that boast a near-perfect performance on all of the percentage-bast
evaluation metrics, the endeavor to visually decode brain signals might seem to nearly solved, with little to no room for improvement for future research. However, upon close inspection, the decoding results, even from the most recent and state-of-the-art models, often
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
fail at reconstructing crucial semantic elements in the original image; e.g., teddy bear may turn into a cat during the reconstruction process. (See Fig.
As this example suggests, observed that current evaluation metrics tend to assign relatively high scores to such flawed reconstructions, potentially misleading researchers and obscuring the true limitations of these models. This leads to the following question: Is the current framework to evaluate visual decoding models aligned with human intuition? To answer that, we first inspected current evaluation metrics and identified few limitations: the dependency on the comparison image pool, insufficient difficulty, and the lack of human-likeness. In addition, existing related metrics, e.g., FID or SSIM (Wang et al., (004)). are unsuitable since the evaluation of decoding models requires the comparison between two images that could be highly dissimilar. Furthermore, we collected human ratings on the semantic similarities of 1,000 ground-truth (GT) and reconstruction image pairs from 22 evaluators. Using these ratings, we revealed that most existing metrics show a low correlation with human evaluation about the semantic similarity of GT and its brain-decoded reconstruction, with the exception of the EffNet (Tan & Le 2019) metric. Our finding underscores the urgent need for improved evaluation criteria.
To that end, inspired by the human visual perception process, we propose a new evaluation met- ric that primarily focuses on the semantic likeness of two images, SEED (Semantic Evaluation for Visual Brain Decoding). SEED is combinatorial metric that integrates two newly proposed met- rics, Object F/ and Cap-Sim, alongside EffNet, a well-established metric, each resembling different stages of the human visual perception pipeline.
More specifically, Object F1 a metric that aims to identify and capture important elements of the image by automatically detecting and comparing the presence of key objects of the scene using open-vocabulary image grounding models. Cap-Sim is a metric that compares the similarity of the generated captions of two images. This metric captures additional semantic factors that might be overlooked by Object FI, such as backgrounds, pose, and color, offering complementary evalua- tion of the high-level image semantics. EffNet is a widely adopted metric leveraging an ImageNet (Deng et al.) 2009) pre-trained EfficientNet (Tan & Le, (019)) model. The metric is known to be particularly well suited to capture the more global and structural aspects of the scene, thus comple- menting Object Fl and Cap-Sim.
By carefully comparing our proposed and existing metrics with the collected human evaluation re- sults, we show that the two new metrics, Object Fl and Cap-Sim, indeed exhibit strong agreement with human evaluation, and our SEED achieves the highest alignment with human evaluation, com- pared to all existing metrics. In order to facilitate future research on developing metrics, plan to release the human evaluation results.
Furthermore, evaluation of recent visual brain decoding models with SEED revealed that even the most advanced models frequently fail to accurately reconstruct key objects of interest, often confusing them with similar ones. Even when key objects are correctly identified, the models often struggle to capture semantic details. We believe these findings can provide valuable guidance for advancing research in visual brain decoding.
BACKGROUND
2.1
VISUAL BRAIN DECODING MODELS
Visual brain decoding refers to the task of reconstructing visual stimuli, such as an image, given the brain signals of human subject that is viewing the said visual stimuli. In the early stages of development of visual decoding models, linear regression-basee approaches demonstrated that visual information can be decoded from brain signals (Kamitani & Tong 2005 Haynes & Rees, 2005). With the development of deep learning techniques, more sophisticated decoding becomes promising, such as GAN (Goodfellow al., 2020) based visual brain decoding (Seeliger et al.)
2018 Ozcelik et al.) 2022). Recent decoding models adopt latent diffusion models (Rombach et al,
2022 to produce high-quality decoded images conditioned by brain embeddings predicted CLIP
(Radford et al.) ) image embeddings from fMRI signals (Scotti et al.] [003) 2024) Wang et al.
2024b a Tian et al. 2025 Gong et al. 2025). Instead of freezing the pre-trained diftusion mod-
2

---

Published as a conference paper at ICLR 2026
els, NeuroPictor (Huo al. 2024) fine-tunes the diffusion model to directly condition the image generation process with brain embeddings.
Beyond the single modality decoding, recent works aim to simultancously reconstruct the multiple modalities, mainly text and images from a fMRI signals (Mai & Zhang 2023 Xia et al, 2024b Shen et al. 2024
Furthermore, we note that there is line of work that mainly focuses on the reconstructing textual information from the fMRI signals (Chen et al. 2025a b), though they not main focus of our work.
Instead of freezing the pre-trained diffusion models, NeuroPictor (Huo et al. 2024 fine-tunes the diffusion model to directly condition the image generation process with brain embeddings.
2.2
CURRENT EVALUATION SCHEMES
Most the recent decoding literature OZzellik VanRullen 2023 Scotti et al. 2023 Liu et al,
2023 Scotti et al. 2024 Wang et al., 2024a| (Shen al.) [024 Huo et al. [024 Wang et al. 2024b
Xia et al.] 2024a) mainly focus on the following eight eva evaluation metrics: PixCorr, SSIM (Wang
al.) 2004), Alex,e AlexNet(5) (Krizhevsky al., 2012), Inception (Szegedy et al. 2015),
CLIP
(Radford et al. 202T), EffNet ("Tan & Le 2019). and SWAV (Caron et al. (020)).
PixCorr refers to the Pearson correlation between the pixel values of the GT and the reconstruction. SSIM refers to the structural similarity index measure between the GT and the reconstruction.
AlexNet(2), AlexNet(5), Inception, and CLIP refer to the accuracy of two-way identification tasks that use the corresponding feature extractor. Specifically, for every GT embedding, the Pearson correlation with its corresponding reconstruction embedding compared against its correlation with each other reconstruction embedding in the test set. The percentage of cases in which the GT embedding closer to its correct reconstruction is reported.
The n-way extension of the task utilizing the brain- brain- generated intermediate CLIP embeddings and the GT CLIP image embeddings, known as image/brain retrieval, is also reported in some works (Scotti) et al. 2023 2024 Lin et al. (2022). However, the retrieval tasks are not applicable to models such as NeuroPictor (Huo et al. 2024) as they require the model to generate brain-derived intermediate CLIP image embeddings during the decoding process.
EffNet and AV refer to the correlation distance between the GT embedding and the reconstruction embedding, utilizing the corresponding feature extractor.
ISSUES WITH EXISTING EVALUATION METHODS
3.1 EMPLOYMENT OF EXISTING RELATED METRICS
When evaluating visual brain decoding models, it is crucial to measure how closely the reconstruc- tion aligns with the GT, acknowledging potential perceptual and semantic deviations. Unlike typical image generation tasks, which lack a fixed GT, decoding tasks involve a predetermined target. Con- sequently, standard metries for image generation, such as FID, are unsuitable, and a measure that directly compares the reconstruction to the known image is required.
In this sense, due to the nature of comparing the similarity of two images, the evaluation of the decoding task more closely resembles traditional image quality assessment, where images are de- graded by compression, transmission, or other processes. This precisely the context for which metrics like SSIM were originally designed, which likely explains why those metrics widely used for the evaluation of visual brain decoding models.
However, key distinction lies in the inherent noisiness of decoding, where reconstructions can be perceptually different from the GT while retaining a similar semantic theme. This can result in metrics like SSIM assigning unusually low scores they are prone as translations and rotations (Nilsson & Akenine-MMil 2020), let alone the larger distortions often found in reconstructions.
small distortions, such
3

---

Published as a conference paper at ICLR 2026
Object
MM. Grounding- Dino
Min
Thrashaid
Razall
(Woman, Marj (Man)
Visuss (taaid
(Horse)
(Horse)
Nome
Recon
GiT
Two people riding horses down dirt road
Sentence Transformer
Cap-sim
Average
SEED
walking across golf course
EffNet
Correlation
Figure 2: The overall process for calculating SEED.
Consequently, although might appear that conventional image quality assessment metrics ide- ally suited to evaluate decoding models, in practice, they are substantially misaligned from human evaluation, demonstrated in Sec. [.1] Therefore, the focus of evaluation should be geared towards assessing the semantic qualities of the reconstructions, due to the noisiness of the decoding process.
3.2 TWO-WAY IDENTIFICATION
Two-way identification metrics (AlexNet(2), AlexNet(5), Inception, CLIP) serve a crucial role in the evaluation decoding models, as they occupy half of the eight-metric evaluation scheme. How- ever, due to their comparative nature, two-way identification metrics contain some inherent flaws. First and foremost, comparing two-way identification scores between models inappropriate, As each reconstruction is compared against other reconstructions generated by the decoding model, the pool of images each reconstruction is compared against differs for each decoding model. This fact renders the direct comparison of two-way identification scores inappropriate, each model would be evaluated under different criteria.
Another issue arises from the difficulty, or lack thereof, of the two-way identification task. Since the reconstruction only needs to closer to the GT than another random example, a reasonable reconstruction easily "wins" the comparison. Due this, recent decoding models already show near-perfect performance for most two-way identification metrics. This makes it difficult to differ- entiate the performance between different decoding models and thus calls for more challenging evaluation task.
3.3 LACK OF HUMAN-LIKENESS
Excluding PixCorr and SSIM, all other evaluation metrics rely on abstract features extracted from pre-trained vision models. Consequently, difficult to interpret the rationale behind each eval- uation from a human perspective, casting doubt on whether they truly align with human percep- tion especially while under scrutiny. Our human survey findings indeed reveal that most com- monly used metrics gauge semantic similarity in ways that deviate notably from human evaluation. Further details are in Sec. 5.1
NEW SEMANTIC EVALUATION METHODS
Given the issues outlined in Sec. there is clear need for evaluation methods that deliver more accurate and generalizable assessments for visual brain decoding. To that end, we borrow inspiration from the human visual attention system to develop
decoding evaluation protocols. Among
neuroscientific literature (Jonides 1983] Treisman 1998 Zhang 2019), the common consensus is that visual perception and attention two-stage process.

---

Published as a conference paper at ICLR 2026
During the first stage, the visual system analyzes basic features of the environment such color, orientation, and brightness. This process occurs in parallel, simultancously dividing attention across the entire visual field.
Although the specifics may vary from theory to theory, the second stage of visual attention involves focused attention, which is crucial for binding the separately processed features into coherent, rec- ognizable objects. In this stage, attention is selectively concentrated on specific locations within the visual field. When attention directed to particular area, the brain integrates the features present at that location into a unified percept.
We noticed that most existing metrics, especially the ones involving convolution model, models that follow a similar process to the first stage, but not the second stage. This observation motivated us develop two different metrics that each resemble different parts of the second stage, as well as metric to unify the two stages, namely: Object F1, Cap-Sim, and SEED.
4.1 OBJECT F1
We first introduce a metric that focuses on key objects, in order to roughly follow the object-oriented attention mechanism of the second stage of visual attention. Object Fl is metric that measures the similarity of two images based on object presence; that is, objects present in the GT should also be present in the reconstruction, and objects not present in the GT should also not be present in the reconstruction. Using image grounding models, it is possible to automatically detect the objects present in both images and quantify the aforementioned criterion into two proposed metrics: Object Recall and Object Precision.
We first run all GT and reconstructed images through image grounding model and obtain the detection results. The results should contain the list of detected objects with information such as the category and the confidence value for each object. Given confidence threshold which the threshold used to determine whether an object is "detected," we define two preliminary metrics for each image: Object Recall, and Object Precision,
Object Recall, measures the proportion of the object categories from the GT that are also present in the reconstruction. This measures the proportion of objects that are successfully "recalled" in the reconstruction, formulated as:
Object Recall, of categories in both GT and recon of categories GT
Similarly, Object Precision, measures the proportion of the object categories from the reconstruc- tion that are also present in the GT. This essentially measures the "precision" of the objects in the reconstruction, formulated as:
of categories in both GT and recon # of categories in recon
Object Precision, =
(2)
During the process, we apply the same threshold value the GT and reconstruction to ensure the ideal reconstruction (i.c., reconstruction identical to the GT) obtains the best possible score. For simplicity, if multiple objects of the same category present in image, we only consider the object with the highest score, as we only check for the existence of each object category.
To remove the reliance on threshold hyperparameter, Object Precision, while moving the threshold, I, between 0 and and obtain the averaged values:
caleulate Object Recall, and
wild
Object Recall
Object Recall, dt
precall valid
(3)
pesticn
Object Precision
Object Precision, dt
precision
valid
cutoff thresholds, corresponding to the highest confidence value present in the GT and reconstruction, respectively. The threshold is cut off in such a way since there would be detected objects for higher threshold values.
where valid -valid
pprecision
5

---

Published as a conference paper at ICLR 2026
The final evaluation metric, Object FI, is the harmonic mean of the averaged Object Recall and Object Precision:
2
Object FI
Object Recall -1 + Object Precision
The threshold-averaging scheme has the added benefit of penalizing reconstructions with objects far apart from the GT in terms confidence, as those objects would be marked as incorrect during the intermediate threshold values. This trait is beneficial for evaluating decoding models, as they often generate distorted objects (Scotti et GT counterparts.
2024) that tend to show lower confidence values than their
We note that the proposed Object F1 fundamentally differs from the Average Precision (AP) in object detection. AP evaluates detection models by comparing bounding boxes based on IoU for a single image, whereas Object measures similarity of two images based on object existence, independent from loU.
To calculate Object F1, we employ AM-Grrunding-DDie (Zhao et al. 2024 to detect 82 object categories; the full list of categories is available in Sec. B.I For Object Recall and Object Precision, to approximate Eq. 3. we move the threshold from by increments of 0.01, up to the cutoff thresholds, and average the values.
4.2 CAP-SIM
Similar to how Object FI emulates the object-oriented attention mechanism of the second stage of visual attention, we introduce metric inspired by the subsequent process within the same stage that identifies and binds relevant features. Cap-Sim is a metric that measures the similarity between captions generated by image captioning models for each GT and reconstruction pair. Instead of relying on abstract features generated by vision models, this approach emphasizes semantic qualities expressible by natural language since the images essentially "compressed" into text before being compared. This method allows us to evaluate semantic factors that are hard to identify through the existence of objects, such as the background information attributes of the detected object (pose, color, etc.). Furthermore, caption- caption-based evaluation provides an interpretable assessment, as captions heman-readable and closely align with how people describe visual content (He et al. 2019).
Formally, Cap-Sine Sim is formulated as:
Cap-Sim = (os(eren(( (c(Ta))) Grexa (c( (reco)))
(5)
where IcT and Trecon are GT and reconstructions, respectively. The functions (-ea(-) and c(-) denote text encoder and caption generator, respectively, for which we use Sentence Transformer (Reimers) & Gurevych]200 2019) and GIT (Wang et al.) 2022). To the best of our knowledge, we note that caption- based evaluation of image similarity has not been previously proposed, despite its simplicity.
4.3 SEED
Building on these metrics, we aim to construct a unified evaluation framework that captures the complementary aspects of human visual attention, each modeled by the individual metries, and serves as a reliable standard for assessing decoding models. To this end, we introduce Semantic Evaluation for Visual Brain Decoding (SEED), composite metric that integrates Object F1, Cap- Sim, and EffNet.
Note that EffNet is slightly modified metric by calculating correlation, correlation distance, converting it into a higher-is better metric like the other two;
EffNet == corr(Cimg (ToT), Cimg (imggTrecoom (Trccon))
where the function (-mg(-) is the image encoder, EffNet.
The overall procedure to compute SEED and its components for a given image pair is depicted in Fig. 2 We simply take the average of the three metrics to calculate SEED:
SEED
(Object F1 Cap-Sim EffNet) /3

---

Published as a conference paper at ICLR 2026
4.4 HUMAN EVALUATION OF IMAGE SIMILARITY
We collected 5-point Likert scale ratings from 22 human evaluators to assess the alignment of current evaluation metrics with human evaluation. They assessed both the semantic and perceptual similarity between GT and their reconstructions for 1,000 test set images in Natural Scenes Dataset (NSD) (Allen et al. 2022) used by [Scotti et al. (2024), where the reconstructions were generated by the MindEye2 model released by the original author, with 250 reconstructions sequentially sampled from each the four subjects (subject 1, 2, 5, and 7), following the order: the first 250 from subject 1, the next 250 from subject 2, and on. The intraclass correlation (ICC(2, n)) (Koch) 2006) between the human evaluation results 0.84 (p = 0), indicating a sufficiently high inter-rater agreement. Further detailed information the collection of human ratings is provided in Sec. A and we will release the survey results to facilitate future research on similar topics.
5 EXPERIMENTAL RESULTS
5.1 ALIGNMENT WITH HUMAN EVALUATION
Table 1: The meta-evaluation results on NSD with MindEye2. The best results are bolded. SwAV was calculated similarly to Eq.
Table 2: The meta-evaluation results of recon- structions of the GOD dataset with Mind- Vis. The best results are bolded.
Metrie
Pairwise Acc.
Kendall
Pearson
Metric
Pairwise Acc.
Kendall
Pearson
PixCorr
53.8%
.075
.117
PixCorr
51.3%
.029
.078
SSIM
54.5%
.090
.112
SSIM
49.2%
-013
-103
A2exNet(2)
55.0%
.185
.187
AlexNet(2)
66.0%
.377
.492
(5)
49.5%
.266
.258
AlexNet(5)
65.8%
.423
.445
Inception
63.8%
.330
.455
Inception
62.6%
.324
.356
CLIP
66.4%
.368
.436
CLIP
63.2%
.338
.309
78.0%
.559
.748
EffNet
72.5%
.453
.661
SWAV
69.7%
.394
.576
SWAV
68.6%
.376
.498
Object 75.8%
.516
.708
Object
66.0%
.431
Cap-Sim 73.8%
.477
.666
Cap-Sim
68.7%
.376
.577
SEED
81.0%
.621
.813
SEED
73.7%
.477
.706
Kendall's Tau-b, and Pearson correlation to meta-evaluate each metric based on the human ratings the semantic similarity between images. We meta-evaluated eight metrics widely used in prior works (Scotti) et al. 2023 2024 Wang et 2022a Tian et al., 2025). Additionally, we explored alternative ap- proaches for measuring the semantic similarity between images based visual question answering models, detailed in Sec. C.2
Following Lin et al. 20244) adopt pairwise accuracy (Deutsch et al., 2023
The meta-evaluation results, presented in Tab. T indicate that most existing metrics exhibit low relation with human evaluation, except for EffNet. Furthermore, the alternative approaches do not perform as effectively as Object Fl or Cap-Sim. Notably, SEED achieves the highest agreement with human evaluation with statistical significance. To assess the statistical significance of the improve- ment of SEED over EffNet, which shows strong alignment among existing metrics, We performed bootstrapping along the evaluator axis (sample size 22) for 1,000 iterations and computed the confidence intervals of the differences each meta-evaluation metric between SEED and EffNet. The 95% confidence intervals for pairwise accuracy, Kendall's Tau-b, and Pearson correlation were [0.03, 0.07]. [0.02, 0.04], and [0.04, 0.08], respectively, all of which do not include zero. These results indicate that the performance improvement SEED over EffNet is statistically significant.
We note that the combination of the three metrics is essential to achieve the highest alignment with human evaluations. detailed analysis provided in Sec.
5.2 ROBUSTNESS OF SEED
Since several factors in SEED may influence the evaluation process, we conduct experiments to examine its robustness under different scenarios.

---

Published as a conference paper at ICLR 2026
Robustness to dataset and decoding model. One major factor affecting meta-evaluation would be the choice of dataset and decoding model that serves as the evaluation target. To perform meta-evaluation
different setting, collected human evaluations from 10 student volunteers for 50 reconstructions generated by Mind- Vis (Chen et al. (023) on the General Object Decod- ing (GOD) dataset (Horikawa & Kamitani, 2017). The ICC values for semantic similarity was 0.93 (p = 0), indicating high agreement among raters. We used the full list of 50 test set class names to compute Object Fl. As shown in Tab. 2 SEED again achieved the highest alignment with human evaluation, demonstrating that it generalizes well across datasets and decoding models.
Robustness the choice of off-the-shelf models. We next evaluated whether SEED's performance depends on the specific choice of image ground- ing model, caption generator c(-), text encoder (.)x((). We substituted the original components with Yolo-Workd World (Cheng et al. 2024) for image ground- ing, BLIP-2 (Li et al. 2023) for caption generation, and Swen3-Embeddiin--D.. for text encoding. Meta-evaluation results across all eight model combinations are summarized in Fig. 3 The barplots indicate that performance dif- ferences across all choices are minimal, confirming that SEED is robust to the selection of these off-the- shelf models.
0.7
0.6
0.5
(Zhang al.] 2025
0.2
0.1
Pairwise Acc.
Kendall
Pearson
Figure 3: Meta-evaluation results with differ- ent choices off-the-shelf models.
5.3 ANALYSIS OF WORST-CASE JUDGMENTS
To understand why SEED improves upon its components, we present case studies of the "worst-case judgments" for each component of SEED, despite their high agreement with hu- man evaluation. In this context, "worst-case judgments" refer to images whose metric-based ranking differs significantly from the human evaluation ranking. Rankings were computed from each metric's numeric scores and from hu- man ratings, where human ratings were normal- ized per evaluator and then averaged per im- age. The examples shown Fig. are chosen among the worst-case judgments for each met- ric. where the other two metrics made a human- aligned decision, which somewhat mitigates the discrepancy. Additional examples are available in Sec. D3
Metric
Rank
Human
230
(a)
Object
995
Cap-Sim
151
EfNet
113
Matric
Rank
Human
107
(b)
Object F1
266
Cap-Sim
Matric
Rank
Human
270
(c)
Object
Cap-Sim
Fig. (a) shows a case where Object Fl sig- nificantly deviates from human evaluation and other metrics by assigning score of O. This disparity arises because Object Fl fails to cap- ture global scene information, relying solely detected animals (sheep in the GT and cow in the reconstruction).
Figure 4: Visualizations (out of 1000 pairs) of worst-case judgments for (a) Object F1, (b) Cap- Sim, and (c) EffNet.
Fig. (b) shows generated by GIT is [A man on skis standing on snowy hill.] and [A woman on skis is waving while skiing.] for the GT and the reconstruction, respectively. The low similarity likely results from the change of gender or the described action, despite other metrics as well as humans assigning a high similarity.
where Cap-Sim significantly deviates from the others, where the caption
Fig. (c) shows a case where EffNet significantly deviates from the others. Although is difficult pin down the exact reason, possible explanation the fact that the images have different
8

---

Published as a conference paper at ICLR 2026
ImageNet Top-1 predictions from the EffNet model: American egret for the GT and Coucal for the reconstruction. We hypothesize that the EffNet tends to over/underestimate the correlation between two images with the same/different class predictions.
To validate this suspicion. we compared the average z-normalized EffNet and the human semantic evaluation scores of the image pairs with the same/different EffNet ImageNet Top-1 predictions. For images from the same class, EffNet yields an average score of 0.755, whereas human evaluators score 0.313 average. For images of different classes, the average scores are -0.333 for EffNet and -0.138 for humans. This indicates that EftNet produces overestimated assessments, depending on the ImageNet classes, and we believe this explains EffNet's s low correlation for cases like Fig. (c).
5.4 FAILURE MODE DISCOVERY
SEED: 0.591 Object F1: 0.949
GT
Recon
MindEye2
NeuroPictor
C7 Canton: and over the aocmn. Recon Captien: ahdd standing on the graae the surn.
MindBridge
UniBrain
BrainGuard
Figure 6: An example of reconstruction which captures objects correctly but misses semantic details.
Figure 5: Examples of the semantic near-miss phenomenon.
Semantic near-miss phenomenon. One common failure mode of current decoding models the semantic near-miss phenomenon, in which the reconstruction misrepresents the specific object cat- egory from the GT, yet still captures the broader supercategory. For example, if the GT contains dog. the reconstruction might include a car or other animals (See Fig. 5). While this car is in the wrong category, it remains within the correct supereategory, animal.
We quantify this by re-using the object detection pipeline used in Object F1. We calculate the Object Recall (Eq. Th and the Relaxed Object Recall, which measures the proportion of the object categories from the GT where its supercategory (instead of the specific category) is present in the reconstruc- tion. The gap between those two represents the rate of the semantic near-miss phenomenon.
We computed the semantic near-miss rate of salient object categories (Xia al. 2024b) confi- dence threshold of 0.3 for five existing decoding models in Sec. E. and observed rates ranging from 17.5% to 20.6%. Such high incidence indicates that current decoding models often struggle with fine-grained object differentiation, capturing only coarse semantic details.
Captured objects while missing semantic details. We identify another failure mode in which the model reconstructs the main objects but overlooks crucial semantic details. To analyze this, we focus on reconstructions with high Object F1 but low overall SEED, specifically those satisfying Object F1 > 0.7 and Object F1 SEED > 0.2. While the exact thresholds are somewhat arbitrary and can be varied, our goal here not to fixate on specific cutoff values but to demonstrate how such criteria enable systematic identification of failure modes. This criterion isolates cases where low Cap-Sim and EffNet scores reduce the SEED average. Such cases indicate that while the model successfully reconstructs objects, it often fails to capture other details such as backgrounds, pose, or color. Fig. Fig. illustrates one such example, where the reconstruction correctly captures a bird but fails to reconstruct the background well as its pose.
Using this criterion, we measured the proportion of reconstructions. The ratio ranges from 8.3% 10.7% across the five decoding models evaluated in Sec. suggesting that sizable fraction

---

Published as a conference paper at ICLR 2026
of reconstructions, while correctly identifying the main objects, still fail recover tne-grained semantic details.
Potential remedies. While we do not propose solutions for these failure modes, we believe that our findings suggest several promising research directions. First, more systematic error analysis with SEED could provide actionable guidance for data collection. For example, if a model reliably reconstructs objects but frequently mismatches backgrounds, this would suggest collecting images with greater background diversity. Similarly, address the semantic could gather datasets containing images with subtle differences between them. Second, training strategies could aim disentangle object reconstruction from semantic detail reconstruction. Most current decoding models use CLIP image embeddings as regression targets, which may conflate these two aspects and contribute the failures. Future methods may therefore benefit from decou- pling object-level supervision from supervision for other details.
miss phenomenon, one
CONCLUSION & LIMITATIONS
In this work, introduce SEED, novel framework designed to assess the semantic decoding performance of decoding models. Through comprehensive experiments, we show that existing eval- uation metrics often diverge from human judgments, whereas our proposed metric exhibits stronger alignment and improved reliability.
Our results reveal growing mismatch between the goals of modern visual brain decoding and the metrics currently used to evaluate it. Although recent diffusion-based models can achieve near- perfect scores on traditional identification metrics and display high similarity scores, our human- aligned analyses show that these models often overlook substantial semantic errors, including miss- ing objects, incorrect categories, and failures to capture contextual details, which are overlooked by traditional metrics. This indicates that the field may be overestimating progress due to evaluation tools that no longer reflect the true complexity of the task.
SEED addresses this gap by providing a more numan-conssitenn measure of semantic fidelity, in- tegrating object-level, caption-level, and other fine-grained semantic cues. Beyond offering more reliable evaluation metric, SEED reveals distinct failure modes, such as semantic near-misses and losses of fine detail, thereby enabling more targeted model development.
More broadly, our findings highlight that as decoding models mature, so too must our evaluation practices. We hope that SEED encourages the community to adopt richer, human- mainand evaluation frameworks and to develop models that capture objects, attributes, and other semantic details in more faithful and robust manner.
Limitations and future work. Nonetheless, approach has its limitations. As SEED depends on the off-the-shelf models, SEED may inherit systematic errors from the existing models. One such example is provided in Sec. D.2] where all metrics of SEED fail to make human-all -alined judgment when an unusual or maltormed image is given as the reconstruction, which turn leads to the failure of SEED. Training evaluation models or devising metrics that scenarios could be a promising future direction.
robust to these
In addition, because SEED was designed with stronger emphasis evaluating image semantics, it may become less effective once precise assessment of perceptual details is required as brain de- coding technology matures. While currently regard accurate semantic decoding as the higher priority, we expect that, as models improve and reliably capture high-level semantics, the focus will naturally shift toward perceptual fidelity. At that stage, an evaluation method better suited to detecting fine- men-inri perceptual aspects should be introduced.
REPRODUCIBILITY STATEMENT
For the reproducibility of our study, detailed the model used for computation of SEED in Sec. and how to compute SEED. In addition, our code and the human evaluation results are available at wtt.ttttcccccooo/ocmrmeeeee
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported in part by National Research Foundation of Korea (NRF) grant [No. 2021R1A2C200788, No. RS-2025-02263628, No. RS-2023002654066] the Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grants [RS-2021-112122 the BK21 FOUR Education and Re- search Program for Future ICT Pioneers (Seoul National University), funded by the Korean govern- ment (MSIT), the Ministry of Education IRS-2024-004333 (024400433777). the National Supercomputing Center SBSC-2025-CRE-044000 the U.S. Department of Energy's 's ASCR Leadership Computing Challenge [m4750-2024], and Hyundai Motor Chung Mong-Kot Foundation.
RS-2022-11220113 RS-2022-11220559, RS-2021-11211344
REFERENCES
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci,, 25:116-126, 2022.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint 2ti.:2502.1322.. 2025.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of the Advances in neural information processing systems (NeurIPS), volume 33, pp. 9912-9924, 2020.
Jiaxuan Chen, Yu Qi, Yueming Wang, and Gang Pan. Bridging the gap between brain and machine in interpreting visual semantics: Towards self-adaptive brain-to-text decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 21938-21948, 2025a.
Jiaxuan Chen, Yu Qi, Yueming Wang, and Gang Pan. Mindgpt: Interpreting what you see with
non non-invasive brain recordings. IEEE Trans. Image Process., 34:328 34:3281-3299 1-3293, 2025b.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Pro- ceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 22710-22722 2023.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time opcabbocabulary object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition (CVPR), pp. 16901-16911, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Daniel Deutsch, George Foster, and Markus Freitag. Ties matter: Meta-evaluating modern metries with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 22914-1292 2023.
Bing Du, Xiaomu Cheng, Yiping Duan, and Huansheng Ning. fmni brain decoding and its applica- tions in brain compuce interface: A survey. Brain. Sci., 12(2):228, 2022.
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Rongtao Xu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 14247-14255, 2025.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM, 63(11): 139-144, 2020.
John-Dy -alla Haynes and Geraint Rees. Predicting the orientation of invisible stimuli from activity in human primary visual cortex, Nat. Neurosci., 8(5):666 6918 2005.

---

Published as a conference paper at ICLR 2026
Sen He, Hamed R Tavakoli, Ali Borji, and Nicolas Pugeault. Human attention in image captioning. Dataset and analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3529-8538, 2019.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun., 8(1):15037, 2017.
Jingyang Huo, Yikai Wang, Yun Wang, Xuelin Qian, Chong Li, Yanwei Fu, and Jianfeng Feng- Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi- level modulation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 56-73, 2024.
John Jonides. Further toward model of the mind's eye's movement. Bull. Psychon. Soc., 21(4): 247-250, 1983.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nat. Neurosci., 8(5):679-66) 685, 2005.
Gary G. Koch. Intraclass Correlation Coefficient. John Wiley & Sons, Ltd, 2006. ISBN 9780471667193. doi: https//66.ooog///000000007766661 onninelibraa
URL https:// pub2
com
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Proceedings of the Advances Neural Information Processing Systems (NeurIPS), volume 25, 2012.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the Inter- national Conference on Machine Learning (ICML), pp. 19730- 19730-19742, 2023.
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 29624 29636, 2022.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 366-384, 2024.
Yulong Liu, Yongqiang Ma, Guibo Zhu, Haodong Jing, and Nanning Zheng. See through their minds: Learning transferable brain decoding models from cross-subject fimri In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 5730-5738, 2025.
Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXivv2308.07422 2023.
Weijian Mai, Jian Zhang, Pengfei Fang. and Zhijun Zhang. Brain-conditional multimodal synthesis: A survey and taxonomy. IEEE Trans. Artif. Intell., 10800090-, 2024.
Jim Nilsson and Tomas Akenine-Möller Understanding ssim. arXiv preprint 1.X14 2020.
Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin'ichi Satoh. Toward verifiable and reproducible human evaluation for text-to- image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14277-14286, 2023.
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using genera- tive latent diffusion. Sci. Rep.. 13(1):15666, 2023.
Furkan Ozcelik. Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of perceived images from fmri patterns and semantic brain exploration using gans. In Proceedings of the 2022 International Joint Conference on Neural Networks (IJCNN, pp. 1-8, 2022.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International conference machine learning (ICML), pp. 8748-8763, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-
IJCNLP), pp. 3982-3992, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022.
Simanto Saha, Khondaker A Mamun, Khawza Ahmed, Raqibul Mostafa, Ganesh R Naik, Sam Darvishi, Ahsan H Khandoker, and Mathias Baumert. Progress in brain computer interface: Chal- lenges and opportunities. Front. Syst. Neurosci, 15:578875, 2021.
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisherg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 24705-24728, 2023.
Paul Steven Scotti, Mihir Tripathy, Cesar Torrico, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu. Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable tional Conference on Machine Learning (ICML), pp. 44039-44059, 2024.
with hour of data. In Proceedings of the Interna-
Katja Seeliger, Umut Güçlüü Luca Ambrogioni, Yagmur Güçlüüüükk, and Marcel AJ Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolm- age, 181:775-785, 2018
Guobin Shen, Dongcheng Zhao, Xiang He, Linghao Feng, Yiting Dong, Jihang Wang, Qian Zhang, and Yi Zeng. Neuro-vision to language: Enhancing brain recording-based visual reconstruction and language interaction. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 98083-98110, 2024.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 1-9, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In Proceedings of the International Conference on Machine Learning (ICML), pp. 6105- 61 6114,20 2019.
Zhibo Tian, Ruijie Quan, Fan Ma, Kun Zhan, and Yi Yang. Brainguard: Privacy-preserving mul- tisubject image reconstructions from brain activities. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 14414-14422 14422, 2025.
Anne Treisman. Feature binding, attention and object perception. Philos. Trans. R. Soc. Lond. Biol. Sci., (53(1333):1295- 1306, 1998.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 2201:22005.1400,, 2022.
Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11333-11342, 2024a.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment:
from error visibility structural similarity. IEEE Trans. Image Process., 13(4):600-612, 2004.
13

---

Published as a conference paper at ICLR 2026
Zicheng Wang. Zhen Zhao, Luping Zhou, and Parashkev Nachev. Unibrain: A unified model for cross-subject brain decoding. arXiv preprint arXi::2412.19487, 2024bb.
Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVE Winter Conference on Applica- tions of Computer Vision ( (WACV), pp. 8226-8235, 2024a.
Weihao Xia, Raoul de Charette, Cengiz. Oztireli, and Jing- Hao Xue. Umbrae: Unified multimodal brain decoding. In Proceedings of the European Conference on Computer Vision (ECCV), pp-
242-259, 2024b.
Jiawei Zhang. Cognitive functions of the brain: perception, attention and memory. arXiv preprint arXiv: 2019.
Yanzhao Zhang, Mingxin Li, Dingkun Long. Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models, arXiv preprint arXiv:2506.0517, 2025.
Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361, 2024.
14

---

Published as a conference paper at ICLR 2026
SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding
Appendix
THE USE OF LARGE LANGUAGE MODELS (LLMs)
We utilized LLMs for the purpose of polishing manuscript only.
A COLLECTION OF HUMAN EVALUATIONS
We used the Amazon Mechanical Turk (MTurk) platform as well as additional student evaluators to collect human ratings on the semantic and perceptual similarity between GT and its reconstruction. A screenshot of the survey window is shown in Fig.
Evaluate the Semantic similarity
Evaluate how simite the main idea meaning of the two
Evaluate the Visual similarity
Evaluate how visually similer the two Consider the textures, colora, object locations, composition and overal appearance.
Not oimilar Motohee
Figure A screenshot of our Amazon MTurk survey window.
Referring to Otani et al. (2023), we applied the following filter for worker requirements when cre- ating the MTurk project: I) Master: prod-pperfrming and granted AMT Masters. Each annotator was paid $0.03 for evaluating the semantic and perceptual similarity of single pair of GT and its reconstruction image. We gathered a total of 22 ratings for each of the 1,000 pairs.
The intraclass correlation (ICC(2, n)) (Koch 2006) for the perceptual similarity evaluation results was 0.79 with p = 0, which indicates high inter-rater agreement.
B CHOOSING CANDIDATE OBJECT CATEGORIES FOR OBJECT DETECTION
B. FULL LIST OF OBJECT CATEGORIES
The list of object categories, which was used for object detection, is composed of 80 COCO cat- egories plus 2 additional human categories (man and woman). The resulting 82 categories can be further classified into 30 "Salient" and 52 "Inconspicuous" objects per et al. (2024b).
The 30 salient objects are: [person, man, woman, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, bicycle, car, motorcycle, airplane, bus, train, truck, boat, bench, chair, couch, bed, dining table, toilet, sink, refrigerator, clock]
The 52 inconspicuous objects are: [traffic light, fire hydrant, stop sign, parking meter, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana,
15

---

Published as a conference paper at ICLR 2026
apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, potted plant, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, book, vase, scissors, teddy bear, hair drier, toothbrush].
B.2 CHOOSING CATEGORIES WITH VLM
The rapid development of vision-language models (VLM) made us wonder if the process of choosing object categories could be delegated to VLMs instead of using fixed set of objects. To answer this question, we use an open-sourced the object categories instead of using the aforementioned 82. We gave the model each GT and reconstruction image separately; we experimented with different text prompts, but the following was the most effective: Generate list of objects and background features that present in the image. Only answer in comma-separaled list of objects. Do nol include any other lext or explanation. With the extracted object categories, and the Object Precision with the categories the reconstruction image, separately for each image pair. Compared to the fixed list of 82 categories, which is the one used in the manuscript, this strategy performed slightly worse, although still significantly outperformed existing metrics.
1-B.5--V--B--Istrrue (Bai et al. 2025) model to extract
calculated the Object Recall with the categories of the GT
Table The meta-evaluation results while using fixed set of 82 categories versus VLM 'LM--generated object categories.
Metric
Pairwise Acc.
Kendall
Pearson
Object F1 Object FI (VLM)
75.8%
.516
.708
73.7%
.473
.658
SEED SEED (VLM)
81.0% 80.4%
.621
.813
.607
.800
C ADDITIONAL ANALYSES
C.1 INCORPORATION OF LOCATION, SIZE, AND NUMBER INFORMATION
Table 4: The meta-evaluation results of Object F1 with incorporation of additional information.
Options
Pairwise Ace.
Kendall
Pearson
Existence
Size
Location
Number
75.8%
.516
.708
75.8%
.517
.709
75.9%
.517
.710
74.7%
.493
.648
We incorporate location, size, and number information into Object to determine whether each factor contributes to the improvement of alignment with human evaluations, outlined below:
Size weighting We weight object categories based on their bounding box size, with larger sizes receiving higher weights. An object that fills the entire image would be weighted twice much as an object with zero area, with scaling linearly.
Location weighting We weight object categories based on their proximity to the center of the image, with objects closer to the center receiving higher weights. An object at the center would be weighted twice much as object at the edge of the image, with scaling linearly.
Number count During recall and precision calculation, each object category receives partial credit if the number of detected object categories is either underestimated overestimated, depending the error.
The results summarized in Tab. 4 Since none of these weighting schemes seemed improve the metric, they were not included the final version order to avoid needlessly complicating the metric.
16

---

Published as a conference paper at ICLR 2026
C.2
ADDITIONAL RESULTS OF SEC. 5.1
Table 5: The meta-evaluation results of each metric. The best results are bolded.
Metric
Pairwise Acc.
Kendall
Pearson
Object
75.8%
.166
.708
Cap-Sim
73.8%
.477
.683
EfiNet
78.0%
.559
.748
Object F1+ Cap-Sim
78.3%
.566
.768
Object F1+ EffNet
80.1%
.602
.794
Cap-Sim+ EffNet
79.2%
.583
.787
BLIP VQAScore GIT VQAScore
71.3%
.427
.666
71.7%
.434
.574
SEED
81.0%
.621
.813
We present additional meta-evaluation results for possible combinations components of SEED in Tab. [5] In addition, we explored alternative options for measuring the semantic similarity: CLIP- FlanTS VQA scores (Lin et al. 2024) with BLIP/GIT generated captions for GT images. Indeed, can be observed that SEED demonstrates the best agreement with human evaluations.
C.3 COMBINATION OF EVALUATION METRICS
0.6
Pixcorr 0.09
0.10
0.18
0.21
0.33
0.36
0.36
0.20
0.44
0.39
SSIM
A2ex(2)
0.18
0.18
0.19
0.22
0.34
0.35
0.22
0.20
0.21
0.21
ASe
0.21
0.22
0.22
0.24
0.36
0.40
0.31
0.28
0.31
0.30
Inception
0.33
0.32
0.36
0.33
0.41
0.34
0.33
CLIP
0.36
0.40
0.37
EffNet
0.36
0.44
0.22
0.31
0.34
0.38
0.56
0.54
0.60
0.58
SwAV
0.20
0.26
0.20
0.28
0.33
0.37
0.54
0.39
0.56
0.53
0.2
Object FI
Cap-Sim
0.39
0.43
0.21
0.30
0.34
0.37
0.58
0.53
0.57
0.48
SSiM Alex(2)Alex(5) Incep.
CLIP
EfNer
SwAV
Cap-Sim
Figure 8: The heatmap of correlations between metric combinations and human evaluation, mea- sured by Kendall's Tau-b. The green outline indicates combinations within current metrics.
To investigate possible candidate metrics that could be included in SEED, we computed the corre- lation with human evaluations for each possible metric combination, as shown in Fig. X The com- bination is calculated by simply averaging the two metrics. The highest-performing metrics come from the combination of Object F1, Cap-Sim, and EffNet, with each combination outperforming the individual components. This result naturally prompts the combination of those three to obtain SEED.
One interesting observation is that it is impossible to create superior evaluation metric by combin- ing existing metrics; all possible combinations within existing metrics are not better than standalone EffNet. A better metric emerges only when combined with Object Fl Cap-Sim. We believe that
17

---

Published as a conference paper at ICLR 2026
Metric
Rank
PixCor
755
())
SSIM
75
(c)
Inception
17
Alax22
157
CLIP
9
Alax(a)
188
SEED
646
Metric
Rank
Human
3
Metric
Rank
331
(d)
964
Alex22
826
SwAV
868
Alax(5)
726
SEED
568
SEED
47
GT
Recon
GT
Recon
Figure 9: Examples of worst-case judgments for other metrics
this is one indirect evidence that our proposed metrics evaluate the reconstructions from different angle from EffNet, making it possible for them to work as complementary metric for each other.
D ADDITIONAL EXAMPLES AND ANALYSIS OF WORST-CASE JUDGMENTS
D.1 WORST-CASE JUDGMENTS FOR OTHER METRICS
Discussions of worst-case judgments in Sec. 5.3 were focused on individual metrics of SEED in order to provide insight as why SEED performed better than components. In Fig we provide some worst-case judgments for the existing metrics (PixCorr, SSIM, AlexNet, Inception, and CLIP) to analyze cases where those metrics make mistakes and how SEED might improve upon them.
Fig. (a) and (b) represent cases where the four low-level metrics, PixCorr, SSIM, Alex(2), and Alex(5), either overestimates or underestimates the similarity of the two images. is fairly straight- forward to why those misjudgments came to be for these low-level metrics: for (a), the reconstruction put a malformed airplane in place of the traffic light while the general shape and the background matches the GT. This semantic mismatch made humans well SEED rank this pair very low, while the metrics ranked this pair relatively high since the general shape and color of these match pretty well. For (b), we can see both pictures depict surfing man, while the specific shape of the waves and the general color tone of the two quite differ. This probably led to humans and SEED to highly rank this pair while the low-level metrics generally rank this pair low.
For the high-level metries, it was more difficult to pinpoint the causes for any mistakes or find reliable pattern between the mistakes, compared the low-level metries, due their abstract nature. Nevertheless, in Fig. (c) and (d), we show the worst-case judgments for the four high-level metrics, further grouped based on their evaluation method. (c) shows worst-case judgment for the 2-way identification methods, Inception and CLIP. We can see that the reconstruction depicts a slightly disfigured hand, while the object held by the hand was changed from remote control to smartphone. This difference likely led to humans and SEED to not favor the reconstruction, while Inception and CLIP might have overvalued the reconstruction since it still features a hand. (d) shows worst-case judgment for the two correlation distance metries. EffNet and SwAV, which is an example brought from Fig. (c). We can see that SwAV made misjudgment similar to EffNet. We suspect the cause for this mistake is similar, since SW also trained using ImageNet.
D.2 WORST-CASE JUDGMENTS FOR SEED
Of course, SEED is not flawless evaluation metric. SEED has the potential to make a misjudgment when its three elements all make misjudgment for one reason another, which is displayed in Fig. 10 Here we can see the GT is an image with a person holding a red umbrella, while the recon- sturction is slightly ambiguous image with yellow/blue umbrella-like object top of wooden object, with a lake on the background. Humans slightly favored this reconstruction since the general pose of the image is similar and the umbrella was somewhat reconstructed. However, all elements
18

---

Published as a conference paper at ICLR 2026
Metric
Rank
Human
314
Object F1
793
Cap-Sim
847
EffNet
773
boat: 42.0
SEED
849
a yellow and blue towel sitting on top of a wooden bench.
person holding red umbrella over wall.
GT
Recon
Figure 10: Example of worst-case judgment for SEED
of SEED undervalued this reconstruction, which consequently led to SEED to also undervalue the reconstruction. we look into the reason, Object F1 gave poor score since the person from the GT is missing while the yellow/blue umbrella was detected as boat instead, probably due to the wooden protrusion and the watery background. Cap-Sim gave a poor score for a similar reason; the person was missing from the reconstruction caption, the yellow/blue umbrella was identified towel, and the wooden bench was added the caption. While it is difficult to know rationale, EffNet gave poor score, presumably due to the background and the color of the umbrella of the reconstruction being different.
As illustrated by this example, SEED has a chance to fail when the reconstruction is distorted or has some unusual features. This essentially puts the models in an oft-o--diittibuttin setting, and they may make decision that is not aligned with typical human judgment. Improving the object grounding model or the image captioning model of SEED to better generalize to these distorted images, advancing the brain decoding models to produce distorted images in the first place would help in these scenarios.
D.3 ADDITIONAL WORST-CASE JUDGMENTS FOR SEED ELEMENTS
Here, we present additional examples of the worst-case judgments discussed in Sec. 5.3
Table 6: Evaluation results with pre-trained models provided by authors. SNM represents the proportion of "semantic near-miss." SDM quantifies the proportion "semantic detail misses", defined the fraction of cases with Object F1 0.7 and Object F1 SEED 0.2. *MindEye2 was evaluated with 18 additional images, following the original work.
Hege-Level
Method
Object FI
Cap-Sim
SEED
SNM
SDM
Incep
CLIP f
EftNet |
SwAV
MindEye2
Scotti et al.
2024
95.1%
93.2%
.677
.340
.577
.42
481
.175
.107
NeuroPictor
Huo et al.
2024
94.6%
93.5%
.637
.350
481
.522
.52
.191
.097
MindBridge
Wang
UniBrain Wang
2024b
92.3%
93.7%
.006
.415
.206
BrainGuard
Tian et
2025
94.8%
94.8%
645
.374
489
.25
.456
.192
.092
19

---

Published as a conference paper at ICLR 2026
Metric
Rank
Human
195
(a)
Object F1
836
Cap-Sim
339
EffNet
355
Metric
Rank
Human
32
Object F1
252
Cap-Sim
776
EffNet
135
Metric
Rank
Human
252
(c)
Object F1
133
Cap-Sim
216
EffNet
GT
Recon
Figure 11: Additional examples of worst-case judgments
Fig. (a) illustrates where Object F1 significantly deviates from human evaluation, assigning a score O. This discrepancy arises because the detected category from the GT is Sink, while the detected category from the reconstruction Toilet. Since Object Fl evaluates similarity based solely on the presence of the detected category, it assigns a zero score, despite the reconstruction successfully generating an image that represents the concept of restroom.
Fig. II (b) illustrates a case where Cap-Sim assigns low similarity score between two images. The captions generated by GIT for the GT and the reconstruction
[A group of people walking across snow covered field.] and [A person riding skis on snowy surface.), respectively. This low similarity is likely due the different actions that people the image are taking, despite human and other evaluation metrics considering them similar.
Fig. II (c) presents a case where the EffNet metric produces an extremely low correlation between two images. The ImageNet Top-1 predictions for the GT and the reconstruction are Container ship and Truffic light, respectively. This example highlights how EffNet can yield an incorrect evaluation due to misclassification.
Although the main objects in both images resemble a yacht-like boat, EffNet assigns them to dif- ferent classes. We believe this occurs because the class yacht is not included in the 1,000 ImageNet categories. Consequently, EffNet predicts the GT as a Container ship, likely focusing on the ship behind the yacht, while misclassifying the reconstruction Traffic light, completely irrelevant class.
E RE-EVALUATION OF EXISTING DECODING MODELS
We report the performance cxisting visual decoding models evaluated with SEED in Tab, 6 We report the evaluation results of five recent decoding models: MindEye2, NeuroPictor, MindBridge, UniBrain, and BrainGuard. We directly evaluated the pre-trained models provided by the authors of each work. The evaluation metrics consist of four existing evaluation metrics alongside our proposed Object FI, Cap-Sim, SEED, and the semantic near-miss rate. Note that MindEye2 was evaluated with 18 additional image pairs as per the original work due the sequential disclosure of the NSD dataset.
20


Reviews:
[{"review_id": "PnKUnG0qB4", "reviewer": "Reviewer_UN34", "summary": "This paper identifies a critical problem: current evaluation metrics are poorly aligned with human judgment of semantic similarity in visual brain information reconstruction. Even if results achieve near-perfect scores on existing metrics, they may still be semantically flawed. To address this, they propose SEED, a new evaluation metric that integrates three complementary components—Object F1, Cap-Sim, and EffNet. Through extensive \"meta-evaluation\" against a new human judgment dataset, they demonstrate that SEED aligns significantly better with human evaluation than all existing metrics.", "questions": "1. In terms of semantic understanding, this is a problem that all generative models encounter. Does this method and insight also work for current generative models? Or is there any new semantic metric in the field of image generation now? How are they being used?\n\n2. Given that multiple large models were used, what was the speed of the inference? What is the total amount of computing resources consumed for the operation? If it is extremely large, it may affect the use of the indicators. Besides, how and why are the weights of multiple models determined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the article is very good. The current evaluation of decoding models indeed has such a problem. Especially, many tasks utilize contrastive learning in the CLIP space and then proceed with image generation. Naturally, this leads to excellent generation scores.\n\n2. The design of SEED is thoughtful. The two new proposed metrics, Object F1 (object-level attention) and Cap-Sim (feature binding into a scene description), offer novel and complementary perspectives. It is indeed something worth noting during the recovery process.\n\n3. The article conducted numerous human alignment experiments and evaluated the results of many models, proving that the proposed indicators are indeed closer to human understanding of image restoration.", "weaknesses": "1. The proposed indicators mainly focus on assessing semantics and may be insensitive to information like color and texture. This is also a significant factor influencing restoration and human perception.\n\n2. The article does not provide a clear description of how human evaluations are conducted. For example, in Figure 1, why is that image ranked 846th out of 1000? How was this ranking determined? The human evaluators rated both \"semantic and perceptual similarity\". For the meta-evaluation, which rating was used?\n\n3. The method used in the article is reasonable but somewhat complex, which is composed of many current models. I'm not sure if there might be any hidden biases here. If one of the models has a problem, will it lead to the failure of the evaluation? This merely presents a feasible solution, but does not discuss whether there are better evaluation methods or directions for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iIPT3iHJ8M", "reviewer": "Reviewer_hyma", "summary": "This paper proposes SEED (Semantic Evaluation for Visual Brain Decoding) — a new metric designed to better evaluate visual brain decoding models in terms of semantic similarity. The authors integrate three components — Object F1, Cap-Sim, and EffNet — to capture complementary aspects of human-like visual perception. Extensive human evaluations demonstrate that SEED aligns more closely with human judgments than existing metrics.", "questions": "* Could SEED be adapted to multimodal decoding (e.g., brain-to-text or cross-modal tasks)?\n\n* Could evaluating the semantic fidelity of reconstructions using fine-grained category labels rather than object detection models provide a more efficient and reliable assessment?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* A more human-like evaluation framework is essential for the brain decoding community.\t\n\n* The use of large-scale human evaluations is impressive.", "weaknesses": "* A key limitation of SEED lies in its reliance on off-the-shelf captioning and detection models (e.g., GIT, Grounding-DINO). These components were not trained to reflect human semantic judgments but to optimize task-specific objectives (caption likelihood or object detection accuracy). As a result, SEED may inherit their systematic errors, leading to misleading evaluations in certain cases.\n\n* The authors argue that in some existing metrics such as n-way identification, decoding models have reached near-ceiling performance, but this is largely due to the small candidate pool typically used in evaluation. If the number of candidate images is substantially increased (e.g., 100-way or 1,000-way identification), most current decoding models exhibit significant performance degradation, revealing substantial room for improvement.\n\n* Moreover, recent progress in visual decoding is moving toward multimodal decoding frameworks (text and image). In such settings, the semantic quality of reconstructed images can already be assessed through predicted text. This trend raises questions about the necessity of using GIT-based caption generation in SEED. Since GIT introduces additional linguistic biases and noise unrelated to the decoding model itself.\n\n* The complexity of SEED may to some extent hinder its widespread adoption. For example, its reliance on multiple models introduces issues such as version differences, model updates, and parameter inconsistencies, which reduce the reproducibility of results and the comparability within the community.\n\nMinor: Several recent works such as [1-2] in brain decoding are not cited or discussed.\n\n[1] Bridging the Gap between Brain and Machine in Interpreting Visual Semantics: Towards Self-adaptive Brain-to-Text Decoding, ICCV 2025.\n\n[2] Mindgpt: Interpreting what you see with non-invasive brain recordings, IEEE TIP 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DqeKihkjaB", "reviewer": "Reviewer_cg7v", "summary": "In “SEED: Towards More Accurate Semantic Evaluation for Visual Brain Decoding” authors proposes a new metric, SEED, to assess the semantic accuracy of images reconstructed from brain activity. They argue that existing metrics (e.g., SSIM, CLIP, Inception) fail to align with human perception of semantic similarity, often overestimating model performance. SEED combines three complementary measures: (1) Object F1, which captures object-level overlap using open-vocabulary grounding models; (2) Cap-Sim, which compares captions of ground-truth and reconstructed images using text embeddings; and (3) EffNet, a feature-based similarity measure. Through medium-scale human evaluations on 1,000 pairs from NSD and additional datasets, SEED metric demonstrates stronger correlation with human judgments than existing metrics and reveals that state-of-the-art decoding models still struggle to reconstruct fine-grained semantics.", "questions": "1) Will this metric/code be released in a easy to access and no-brainer way to encourage wide usage? I think this is really the keypoint here, otherwise the whole thing loses a lot of traction.\n\n2) In the field everyone is competing to have his own line in bold. Your new metric was evaluated on some famous approach, but the best-in class and the order didn't change much. Since the ranking was mostly unaffected, could you highlight better the cases where this metric is telling something more? I liked the failure cases ecc, but I really would love to see more example and use cases where errors or successes were hided by other metrics and SEED is able to point them out\n\n3) The sample size for human evaluation is kind of limited but I understand this is difficult to solve. There is a lot of work on THINGs dataset with million of annotations and human preferences between categories and images ecc, as well as many models trained on them, Does it make sense to compare SEED with these models?\n\n4) One thing that could blow my mind is whether you think SEED could became an objective loss function to be optimized. Right now, as far as I understood it involves several non-differentiable operation that limit the use as a metric alone, but do you think this could be extended as a training objective?\n\n5) Discussion for a position paper like this is a bit short and dry. What are the implication? What's the significance? Why it's needed? Plus general reflection on the field would improve the maturity of the work.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a very timely contribution: Evaluation misalignment between metrics and human perception is a major issue in brain-to-image decoding. In the last 2-3 years I have worked in the field and seen a significant amount of brain decoding works, always competing for few fractions of percentage points of saturated metrics that not always reflect real performance.\nHere, SEED addresses a crucial gap that could significantly impact future research.\n\nThe metric is well-motivated, interpretable, and partially grounded in both neuroscience and computer vision (object-based attention and semantic binding).\n\nThe authors conduct meta-evaluations, robustness tests across models/datasets, and qualitative analyses (worst-case examples, semantic near-miss, and failure modes).\n\nA very good point is the commitment to release the human evaluation dataset adds substantial reproducibility and community value. I'd also suggest the authors to make this metric a very easy to use (ideally a one-liner python function) to encourage reproducibility.", "weaknesses": "Introducing the metric is an important point, but I felt something is missing in this work:\n\nSome literature is absent, the number of work and approaches has become very big. Many are missing. To position in the field and constructively criticize the practices I think extended literature review is needed. \n\nThis metric is better aligned with human judgement, but still has some bias from the model and limitations. Could the author discuss more in detail these limitations and future work to be improved? Other metrics, ecc.\n\nOverall, this metric is a linear combination of a bunch of others, all relying on some models. In appreciate the point of view even if the technical contribution is somehow limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)