Generating metamers of human scene understanding
Authors: Ritik Raina
Abe Leite
Alexandros Graikos
Seoyoung Ahn
Dimitris Samaras
Greg Zelinsky
Venue: ICLR 2026 Oral (2026)
Keywords: human scene understanding
generative modeling
Primary area: applications to neuroscience & cognitive science

Abstract: Human vision combines low-resolution “gist” information from the visual periphery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce MetamerGen, a tool for generating scenes that are aligned with latent human scene representations. MetamerGen is a latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. “foveated”) inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, we conducted a same-different behavioral experiment where participants were asked for a “same” or “different” response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding. Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned on viewers’ own fixated regions.

Published as a conference paper at ICLR 2026
GENERATING METAMERS OF HUMAN SCENE UNDER- STANDING
Ritik Raina' Abe Leite'
Alexandros Graikos'
Seoyoung Ahn? Dimitris Samaras'
Gregory J. Zelinsky!
!Stony Brook University UCC Berkeley
ABSTRACT
Human vision combines low cww-resolution "gist" information from the visual periph- ery with sparse but high-resolution information from fixated locations to construct a coherent understanding of a visual scene. In this paper, we introduce Metamer- Gen, a tool for generating scenes that are aligned with latent human scene rep- resentations. MetamerGen is latent diffusion model that combines peripherally obtained scene gist information with information obtained from scene-viewing fixations to generate image metamers for what humans understand after viewing a scene. Generating images from both high and low resolution (i.e. "foveated") inputs constitutes a novel image-to-image synthesis problem, which we tackle by introducing a dual-stream representation of the foveated scenes consisting of DINOv2 tokens that fuse detailed features from fixated areas with peripherally degraded features capturing scene context. To evaluate the perceptual alignment of MetamerGen generated images to latent human scene representations, ducted a same-different behavioral experiment where participants were asked for "same" "different" response between the generated and the original image. With that, we identify scene generations that are indeed metamers for the latent scene representations formed by the viewers. MetamerGen is a powerful tool for understanding scene understanding, Our proof-of-concept analyses uncovered specific features at multiple levels of visual processing that contributed to human judgments. While it can generate metamers even conditioned on random fixations, we find that high-level semantic alignment most strongly predicts metamerism when the generated scenes are conditioned viewers' own fixated regions.'
con-
INTRODUCTION
Understanding the latent representation of a scene formed by humans after viewing remains a fun- damental unanswered challenge in cognitive science (Fisttein & Baker, 2019; Bonner & Epstein, 2021; Malcolm et al., 2016; Vo, 2021). What clear is that humans represent coherent scenes by mixture of "gist" information encoded from peripheral vision (Potter, 1975; Greene & Oliva, 2009) with high-resolution but sparse information that humans extract during their scene viewing fixations (Larson & Loschky, 2009; Larson al., 2014; Eberhardt et al., 2016). Related recent work on scene perception has focused on the concept of object and scene metamers generated stimuli that, although physically different from originals, cannot be discriminated as different by humans when viewed under constrained experimental conditions. (Freeman & Simoncelli, 201 1; Balas et al., 2009; Rosenholtz et al., 2012). Understanding scene metamerism is important because metamers tell us the level of misalignment between an actual and generated image that is tolerated by hu- mans and judged to be the same. Generated scenes that fail to become metamers also reveal the details that are important to scene's 's representation and that, if changed, result in the generation being detected. However, although several paradigms have been used to identify metamers (e.g., same-different tasks, A/B/X tasks, oddity judgment tasks) (Rosenholtz, 2020), this work scene perception used simple generative models to synthesize textures and shapes that were shown in be- havioral experiments to be metameric with what humans perceive in their visual periphery when
Project Link:

---

Published as a conference paper at ICLR 2026
eye position is fixed. These paradigms were not designed to study how post-gist changes in fixation affect scene metamerism or what objects a person believes to exist in their blurred peripheral view of scene, which are the problems we engage.
Inspired by these previous studies showing that generated textures and shapes can become metamers for human scene perception, introduce MetamerGen, state-of-the-an generative model that ex- tends the metamer generation approach to human scene understanding. Rather than seeking to generate simple patterns that share low-level statistics with peripheral vision, MetamerGen bet- ter captures a post-gist level of representation reflecting multiple free-viewing fixations. We see this topic as closer to scene understanding, and we consider the scenes generated by MetamerGen as hy- potheses for what people believe to be in their peripheral vision. In this paper, we will therefore use the term scene metamer to refer to two scenes that have an equivalent understanding. Our approach combines gist-level scene representation extracted from peripherally blurred pixels with higher- resolution and -raccfi--ppccficc "foveal" representations corresponding to scene-viewing fixations. Scene gist and the objects fixated during viewing are used to generate in the non-fixated blurred pixels a scene context that is aligned with what a human understands to be in their peripheral vision.
We not only show that many of the scenes generated by MetamerGen are metamers for human scene understanding, we also model the dynamic evolution of this understanding by leveraging the capability of a latent diffusion model (Rombach et al., 2022) to generate photorealistic images from diverse conditioning signals (Zhang et al., 2023; Ramesh et al., 2022). MetamerGen is a latent diffusion model (Stable Diffusion; Rombach et al., 2022), thus we use each viewing fixation conditioning signal to obtain an incremental exation-by-fixate
understanding of scene (Figure I).
To adapt Stable Diffusion to our task of generating scene in blurred peripheral pixels, we introduce a dual-stream representation of foveated scenes (i.e., ones with a high-resolution center and blurred periphery) using a self--uper image encoder (DINOv2) (Caron et al., 2021; Oquab al., 2024; Darect et al., 2024). We utilize an adapter-based framework (Mou et al.. 2023), where we condi- tion a pre-trained text-to-image diffusion model on exxationgroundee features extracted by DINOv2 feature representations obtained at each of the fixation locations. We complement the fixation rep- resentations with the scene context available from peripheral information by adding a second source of conditioning that uses DINOv2 tokens extracted from blurred version of the same image.
Our conditioning mechanism allows plausible scene hypotheses to be generated from variable input information. We expect more foveal glimpses of scene will lead to richer DINOv2 representation that will in turn enable MetamerGen to generate increasingly plausible and contextually appropriate content at the non- non-fixated scene locations, analogous to how human scene understanding becomes more elaborate with more viewing fixations. We see MetamerGen as a tool for generating fixation- specific scene understanding hypotheses that cognitive scientists can test in behavioral studies.
We integrated MetamerGen into a same-differen behavioral paradigm and conducted experiments to identify the generated scenes that are metamers for human scene understanding. In our paradigm, participants viewed a scene for a variable number of fixations (i.e., gaze contingent), followed by a 5-second delay (during which MetamerGen generated a scene from the viewing behavior) and then briefly viewed a second scene (200 msec). Their task was to judge whether this second scene was the same or different from the first. We define a scene metamer as a generation that a participant judges to be the same as the real scene that was first viewed. Our post-hoe analysis showed that while all features throughout the visual hierarchy contributed to the understanding of a scene, high-level semantic features emerged as the strongest predictors of scene understanding metamers.
PRELIMINARIES
2.1 IMAGE GENERATION USING LATENT DIFFUSION MODELS
et al., 2015; Ho et al., 2020) comprise two opposing processes diffusion process that gradually corrupts data and a denoising process that restores information. The diffusion process relies Gaussian noise of increasing intensity at every step, while the denoising process uses a learned denoiser model to reverse the degradation. By iterating this process, starting from random Gaussian noise, diffusion models generate new samples.
Diffusion models
2

---

Published as a conference paper at ICLR 2026
Latent diffusion models (LDMs) (Rombach al., 2022) reduce the overall cost by applying the dif- fusion processes in the latent space of a variational autoencoder (VAE) (Kingma & Welling, 2013). Stable Diffusion (Rombach et al., 2022) uses pre-trained VAE that spatially compresses images 8x with its encoder and decompresses latent diffusion samples with the decoder. The denoiser co(+) UNet (Ronneherger al., 2015) with pairs of down and up-sampling blocks, well as a middle bottleneck block. Each network block consists of ResNet (He et al., 2015), spatial self-attention, and cross-attention layers, with the latter introducing the conditioning information.
The cross-attention layers condition the denoising process by computing relationships between in- termediate image features during denoising and set of given conditioning embeddings. usually text. When F E Rxx represents the intermediate image features during denoising (reshaped to hwr x e for attention computation) and Raxd are the conditioning embeddings, the cross- attention mechanism first projects features into queries and embeddings into keys and values
FWQ, K eWk, V = eWW, Q Rlwxd, K E Raxds V E Rxdd
(1)
where We Rexdx WK E Rdxde, and W Raxd, are learned projection matrices. The cross-attention output is then computed as:
(QKT Vax
CrossAttention softmax
V
This mechanism allows each spatial location in the image (rows in Q) to attend to relevant parts of the conditioning (rows in K), with the attention weights determining how much the information in each conditioning embedding contributes to the denoising process at each spatial location.
2.2 ADAPTING LATENT DIFFUSION MODELS TO NEW CONDITIONS
In text-to-image LDMs (e.g., Stable Diffusion), cross-attention layers condition image features on text embeddings. An efficient approach for incorporating additional conditioning types, without retraining the model from scratch, can be achieved through adapter-based frameworks (Mou al., 2023). These adapters re-use the learned text conditioning pathways in the LDM to introduce other modalities of conditioning. This done by introducing trainable components that transform and project new condition signals into a format compatible with the UNet's existing cross-attention mechanisms. This approach has proven particularly effective for incorporating visual conditioning into text-to-image models (Ye et al., 2023; Wang & Shi, 2023; Ye et al., 2025).
2.3 SELF-SUPERVISED IMAGE ENCODERS
DINOv2 (Caron et al., 2021; Oquab et al., 2024) self-supervised vision transformer trained for hierarchical visual representation learning without manual annotations. Using multiple self- supervised objectives, including a contrastive loss that causes image features that appear together to have similar embeddings and reconstruction loss that induces patches redundantly encode information about their surrounding context, DINOv2 represents both local visual details and higher- level semanties. These properties make it excellent tool study byyfitin human scene understanding. Adeli et al. (2023; 2025) showed that self-supervised encoders are capable of capturing object-centrie representations without labels and provide backbone capable of predicting high-level neural activity in the brain.
3 MetamerGen: PERCEPTUALLY-INFORE CONDITIONING
3.1 REPRESENTING FOVEAL PERIPHERAL VISUAL FEATURES
Given image and set of fixation locations, such as those fixated by human during free-viewing, we first aim to extract the foveal information from the image at each fixation location and peripheral information capturing the overall context. We employ a DINOv2-Ban feature extractor to obtain these two sources of information. In Appendix Section A. we validate choice of DINOv2 the feature extractor for MetamerGen by showing its superiority to CLIP.
model (with registers) the
DINOv2 processes 448 x 448 images with a patch size 14 x 14, yielding 1024 tokens (32 x 32 grid), each embedded in 768 dimensions (along with CL.S token and four register tokens encoding
3

---

Published as a conference paper at ICLR 2026
fixation querynoo
Soll on mosk
Inggressnggg
visual
fixation features
periphoral
paripheral features
pre trained text-to-lmage
modified MetamerGen UNet block
L
foution liberr
input
residual
output
features
self-att sell-attention
KV
feedforward
Fivetion cross attortion
pertpheral
Figure 1: MetamerGen model architecture. High-resolution and blurred, low-resolution images are processed through EINOv2-Base to extract patch tokens. Foveal features are obtained by apply- ing binary masks high-resolution patch tokens, retaining only fixated regions. Both foveal and peripheral patch tokens are processed through separate Perceiver-based query networks that com- press features into conditioning tokens compatible with Stable Diffusion's 's cross-attention mecha- nism. The resulting dual conditioning streams are integrated into the pretrained UNet.
general information about the image). The patch token specific location encodes detailed visual and semantic information about that location, analogous to the high-resolution information sampled by the fovea during fixation. also encodes local contextual information from around the fixated locations (Adeli et al., 2023; 2025), analogous to the low-resolution parafoveal information extracted by humans. To model the information gathered during a series of fixations, we apply a binary mask Mixation to the patch tokens extracted from a scene image I, corresponding to the image locations fixated by humans, zeroing out all non-fixe non-fixated image patches.
To obtain peripheral visual features, we downsample the image and then upsample it back to 448 x 448. This blurred image, Ipenipterats is also processed with DINOv2, but now retaining all output patch tokens without masking. These peripheral tokens encode uncertain visual representations across the entire scene, capturing the noisy information available in peripheral vision that requires validation through targeted foveal fixations (Srikantharajah & Ellard, 2022; Michel & Geisler, 2011).
3.2 FOVEAL & PERIPHERAL. CONDITIONING ADAPTERS
We develop foveal and peripheral conditioning adapters to integrate visual information as additional conditioning signals in Stable Diffusion. Similar to IP-adapters (Ye al.. 2023), which integrate CLIP image embeddings into Stable Diffusion, we learn how to incorporate DINOv2 patch embed- dings into the gross-attention mechanism of the text-to-image Stable Diffusion model.
Both foveal and peripheral DINOv2 embeddings are first processed through separate Perceiver- based resampler networks R(-) (Alayrac et al., 2022; Jaegle et al., 2021) that compress the 1024 DINOv2 embeddings into 32 conditioning tokens compatible with the pre-trained UNet's cross- attention (see Appendix A.3 for more information):
Gloveal R (DINOv2 Turiginal)
Mixation) Cperipheral Reperipheaal (DINOv2( (T downsample))
4

---

Published as a conference paper at ICLR 2026
The conditions are then integrated through separate cross- coos-attention mechanisms. For each condi- tioning source (text, foveal, peripheral) we project separately into keys and values
K- = c. V. = c. Ke Ve E R c (text, foveal, peripheral}
(4)
which we then combine additively into the denoising through cross-attention:
QK text
Attention( K, V) softmax
Viest + Afoveal softmax
V Vioveal
ViTe
vdla
(5)
QQT Q
peripheral
+ perripheral softmax
Vperipherni
dde
Aroveal and Aperipheral are scaling factors that control the contribution of either foveal or peripheral visual features to the generation process. practice we "freeze" the text conditioning, by setting the text caption for all images to empty string
3.3 TRAINING AND INFERENCE
We start from a pre-trained Stable Diffusion 1.5 network (Rombach et al., 2022). The trainable components of MetamerGen are the foveal and peripheral resampler networks and their associated key-value projection matrices. Training is conducted on the complete MS-COCO training set (Lin al.. 2015) of approximately 118, 000 images. For foveal conditioning, we apply binary masks that randomly retain {1, 2, 3. 5, 10} DINOv2 patch tokens while zeroing all others. This sampling strategy ensures compatibility with our free-viewing behavioral experiments, which constrain scene viewing to a maximum of 10 fixations. At training time, the foveal locations are randomly sam- pled, whereas at inference time, they are determined by the experimental design. For peripheral conditioning, blur the images by downsampling to {0.0625x, 0.125x, 0.25x, 0.5x, 1x) of the original resolution.
To enable robust conditioning in inference, we randomly drop conditions with probabilities Pfoveal 0.05 and Pperipheral = 0.10. The higher peripheral dropout rate prevents over-reliance on peripheral features, which, despite being blurred, retain substantial visual information compared the sparse foveal features. We employ DDIM (Song et al., 2022) for 50 timesteps, with CFG++ (Chung et al., 2025) and set Atoveal 1.2 and Aperiphena 0.7 balance detail generation with scene plausibility.
We point out that, although MetamerGen is conditioned dense DINOv2 representations of an image (periph- and fixation patch embeddings), the model does not simply reconstruct input images verbatim. We attribute this to the lossiness introduced by the DINOv2 embed- dings, as well as stochasticity in the sampling process. We demonstrate this further in Appendices A.7 and A.8.
0008
Blur level
05
3.4 IMAGE GENERATION QUALITY
FID
25
We first evaluate the image quality of samples from our model using Fréchet Inception Distance (FID; Heusel et al., 2017) between images generated from Metamer- Gen and COCO-10k-test. Figure 2 shows the results us- ing a single central fixation. Green: We fix the blur level 0.25x, matching the blur level used in our behavioral paradigm, and evaluate how peripheral context affects generation quality by varying the peripheral scale. pe- ripheral scale increases, FID scores improve as the model better integrates the context coming from the peripheral DINOv2 representations. Red: We evaluated the effect of blur level and found that our model consistently generates plausible scenes for all levels of blur in evaluation. Blue: We include a text-to-image baseline using SD-1.5 with 10k random captions from the COCO training set.
20
15
02
00
pertoherl
Figure 2: FID values for different in-
put parameters of MetamerGen. Lower FID indicates closer alignment with real images and better quality.
5

---

Published as a conference paper at ICLR 2026
MetamerGen, fine-tuned the COCO images, consistently outperforms the text-to-image model, proving that we have successfully integrated images of variable resolution into the conditioning mechanism of Stable Diffusion.
4 BEHAVIORALLY--CNDIIOONN SCENE METAMERS
4.1 PROBING LATENT SCENE REPRESENTATIONS THROOGH SAME/DIFFERENT JUDGMENTS
Metamerism, term initially established in the color vision literature, has since been applied to questions in texture perception and visual crowding to infer underlying perceptual representations (Freeman & Simoncelli, 2011; Balas al., 2009; Rosenholtz et al., 2012). In the context of scene understanding, metamers offer a unique opportunity to probe what the visual system encodes and retains after viewing complex natural scenes.
Scene understanding requires the extraction of meaningful structures from complex inputs, including spatial layout, object relations, and global context (Oliva & Torralba, 2006). It is shaped by what the visual system extracts from the input and not the input itself. When person views scene A and forms internal representationa, then later sees a different scene B and forms representationg. they may believe that these scenes are the same if these representations are aligned. We infer aligned representations when a person responds "same" to generated version of the scene. These scene metamers therefore reveal what information the brain has encoded and retained from an originally viewed scene, and by exploring the factors causing generation to become a metamer we are able to investigate the structure of human scene understanding.
4.2 REAL-TIME BEHIAVIORAL PARADIGM
We developed a real-time same-different behavioral paradigm to evaluate whether MetamerGen generates perceptually convincing scene metamers. This paradigm directly tests whether images reconstructed from sparse fixational sampling can achieve perceptual equivalence with the original, thereby revealing the sufficiency of fixated information for scene representation.
Experimental Design Participants (n=45) engaged in paradigm requiring them to first freely view a scene and then make same-different judgment about that scene. Each of the 300 exper- imental trials followed the same structured sequence, and eye gaze was tracked throughout each trial (see Appendix A. for additional details and a figure illustrating the paradigm). Participants completed drift check (to maintain eye-data quality) and then freely viewed natural scene image until reaching a predetermined fixation count (1. 2, 5, 10), after which the image was removed. Critically, participants chose their own fixation locations. We systematically varied information availability by manipulating fixation count, testing how additional visual information influenced the generation quality and the rate of metamerism. A 5-second interval followed the offset of the scene during which the participant maintained fixation on an otherwise blank screen. During this inter- val, MetamerGen generated a new version of the original image in real time based on the viewing fixations. Participants were then shown a second image for 200 milliseconds, too brief for an eye movement but sufficient for a perceptual judgment (Broderick et al., 2023; Wallis et al., 2019), and asked to indicate whether this second image was the same different from the image viewed orig- inally. Although our primary experimental manipulation was whether the second image was the original generation from the participant's own fixations, also had "random" condition con- sisting of 12 other participants making a similar same or different judgment, only now between a just-viewed original image and generation based random sampling of fixations in the image.
Stimulus Selection Our stimuli were 300 images from the Visual Genome dataset (Krishna et al., 2017), subset of the YFCC100M dataset (Thomee et al., 2016). We chose this dataset to avoid overlap with the COCO data used MetamerGen training. To maximize visual diversity in our 300 image sample, we used DreamSim (Fu al., 2023) to cluster Visual Genome images in semantic representational space and selected one representative image per cluster. Images were filtered to exclude challenging elements for current diffusion models (e.g., humans, printed text, clocks).

---

Published as a conference paper at ICLR 2026
"SAME" judgment metamers
"DIFFERENT" judgment
Figure 3: Metameric vs. non-metameric judgments. (Left) Original images with human fixations overlaid in red paired with generated images judged as the "same" by participants. (Right) Original images with fixations and generated images judged as "different" by participants. More examples, including generations from random fixations, can be found in Appendix A.6.
5 RESULTS: PROBING THE FEATURES OF SCENE METAMERS
As shown in Figure 3, MetamerGen'ss 's conditioning on a viewer's own fixations enables it generate rich and plausible hypotheses for person's 's scene understanding. However, from such qualitative results is difficult to know the features responsible for a generated scene becoming a metamer. We conducted three quantitative analyses to answer this question. First, we compared visual similarity from neurally grounded CNN features with human same-different responses. Our second analysis focused on interpretability by asking which visual features Branuigs-ranging from low to high levels- shape metameric judgments. Third, we collected new behavioral data to determine the contribution of foveal versus peripheral information by ablating each from the combined model and observing the effect on metamerism rates.
5.1 NEURALLY-GROUNDE FEATURE MAPS
To investigate the level of visual processing in the human brain that is most responsible for scene becoming a metamer, we compared our behavioral judgments to an AlexNet that was trained to be robust image blur and demonstrated to have internal representations aligned with neural responses from visual brain areas ranging from V1 to inferotemporal cortex (TT) (Jang & Tong, 2024). Using this neurally- grounded model, we isolated contributions from early to late levels of the visual hierar- chy to metameric perception. As illustrated in Figure 4, our analysis pipeline treats early, middle, and late layers as proxies for different stages of visual processing. For each layer, we extracted feature maps from both original and generated images and computed cosine similarity to quantify alignment across the visual hierarchy. For attention attention-guided -uided generations, we found that as feature similarity in- creased, the proportion of participants judging images as metameric also increased. This relationship held consistently across layers of the network, from early visual features through high-level rep- resentations. These results suggest that metamerism spans the entire hierarchy of visual processing rather than being confined to a single processing stage, and that in order for a scene to be judged a metamer, there must be broad representational alignment across low and high-level visual features.
Although scenes generated from actual fixations and random fixations yielded roughly the same metamer rates (29.4% and 27.7%, respectively, p 0.24), closer comparison hints at interaction in the mid- and late-layer similarities. For attention -uided metamers, higher similarity to the origi- nal predicted more "same" judgments in consistently linear trend. However, for randomly-guides
7

---

Published as a conference paper at ICLR 2026
Low-level
Mid-level
High-
level
original
socing
correlations with human
poong
neuronal responses
generated
low
mid
high
0.6
0.6
0.6
1-0.00 p-1 66 p.0.3
r-0,00 p-3t-000 r-0.08 p=0.087
n-0.08 px2.5e
--0.05 -00.05, po0.666
Proportion "same"
genwalions
Proportion "same"
Proportion "same"
0.4
0.2
0.2
0.2
generstirn from random ftastions
0.0
0.0
0.3
0.4
0.5
0.6
0.4
0.5
0.6
0.7
0.8
Early-layer cosine similarity
Mid-layer cosine similarity
Late-layer cosine similarity
Figure 4: Multi-level feature analysis pipeline using neurally-grounded model: (Top) Early, mid, and late network layers serve proxies for different stages of processing actoss the hierarchy of visual brain areas. (Bottom) Results show that as feature similarity increased at these different pro- cessing levels, the proportion of participants judging generated images metamerie also increased. These effects were clearer when metamers were generated based on a viewer's own fixated locations (salmon) than randomly-sampled locations (turquoise).
generations, metamerism rates decreased when feature similarity was high. We interpret this trend as suggesting that realistic details in non-fixated regions may expose inconsistencies with the viewer's internal scene representation that make the generated scene discriminable from the original.
5.2 INTERPRETABLE VISUAL FEATURE ANALYSIS
Having demonstrated that metameric judgments align with neurally grounded feature similarity, next performed analyses aimed at making our results more interpretable. We did this by focusing on specific features likely to reflect low-, mid-, and high-level visual processing. We considered diverse set of features: low-level (e.g., edges, Gabor filters, color), mid-level (c.g.. depth cues, proto- object structure), and high-level (e.g., object, semanties, overall perceptual similarity). Because many of these features are correlated, we applied forward stepwise regression model to identify the most predictive subset (R2 0.039), which we focus on the main text. Detailed contributions of each feature can be found in Appendix A.4.
5.2.1 LOW-LEVEL VISUAL FEATURES
We compared human "same" judgments function of Gabor filter intensities and (ii) Sobel edge density response differences between the generated and original images to assess how low- level texture features affect similarity judgments. We used normalized responses from Gabor filters oriented at 09, 45, 90", and 1355. Surprisingly, we found that positive differences in Gabor filter responses-wiet generated images showed stronger texture responses than originals-ccorelattee with more "same" judgments. This suggests that enhanced texture definition, which makes bound- aries more distinctive, increases the perceived realism of generated images, even when they differ substantially from the originals (Ho al., 2012). We also found that greater Sobel edge density sponses (Kanopoulos et al., 1988) led to greater "same" judgments, though this effect was redundant with the Gabor filter effect (see A.5).
8

---

Published as a conference paper at ICLR 2026
mid-level depth & proto-object segmentation
high-level visuo-linguistic alignmmnt
0.6
1000L M077
Proportion "samn"
Proportion "same"
Proporbon "sme"
0.4
Proportion "samme
0.3
0.2
0.1
0.0
0.5 3.5
1.5
2.0
2,5 3.0
0.3
0.4 level segmentattion
0.5
0.6
5.7
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4
Depth Extimation (SiLog)
CLIP cosine similarity
DreamSin distance
Figure 5: (Left) Mid-level visual features driving metamer judgments: For metamers gener- ated from the viewer's own fixation locations (salmon), changes in monocular depth estimates of scene structure strongly predicted "same" judgments to generations. The alignment in proto- object segmentation between original and generated scenes, quantified by mloU, similarly predicted metamerism rate (higher mloU scores correlating with higher proportions of "same" judgments), although here the relationship was less pronounced. (Right) High-level visual features driving metamer judgments: Semantic similarity strongly predicts metameric scene understanding, with larger DreamSim distances corresponding to reduced perceptual alignment. CLIP similarity shows similar trend, but when scenes were generated from randomly-sampled locations (turquoise).
5.2.2 MID-LEVEL. VISUAL FEATURES
We tested two different mid-level visual features that represent local scene layout information prior to full scene segmentation: (i) relative depth and (ii) proto-object segmentation. We used Depth Anything (Yang et al., 2024a) to obtain depth maps from the original and generated images and compared them using Scale-Invarian Log error (SiLog) (Lee et al., 2021; Eigen et al., 2014). As shown in the leftmost plot from Figure 5, as discrepancies between depth maps increased, the pro- portion of "same" metameric judgments systematically decreased. This pronounced effect of depth on metamerism rate suggests that a mid-level representation of the depth relationships in a scene, essential for capturing scene layout (Verhoef et al., 2016), is an important factor in determining whether a generated scene becomes a metamer.
To analyze the effect of mid-level grouping metamer rates, we extracted proto-object segmen- tations using the conv3 layer of the blur-trained AlexNet (Jang & Tong, 2024). These mid-layer representations are important the formation of robust "proto-objects" (Finkel & Sajda, 1992; Rensink, 2000; Yu et al., 2014), which are the pre-semantic building blocks of visual objects created by grouping local features into simple shapes. We found that greater mloU in proto-object segmen- tation predicted more "same" judgments (Figure 5), demonstrating that similarity in proto- object segmentation also affects scene metamerism. Of the two mid-level features tested, however, the effect of depth seemed the clearer.
5.2.3 HIGH-LEVEL VISUAL FEATURES
To analyze the effect of high-level semantic features on metamerism rate, we used both (i) CLIP (Radford et al., 2021) and (ii) DreamSim (Fu et al., 2023) as learned semantic similarity models. Higher CLIP values indicate greater semantic aligmment between original and generated scenes, whereas for DreamSim greater alignment is indicated by lower values.
DreamSim distance provides the clearest evidence for high-level semantic alignment contributing to metameric judgment. We found that smaller distance values predict more "same" responses to generations (Figure 5, rightmost plot). This is likely because DreamSim was specifically trained on human judgments using two-altrmative forced-choice paradigm to capture human-like notions of visual similarity. CLIP also predicted this relationship between semantic alignment and the likeli- hood of generation becoming metamer (Figure 5). However, metamerism rate increased with semantic alignment only when generations were based on the viewer's 's own fixations. When gener- ations were from random fixations, higher CLIP similarity did not predict more "same" metameric

---

Published as a conference paper at ICLR 2026
judgments. We suggest that this interaction may be due to random fixations often falling on contextu- ally irrelevant regions, thereby exposing semantic details that are not aligned with the viewer's s inter- nal scene representation. A similar interaction between own-fixation and random conditions appears in DreamSim, but only at high distances. Together, we interpret our results to mean that scenes gen- erated from viewer's s attention have hetter high-level semantic alignment with the viewer's internal scene representation. Additional object-level visual feature analyses are discussed in Appendix A.5.
5.3 FOVEAL AND PERIPHERAL FEATURES BOTH CONTRIBUTE TO METAMERIC JUDGMENTS
We ran ablation experiment to isolate the contribution of foveal and peripheral conditioning in MetamerGen. We recruited 10 additional viewers to participate in a same-different task similar to the primary experiment reported above, but with four second-image conditions that systematically assessed the impact of conditioning: identical original images (true "same" responses), generated images using both foveal and peripheral conditioning (as in the primary experiment), generated images using peripheral-only, and generated images using foveal-only conditioning. We found that while both foveal and peripheral conditioning played role in whether a generation becomes a metamer, the role played by peripheral conditioning was greater. As expected. the full model had the highest metamer rate of 54.5%, compared to the second highest rate of 45.8% in the peripheral- only generation condition. The metamer rate in the foveal-only condition was only 8.4%. This much lower rate was due to the generations in that condition correctly capturing fixated details but differing from originals substantially in the non-fixated regions. In contrast, generations in the peripheral-only condition were more likely to be judged metamers because they better captured the global scene structure and layout. Nevertheless, this ablation analysis shows that the additional conditioning from foveal inputs, when combined with the peripheral conditioning, contributes visual and semantic information that leads to generations better aligned with human scene understanding than generations produced from peripheral conditioning alone. We replicated the multi-level visual feature analysis from Section 5.2 for each condition of this ablation experiment in Appendix A.9.
LIMITATIONS
While MetamerGen is effective at generating semantically coherent scenes from sparse visual in- puts, inherits limitations from the pre-trained Stable Diffusion model on which it is built. In our work, we identified two main limitations: (1) difficulties producing ane-grained facial details and accurate limb articulations (Narasimhaswamy et al., 2024; Wang et al., 2025), and (2) generations of text were often unreadable (Yang et al., 2024b) even when directly fixated. To mitigate the effects of these model weaknesses on our behavioral experiment, we excluded images containing such prob- lematic elements. Including these elements would have caused participants to respond "different" due to Stable Diffusion artifacts rather than differences in their own scene representations.
7 DISCUSSION
This paper introduces MetamerGen, a latent diffusion model for generating images that are aligned with human scene representations enough to become scene metamers. MetamerGen demonstrates that sparse conditioning based on peripheral gist and fixation-specific details is sufficient to produce human-aligned scene representations, and should be of broad interest to communities ranging from cognitive scientists to machine learning researchers. For cognitive scientists, MetamerGen becomes a powerful tool for exploring the human representation of scenes. It enables the testing of fixation- specific hypotheses about what person's 's scene understanding would be given their allocation of at- tention during scene viewing. For machine learning researchers, MetamerGen advances generative modeling by leveraging sparsely sampled inputs to produce semantically coherent scene generations that are broadly aligned with human scene representations. This greater human alignment will con- tribute to next-generation assistive technologies capable of interacting more naturally with humans. Lastly, while we found that attention -guided conditioning yielded stronger correlations across all feature hierarchies (Figs. 4, 5), also found no significant difference in metamerism rates be- tween random and attention aatention-guided conditions and this has practical implications for the collection of large-scale datasets of scene metamers because it removes the necessity for eye-data collection.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
The behavioral data presented in this work was collected in accordance with ethical guidelines deter- mined by the review board Stony Brook University responsible for overseeing research on human subjects. All subjects provided informed consent before participating in the behavioral experiment accordance with board-approved protocols and forms. Participation was entirely voluntary, and participants were free to withdraw at any time without penalty. All behavioral data collected for this study, including the gaze-fixation data collected using the EyeLink 1000 eye-tracker, were de- identified (codified) prior to analysis that there no link back the individual participants.
ACKNOWLEDGMENTS
We would like to thank the National Science Foundation for supporting this work through awards 2123920 and 2444540 to GJZ, and the National Institutes of Health through their award RO1EY030669, also to GJZ. AL is supported by NSF-GREP award 2234683, AG supported by NSF grants IIS-2123920 IIS 25-22120066 awarded to DS. Any opinions, findings, conclusions, rec- ommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agency.
REFERENCES
Hossein Adeli, Scoyoung Ahn, Nikolaus Kriegeskorte, and Gregory Zelinsky. Affinity-based at- tention in sepersuue transformers predicts dynamics of object grouping in humans. arXiv preprint Stti:23306002994 2023.
Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses. arXiv preprint arXiv:2505. 2025.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022. URL https:::p..........ttt /aariaivvvoooo orgg 2204 141.1499 98.
Benjamin Balas, Lisa Nakano, and Ruth Rosenholtz. A summary--tattisic representation in periph- cral vision explains visual crowding. Journal of Vision, 9(12):3311-13118 2009.
Luca Barsellotti, Lorenzo Bianchi, Nicola Messina, Fabio Carrara, Marcella Cornia, Lorenzo Baraldi, Fabrizio Falchi, and Rita Cucchiara. Talking to dino: Bridging self-supervised vision backbones with language for open-vocabulary segmentation. Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 22025-2201 2025.
Michael F. Bonner and Russell A. Epstein. Object representations the human brain reflect the co-oceurrence statistics of vision and language. Nature Communications, 12:4081, 2021. doi: 10. 10.1038//41467--0124446666
William F. Broderick, Gizem Rufo, Jonathan Winawer, and Eero Simoncelli. Foveated metamers of the early visual system. eLife, 12:RP90554, 2023. doi: 10. 10.7554/eLife.905544 URL https: / //d.i..ogg//ii 10. 7554/ 90554.1.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. CFG++: Manifold-conste constrainndd classifier free guidance for diffusion models. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. URL https: https://openreviii http///peeeeew.eeeeeeee nett/ forum?

---

Published as a conference paper at ICLR 2026
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2024. URL https: /arriiiiiv..///.i///a//////aiavm wwg/oo///ooooooooooo...o.. abs 2209 16588.
S. Eberhardt, C. Zetzsche, and K. Schill. Peripheral pooling is tuned to the localization task. Journal of Vision, 16(2):14, 2016. doi: 10.1167/16.2.14, URL https www.:ooo.........
2.14.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network, 2014. URL https:: //axiii. org/ abs 1406. 2283.
Russell Epstein and Chris I. Baker. Scene perception in the human brain. Annual Review of Vision Science, 5:373-397, 2019, doi: 10.1 http://urree-viiioo0911111111-11
L. H. Finkel and P. Sajda. Proto-objects: an intermediate-level visual representation. In Optical Society of America Annual Meeting, Technical Digest Series. Optica Publishing Group, 1992. paper FO1.
Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. Nature Neuroscience, 14 (9):1195-1200 1201, 2011.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, volume 36, pp. 50742-50768, 2023.
M. R. Greene and A. Oliva. The briefest of glances: The time course of natural scene understanding. Psychological Science, 20(4):464 472, 2009, doi: 10.11 (0.11111) (0.1111/j.1467-91 00066799800020902211666
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL ht https:::/axxii.rrgg abs 111111311311
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.1m 11239, 2020.
Tiffany C. Ho, Scott Brown, Newton A. Abuyo, Eun-Hae J. Ku, and John T. Serences. Perceptual consequences of feature-based attentional enhancement and suppression. Journal of Vision, 12 (8):15-15, 08 2012. ISSN 1534-7362, doi: 10.1167/12.8.15, URL https::///// /ttp://ooi.ooo org/10. 1167/12.8. 8.2.2...1 15.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021. URL https: http:/://axxvvoo/// abs/2103, 03206.
Hyodong Jang and Frank Tong. Improved modeling of human vision by incorporating robustness to blur in convolutional neural networks. Nature Communications, 15(1989), 2024. doi: 10310
$41467-024-45679.0 URL https: //t.///o.iio/ 1tt///...00///111111111 020-44677--2--4790000
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLOv8, 2023.
URL https:
//github. cot.b/.comlllio http////aaaalltssssrrlaalaaa
Nick Kanopoulos, Nagesh Vasanthavada, and Robert L Baker. Design of an image edge detection filter using the sobel operator. IEEE Journal of solid-state circuits, 23(2):358-367, 1988.
Diederik P Kingma and Max Welling. arXiv:1312.00 2013.
Auto-encoding variational bayes.
arXiv preprint
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Interna- fional Journal of Computer Vision, 123(1):32-7,, 2017.
12

---

Published as a conference paper at ICLR 2026
A. M. Larson and L. C. Loschky. The contributions of central versus peripheral vision to scene gist recognition. Journal of Vision, 9(10):1-16, 2009. doi: 10.1167/9.10.6. URL https:
//doi. wow.iorg//i..oooooooooooooo wt./g//1.111111 1ww.11....11..11
6.
A. M. Larson, T. E. Freeman, R. V. Ringer, and L. C. Loschky. The spatiotemporal dynamics of scene gist recognition. Journal of Experimental Psychology: Human Perception and Per- formance, 40(2):471, 2014. doi: 10.1037/a0033986. URL https: httpps:/aci..rgg/ 1037/ a0034986.
Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation, 2021. URL https: /ttx/v.xiivv..//////..////aaalla wwxxivvorrg org/ abs/ 0tt//997..1030
Yihao Li, Saeed Salehi, Lyle Ungar, and Konrad P. Kording. Does object binding naturally emerge in large pretrained vision transformers?, 2025. URL 24709.
org/ abs/ 2510.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015, URL htwpthttt/ttt //ttpps://axiv.... wtt/aas/////oo/oooo/o..//t...sre 11405 0312.
George I. Malcolm, IIA Groen, and Chris I. Baker. Making sense of real-world scenes. Trends in Cognitive Sciences, (0(11):843-28 2016. doi: (.006/..iiss.211..99.
Melchi Michel and Wilson S. Geisler. Intrinsic position uncertainty explains detection and localiza- tion performance in peripheral vision. Journal of Vision, 11(1):18, 2011. doi: 10.1167/11.1.18. URL htt.t:tt////ll
org/
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302. 08453, 2023.
Supreeth Narasimhaswamy, Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, and Minh Hoai. Handiffuser: Text-to-image generation with realistic hand appearances. In 2024 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2468-2479. IEEE, June 2024. doi: 10. 111109/cpp55733.222400000
URL http://dx: doi org/
1109/CVPR52TT 2024 00239.
Aude Oliva and Antonio Torralba. Building the gist of scene: the role of global image features in recognition. Progress in Brain Research, 155:23-36, 2006.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Werr Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL https: http/aiiivv.////// org abs 22104.334000 07193.
Mary C. Potter. Meaning in visual search. Science, 187:965-966, 1975. doi: 10. 10.1126/science. 1145183.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://azze org/ wttgg/ab///uu/ /213 00020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL 2204.00 06125.
Ronald A Rensink. The dynamic representation of scenes. Visual Cognition, 7(1-3):17-42 2000. doi: 00080/13506800394666
13

---

Published as a conference paper at ICLR 2026
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, 2015. URL https: / //wpvvvvvvvv//////////////////vl org/ abs 10500 04597.
Ruth Rosenholtz. Demystifying visual awareness: Peripheral encoding plus limited decision com- plexity resolve the paradox of rich visual experience and curious perceptual failures. Attention, Perception, & Psychophysics, 82(3)901-9225, 2020.
Ruth Rosenholtz, Jie Huang, Alvin Raj. Benjamin Balas, and Livia Ilie. A summary statistic representation in peripheral vision explains visual search. Journal of Vision, 12(4):14, 2012.
Jascha Sohl-Dickster Dicksteein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 1256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL m.r.press / vv7
html.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL
https::/aaxat otg/ httg/bb/////eoo. 2000000000000 02502.
SR Research Ltd. Eyelink 1000.
[Apparatus and software]. 2006. URL https: www.ssrreeerrc........
/ //web. archive
/twee///2200001550//0////// com/ fixed_iaiddmi..piie php.
Jatheesh Srikantharajah and Colin Ellard. How central and peripheral vision influence focal and ambient processing during scene viewing. Journal of Vision, 22(12):4, 2022. doi: 10. 10.1167/jov. 22. 12.4. URL https:t////o : doi. /ttp//o........1/o////.
22. 12.....4.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: the new data in multimedia research. Commun. ACM, 59(2): 73, January 2016. ISSN 0001-0782. org/10.111 0te/1001145/2812288
doi: 10.1 145/2812802 URL https://doi.
Bram-Ernst Verhoef, Rufin Vogels, and Peter Janssen. Binocular depth processing in the ventral visual pathway. Philosophical Transactions of the Royal Society B: Biological Sciences, 371(1697)200100559, 2016, doi: 10. (098/rstb2215500255
URL https://
royalsocietypubllatt
org/ /d/// atw.sas...... 1098 /0000/1/0000 rttpb 2015 0259.
Melissa Le-Hoa Vo. The meaning and structure of scenes. Vision Research, 181:10-20, 2021.
ISSN 0042-6989. doi: http:://oi.or//ll..ooooo.sssssss
URL https: httt::///ww..
Thomas SA Wallis, Christina M Funke, Alexander S Ecker, Leon Gatys, Felix A Wichmann, and Matthias Bethge. Image content is more important than bouma's law for scene metamers. elife, 8:e42512, 2019. 42512.
doi: 10.75544eLifc.42512. URL https:://odt org 7554/eLife.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference, 2024. URL https://///......t. /waxivvvvv/./////////a/////aaava wwt.voorgga//////e 2.12.. 01597.
Kaihong Wang, Lingzhi Zhang, and Jianming Zhang. Detecting human artifacts from rext-to-image models, 2025. URL https::tp//c htttps///arxiw.orr//aasssss
138422.
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation, 2023. URL https: : //t//aiivv////////////.////aatva http:///wwwwwwwwwwaassesce/////e wttp:/ww.wrgg/bb//2/...0.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a.
Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, and Kai Chen. Glyphcontrol: Glyph conditional control for visual text generation. Advances in Neural Informa- fion Processing Systems, 36, 2024b.
14

---

Published as a conference paper at ICLR 2026
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image image diffusion models. arXiv preprint arXiv:2338..677 06721,, 2023.
Zhipeng Ye, Feng Jiang, Qiufeng Wang, Kaizhu Huang, and Jiaqi Huang. Idea: Image description abs 2501 08816.
enhanced clip-adapter, 2025. URL atttps://arxi........
Chen-Ping Yu, Dimitris Samaras, and Gregory J. Zelinsky. Modeling visual clutter perception using proto-object segmentation. Journal of Vision, 14(7):4, 2014. doi: 10.1167/14.7.4. URL https:
//doi..rg//0/ 1167/14 7.4.7...
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. URL https: /ttp//aaw..g/.abaaaa/// htt.://///22.20......
15

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 EXPERIMENT DETAILS
MetamerGen
same
diff
+
+
check
fixation
original image It. fixs.]
fixation cross
generated image (209 ms)
decision stage [19 s]
trial execution
Figure Real-time paradigm for determining scene metamers. Each trial begins with drift correction and central fixation, followed by free viewing of an original scene for a predetermined number of fixations. After image offset, participants maintain central fixation for seconds while fixation coordinates are transmitted via API to MetamerGen for real-time image generation. The generated image (or original, depending on the condition) is then presented the viewer for 200ms, followed by an enforced same-different behavioral judgment via gamepad within 10-second response window.
For each behavioral trial, we map the fixation coordinate (x., y) of each viewing fixation to the corresponding patch token DINOv2's 32 x 32 grid. For 448 448 input images, each patch token represents a 14 14 pixel region (roughly 1.2 x 1.2" visual angle). Fixation coordinates are normalized to this grid space, with the nearest patch token selected and all others zeroed out, forcing the model reconstruct the entire scene from sparse fixation inputs.
Eye movements were recorded using an EycLink 1000 eye-tracker (SR Research Ltd., 2006) in tower-mount configuration. This configuration positions the infrared camera above the participant via a mirror, providing an unobstructed view while enabling monocular tracking over a wide visual angle (559 horizontal and 45 vertical). The participant, whose head fixed by chin head restraint, viewed stimuli on a 27 inch (2560 x 1440 resolution) 240Hz OLED monitor posi- tioned 24 inches from their eyes, which subtended approximately 55% 30 degrees of visual angle on their retina. Prior to each experimental session, a standard 13-point calibration procedure was performed to ensure accurate eye-gaze tracking. Default algorithms were used detect eye move- ments and fixations during both the fixation-depender free viewing of scenes and all subsequent analyses of eye-gaze fixations.
and
A.2 MetamerGen TRAINING AND INFERENCE DETAILS
We train following the configuration of Stable Diffusion 1.5 (linear scheduler, fixed variance) for 200 steps with a batch size of 32, distributed across NVIDIA H100 GPUs, using the AdamW optimizer with learning rate of 10-4 and weight decay of 0.01. Images from the dataset padded with Os to preserve aspect ratios. The model generates output RGB images of size 512 x 512.
A.3 PERCEIVER-BASED RESAMPLER ARCHITECTURE
The Perceiver-based resampler networks R(-) compress variable-length visual embeddings into a fixed number of conditioning tokens suitable for cross-attention in the pre-trained UNet of Stable Diffusion. This architecture is adapted from Alayrac et al. (2022) and Jaegle et al. (2021) Alternative approaches than resamplers like mean pooling convolutional downsampling would lose spatial relationships and semantic structure in the conditioning tokens (c.g. our case DINOv2) that are crucial for high-quality image generation.
16

---

Published as a conference paper at ICLR 2026
Perceiver Attention The component is cross-attention mechanism that allows fixed set of learned latent queries to attend to variable-length input sequences (DINOv2 tokens). Given input features 1nxd and latent queries l' gm the Perceiver attention computes:
(WQ K, V concat x. WV
(6)
QQT
PerceiverAttn (z, E = softmax
V
ddz
The key insight is that queries come solely from the learned latents L, while keys and values are computed from both input features and latents concatenated together. This allows the latents to attend to relevant information the input sequence while maintaining their learned structure.
Resampler Architecture
The full resampler consists of:
Learned latents: m 32 learned query vectors initialized from (o d=0.5)
Input projection: Linear layer mapping from DINOv2 embedding dimension (1024) to internal dimension d
Attention layers: L = 8 layers of Perceiver attention followed by feedforward networks with residual connections
Output projection: Final linear projection match UNet's cross-attention dimension
The resampler processes the 1024 DINOv2 patch embeddings (whether it is via high-resolution fixations or low-resolution peripheral images) and outputs exactly 32 conditioning tokens regardless of input length.
A.4 STEPWISE REGRESSION MODEL DETAILS
We performed forward stepwise linear regression analysis to measure the extent to which human judgments could be explained by feature differences in our primary behavioral experiment. The resulting linear model had an Rº value of 0.039, representing small but meaningful effect size (in psychological terms). This model incorporated variables, and we evaluated their importance the model by comparing the full linear model model omitting each of them and reporting the change in Ri2 for each. In descending importance, these variables were: DreamSim distance (AR2 0.10), vertical Gabor intensity (AR² 0.006), predicted depth map RMSE (AR² 0.003), D3 (Percentage of pixels with depth error < 1.25 threshold) (AR = 0.003), mid- level blur-trained CNN feature similarity (AR² 0.002), CLIP feature similarity (last hidden layer) (AR² = 0.001), CLIP image similarity (CLS) (AR² 0.001), and D0.25 (Percentage of pixels with depth error < 1.250.25 threshold) R 0.001). These results highlight that human scene similarity judgments depend on independent features distributed across the levels of visual processing, and indeed the three most important features in this regression included low-level, mid- level, and high-level measures.
For comparison, we also ran a stepwise regression on the generations conditioned on random fixa- tions. The resulting linear model had an R² value of 0.031, meaning that in spite of the generations' variability, we were able to begin explaining scene judgments in this case. However, consistent with earlier findings that these generations differed from the original image in such unpredictable ways that interpretable predictors were no longer significant, this regression only found significant regressors: DreamSim distance RAR2~ 0.016) and 135 Gabor intensity RAR2 0.014).
17

---

Published as a conference paper at ICLR 2026
A.5
ADDITIONAL DRIVERS METAMERIC JUDGMENT
A.5.1
LOW-LEVEL TEXTURE FEATURES
Gabor filter response differences
1.0
1.0
1.0
0.8
0.8
0.8
Proportion "aaeer
Proportion "same""
0.6
0.6
0.6
Proportion
0.4
0.4
0.2
0.0
0.0
0.000 0.005 0.010
0,000
0,005
0. 1.000
0.005
Gebor (6 degrnes) difference
Gabor (45 degrees) degreee difference
Gabor (135 degrenc) diffrence
increeoed reeparaee magnituuee deferreccess
Figure 7: Stronger Gabor texture responses than originals coincided with greater proportions of metameric judgments. This suggests that enhanced texture definition, like enhanced edge infor- mation, contributes to the perceived realism of generated metamers across multiple spatial frequen- cies and orientations.
As discussed in Section 5.2, we found that greater oriented edge contrast the generated image (as measured by Gabor filters) predicts more "same" judgments. Figure 7 presents the results of this analysis visually, demonstrating the effect for 0%, 45, and 135 filters. (The effect was not significant for the 90'horizontal filter.) The right side of the figure visualizes specifie generated images with high or low filter responses at each orientation.
A.5.2 MULTI-OBJECT RECOGNITION & LOCALIZATION
multi-object recognitton + localization
P-000 F-0
F-000 000
Proportin "samee
0.4
0.4
0.4
Proportion
Proportion
0.25
0.2
0.4
0.8
0.0
(60%)
(70%)
mAP (80%)
0.6
0.6
px2.00
Propurton "uama""
Proportinn "uamea"
0.4
0.4
0.4
Proportinn
0.2
0.2
Object Det. (Precision)
Object Det, (Recall)
Object Det. (PI)
Figure 8: Object detection errors predict metameric perception: (Top) mAP scores demon- strate that higher precision accuracies (from mAP 60% to mAP 80%) with better alignment at strict localization boundaries correlate with increased "same" metameric judgments. (Bottom) Object de- tection metrics show positive relationship where improvements in model precision, recall, and F1 scores correspond to increased "same" metameric judgments.
18

---

Published as a conference paper at ICLR 2026
To analyze object-level scene understanding, employed YOLOv8 (Jocher al., 2023) extract object detection bounding boxes and class predictions from both original and generated images. Our pipeline compared object inventories between image pairs, quantifying detection errors across mul- tiple metrics: precision (avoiding extra objects), recall (retaining original objects), and localization accuracy measured by Average Precision (mAP) different IoU thresholds.
Analysis revealed that object-level localization inconsistencies systematically impacted metameric perception (Figure 8 (Left). Localization accuracy showed consistent relationships with metameric perception. As we required increasingly precise object positioning (shown by increasing mAP thresholds), then the gap between human- auman-guided and random fixation conditions systematically widened. This suggests that extremely precise spatial localization becomes increasingly critical for metameric judgments, and that can best be exemplified using human fixations.
A.6 ADDITIONAL GENERATION VISUALIZATIONS BASED ON FIXATED INPUTS
"SAME" judgment metamers
"DIFFERENT" judgment
Figure 9: Additional metameric vs, non-metameric judgment example images based on human fixations. (Left) Original images with human fixations overlaid in red and corresponding generated images judged as "same" by participants. (Right) Original images with fixations and generated images judged as "different" by participants.
19

---

Published as a conference paper at ICLR 2026
"SAME" judgment - metamers
"DIFFERENT" judgment
AAA
m
CAB
Figure 10: Additional metameric vs. non-metameric judgment example images based on randomly-sampled fixations. (Left) Original images with randomly-sampled fixations overlaid red and corresponding generated images judged as "same" by participants. (Right) Original im- ages with fixations and generated images judged as "different" by participants.
20

---

Published as a conference paper at ICLR 2026
A.7 EFFECT OF PERIPHERAL BLUR AND FOVEAL TOKENS ON IMAGE GENERATION QUALITY
We ran two computational ablation studies to measure how image generation quality on the COCO- 10k-test is affected by (1) peripheral blur level and (2) foveal token count.
For the study of blur levels (Figure 11, Top), we provided peripheral features as the sole input (at varying levels of downsampling) and masked the foveal tokens. We found that greater downsampling blur yields lower CLIP similarities and higher DreamSim distances, as well as higher (worse) FID, though downsampling 0.25x to 112 112 seems have little effect quality compared keeping the original 448 x 448 image, perhaps due to the limited capacity of the DINOv2 embedding and the stochasticity in sampling. Empirically, we find that this introduced just enough uncertainty in the reconstruction that it still resembles the original (CLIP similarity ~ 0.888 without maintaining fine details. This is the blur level we choose in our human experiments.
For the study of foveal token count (Figure 11, Bottom), we instead varied the number of randomly- positioned foveal tokens, and provided them as the sole input to MetamerGen, masking the periph- eral tokens entirely. We found that increasing number of foveal tokens improves CLIP, DreamSim, and FID scores, which nevertheless remain worse than the scores reconstructions attained by peripheral features only. This highlights the importance of the peripheral conditioning, which behaviorally confirmed Section 5.3 and Appendix A.9.
CUIP cosine similarity
0.90
DreamSim distance
0.40
0.35
2.80
10
3.70
0.15
20h28
16-36
112-112
224-224
448-448
28-28
96-06
12412
224-2M
418-648
20-28
50:50
112-122
224--22
445-148
Downsampling blur
Downsampling
Downsampling level
0.85
CLIP cosine similarity
DreamSim distance
27.2
0.80
0.00
0.55
26.8
20.6
0.50
0.70
0.45
26.2
2
foveal tokens
foveal tokens
foveal tokens
Figure 11: Influence of blur-level and foveal token count on image generation quality. (Top Row) Image generation quality decreases as greater blur degrades the base image. (Bottom Row) Image generation quality increases function of increasing foveal tokens.
A.8 SEMANTIC SIMILARITY IS MORE IMPORTANT THAN PHYSICAL DISTANCE
While MetamerGen was trained denoise the original image from sparse visual inputs, we found that it never precisely recreated the original images, even when presented with all patch tokens. More importantly, we found that pixel-level similarities between the generated and original images had no effect whether an image was judged metamer. Instead, this judgment was predominantly driven by high-level semantic similarities between the generated and original images.
This result indicates that observers rely primarily conceptual and semantic understanding of the scene rather than on low-level pixel features when making metameric judgments. In Figure 12(A), found that the pixel-level similarities measured by PSNR did not predict whether image would be a metamer; Figure 12(B), we establish that a SONNR-reamSing plot, low DreamSim distances predict "same" judgments, but high PSNR values do not, with examples of each included.
21

---

Published as a conference paper at ICLR 2026
We conclude that if a researcher wishes to titrate the rate of similarity judgments, they should do so by selecting images based on DreamSim scores, not physical stimulus distance.
54
original
generated
2200
Sorre
Dificrent
B fierent
3.75
DreamSim distance
3150
Density
0.125
0.3
pinc siiiaity, Somartt similiity
0100
3.03
3,50
3/25
3.000
10.0
12.5
15.0
17.5
20
PSNR (dB)
PSNR (dB)
original
generated
Figure 12: Comparison of pixel-level (PSNR) and semantic (DreamSim) similarities in metameric judgments. (A, Left) portrays the histograms of PSNR values for generated images judged as "same" "different" in the behavioral task, with nearly identical distributions between the two groups. (B, Right) shows the relationship between PSNR and DreamSim distances for all image pairs. There clear vertical separation by DreamSim distance that corresponds with metameric judgments, while PSNR values do not discriminate between what is considered metameric.
A.9 FOVEAL AND PERIPHERAL TOKEN CONTRIBUTIONS TO METAMERIC JUDGMENTS
We present further statistics and analyses of the behavioral ablation experiment presented Sec- tion 5.3. Figure 13 provides qualitative examples of generations from each treatment that were judged "same" and "different", together with a plot of the fool rate each condition. The gener- ations in the full-moda condition are both qualitatively the highest-quality and quantitatively the most likely to fool participants.
One subtlety that emerged during this experiment was that, despite the same participant population and an identical model, participants were substantially more likely to judge images generated by MetamerGen "same" (54.5%) than in the primary experiment (29.4%). Our interpretation of this unexpected pattern of results is that the foveal-only condition, which was generally easy for participants distinguish from the original image, acted low anchor on image similarity, thus lowering participants' threshold for making a same judgment. Because this increases the amount of variability in judgments that can be explained (see below), we see this as 'feature' rather than "bug".
Figure presents our exhaustive replication of the multi-scale feature correlation analysis from the primary experiment under each condition of the ablation experiment. The main summarizes the two conclusions following from this analysis, which we explain in more detail here.
First, under nearly all metrics, full-model (foveal peripheral) generations fool participants at greater rate than peripheral-only generation, even at the same metric scores. This is evidenced by the clear separation between the green and orange lines in most Fig 14 panels. The only features where this gap was not apparent were late-layer cosine similarity and DreamSim distance, suggesting that these metrics may capture large proportion of the factors that caused humans judge full-model generations as "same" more than peripheral-on/) generations.
Second, participant judgments full-model generations more explainable than judgments peripheral-on., generations. Along nearly all feature axes (apart from early-layer cosine similar- ity and mid-level proto-object segmentation mloU), judgments were more strongly correlated with
22

---

Published as a conference paper at ICLR 2026
Praportion "same"
0.543
Input visual stream
original
generated
original
generated
original
generated
Ruel
using foveal peripheral features
using peripheral features only
using foveal features only
Figure 13: Higher metameric (same) judgments for images incorporating both peripheral and foveal information. Using both foveal and peripheral features produced the highest fooling rates. Peripheral-only conditioning yielded the second best results, while foveal-only generations lagged significantly behind. Although the difference between peripheral-only and combined foveal- peripheral conditioning is small, meaningful: the additional high-resolution details from fixa- tions lead participants to be more easily fooled.
features under the foveal + peripheral condition. (Under some metrics, they were actually more correlated than they were during the main experiment, thanks to the increased meaningful variance given this set of conditions.) Foveal-only generations were even more poorly explained by these metrics. In fact, the only metric which could explain judgments of foveal-only generations sta- tistical significance was DreamSim, indicating that these generations, which lacked the gross scene structure and layout provided from the periphery, were far from the original image that ordinarily important feature axes did not influence judgments.
23

---

Published as a conference paper at ICLR 2026
003
PHO.054 TH0000 p.0.0
0.6
0.6
0.6
Proportion "same"
Proportion "same"
Proportion "same"
0.4
0.4
0.4
0.
0.2
0.2
0.0
0.0
0.0
Early-layer cosine similarity
Mid-layer cosine similarity
Late-layer cosine similarity
0.7
n--004 p-0
poo 1902
0.7
--..0, prooren 0-0012 1w p=8677
foveal + periphoral
Proportion "same"
Proportion "same"
0.5
0.5-
peripheral only
foveal only
0.3-
0.3-
0.1
0.1
0.4
0.6
0.8
1.0
1.2
1.4
0.2
0.3
0.4
0.5
Depth Estimation (SiLog)
Mid-level segmentation mloU
p-0.00009
mo.t p-0.00237
rt 0.17, p77.
nid al pioc.0
ri-4 16, 441
Proportion "same"
Proportion "same"
0.50
CLIP cosine similarity
DreamSim distance
Figure 14: Impact of input visual streams hierarchical feature analyses. (Top Row) Multi- level feature analysis using neurally-grounde model (Jang & Tong, 2024) on driving metameric judgments. (Middle Row) Mid-level visual features driving metamerie judgments (mid-level sege- mention mloU and SiLog depth estimation). (Bottom Row) High-level visual features driving metameric judgments (CLIP cosine similarity and DreamSim distance).
24

---

Published as a conference paper at ICLR 2026
A.10 DINOV2 VERSUS CLIP AS THE VISION ENCODER OF MetamerGen
Previous adapter-based approaches, like IP-Adapter Ye et al. (2023), have utilized CLIP embed- dings image conditioning inputs for Stable Diffusion. We choose DINOv2 the visual encoder for foveal and peripheral feature extraction because DINOv2 patch tokens have been shown to better encode both local and contextual information. This contextual encoding emerges from DINOv2's self-supervised training objectives: its reconstruction loss encourages patches to redundantly en- code information about their surroundings, while contrastive loss causes semantically related patches to have similar embeddings (via object and scene structure) (Barsellotti et al., 2025; Adeli et al., 2023). This means a single DINOv2 patch token naturally captures both foveal detail (local information) and parafoveal context (relationships to nearby regions) precisely the type of repre- sentation needed for modeling human fixations.
On the other hand, vision-language models like CLIP optimize for global image-text alignment, which limits their patch-level spatial selectivity and the spatial relationships modeled in their deep layers (Wang et al., 2024; Li et al., 2025). (CNN-based encoders lack emergent representations of patch context - ie, parafoveal information - entirely.)
To empirically validate our choice of DINOv2, retrained MetamerGen, using CLIP vision encoder, for 100K steps. Using the -LIP-conditioned model, we conducted two image generation ablations using COCO-10K-test images: (1) varying the blur levels for peripheral-only image generation, and (2) varying the number of foveal tokens for a foveal-only image generation. This ablation directly comparable to the DINOv2 ablation presented in Appendix A.7.
We observed that CLIP features perform significantly worse at encoding peripheral information, re- flected by the higher FID values obtained in Figure 15 (left), and Figure 16. In contrast to DINOv2 tokens, the FID value does not show significant decrease when we reduce the blur level, mean- ing that CLIP encodes similar (impoverished) information in its patch tokens for the blurred and non-blurred images, focusing on global scene category, at the expense of more fine-grained scene structure. This is consistent with previous observations made regarding CLIP's relatively low ability to encode contextual information with its patch tokens (Li et al., 2025). For foveal information, increasing the number of tokens has a similar effect for both encoders, though single DINOv2 token seems to encode more information than a single CLIP token. Overall, our choice of DINOv2 is mainly motivated by its ability to accurately encode both the peripheral and foveal information present in the image.
22
20
DreamSim distance distance
0.65
18
DINOV2-Base
0.60
DINO22
16-
0.55
12-
0.45
10
0.40
28.28
Mix56
12N-12
204x204
448-448
Downsampling blur level
foveal tokens
Figure 15: FID and DreamSim evaluations based on DINOy2 and CLIP as vision encoders for foveal and peripheral feature extraction. (Left) The image generation quality (FID) for DINOv2- based peripheral generations consistently better than CLIP patch embeddings. For DINOv2, we observe sharp drop when decreasing the blur level, showing how decreasing blur results in the model encoding different, more accurate image features. This is not true for CLIP patch tokens, which seem to encode the same limited information across all blur levels. (Right) With increasing numbers of foveal token inputs, the DreamSim distance for both DINOv2 and CLiP--bsed embed- dings decreases. However, DINOv2-based generations yield greater semantic similarities with the original images, especially at low token counts.
25

---

Published as a conference paper at ICLR 2026
Downsampling
28-28
S6x56
112-112
224+224
445.448
Original
DINOV2
CLIP
Original
DINOV2
CLIP
Figure 16: Image generation examples across blur levels using DINOv2 and CLIP as vision encoders. DINOv2-based peripheral generations resemble the original images more than CLIP- based generations, even at low blur levels. As the rate of downsampling decreases (28 x 28 > 448 448), DINOv2-based generations continue to show substantial improvements while CLIP- based generations exhibit minimal improvements. For the bottom two pairs, DINOv2-based generations are able to keep the size of the plane (as well as its spatial position) intact irregardless of blur level input. However, that is not the case for the CLIP-basee image generations.
26


Reviews:
[{"review_id": "imY4QUheBS", "reviewer": "Reviewer_jZsD", "summary": "This paper introduces MetamerGen, a computational model designed to generate visual \"metamers\"—images that are physically different from an original but are perceived as the same by a human observer. The model, based on a latent diffusion architecture, is conditioned using a dual-stream input that mimics human vision: sparse, high-resolution features from specific fixation points (foveal vision) and blurred, low-resolution features from the entire scene (peripheral vision). This paper utilized a latent diffusion transformer to reveal how humans perceive visual information.\n\nTo validate the model, the authors conducted a real-time behavioral experiment where a participant's eye movements were tracked while viewing an image. These fixation data were then used to generate a new image with MetamerGen, which the participant had to judge as \"same\" or \"different\" from the original. The study's analysis reveals that high-level semantic similarity between the original and the generated image is the strongest predictor for a \"same\" judgment.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduced a method to understand how humans process visual information, and what kind of information/feature matters in visual perception.\n\nThis paper designs a reasonable experimental setting to interpret human scene understanding by tracking eye movement and leveraging the movement to generate \"fake\" images using LDM, and asks the tester to justify the fake images.\n\nThe findings provide strong evidence that human memory and understanding of a scene rely heavily on high-level semantic information (e.g., object identity and spatial relationships) rather than on precise, pixel-level details.", "weaknesses": "This paper only considers a fixed-resolution scenario. It would be interesting to investigate whether varying aspect ratio has an impact on human scene understanding. For example, give a larger resolution version of the original image or give a 9:16 image (the original is 1:1), which adds or removes some side content and maintains the center content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RVK1cpf9Ef", "reviewer": "Reviewer_KkGc", "summary": "The paper is about a very interesting neuroAI study. The problem is to understand when humans consider two pictures to be the same. The authors train a DiT model to generate a different picture (metamer) from an original picture that can fool human subjects into considering they are the same. The DiT is fine-tuned from a pretrained model, adding cross-attention conditioning on the original picture’s features. The features are obtained from DINO. The authors do several in-depth analyses on comparing behavior patterns between different conditions, and between human and CNN and CLIP. The authors also identified that it is the high-level feature that contributes to the metamer perception. Overall, the study provides a novel paradigm to study human scene understanding and shows mechanistic similarity between human scene perception and DNN scene perception.", "questions": "**Physical distance between metamer and original image**\n\nThe training and inference process does not regularize the distance between the generated image and the original image. Is it possible their training process learn to recover the original image based on fixation and peripheral condition. So, the generated images are for sure metamers because they are the same or very similar to the original image? Given that the generated images are not always metamers, is there a way to regularize, such that by controlling the distance in some metric space, the generated images are more likely to be metamers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, this is a solid study with rigorous human subject experiments and valid deep learning algorithms. It provides a novel paradigm to study human visual scene perception. The analysis provides insights into the similarity between DNN and human scene perception, and which level of features is critical.", "weaknesses": "**Insufficient background on human visual scene perception**\n\nConsidering the paper is submitted to a machine learning venue, neuroscience/psychology jargon should be explained. Terms like \"metamer\" should be defined clearly early on, and traditional study paradigms on them should be introduced.\n\n**Figure 1**\n\nInconsistent color in Figure 1. in the top part, the fixation is colored blue, but in the bottom part, it is red.\n\n**Structure**\n\nImportant results, such as line 478-479 \"While MetamerGen was trained to predict images from randomly sampled locations—and\nsuch generations fooled participants at a rate comparable to those conditioned on their own fixations\n(27.7% vs. 29.4%)\" is hidden in the discussion, rather than in the result section. This is important information for knowing how effective the generated metamers are under different conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXRoGe0oVO", "reviewer": "Reviewer_y8G9", "summary": "The paper introduces MetamerGen, an image-to-image framework that synthesizes scene understanding metamers for a viewer’s internal scene representation by conditioning a Stable Diffusion backbone on (i) DINOv2 patch tokens extracted from fixated (high-res) regions and (ii) DINOv2 tokens from a blurred peripheral image. The model is evaluated with a gaze-contingent same-/different behavioral paradigm that measures whether humans judge generated scene maetamers as the “same” scene after a brief re-presentation. The paper further analyzes which levels of visual features (low, mid, high) predict metameric judgments.", "questions": "There is a tradition of visual metamers that could be used as a baseline. There is the Freeman and Simoncelli (Nat Neuro) traditional methods, and also newer methods that were developed based on style transfer by Deze at al., 2019 (Neurips).\n In principle, the semantic methods should be better than the pure visual for some tasks (unsure if the one used by the authors). \nThis should be an important control.  \n\nThis leads to the next question.  Couldn’t we have two scenes that are clearly different under unlimited viewing, such that observers can easily tell them apart, yet they are still scene metamers because the differences are irrelevant to scene understanding? For example, consider two scenes that are visually distinct but convey the same high-level meaning: a white truck driving on the highway versus a black truck driving on the freeway. Although their low-level image statistics differ, both scenes represent the same event or concept. In that sense, metamerism could arise from equivalence in interpretation, not appearance, suggesting that scene metamers might exist even without any gaze or foveation constraints.\n\n\nAblations needed to strengthen the paper:\nNo-fovea baseline (peripheral-only)\nNo-periphery baseline (fovea-only)\nTests of DINOv2 suitability", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Formulation of a foveated image-to-image synthesis task that fuses sparse foveal tokens and blurry peripheral tokens into a single latent diffusion generator.  \n\nA real-time gaze-contingent same/different behavioral paradigm (45 participants) showing that many generated images are judged “same” (i.e., metameric) and that semantic/high-level alignment best predicts metamerism for human-fixation conditioning.  \n\nThe paper repurposes the concept of metamers (historically a low-level vision/color concept) as a latent generative modeling problem conditioned on gaze. This is a clever insight that links cognitive science, visual neuroscience, and modern diffusion modeling.\nThe “foveal + peripheral token fusion” formulation is conceptually neat\nUsing human fixation maps as structured conditioning signals is a creative way to connect psychophysical data with generative models, and to probe which representations are perceptually sufficient for “scene sameness.”\n\nThe method is technically up-to-date and grounded in state-of-the-art visual encoders (DINOv2) and latent diffusion architectures (Stable Diffusion).", "weaknesses": "Role / necessity of foveation is not fully ablated or isolated: \nThe paper reports that MetamerGen can generate metamers conditioned on random fixations as well as human fixations and reports similar overall fooling rates (27.7% vs 29.4%) but then emphasizes the scientific value of human fixations because they produce stronger correlations with feature hierarchies.  This raises two concerns.  If random fixations can produce similar fool rates, how much unique value does foveation / fixation conditioning add for producing perceptually aligned hypotheses (not just for fooling)?\nThere is no explicit ablation that removes the foveal stream entirely (i.e., peripheral only) or removes the peripheral stream (foveal only) and compares behavioral / feature alignment. A single-factor ablation is missing.\n\nSuitability of DINOv2 patch tokens as an explicit model of foveal or parafoveal signals.\nThe approach treats a DINOv2 patch token (~14x14 pixels at 448 input → 32x32 grid) as if it corresponds to the information a foveal fixation provides. But DINOv2 was trained on normal (non-gaze-conditioned) images; the authors acknowledge that patch tokens also encode some context. The paper does not show that DINOv2 patch tokens (or their masked/compressed form) are the best representation for foveal information vs alternatives (CLIP token, a CNN trained with foveated inputs, or even raw pixel crops).  \n\nPatch ↔ fovea geometric mapping appears simplistic.\nA fixation is mapped to the nearest single 14x14 patch token. The visual system’s foveal footprint (receptor sampling + acuity falloff) is more continuous and covers multiple patches; using a single patch may under/overestimate the foveal information.  The authors also need to justify the 14x14 path representing the fovea’s DVA. \n\nPossible confounds in stimuli & filtering\nThe stimulus set excluded faces/hands etc. to accommodate Stable Diffusions limitations (of course, reasonable), but this restricts generality to many real-world scenes.\n\nVisual vs. Semantic Metamers:\nTheir psychophysical validation is a same/different task.  Although the paper makes important claims about the semantic nature of the metamers, how are they different from the classic visual metamers?  Would the proposed semantic metamers result in lower ability of observers to tell them apart than using visual metamers?\nThis seems to be missing in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)