Conjuring Semantic Similarity
Authors: Tian Yu Liu
Stefano Soatto
Venue: ICLR 2026 Poster (2026)
Keywords: Meaning Representation
Semantic Similarity
Diffusion Model
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a method to quantify, visualize, and measure the alignment of, semantic representations learned by text-to-image diffusion models.

Abstract: The semantic similarity between sample expressions measures the distance between their latent `meaning'.These meanings are themselves typically represented by textual expressions.  We propose a novel approach whereby the semantic similarity among textual expressions is based not on other expressions they can be rephrased as, but rather based on the imagery they evoke. While this is not possible with humans, generative models allow us to easily visualize and compare generated images, or their distribution, evoked by a textual prompt. Therefore, we characterize the semantic similarity between two textual expressions simply as the distance between image distributions they induce, or 'conjure.' We show that by choosing the Jeffreys divergence between the reverse-time diffusion stochastic differential equations (SDEs) induced by each textual expression, this can be directly computed via Monte-Carlo sampling. Our method contributes a novel perspective on semantic similarity that not only aligns with human-annotated scores, but also opens up new avenues for the evaluation of text-conditioned generative models while offering better interpretability of their learnt representations.

Published as a conference paper at ICLR 2026
CONJURING SEMANTIC SIMILARITY
Tian Yu Liu
Stefano Soatto
Department of Computer Science University of California, Los Angeles i tianyu@cs. wwwpilsssslll.......... edu
Department of Computer Science
University of California, Los Angeles soattolcs ucla. edu
ABSTRACT
The semantic similarity between sample expressions measures the distance be- tween their latent 'meaning'. These meanings themselves typically repre- sented by textual expressions. We propose novel approach whereby the semantic similarity among textual expressions is based on other expressions they can be rephrased as, but rather based on the imagery they evoke. While this is not pos- sible with humans, generative models allow generated images, or their distribution, evoked by textual prompt. Therefore, we characterize the semantic similarity between two textual expressions simply as the distance between image distributions they induce, or "conjure." We show that by choosing the Jeffreys divergence between the reverse-time diffusion stochastic dif- ferential equations (SDEs) induced by each textual expression, this can be directly computed via Monte-Carle sampling. Our method contributes a novel perspective semantic similarity that not only aligns with human human-annotuan annotated scores, but also opens up new avenues for the evaluation text-conditioned generative models while offering better interpretability of their learnt representations.
easily visualize and compare
INTRODUCTION
Semantic similarity is about comparing data not directly, but based on their underlying 'concepts' or 'meanings'. Since meanings are most commonly expressed through natural language, various methods have attempted compute them in this space. Words have been often compared based on the occurrences of other words that surround them, and images have likewise been compared 'semantically' by using the text captions that describe them.
While measuring semantic similarity comes natural to humans who share significant knowledge and experience, defining semantic similarity for trained models non-trivial. For the class of Large Language Models (LLMs), Liu et al. (2023) defined the space of meanings for autoregressive models the distribution over model continuations for any given input sequence. Semantic alignment of these models with humans can be directly quantified by comparing the geometric and algebraic properties of this space for instance, measuring whether prompts semantically similar to humans are also closely embedded in this space of meanings.
In this work, we study the converse: Interpreting the learnt semantic space of image (rather than text) generation models, and how well they align with that of human annotators. Instead of comparing images by the captions that describe them, we compare textual expressions terms of the images they conjure. In other words, we propose an expanded notion of meaning that is purely "visually- grounded". This would be hard if not impossible for humans, since the process requires visualizing and comparing 'mental images' each individual can conceive, but it is straightforward for trained models.
We focus the class of text-conditioned diffusion models. Under our definition, the semantic similarity between two text passages for a model can simply be measured by the similarity of image distributions generated by the model, conditioned on those passages.
There is a technical nugget that needs to be developed for the method to be viable, which is how compare diffusions in the space of images. To address this, propose leverage the Jeffreys

---

Published as a conference paper at ICLR 2026
Denoising with "Snow Leugard"
Noisy sequence "Snow Leopard"
t=T
t=
Denolsing "Bengal Tiger"
Denoising with "Snow Leugard"
Noisy sequence "Bengal Tiger"
t=T
t=0
Denolsing with
"Bengal Tiger"
Figure 1: We illustrate the process of conjuring semantic similarity between textual expressions "Snow Leopard" and "Bengal Tiger". We denoise each sequence of noisy images (middle row of both halves of figure) with both prompts (top and bottom row of both halves of figure). Our method can be interpreted as taking the Euclidean distance between the resulting images sequences of noisy images of Figure) starting from a Gaussian prior (t 7). Observing cells highlighted red, we see that the model converts pictures of Snow Leopards into Bengal Tigers by changing their characteristic spotted coats into stripes, and adding striped textures to the animal's face (top half of Figure), and conversely converts Bengal Tigers into Snow Leopards by changing their characteristic stripes into spotted coats (bottom half of Figure). This enables interpretability of their semantic differences via changes in their evoked imageries.
two rows. The obtained with either of the two text expressions (top / bottom halves
Divergence between the stochastic differential equations (SDEs) that govern the flow of the diffusion model, which we will show to computable using a Monte-Carlo sampling approach.
Although our goal is primarily to introduce method to quantify and visualize the semantic rep- resentations learnt by image generation models, will show that our simple choice of distance already leads to results comparable to zero-shot approaches based on large language models. To validate our proposed definitions, we further conducted ablation studies on several components of our method, demonstrating robustness to the specific decoding algorithms used for multiple choices of diffusion models.
To summarize the contributions of this work, we propose approach for evaluating semantic sim- ilarity between text expressions that grounded in the space visual images. Our method has unique advantage over traditional language-based methods that, in addition to providing numerical score, it also provides a visual expression, or 'explanation', for comparison, enabling better inter- pretability of the leantt representations (Figure I]. Additionally, quantifying the alignment of semantic representations learnt by diffusion models compared to that of humans, which open up avenues for the evaluation text-conditioned diffusion models.
method is the first to enable
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Cext-Conditioned Image Generative Models. We begin by briefly surveying the literature on text-conditioned image generation. Goodfellow et al. (2014) proposed the Generative Adversarial
Network (GAN). a deep learning- cearning-based approach for image generation. [Mizaa & Osindero (2014) proposed method for conditioning GANs based on specified labels. Works such as VQ- VAE (Van Den Oord et af.) 2017) and VQ-VAR-2 (Razavi et al.; 2019) also built upon the foundational works of Variational Auto Encoders (VAEs) (Kingma, 2013) 2) to learn discrete representations used to generate high quality images, among other outputs, when paired with an autoregressive prior. DALL-E (Ramesh et al.) 2021) has also been developed as effective text-to-image generator leveraging autoregressive Transformers [Vaswani 2017).
Our work focuses on diffusion models (Sohl-Dickstein et al. 2015). which have achieved state- of-the-art results among modern image generation models. These models can be viewed from the perspective of score-based generative models (Song et al. (020b), which represent the distribution of
data via gradients, and are sampled from using langevin dynamics (Welling & Teh 2011). Diffusion
models have been trained by optimizing the variational bound data likelihood (Ho et al.. 2020)_
Modeled as stochastic differential equations (Song et al. 2020b). the same training objective has also been shown to enable maximum-likelihood training under specific weighing schemes (Song et al.) 2021 Diffusion models have also demonstrated strong results on conditional image generation tasks. Sick-sickstein et al. (2015) and [Song et al. (2020b) showed that gradients of a classifier can be used to condition a pre-trained diffusion model. Dhariwal & Nichol (202I) introduced Classifier- guidance, which achieved state-of-the-art results in image synthesis at the time it was released. Song et al. (020a) introduced Denoising Diffusion Implicit Models (DDIM), greatly accelerating the process of sampling from trained diffusion models as compared to DDPM (Ho et al. 2020). Karras et al. (2022) presented unified view of existing diffusion models from a practical standpoint, enabling develop improved sampling and training techniques to obtain greatly improved results.
Semantic Space of Generative Models. The Distributional Hypothesis (Harris 1954) forms the basis for statistical semantics, characterizing the meaning of linguistic items based on their usage distributions. This is also closely related to Wittgenstein'' use theory of meaning (Wittgenstein) 1953), often popularized as "meaning is use". Many methods have been developed in machine learn- ing and Natural Language Processing (NLP) literature compute these semantic spaces, including Werd2Vec (Mikolov et al.] 2013). In light of modern Large Language Models (LLMs), Liu et al. (023) defined the space of meanings for autoregressive models be the distribution over model continuations for any input sequence. This has been used to define notions of semantic distances and semantic containment between textual inputs. Achille et al. (2024) further defined conceptual simi- larity between images by projecting them into the space of distributions over complexxty-constrained captions, producing similarity scores that strongly correlate with human annotations. /Saato et al.) 2023) generalizes these definitions by considering meanings as equivalence classes, where parti- tions can be induced by either an external agent or the model itself. To evaluate generative distribu- tions, Pillutla et al. (2023) quantifies the gap between target and model distributions by estimating the area under divergence frontiers in an embedding space. Vector-based representations such as those obtained from CLIP Radford al. (2021), or general any sentence embedding model (De- vlin et al., 2018 Opitz& & Frank 2022) have also been defined specifically for the computation of semantic distances. Such representations, however, are often difficult to interpret.
In contrast to these works, Bender & Koller (2020) argues that training on language alone is insuf- ficient to capture semantics, which they argue requires a notion of "communicative intents" that are external to language. In this paper, we explore a notion of meaning that is grounded in the distribu- tion of evoked imageries, and present a simple algorithm to compute interpretable distances this space for the class of text-conditioned diffusion models.
Evaluation and Interpretation of Diffusson Models. Common metrics used to evaluate diffusion models are, among many others, the widely-used FID (Heusel et al.) 2017) score, Kernel Inception Distance (Binkowski et al. 2018). and the CLIP score (Hessel et al. 2021). Despite these choices, Stein et al. (2024) recently discovered that no existing metric used to evaluate diffusion models correlates strongly with human evaluations. While existing evaluations often focus on the quality and diversity of generations, method is the first to evaluate semantic alignment of the repre-
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Conjuring Semantic Similarity
Require: Diffusion model sø, Prompts y1, V2, Monte-Carlo steps &
Initialize d =0
for i 1... do
f Sample from initial distribution
.T...., 50 f Denoise ø7 conditioned on y/
TT. IT...... -o f Denoise ør conditioned on 1/2
d d + T
T ||so (+,, fly1) so fly2)
- d +
T 1= (T,, 411) So A/22)
end for return d/k
>> Return similarity score
sentations learnt by diffusion models. Several techniques have also been developed interpret the generation of diffusion models. Kwon et al. (2022); Park et al. (2023) edit the bottleneck representa- tions within the U-Net architecture of diffusion models for semantic image manipulation. (Gandikota
et al. (2023) identifies low-rank directions corresponding to various semantic concepts. and Li al. (2023) shows that diffusion models can be used for image classification. [Kong et al. (2023a b frame diffusion models using information theory, improving interpretability of their learnt semantic relations. Orthogonal to these works, our method enables visualizing the semantic relations be- tween textual prompts in natural language learnt by diffusion models via the distributions over their generated imageries.
3 METHOD
We will present a short preliminary on (conditional) diffusion models in Section 3.1 and derive our algorithm for computing semantic similarity Section 3.2
3.1 PRELIMINARY
Our derivations will leverage Song et al. (2020b)ss 's the SDE formulation of diffusion models when viewed from the lens of score-based generaive modeling. particular, a (forward) diffusion process {T{t)] can be modeled as the solution to the following SDE:
dæ- = (ax t)dt + g(()dw,
with drift coefficient f Rd x Rd and (scalar) diffusion coefficient g IR H R, where w, is standard Brownian motion. We constrain the timesteps such that f E [0, 7]. where x=() represents the distribution of "fully-denoised" images generated by the diffusion model. We also assume that by construction, the prior time T known and distributed according (T) π, where often I0.
Once trained, we can view text-conditioned diffusion models map se (2, tly) parameterized by 0 and conditioned on textual prompt E y that is used to approximate the score function (Song al. 2020b), where so(-ttly) _d + F2 and is the set of textual expressions. As such, each conditional model so (xx fly) defines riverse-time SDE given by:
dæ [f t) g(t)²so (x, tly)]dt + g(t)
(1)
where W, is the Brownian motion running backwards in time from T to 0. For ease of notation, we will denote no (0 t, y) [f t) g(t)²
3.2 CONSTRUCTION
Given two textual prompts y1 and 1/2, we obtain two separate diffusion SDEs in the space of images using eq. (T), which are given by
dæ = No( t, y1 yt + (t))
(2)
dx = (2022 v2) (())
(3)
4

---

Published as a conference paper at ICLR 2026
We assume the standard conditions for existence and uniqueness of their solutions, in particular for all E [0, we have E [t)]] I co, and for all y E y and I, E R there exists and (1, L,
constants C, such that I/ (z, i, y +l S C(1++|<>]le)
(z.
y) --oooo v)lla S
D||z z' 2. We further assume Novikov's Condition holds for all pairs y1, 1/2 E y, in particular we are guaranteed the following: E exp 2 So ( t, y2) ua (x, t, (, )1)
< 80.
Since our goal is to define semantic distance between textual prompts y1 and y/2 by comparing the distributions over images that they conjure, we can achieve this by computing discrepancy function between the respective SDEs that they induce. In particular, we will use the Jeffreys divergence, which is simply the symmetrized Kullback-Leibler (KL) divergence between two SDEs. In the following, we show how this divergence can be computed via a Monte-Carlo approach.
Denote the path measures associated with eq. 22 and eq. respectively to be P1 and P2. Then, the KL divergence between the two SDEs are defined via
dIP
DKL (K2(PP))P1) = -Ep, log
dIP
dP, is the Radon-Nikodym derivative. By Girsanov theorem Girsanov where JP, pute this derivative as the stochastic exponential given by
1960), we can com-
exp - g(t) (2o t, t1 ) pe(a, y2)) dwū,
1 l
g(t)2 (2 t,,, - (2. t, y2) 12dt
2
Next, paralleling the derivation in Song et al.
2021
Novikov's Condition guarantees that the
stochastic integrand term s(t) (pe az f, y1)
Ho E,,,,,,)) y2)) is a Martingale. As result, taking ex-
pectation under IP2 would reduce it to zero, leaving only the drift term - f wyey I1 (2r, I, v1)
no(2,,, y2) de... Consequently, the KL divergence between the two SDEs can be simplified as
DRL(P2|PP:) = Er, J s(t)2 1,o vn) - ns (2c, 1/2 dl
J/ (10 s0(2, t122))ladt
Ep,
We can symmetrize to form our desired distance function:
doure Vooro(V1.) 322 = Dxz(P2|(P:) + (KI(Pi
which, ignoring constants, can be written
dours (2) = E un-iii(( (xll)) (o [o(1) (m fuv1) sa 4u2)
where Pr(-l)) is the distribution of noisy images at timestep t. Similar to how losses at differ- ent timesteps are weighted uniformly in the training real-world diffusion models (e.g. Liimple proposed by Ho et al. (2020)), we set gt) to be constant, in particular g(t) 1, to simplify our algorithm such that it does not have to be tailored specifically each choice of scheduler.
We can compute this resulting semantic distance using Monte-Carlo by discretizing the timesteps (1....: T). In particular, this computed practice via sampling an initial noise vector x(T") ग, and denoising it with both yı and y2 to obtain sequence of samples 2, 's, and computing the difference in predictions [[, (1y)) se (x t/y2) at each denoising timestep. We describe
.......),
this process in Algorithm
EXPERIMENTS
We describe implementation details in Section
4.1 empirical validation for our definitions Sec-
tion 4.3] and ablations in Section 4.4
5

---

Published as a conference paper at ICLR 2026
4.1 IMPLEMENTATION DETAILS
We use Stable Diffusion v1.4 (Rombach et al. 2022), text-conditioned diffusion model, for all experiments. For sampling, we use classifier-frre guidance (Ho & Salimans 2022) with guidance scale of 7.5, and sample using the LMS Scheduler (Karras et al. 2022). We specity image sizes be 512 x 512, but note that Stable Diffusion v1.4 uses latent diffusion, as such model predictions are in practice of dimension 64 64. We compute the Euclidean distance directly in this space. However, for visualization experiments such as in Figure 1 we decode the predictions using the VAE before plotting them. We set T 10 in our experiments to ensure computational feasibility, after ablating over other choices in Table We perform all experiments with single RTX 4090 GPU, and each Monte- Carlo step takes around 2.0s to complete in our naive implementation.
As technicality, while model the denoising direction term specified in the reverse- time SDE in eq. (T) as the ((ext-conditioned) model output so( (-1y), the exact implementation varies depending on the rext-conditionigg method used. However, we note that in the case of classifier-free guidance, the resulting distance computed using the model output as sp s-(y)) is equivalent to that computed using cussifier-guidange directions up to proportionality.
4.2 BASELINES
While there exists comparable baselines for quantifying semantic similarity in text-conditioned diffusion models at the time writing, we present several derivatives of our method below which are directly comparable against:
Prediction at initial timestep: We compare the one-step predicted noise vector at the initial denois- ing timestep. This is defined as dinitial := Ex~= []|= ' (11) sg(2, y2)
Prediction at final timestep: We compare the predicted noise vector the final denoising timestep.
This is defined as drinal := Be- (oor))(((ee)))+ttt))) (po(a=)))))))) [[]s0(2>, 0/8r) sa(æ, 0l12)
directly compute the differ- ence between the images produced by the two different labels. This is defined as doupat == Ex>z (<(l))))))) va [1]] where v-l(lly) : R² + Rd represents the full reverse diffusion process from the noise prior time T to the output distribution at time 0.
Direct output comparisons: For the same initial condition,
KL--Dvergence: We evaluate the non-symmetrized version of our method, computed via the KL divergence between the SDEs obtained from different prompts: dours KL (111 V2)
(0=1) [[(()=|s=e<<,, fly1) se( 41v2)1
Using the same parameters as our proposed method, we implement all baselines via Monte-Carlo sampling.
4.3 EMPIRICAL VALIDATION
While our work defines a notion of semantic distance grounded in evoked imagery, the validity of this definition hinges on its use measure of similarity that aligns with humans'. In particular, we should expect our definition to produce measurements of similarity that agree often with human annotators (which can be viewed as "ground-truth"
To quantify this, we use the Semantic Textual Similarity (STS) (Agirre et al., 2012: 2013; 2014 2015 2016 Cer al.] 2017) and Sentences Involving Compositional Knowledge (SICK -) (Marelli et al. 2014) datasets, containing pairs of sentences each labelled by human annotators with a se- mantic similarity score ranging from 015. We then use our method compute the image-grounded similarity score, and measure their resulting Spearman Correlation with the annotations.
Interestingly, our experiments in Table [ show that our visually-grounded similarity scores exhibit significant degrees of correlation with that annotated by humans. While, expectedly, our method presently lags behind embedding models trained specifically for semantic comparison tasks, we essuall-grounder similarity scores can rival that produced by existing large language
show that models up to 33B in size. Since our work is the first to formalize and evaluate semantic alignment for this class of conditioned diffusion models, have also included results from baselines with which

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with zero-shot methods Semantic Textual Similarity benchmarks, evaluated via Spearman Correlation (higher is better). *; f: indicate results taken from Ni et al. (2021) Gao
al. 202T: Liu et 2023) respectively. Expectedly. our zero-shot approach does not perform as well as embedding models such as CLIP (Radford et al. 2021) and SimCSE-BERT (Gao et al.) 2021]. which are trained specifically for semantic comparison tasks. Nevertheless, semantic struc- tures extracted from lext-conditioned diffusion models (StableDiffusion) using our method are still well-aligned with human annotators, rivaling those extracted from autoregressive Large Language Models while outperforming encoder- seoodrr-based language models such as BERT (Devlin et al. 2018).
STS-B STS12 STS13 STS14 STSIS STSI6 SICK-R
Avg
Tontrastive-Trainedd Embedding Models
CLIP-VITL14* RRdfford al.
2021
65,5
67.7
68.5
58.0
67.1
73.6
68.6
67.0 + 4.3
IS-BERT" Zhang 2020
56.8
69.2
61.2
75.2
70.2
69.2
64.3
66.6
5.7
MECCSE-BERT' (Gao et al. 2021
68.4
82.4
74.4
80.9
78.6
76.9
72.2
76.3 + 4.6
Zero-Shot Encoder-based Models
BERT-CLS" (Devlin et al.
2018
16.5
20.2
30.0
20.1
36.9
38.1
42.6
29.2 9.6
BERT-mean' (Devlin et al. 2018
45.4
38.8
58.0
58.0
63.1
61.1
58.4
54.8 8.3
BERT Large-mean (Deviin
2018
47.0
27.7
55.8
44.5
51.7
61.9
53.9
48.9
10.2
RoBERTa Large-mean
2019
50.6
33.6
57.2
45.7
63.0
61.2
58.4
52.8
STS-Hnc-mean aaage
2021
56.3
28.0
52.6
41.4
61.3
63.6
59.5
51.8
11,9
STS-Ene-mean (1 (II Ni
2021
62.8
35.0
60.2
47.6
66.4
70.6
63.6
58.0
11.5
Autoregressive Models (Meanings as Trujectories
Liu et al.
2023)
GPT-2-
55.2
39.9
42.6
30.5
52.4
62.7
62,0
49.3
+ 11.2
GPT-2-XL
62.1
43.6
54.8
37.7
61.3
68.2
68.4
56.5
11.1
Falcon-7B#
67.7
56.3
66.5
53.0
67.4
75.5
73.5
65.7 7.7
65.7
LLaMA-13B
70.6
52.5
65.9
53.2
67.8
74.1
73.0
65.3 - 8.3
LLaMA-33B*
71.5
52.5
70.6
54.6
75.2
8.5
Text-Conditioned Diffusion Models (StableDiffusion)
Initial Timestep Prediction
55.8
46.7
53.4
47.2
54.3
57.9
56.0
$3.0 + 4.1
Final Timestep Prediction
64.9
41.6
56.4
51.0
65.2
60.2
58.9
56.9
7.7
Direct Output Comparison
57.0
44.7
45.6
43.3
58.5
56.2
53.5
51.3 + 6.0
Conjuring Semantic Similarity (KL-Div)
69.1
56.9
60.6
59.5
71.5
65.7
64.8
64.0 + 4.9
Conjuring Semantic Similarity
70.3
57.9
61.0
60.8
73.6
67.9
66.0
65.4 + 5.3
our approach is directly comparable against. Our method convincingly outperforms all baseline methods for evaluating semantic similarity in diffusion models.
We further remark that in most existing text-conditioned diffusion models, the representation struc- tures that can be captured by our method limited by those learnt by text-encoder models such as CLIP (Radford et al. 202T), since these encoders are often used to pre-process textual prompts. In light of this, the experiments also suggest that our method can be an effective metric quantify how well representation structures learnt by these text-encoders have been distilled to the resulting diffusion model. We explore this in the following section, where we study how faithfully learnt semantic relations between words transfer from text-encoders to the full diffusion model.
In Figure we also qualitatively evaluate our method on measuring semantic similarity between various words. From the resulting pairwise similarity matrices, we can observe that words are closer in terms of their common hypernym class tend to cluster together, showing that our method can effectively capture word taxonomies. For instance, nouns that describe types of dogs cluster together, and nouns that describe types marine animals similarly form another cluster. These two clusters separate, in the sense that distances of words across these clusters are large than those within each cluster.
4.4 EMPIRICAL ANALYSIS
In this section, we run ablation studies using the STS-B dataset as benchmark to explore the design space of our method, and improve its computational efficiency. We further analyze the failure modes

---

Published as a conference paper at ICLR 2026
445
F25
831
333
227
0.26
93
114
634
6:1
115
1:15
2:4
0.10
3.22
C16
E31
8
0.20
2.2
5:44
8
12
#
255
9.25
22
Figure 2: Qualitative evaluation of conjured semantic similarity, visualized using pairwise distance matrices and hierarchial clustering. (Left) shows that nouns cluster based on shared hypernym classes: Dogs (puppy. poodle, dalmatian, pug) form a visible cluster in the nottom-right 4x4 block, while marine animals (whale, shark, dolphin, sealion) form another cluster in the top-left 4x4 block. (Right) shows that the same pattern holds for flying-related action verbs (elevate, ascend, soar, glide) v.8. negative stative verbs (disappoint, grieve, worry, regret).
of diffusion models through comparing semantic alignment of words belonging to different parts of speech.
70.0
Ssarman serdetton
67.5
b-2 loardaati
Speammn Comelction
60
65.0
1xd laa
02.5
60.0
3nd
30
57.5
4dg Postn
55.0
52.5
Timesstop
Stable Diffusion Medium (512
Figure 3: (Left:) We ablate over different choices of priors over timesteps uniform distribution over timesteps (T',.... T} where T' < T 10, represented by the blue line (cumulative), and the Direc Delta any particular timestep 7' {1, ... I'), represented by the orange line (pointwise). We show that a uniform prior over all timesteps gives the best results. The same plot also ablates over the number of Monte-Carlo samples, & E {I....,5] .5,, where conclude that only few iterations are required for convergence. (Right:) We further ablate over different choices of diffusion models, and show that results remain relatively consistent across the tested choices.
Prior over timestep distribution. In algorithm, placed uniform prior over timesteps (1....,T) T). In Figure 3] we show that this simple choice works best when evaluated on alignment with human annotators
STS-B, compared other choices such considering only uniform prior over the subset (T" (T'....,7] TT (cumulative from T' to T') where T' < T, or a Direc delta on any particular timestep I" [1, [L....,T] T' (pointwise).
Number of Monte-Carla Steps. The computational feasibility of our method depends the number of Monte- Carlo steps (i.e. l in Algorithm (1) required to produce a reliable approximate of the desired distance. We ablate choices of T (1....,5} in Figure 3 and show that the
8

---

Published as a conference paper at ICLR 2026
deviation of scores when evaluated the STS-B B dataset is small (#0.77) across different choices of k. This finding is promising, as it implies that our method can be computationally efficient, requiring only small number of Monte- Carlo iterations to converge.
Choice of Stable Diffusion Model. On the right of Figure B we also show that our results are relatively consistent across several versions of Stable Diffusion models, including Stable Diffusion XL and Stable Diffusion 3 Medium.
Error Analysis. Here, we evaluate our method on word similarity datasets, categorized by the part of speech (POS) from which the words originate. We use RG65 (Rubenstein & Goodenough, 1963) and SimLex-999 (Hill et al. 2013) for our evaluation, which contains pairs words and their semantic similarity score annotated by humans. The former consists of nouns, while the latter includes adjectives, nouns, and verbs. Our analysis in Table compares the representations learnt by the text encoder of the diffusion model to that of the model itself (extracted via our approach). Interestingly, we observe that while the semantic relations between nouns are largely preserved, the semantic relations between verbs and adjectives tend to deteriorate after learning the reverse diffusion process for image generation.
We evaluate the spearman correlation between semantic similarity scores obtained via our method, and that labeled by human annotators. We use the text encoder bottleneck of the StableD- iffusion model as the paragon. While the semantic properties of nouns are largely preserved from learning the diffusion process, this comes at the expense of semantics of adjectives and verbs.
Table 2:
RG65
SimLex
SimLex (Adj) SimLex (Noun) SimLex (Verb)
Paragon StableDiffusion
77.5
34.2
40.6
42.8
10.3
60.2
20.5
34.0
30.0
-14.3
Table 3: Ablation choices for I on the STS-B dataset: We show that variance with respect to the choice of T is small, allowing semantic distances to be computed efficiently by using lower values.
= 5
T
10
T
15
T 20
T 30
T 50
Spearman Corr. 70.1
70.3
70.1
68.9
70.2
69.5
5 DISCUSSION AND LIMITATIONS
Our method has several limitations. Our work does not, nor is meant to, provide a superior, ceneral-purposs metric for computing semantic similarity (e.g. for applications such retrieval). Indeed, imageries might indeed not be sufficient to fully capture the meaning of certain expressions, such mathematical abstractions (like 'imaginary numbers') and metaphysical concepts (like 'con- science'). ). Furthermore, many modern diffusion models use pre-trained text-encoder to pre-process textual prompts. This means that representation structures obtained from the diffusion model outputs would be bottle-necked by those learnt by the text encoder. The meanings of textual expressions can often be ambiguous or open multiple (subjective) interpretations. Our method does not resolve this linguistic ambiguity, since generated images are conditioned on the same potentially ambiguous text they are meant to clarify. Instead, simply reveals which visual interpretation the model has associated with that expression.
Finally, computation costs also remain a key limitation of our method, since it requires several in- ference passes through the diffusion model to compute single semantic similarity score, mitigated only partially by the conclusions of our ablation study on the required number of iterations.
Nevertheless, our method is the first to show that textual representations can be meaningfully com- pared for diffusion models by "grounding" them in the space of conjured images. We introduce the first method for evaluating semantic alignment of text-conditioned diffusion models. In par- ticular, our work enables not only qualifying (via visual 'explanations', ), but also quantifying the alignment of this resulting semantic space with that of human annotators. Our method also enables

---

Published as a conference paper at ICLR 2026
fine-gri fine-grained analysis of the failure modes of existing diffusion models, pinpointing specifie areas where they align poorly with human annotators. Our general framework of conjuring semantic sim- ilarity is applicable to the broader class of image generative models, and we leave the exploration of applications beyond diffusion models to future work.
REFERENCES
Alessandro Achille, Greg Ver Steeg, Tian Yu Liu, Matthew Trager, Carson Klingenberg, and Stefano Soatto. Interpretable measures of conceptual similarity by complexity-oonstrainee descriptive auto-encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11062-11071 2024.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. Semeval-200 2012 task 6: A pilot on semantic textual similarity. In * SEM 2012: The First Joint Conference Lexical and Computa- tional ceanannicss-olumee 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385-393, 2012.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational se- mantics (* SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pp. 32 43, 2013.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th international workshop on semantic evalu- ation (SemEval 2014). pp. 81-91 2014.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, Wei- wei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, al. Semeval-2011 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In Proceedings of the 9th international workshop semantic evaluation (SemEval 2015), pp. 252-263, 2015.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17: San Diego, CA. Stroudsburg (PA); ACL; 2016. 497-511. ACL (Association for Computational Linguistics), 2016.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5185-5198, 2020.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 180::1800000000 01401, 2018
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarity- muutiiiiuulll and cross-lingual focused evaluation. arXiv preprint arXiv: 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Rohit Gandikota. Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311 12002, 2023.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv: 20xv:2104.0882/ 2021.
10

---

Published as a conference paper at ICLR 2026
Igor Vladimirovich Girsanov. On transforming certain class of stochastic processes by absolutely continuous substitution measures. Theory of Probability & Its Applications, 5(3):285-301, 1960.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Zellig S Harris. Distributional structure. Word, 10(2-)):14-112,, 1954.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999 Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41(4):665-695, 2015.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207.122 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 265777755-6677,, 2022.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theorrticc diffusion. arXiv preprint arXiv:2302.03799 2023a.
Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable diffusion via information decomposition. arXiv preprint 1rXiv:2310.07972, 2023b.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. arXiv preprint arXiv:2210. 100.:000000 2022.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your dif- fusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2206-2217, 2023.
Tian Yu Liu, Matthew Trager, Alessandro Achille, Pramuditha Perera, Luca Zancato, and Ste- fano Soatto. Meaning representations from trajectories in autoregressive models. arXiv preprint arXiv:2310.18.3 2023.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 11622 2019.
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pp. 1-8, 2014.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-3119, 2013.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 1784, 2014.
11

---

Published as a conference paper at ICLR 2026
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yin- fei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.0888 2021.
Juri Opitz and Anette Frank. Sbert studies meaning representations: Decomposing sentence embed- dings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Con- ference Natural Language Processing, pp. 625-638, 2022.
Yong-Hyun Park, Mingi Kwon, Junghyo Jo, and Youngjung Uh. Unsupervised discovery of seman- tic latent directions in diffusion models. arXiv preprint arXiv:2302.12466. 2023.
Krishna Pillutla, Lang Liu, John Thickstun, Sean Welleck, Swabha Swayamdipta, Rowan Zellers, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Mauve scores for generative models: Theory and practice. Journal of Machine Learning Research, 24(356): 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning. pp. 8821-8831. Pmlr, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
Herbert Rubenstein and John B Goodenough. Contextual correlates synonymy. Communications of the ACM, 8 8(1)):27-633, 1965.
Stefano Soatto, Paulo Tabuada, Pratik Chaudhari, and Tian Yu Liu. Taming Al Bots: Controllability of Neural States in Large Language Models, May 2023, URL http::/aaxxv..rrg/ 23C5 18449 arXiv:2305. 18449 [cs, eess].
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference on machine learn- ing. pp. 2256-226622222 2265. PMLR, 2015.
Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganena Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2026
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. Citeseer, 2011.
Ludwig Wittgenstein. Philosophical Investigations. Macmillan Publishing Company, edition, 1953.
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An unsupervised sentence embedding method by mutual information maximization, arXiv preprint arXiv:2009 /2061, 2020.
13

---

Published as a conference paper at ICLR 2026
A
ADDITIONAL VISUALIZATIONS
Denoising with "Merlion"
Nolsy Image Sequence
t=t
t=T
F=
t=0
Denoising with "Mermaid Lion"
Figure 4: "Merlion" vs "Mermaid Lion": While both prompts express compositions of the same of objects, the model associates different meanings with "Merlion" as opposed to "Mermaid Lion", where the former is associated to the mascot of Singapore, while the latter is a mermaid with hair resembling a lion's 's mane.
Denoising with "Bag of Chips"
Noisy Image Sequence Denoising with "Bag of Fries"
t=T
t=T [=0
t=0
Figure 5: "Bag of Chips" vs "Bag of Fries": The interpretation of "chips" depends on cultural background (US vs UK), but the interpretation of "fries" is relatively non-ambiguous. Interestingly, this observation can be visualized when computing semantic similarity with our method. We see that the left of the figure (second image column), the model attempts to convert a picture of chips (US) into fries by changing the rounded textures into sharper rectangular ones, when denoised with "Bag of Fries". On the other hand, pictures of fries still remain relatively identifiable as fries (fifth image column) when denoised using "Bag of Chips".
B ADDITIONAL EXPERIMENT DETAILS
We detail the datasets used below:
Semantic Textual Similarity (STS) (Agirre et al. 2012 2013 2014 2015 2016 Cer et al. 2017):
Each of the STS datasets provides pairs of sentences, and human-labele score between 0-5 based on how semantic similar each pair is. We use the validation splits in STS-B. B, containing 1.5K exam- ple pairs, and the test splits in STS-12 to STS-16, containing 3.11K, 1.5K. 3.75K, 3K, and 1.19K example pairs respectively. We use the Spearman coefficient (scaled by 100x) to evaluate correla- tion with these human-annotated similarity scores.
14

---

Published as a conference paper at ICLR 2026
Sentences Involving Compositional Knowledge (SICK-R) (Marelli al. 2014): : The SICK-R dataset contains 9.93K pairs of examples which involve compositional knowledge, and similarly provides human-labeled semantic similarity score for each pair. Below contains example:
Sentence A: A man with jersey is dunking the ball at a baskethall game
Sentence B: The ball being dunked by man with jersey at a basketball game Relatedness Score: 4.9
RG65 Rubenstein & Goodenough 1965 contains human similarity ratings for 65 English noun pairs.
fimmLe-999 er al, 2015) contains 999 different concrete and abstract pairs of adjectives, nouns, and verbs. Each pair also scored by annotators based on their semantic similarity.
15


Reviews:
[{"review_id": "8XbFyGMMQz", "reviewer": "Reviewer_g9v9", "summary": "The authors propose to evaluate semantic similarity via the image that the text evokes, which is achieved by performing diffusions and then evaluating the JS divergence.", "questions": "1. In principle, what limits this approach to diffusions? Could flows, or any other conditional text to image generative model also work in principle?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: I find the core idea of using images to evaluate semantic similarity to be highly original and interesting. This is a major strength of the paper. \n\nClarity: the paper is written and motivated in a clear way that is broadly accessible to a general interest ML/AI audience. \n\nQuality: I find the mathematical/technical portion of the paper to be sufficiently in depth for ICLR. The experiments performed are rather comprehensive and present evidence that supports the claims of the paper and the proposed method. \n\nSignificance: The idea/topic of evaluating semantic similarity of generative models is timely and interesting. The main contribution of the authors approach is that it provides a new form of interpretability, which is sufficiently significant for ICLR.", "weaknesses": "My overall impression of the paper is largely positive, and I don't find any major weaknesses along quality/clarity/originality/significance in this paper. I raise some follow up questions in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I5moDzXNGM", "reviewer": "Reviewer_fA2Z", "summary": "The paper proposes a method for defining semantic similarity between text expressions, not through textual embeddings such as BERT or CLIP, but by comparing the image distributions generated by the texts when used as prompts in a text-conditioned diffusion model. Instead of operating in embedding space, the method treats the diffusion model’s denoising process as a stochastic differential equation and, for two prompts y1 and y2, computes the Jensen–Shannon divergence between their corresponding reverse-time SDEs, representing the model’s denoising trajectories. This divergence, estimated through Monte Carlo sampling, serves as a visually grounded measure of semantic distance, meaning that two texts are considered semantically similar if they produce comparable denoising trajectories or similar image distributions in the diffusion model.", "questions": "- L107 – Missing citation for the original GAN paper [1].\n- Clarification needed on STS datasets: The paper mentions several variants of the Semantic Textual Similarity (STS) benchmark (e.g., STS-B, STS-12, STS-13, etc.). Still, it remains unclear which specific datasets were used and how they differ in terms of setup or evaluation.\n- Possible extension: Could a variant of the proposed method be designed for image–image matching, similar to Maximum Mean Discrepancy (MMD)-based distance measures such as FID or KID?\n- Relation to existing perceptual metrics: Since the approach compares image distributions during the denoising process to estimate semantic similarity, is there a conceptual or mathematical connection to the Learned Perceptual Image Patch Similarity (LPIPS) metric [2]?\n\n[1] Goodfellow, Ian J., et al. *“Generative Adversarial Nets.”* *Advances in Neural Information Processing Systems* 27 (2014)\n\n[2] Zhang, Richard, et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper introduces a novel perspective on semantic similarity by grounding it in the visual space of diffusion models rather than the traditional textual or embedding space.\n- By formulating distances between diffusion trajectories using stochastic differential equations and the Jensen–Shannon divergence, the method establishes a sound mathematical foundation instead of depending on heuristic measures of similarity.\n- The proposed idea yields interpretable visual explanations showing how two prompts differ semantically through their generated imagery.", "weaknesses": "- One of the major shortcomings of the paper is that it does not clearly justify the need for its proposed methodology. Although the authors argue that contrastive-trained embedding models such as CLIP fail to capture textual semantic similarity fully, they neither provide concrete examples nor analyses demonstrating when or why these models fail nor explain how their approach remedies these limitations. Furthermore, it is not evident from the results or discussion that their method offers any improvement over existing embedding-based models, making the motivation for introducing a new framework less convincing.                  \n    - The proposed method also underperforms these contrastive models by roughly 10%, which further weakens the argument for its necessity. Moreover, there is no quantitative or qualitative evidence presented in the paper demonstrating situations where traditional methods break down, but the proposed technique succeeds. \n    - The authors’ statement in line 317, \"that large standard deviations imply that correlation with human annotators matters more than achieving state-of-the-art scores,\" raises additional concerns. If benchmark reliability is questioned, then the validity of their evaluation results becomes uncertain, making the overall empirical evidence for the method less convincing.\n- Missing technical details: A central component of the proposed approach is the Monte Carlo (MC) sampling procedure, yet the paper provides very limited discussion of it. The authors report using up to only five MC samples, which appears insufficient for accurately estimating an expectation. Moreover, in Figure 3, the reported Spearman correlation is higher for three samples than for five, which raises several important questions:\n     - How can the empirical mean be reliably estimated with as few as three samples? Wouldn’t such small sample sizes produce highly noisy estimates?\n     - Why does the correlation decrease when the number of samples increases to five? Five samples are still far too few to ensure convergence.\n    - Should a baseline or variance-reduction technique be introduced, as is common in Monte Carlo estimation?\n\nBecause these issues are never addressed in the paper, it becomes difficult to assess the soundness and stability of the proposed method.\n\n- (Minor) Ambiguity: Some parts of the paper are unclear or lack sufficient explanation. For example, the metric reported in Table 1 is not described in the caption, leaving readers unsure whether higher or lower values indicate better performance. Similarly, Figure 2 presents distance matrices without clarifying that smaller values correspond to higher semantic similarity, which makes the figure confusing at first glance and difficult to interpret without additional context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rNeHc6hLut", "reviewer": "Reviewer_zye1", "summary": "The paper proposes to measure the semantic similarity between text phrases by comparing the image trajectories produced by pre-trained text-to-image generative diffusion models. They propose doing so with a symmetrized KL divergence and shows how it compares to human-annotated scores and previous approaches in the literature.", "questions": "* How does one obtain the expression in line 234 for the KL divergence? It would be good to see a formal derivation, at least in the appendix. Unrelated side question: what other divergences can be written in this form?\n* It would be good to see some details about the data, e.g. domain, how many data points, etc. Also, some examples would be nice.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper gives a refreshing and novel approach to measure semantic similarity. The paper is well-written and was fun to read. Experiments are fairly well-done and convincing.", "weaknesses": "I have some minor concerns, which should be easy to address.\n\n* The paper notes that the proposed measure has a large standard deviation in line 318. I wonder if this can be reduced by averaging over multiple reverse SDE trajectories. Regardless, it would be good to see some experiments on this, as $p(\\text{image} | \\text{text})$ has some inherent variability. \n* I find it hard to interpret figure 2. It would easier to interpret based on the output of some clustering algorithm (e.g. hierarchical clustering) on this matrix to visualize the clusters.\n* It would be interesting to see how the proposed approach is better than computing the KL divergence between $p(\\cdot | y_1)$ and $p(\\cdot | y_2)$ directly in embedding space, similar to FID. Concretely, a suggested procedure is:\n     - sample a few images from $p(\\cdot | y_1)$ and $p(\\cdot | y_2)$\n     - get their embeddings from CLIP or a some other model and reduce their dimensionality with PCA (since we probably do not have too many samples)\n     - Calculate their means and covariances $(\\mu_1, \\Sigma_1), (\\mu_2, \\Sigma_2)$, and use the formula for KL between Gaussians $N(\\mu_1, \\Sigma_1)$ and $N(\\mu_2, \\Sigma_2)$. \n\nFor example, see the parametric approximation method of https://arxiv.org/pdf/2212.14578 (this citation is also probably worth adding in the related work). \n* The divergence in line 242, namely $1/2 \\big( KL(P_1 \\Vert P_2) + KL(P_2 \\Vert P_1)\\big)$, is known as the Jeffreys divergence. This is not the Jensen-Shannon divergence, which is defined as $1/2 \\big( KL(P_1 \\Vert M) + KL(P_2 \\Vert M)\\big)$ where $M = (P_1 + P_2)/2$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ZlcTrxBW4", "reviewer": "Reviewer_1bYa", "summary": "This paper introduces a novel, visually grounded approach to semantic similarity between textual expressions. Rather than defining similarity based on rephrasing or textual embeddings, the authors propose measuring the distance between the image distributions “conjured” by a text-to-image diffusion model when conditioned on different prompts. Formally, the method computes the Jensen–Shannon divergence between the reverse-time SDEs of two conditional diffusion processes, estimated via Monte Carlo sampling. The resulting metric — termed Conjuring Semantic Similarity (CSS) — enables interpretable visualization of semantic relations and quantifies the alignment between a diffusion model’s learned semantic space and that of human annotators.", "questions": "- Your experiments are limited to Stable Diffusion v1.4, which relies on an older CLIP text encoder. Given that SDXL, SD3, or Flux employ stronger text encoders (e.g., OpenCLIP-L or T5-XXL), how would you expect your Conjuring Semantic Similarity metric to behave when the underlying latent space better captures linguistic semantics? Would the observed moderate correlation (~0.65) improve, or do you think the limitation is intrinsic to visually grounded similarity?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- It suggests a fresh and intellectually stimulating inversion of the usual approach: comparing texts via images rather than images via text.\n- The use of JS divergence between SDEs is mathematically principled and connects diffusion dynamics with semantic distance.\n- The qualitative visualizations (e.g., “Snow Leopard” ↔ “Bengal Tiger”) concretely illustrate semantic drift in the image domain.", "weaknesses": "- Comparisons are limited to Stable Diffusion v1.4; modern models (e.g., SDXL, Flux, SD3 Large) or text encoders like T5/CLIP-L) are not systematically tested. I believe Stable Diffusion v1.4 cannot reflect the latent space sufficiently.\n- The correlations to human similarity scores, while encouraging, are moderate and inconsistent across datasets. Also, the Spearman correlation 0.65 is not enough for me.\n- The paper primarily showcases successful examples (e.g., “Snow Leopard” vs. “Bengal Tiger,” “Bag of Chips” vs. “Bag of Fries”) where semantic differences are clearly visually representable. However, it provides no concrete failure analyses demonstrating when the proposed Conjuring Semantic Similarity method breaks down (e.g., Abstract or non-visual concepts, Metaphorical or figurative expressions, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)