Conjuring Semantic Similarity

## Abstract

Abstract The semantic similarity between sample expressions measures the distance between their latent ‘meaning’. Such meanings are themselves typically represented by textual expressions, often insufficient to differentiate concepts at fine granularity. We propose a novel approach whereby the semantic similarity among textual expressions is based not on other expressions they can be rephrased as, but rather based on the imagery they evoke.
While this is not possible with humans, generative models allow us to easily visualize and compare generated images, or their distribution, evoked by a textual prompt. Therefore, we characterize the semantic similarity between two textual expressions simply as the distance between image distributions they induce, or ‘conjure.’ We show that by choosing the Jensen-Shannon divergence between the reverse-time diffusion stochastic differential equations (SDEs) induced by each textual expression, this can be directly computed via Monte-Carlo sampling. Our method contributes a novel perspective on semantic similarity that not only aligns with human-annotated scores, but also opens up new avenues for the evaluation of text-conditioned generative models while offering better interpretability of their learnt representations.

## 1 Introduction

Semantic similarity is about comparing data not directly, but based on their underlying ‘concepts’ or ‘meanings’. Since meanings are most commonly expressed through natural language, various methods have attempted to compute them in this space. Words have been often compared based on the occurrences of other words that surround them, and images have likewise been compared ‘semantically’ by using the text captions that describe them.

While measuring semantic similarity comes natural to humans who share significant knowledge and experience, defining semantic similarity for trained models is non-trivial since text can often be ambiguous or open to multiple (subjective) interpretations. On the other hand, visual elements often transcend communication barriers and allow for comparison at finer granularity. Comparing images is also relatively simple – unlike words, pixel values do not depend on distant knowledge or context.

Hence, instead of comparing images by the captions which describe them, we propose the converse: comparing textual expressions in terms of the images they conjure. In other words, we propose an expanded notion of meaning that is purely “visually-grounded”. This would be hard if not impossible for humans, since the process requires visualizing and comparing ‘mental images’ each individual can conceive, but it is straightforward for trained models.

We propose to leverage modern image generative models, in particular the class of text-conditioned diffusion models, for this purpose. By doing so, the semantic similarity between two text passages can simply be measured by the similarity of image distributions generated by a model conditioned on those passages.

There is a technical nugget that needs to be developed for the method to be viable, which is how to compare diffusions in the space of images. To address this, we propose to leverage the Jensen Shannon Divergence between the stochastic differential equations (SDEs) that govern the flow of the diffusion model, which we will show to be computable using a Monte-Carlo sampling approach.

We will show that our simple choice of distance already leads to results comparable to zero-shot approaches based on large language models. To validate our proposed definitions, we further run ablation studies over several components of our method, demonstrating robustness to specific choices of diffusion models and their inference algorithms.

To summarize the contributions of this work, we propose an approach for evaluating semantic similarity between text expressions that is grounded in the space of visual images. Our method has a unique advantage over traditional language-based methods that, in addition to providing a numerical score, it also provides a visual expression, or ‘explanation’, for comparison, enabling better interpretability of the learnt representations (Figure 1). Additionally, our method is the first to enable quantifying the alignment of semantic representations learnt by diffusion models compared to that of humans, which can open up new avenues for the evaluation text-conditioned diffusion models. Finally, we note that our approach can be used to derive many possible variants based on the metric used to compare images, which we leave for future exploration.

Figure: Figure 1: We illustrate the process of conjuring semantic similarity between textual expressions “Snow Leopard” and “Bengal Tiger”. We denoise each sequence of noisy images (middle row) with both prompts (top and bottom row). Our method can be interpreted as taking the Euclidean distance between the resulting images in the two rows. The sequences of noisy images are obtained with either of the two text expressions (left and right halves of Figure) starting from a Gaussian prior ($t=T$). Observing cells highlighted in red, we see that the model converts pictures of Snow Leopards into Bengal Tigers by changing their characteristic spotted coats into stripes, and adding striped textures to the animal’s face (left half of Figure), and conversely converts Bengal Tigers into Snow Leopards by changing their characteristic stripes into spotted coats (right half of Figure). This enables interpretability of their semantic differences via changes in their evoked imageries.
Refer to caption: x1.png

## 2 Related Works

#### Text-Conditioned Image Generative Models.

We begin by briefly surveying the literature on text-conditioned image generation.
Goodfellow et al. () proposed the Generative Adversarial Network (GAN), a deep learning-based approach for image generation.
Mirza & Osindero () proposed a method for conditioning GANs based on specified labels. Works such as VQ-VAE (Van Den Oord et al., ) and VQ-VAE-2 (Razavi et al., ) also built upon the foundational works of Variational Auto Encoders (VAEs) (Kingma, ) to learn discrete representations used to generate high quality images, among other outputs, when paired with an autoregressive prior. DALL-E (Ramesh et al., ) has also been developed as an effective text-to-image generator leveraging autoregressive Transformers (Vaswani, ).

Our work focuses on diffusion models (Sohl-Dickstein et al., ), which have achieved state-of-the-art results among modern image generation models. These models can be viewed from the perspective of score-based generative models (Song et al., ), which represent the distribution of data via gradients, and are sampled from using Langevin dynamics (Welling & Teh, ). Diffusion models have been trained by optimizing the variational bound on data likelihood (Ho et al., ). Modeled as stochastic differential equations (Song et al., ), the same training objective has also been shown to enable maximum-likelihood training under specific weighing schemes (Song et al., ).
Diffusion models have also demonstrated strong results on conditional image generation tasks. Sohl-Dickstein et al. () and Song et al. () showed that gradients of a classifier can be used to condition a pre-trained diffusion model. Dhariwal & Nichol () introduced Classifier-guidance, which achieved state-of-the-art results in image synthesis at the time it was released. Song et al. () introduced Denoising Diffusion Implicit Models (DDIM), greatly accelerating the process of sampling from trained diffusion models as compared to DDPM (Ho et al., ). Karras et al. () presented a unified view of existing diffusion models from a practical standpoint, enabling develop improved sampling and training techniques to obtain greatly improved results.

#### Semantic Space of Generative Models.

The Distributional Hypothesis (Harris, ) forms the basis for statistical semantics, characterizing the meaning of linguistic items based on their usage distributions. This is also closely related to Wittgenstein’s use theory of meaning (Wittgenstein, ), often popularized as “meaning is use”. Many methods have been developed in machine learning and Natural Language Processing (NLP) literature to compute these semantic spaces, including Word2Vec
(Mikolov et al., ). In light of modern Large Language Models (LLMs), Liu et al. () defined the space of meanings for autoregressive models to be the distribution over model continuations for any input sequence. This has been used to define notions of semantic distances and semantic containment between textual inputs. Achille et al. () further defined conceptual similarity between images by projecting them into the space of distributions over
complexity-constrained captions, producing similarity scores that strongly correlate with human annotations. Soatto et al. () generalizes these definitions by considering meanings as equivalence classes, where partitions can be induced by either an external agent or the model itself. Vector-based representations such as those obtained from CLIP Radford et al. (), or in general any sentence embedding model (Devlin et al., ; Opitz & Frank, ) have also been
defined specifically for the computation of semantic distances. Such representations, however, are often difficult to interpret.

In contrast to these works, Bender & Koller () argues that training on language alone is insufficient to capture semantics, which they argue requires a notion of “communicative intents” that are external to language. In this paper, we explore a notion of meaning that is grounded in the distribution of evoked imageries, and present a simple algorithm to compute interpretable distances in this space for the class of text-conditioned diffusion models.

#### Evaluation and Interpretation of Diffusion Models.

Common metrics used to evaluate diffusion models are, among many others, the widely-used FID (Heusel et al., ) score, Kernel Inception Distance (Bińkowski et al., ), and the CLIP score (Hessel et al., ). Despite these choices, Stein et al. () recently discovered that no existing metric used to evaluate diffusion models correlates strongly with human evaluations.
Several techniques have also been developed to interpret the generation of diffusion models. Kwon et al. (); Park et al. () edit the bottleneck representations within the U-Net architecture of diffusion models for semantic image manipulation. Gandikota et al. () identifies low-rank directions corresponding to various semantic concepts, and Li et al. () shows that diffusion models can be used for image classification. Kong et al. (, ) frame diffusion models using information theory, improving interpretability of their learnt semantic relations. Orthogonal to these works, our method enables visualizing the semantic relations between textual prompts in natural language learnt by diffusion models via the distributions over their generated imageries.

## 3 Method

We will present a short preliminary on (conditional) diffusion models in Section 3.1, and derive our algorithm for computing semantic similarity in Section 3.2.

### 3.1 Preliminary

Our derivations will leverage Song et al. ()’s the SDE formulation of diffusion models when viewed from the lens of score-based generaive modeling. In particular, a (forward) diffusion process $\{\bm{x}(t)\}_{t=0}^{T}$ can be modeled as the solution to the following SDE:

$$ $\displaystyle d\bm{x}=\bm{f}(\bm{x},t)dt+g(t)d\bm{w}_{t}$ $$

with drift coefficient $\bm{f}:\mathbb{R}^{d}\times\mathbb{R}\mapsto\mathbb{R}^{d}$ and (scalar) diffusion coefficient $g:\mathbb{R}\mapsto\mathbb{R}$, where $\bm{w}_{t}$ is standard Brownian motion. We constrain the timesteps $t$ such that $t\in[0,T]$, where $\bm{x}(0)$ represents the distribution of “fully-denoised” images generated by the diffusion model. We also assume that by construction, the prior at time $T$ is known and distributed according to $\bm{x}(T)\sim\pi$, where often $\pi=N(0,I)$.

Once trained, we can view text-conditioned diffusion models as a map $s_{\theta}(\bm{x},t|y)$ parameterized by $\theta$ and conditioned on a textual prompt $y\in\mathcal{Y}$ that is used to approximate the score function (Song et al., ), where $s_{\theta}(\cdot,t|y):\mathbb{R}^{d}\mapsto\mathbb{R}^{d}$ and $\mathcal{Y}$ is the set of textual expressions. As such, each conditional model $s_{\theta}(\bm{x},t|y)$ defines a reverse-time SDE given by:

$$ $\displaystyle d\bm{x}=[f(\bm{x},t)-g(t)^{2}s_{\theta}(\bm{x},t|y)]dt+g(t)d\bar {\bm{w}}_{t}$ (1) $$

where $\bar{\bm{w}}_{t}$ is the Brownian motion running backwards in time from $t=T$ to $t=0$. For ease of notation, we will denote $\mu_{\theta}(\bm{x},t,y):=[f(\bm{x},t)-g(t)^{2}s_{\theta}(\bm{x},t|y)]$.

### 3.2 Construction

Figure: Algorithm 1 Conjuring Semantic Similarity

Given two textual prompts $y_{1}$ and $y_{2}$, we obtain two separate diffusion SDEs in the space of images using eq. 1, which are given by

$$ $\displaystyle d\bm{x}_{1}=\mu_{\theta}(\bm{x}_{1},t,y_{1})dt+g(t)d\bar{\bm{w}} _{t}$ (2) $\displaystyle d\bm{x}_{2}=\mu_{\theta}(\bm{x}_{2},t,y_{2})dt+g(t)d\bar{\bm{w}} _{t}$ (3) $$

We assume the standard conditions for existence and uniqueness of their solutions, in particular for all $t\in[0,T]$ we have $\mathbb{E}\left[\|\bm{x}(t)\|_{2}^{2}\right]\leq\infty$, and for all $y\in\mathcal{Y}$ and $x,x^{\prime}\in\mathbb{R}^{d}$, there exists constants $C,D$ such that $\|\mu_{\theta}(x,t,y)\|_{2}+\|g(t)\|_{2}\leq C(1+\|x\|_{2})$ and $\|\mu_{\theta}(x,t,y)-\mu_{\theta}(x^{\prime},t,y)\|_{2}\leq D\|x-x^{\prime}\|
_{2}$. We further assume Novikov’s Condition holds for all pairs $y_{1},y_{2}\in\mathcal{Y}$, in particular we are guaranteed the following: $\mathbb{E}\left[\exp\left({\frac{1}{2}\int_{0}^{T}\|\mu_{\theta}(\bm{x},t,y_{2
})-\mu_{\theta}(\bm{x},t,y_{1})\|_{2}^{2}dt}\right)\right]<\infty$.

Since our goal is to define a semantic distance between textual prompts $y_{1}$ and $y_{2}$ by comparing the distributions over images that they conjure, we can achieve this by computing a discrepancy function between the respective SDEs that they induce. In particular, we will use the Jensen–Shannon (JS) divergence, which is simply the symmetrized Kullback–Leibler (KL) divergence between two SDEs. In the following, we show how this divergence can be computed via a Monte-Carlo approach.

Denote the path measures associated with eq. 2 and eq. 3 respectively to be $\mathbb{P}_{1}$ and $\mathbb{P}_{2}$. Then, the KL divergence between the two SDEs are defined via

$$ $\displaystyle D_{KL}(\mathbb{P}_{2}||\mathbb{P}_{1})=-\mathbb{E}_{\mathbb{P}_{ 2}}\log\left(\frac{d\mathbb{P}_{1}}{d\mathbb{P}_{2}}\right)$ $$

where $\frac{d\mathbb{P}_{1}}{d\mathbb{P}_{2}}$ is the Radon-Nikodym derivative.
By Girsanov theorem (Girsanov, ), we can compute this derivative as the stochastic exponential given by

$$ $\displaystyle\exp{\left(\int_{0}^{T}-\frac{1}{g(t)}\left(\mu_{\theta}(\bm{x},t ,y_{1})-\mu_{\theta}(\bm{x},t,y_{2})\right)d\bar{\bm{w}}_{t}-\frac{1}{2}\int_{ 0}^{T}\frac{1}{g(t)^{2}}\|\mu_{\theta}(\bm{x},t,y_{1})-\mu_{\theta}(\bm{x},t,y _{2})\|_{2}^{2}dt\right)}$ $$

Novikov’s Condition guarantees that this term is a Martingale so, following the approach of Song et al. (), the KL divergence between the two SDEs can be simplified as

$$ $\displaystyle D_{KL}(\mathbb{P}_{2}||\mathbb{P}_{1})$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\mathbb{P}_{2}}\left[\int_{0}^{T}\frac{1} {g(t)^{2}}\|\mu_{\theta}(\bm{x},t,y_{1})-\mu_{\theta}(\bm{x},t,y_{2})\|_{2}^{2 }dt\right]$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\mathbb{P}_{2}}\left[\int_{0}^{T}g(t)^{2} \|s_{\theta}(\bm{x},t|y_{1})-s_{\theta}(\bm{x},t|y_{2})\|_{2}^{2}dt\right]$ $$

We can symmetrize to form our desired distance function:

$$ $\displaystyle d_{ours}(y_{1},y_{2}):=JSM(\mathbb{P}_{1}||\mathbb{P}_{2})=\frac {1}{2}D_{KL}(\mathbb{P}_{2}||\mathbb{P}_{1})+\frac{1}{2}D_{KL}(\mathbb{P}_{1}| |\mathbb{P}_{2})$ $$

which, ignoring constants, can be written as

$$ $\displaystyle d_{ours}(y_{1},y_{2})$ $\displaystyle=\mathbb{E}_{t\sim\text{unif}([0,T]),\bm{x}\sim\frac{1}{2}p_{t}( \bm{x}|y_{1})+\frac{1}{2}p_{t}(\bm{x}|y_{2})}\left[g(t)^{2}\|s_{\theta}(\bm{x} ,t|y_{1})-s_{\theta}(\bm{x},t|y_{2})\|_{2}^{2}\right]$ $$

where $p_{t}(\cdot|y)$ is the distribution of noisy images at timestep $t$. Similar to how losses at different timesteps are weighted uniformly in the training of real-world diffusion models (e.g. $L_{simple}$ proposed by Ho et al. ()), we set $g(t)$ to be constant, in particular $g(t)=1$, to simplify our algorithm such that it does not have to be tailored specifically to each choice of scheduler.

We can compute this resulting semantic distance using Monte-Carlo by discretizing the timesteps to $t\in\{1,\ldots,T\}$. In particular, this is computed in practice via sampling an initial noise vector $\bm{x}(T)\sim\pi$, and denoising it with either $y_{1}$ and $y_{2}$ to obtain a sequence of samples $x_{t}$’s, and computing the difference in predictions $\|s_{\theta}(x_{t},t|y_{1})-s_{\theta}(x_{t},t|y_{2})\|_{2}^{2}$ at each denoising timestep. We describe this process in Algorithm 1.

## 4 Experiments

We describe implementation details in Section 4.1, empirical validation for our definitions in Section 4.2, and ablations in Section 4.3.

### 4.1 Implementation Details

We use Stable Diffusion v1.4 (Rombach et al., ), a text-conditioned diffusion model, for all our experiments. For sampling, we use classifier-free guidance (Ho & Salimans, ) with guidance scale of $7.5$, and sample using the LMS Scheduler (Karras et al., ). We specify image sizes to be $512\times 512$, but note that Stable Diffusion v1.4 uses latent diffusion, as such model predictions are in practice of dimension $64\times 64$. We compute the Euclidean distance directly in this space. However, for visualization experiments such as in Figure 1, we decode the predictions using the VAE before plotting them. We set $T=10$ in our experiments to ensure computational feasibility, after ablating over other choices in Table 2.

As a technicality, while we model the denoising direction term specified in the reverse-time SDE in eq. 1 as the (text-conditioned) model output $s_{\theta}(\cdot|y)$, we note that they do not usually correspond exactly, since this depends on the text-conditioning method used. However, we note that in the case of classifier-free guidance, the resulting distance computed using the model output as $s_{\theta}(\cdot|y)$ is equivalent to that computed using classifier-guidance directions up to proportionality.

### 4.2 Empirical Validation

**Table 1: Comparison with zero-shot methods on Semantic Textual Similarity benchmarks. $\ast;\dagger;\ddagger$ indicate results taken from Ni et al. (); Gao et al. (); Liu et al. () respectively. Expectedly, our zero-shot approach does not perform as well as embedding models such as CLIP (Radford et al., ) and SimCSE-BERT (Gao et al., ), which are trained specifically for semantic comparison tasks. Nevertheless, semantic structures extracted from text-conditioned diffusion models using our method are still relatively aligned with that of human annotators, rivaling those extracted from autoregressive Large Language Models while outperforming encoder-based language models such as BERT (Devlin et al., ).**
|  | STS-B | STS12 | STS13 | STS14 | STS15 | STS16 | SICK-R | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Contrastive-Trained Embedding Models |  |  |  |  |  |  |  |  |
| CLIP-ViTL14^‡ (Radford et al., ) | 65.5 | 67.7 | 68.5 | 58.0 | 67.1 | 73.6 | 68.6 | 67.0 $\pm$ 4.3 |
| IS-BERT^† (Zhang et al., ) | 56.8 | 69.2 | 61.2 | 75.2 | 70.2 | 69.2 | 64.3 | 66.6 $\pm$ 5.7 |
| SimCSE-BERT^† (Gao et al., ) | 68.4 | 82.4 | 74.4 | 80.9 | 78.6 | 76.9 | 72.2 | 76.3 $\pm$ 4.6 |
| Zero-Shot Encoder-based Models |  |  |  |  |  |  |  |  |
| BERT-CLS^∗ (Devlin et al., ) | 16.5 | 20.2 | 30.0 | 20.1 | 36.9 | 38.1 | 42.6 | 29.2 $\pm$ 9.6 |
| BERT-mean^∗ (Devlin et al., ) | 45.4 | 38.8 | 58.0 | 58.0 | 63.1 | 61.1 | 58.4 | 54.8 $\pm$ 8.3 |
| BERT Large-mean^∗ (Devlin et al., ) | 47.0 | 27.7 | 55.8 | 44.5 | 51.7 | 61.9 | 53.9 | 48.9 $\pm$ 10.2 |
| RoBERTa Large-mean^∗ (Liu et al., ) | 50.6 | 33.6 | 57.2 | 45.7 | 63.0 | 61.2 | 58.4 | 52.8 $\pm$ 9.6 |
| ST5-Enc-mean (Large)^∗ (Ni et al., ) | 56.3 | 28.0 | 52.6 | 41.4 | 61.3 | 63.6 | 59.5 | 51.8 $\pm$ 11.9 |
| ST5-Enc-mean (11B)^∗ (Ni et al., ) | 62.8 | 35.0 | 60.2 | 47.6 | 66.4 | 70.6 | 63.6 | 58.0 $\pm$ 11.5 |
| Autoregressive Models (Meanings as Trajectories (Liu et al., )) |  |  |  |  |  |  |  |  |
| GPT-2^‡ | 55.2 | 39.9 | 42.6 | 30.5 | 52.4 | 62.7 | 62.0 | 49.3 $\pm$ 11.2 |
| GPT-2-XL^‡ | 62.1 | 43.6 | 54.8 | 37.7 | 61.3 | 68.2 | 68.4 | 56.5 $\pm$ 11.1 |
| Falcon-7B^‡ | 67.7 | 56.3 | 66.5 | 53.0 | 67.4 | 75.5 | 73.5 | 65.7 $\pm$ 7.7 |
| LLaMA-13B^‡ | 70.6 | 52.5 | 65.9 | 53.2 | 67.8 | 74.1 | 73.0 | 65.3 $\pm$ 8.3 |
| LLaMA-33B^‡ | 71.5 | 52.5 | 70.6 | 54.6 | 69.1 | 75.2 | 73.0 | 66.6 $\pm$ 8.5 |
| Text-Conditioned Diffusion Models (Ours) |  |  |  |  |  |  |  |  |
| Stable Diffusion | 70.3 | 57.9 | 61.0 | 60.8 | 73.6 | 67.9 | 66.0 | 65.4 $\pm$ 5.3 |

While our work defines a notion of semantic distance grounded in evoked imagery, the validity of this definition hinges on its use as a measure of similarity that aligns with humans. In particular, we should expect our definition to produce measurements of similarity that agree often with human annotators (which can be viewed as “ground-truth”).

To quantify this, we use the Semantic Textual Similarity (STS) (Agirre et al., , , , , ; Cer et al., ) and Sentences Involving Compositional Knowledge (SICK-R) (Marelli et al., ) datasets, containing pairs of sentences each labelled by human annotators with a semantic similarity score ranging from 0-5. We then use our method to compute the image-grounded similarity score, and measure their resulting Spearman Correlation with the annotations.

Interestingly, our experiments in Table 1 show that our visually-grounded similarity scores exhibit significant degrees of correlation with that annotated by humans. While, expectedly, our method presently lags behind embedding models trained specifically for semantic comparison tasks, we show that our visually-grounded similarity scores can rival that produced by existing large language models up to 33B in size. As a remark, we note the large standard deviations in these scores across datasets for all methods, suggesting that what matters for validation purposes is that our distances correlate significantly with that of annotators, rather than obtaining “state-of-the-art” alignment scores since ‘ground truth’ is fundamentally subjective in each of these benchmarks.

We further remark that in most existing text-conditioned diffusion models, the representation structures that can be captured by our method is limited by those learnt by text-encoder models such as CLIP (Radford et al., ), since these encoders are often used to pre-process textual prompts. In light of this, the experiments also suggest that our method can be an effective metric to quantify how well representation structures learnt by these text-encoders have been distilled to the resulting diffusion model.

In Figure 2, we also qualitatively evaluate our method on measuring semantic similarity between various words. From the resulting pairwise similarity matrices, we can observe that words are closer in terms of their common hypernym class tend to cluster together, showing that our method can effectively capture word taxonomies. For instance, nouns that describe types of dogs cluster together, and nouns that describe types of marine animals similarly
form another cluster. These two clusters are separate, in the sense that distances of words across these clusters are large than those within each cluster.

Figure: Figure 2: Qualitative evaluation of conjured semantic similarity. (Left) shows that nouns cluster based on shared hypernym classes: Dogs (puppy, poodle, dalmatian, pug) form a visible cluster in the top-left 4x4 block, while marine animals (whale, shark, dolphin, sealion) form another cluster in the bottom-right 4x4 block. (Right) shows that the same pattern holds for flying-related action verbs (elevate, ascend, soar, glide) v.s. negative stative verbs (disappoint, grieve, worry, regret).
Refer to caption: x2.png

### 4.3 Ablations

In this section, we run ablation studies using the STS-B dataset as a benchmark to explore the design space of our method, and improve its computational efficiency.

Figure: Figure 3: (Left:) We ablate over different choices of priors over timesteps – a uniform distribution over timesteps $\{T^{\prime},\ldots,T\}$ where $T^{\prime}\leq T=10$, represented by the blue line (cumulative), and the Direc Delta on any particular timestep $T^{\prime}\in\{1,\ldots,T\}$, represented by the orange line (pointwise). We show that a uniform prior over all timesteps gives the best results. The same plot also ablates over the number of Monte-Carlo samples, $k\in\{1,\ldots,5\}$, where we conclude that only few iterations are required for convergence. (Right:) We further ablate over different choices of diffusion models, and show that results remain relatively consistent across the tested choices.
Refer to caption: x4.png

#### Prior over timestep distribution.

In our algorithm, we placed a uniform prior over timesteps $\{1,\ldots,T\}$. In Figure 3, we show that this simple choice works best when evaluated on alignment with human annotators on STS-B, as compared to other choices such as considering only a uniform prior over the subset $\{T^{\prime},\ldots,T\}$ (cumulative from $T^{\prime}$ to $T$) where $T^{\prime}\leq T$, or a Direc delta on any particular timestep $T^{\prime}\in\{1,\ldots,T\}$ (pointwise).

#### Number of Monte-Carlo Steps.

The computational feasibility of our method depends on the number of Monte-Carlo steps (i.e. $k$ in Algorithm 1) required to produce a reliable approximate of the desired distance. We ablate over choices of $k\in\{1,\ldots,5\}$ in Figure 3 and show that the deviation of scores when evaluated on the STS-B dataset is small ($\pm 0.77$) across different choices of $k$. This finding is promising, as it implies that our method can be computationally efficient, requiring only a small number of Monte-Carlo iterations to converge.

#### Choice of Stable Diffusion Model.

On the right of Figure 3, we also show that our results are relatively consistent across several versions of Stable Diffusion models, including Stable Diffusion XL and Stable Diffusion 3 Medium.

**Table 2: Ablation on choices for $T$ on the STS-B dataset: We show that variance with respect to the choice of $T$ is small, allowing semantic distances to be computed efficiently by using lower values.**
|  | $T=5$ | $T=10$ | $T=15$ | $T=20$ | $T=30$ | $T=50$ |
| --- | --- | --- | --- | --- | --- | --- |
| Spearman Corr. | $70.1$ | $70.3$ | $70.1$ | $68.9$ | $70.2$ | $69.5$ |

## 5 Discussion and Limitations

Our method has several limitations. First, imageries might indeed not be sufficient to fully capture the meaning of certain expressions, such as mathematical abstractions (like ‘imaginary numbers’) and meta-physical concepts (like ‘conscience’).
Furthermore, many modern diffusion models use a pre-trained text-encoder to pre-process textual prompts. This means that representation structures obtained from the diffusion model outputs would be bottle-necked by those learnt by the text encoder. However, this limitation can be mitigated by the development of better encoders, such as those based on LLMs (BehnamGhader et al., ). Additionally, these encoder-based (vector) representations are often difficult to interpret, while our
proposed method offers a way to visualize and interpret the learnt semantic similarities between textual expressions.
Lastly, computation costs also remain a key limitation of our method, since it requires several inference passes through the diffusion model to compute a single semantic similarity score, mitigated only partially by the conclusions of our ablation study on required number of iterations.

Nevertheless, our method is the first to show that textual representations can be meaningfully compared for diffusion models by “grounding” them in the space of conjured images. Our work is also the first to enable not only qualifying (via visual ‘explanations’), but also quantifying the alignment of this resulting semantic space with that of human annotators, which can potentially be adapted as an additional metric to evaluate text-conditioned diffusion models.

## Appendix A Additional Visualizations

Figure: Figure 4: “Merlion” vs “Mermaid Lion”: While both prompts express compositions of the same set of objects, the model associates different meanings with “Merlion” as opposed to “Mermaid + Lion”, where the former is associated to the mascot of Singapore, while the latter is a mermaid with hair resembling a lion’s mane.
Refer to caption: x6.png

Figure: Figure 5: “Bag of Chips” vs “Bag of Fries”: The interpretation of “chips” depends on cultural background (US vs UK), but the interpretation of “fries” is relatively non-ambiguous. Interestingly, this observation can be visualized when computing semantic similarity with our method. We see that on the left of the figure (second image column), the model attempts to convert a picture of chips (US) into fries by changing the rounded textures into sharper rectangular ones, when denoised with “Bag of Fries”. On the other hand, pictures of fries still remain relatively identifiable as fries (fifth image column) when denoised using “Bag of Chips”.
Refer to caption: x7.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: **Questions:**
Is it possible to detect poor representations automatically in the underlying text encoder using this similarity measure?


**Other:**
I had a hard time understanding Figure 1, consider redesigning it or breaking it down to two figures
soundness: 3
strengths: **S1:** The idea of learning semantic textual similarity through the images such expressions evoke is creative, original, and intriguing.

**S2:** Related Work provides a detailed and relevant account of existing work in the subject.
summary: The paper proposes a novel similarity measurement–textual similarity based on the imagery the texts evoke. They propose learning this similarity by computing the Jensen-Shannon divergence between diffusion processes conditioned on the two compared prompts, using Monte-Carlo sampling. The method is then compared to existing similarity measurements on the STS benchmarks and is ablated.
weaknesses: **W1:** The premise of the paper–obtaining a similarity measurement based on the imagery texts evoke–is unique and interesting, but I don’t understand what use-case is it tailored to address. If the use-case is no different than measuring similarities in text-only environments, I don’t see why it is preferable over methods that are inherently text-only, are easier to scale, and are equipped to represent abstract notions, which are difficult to visualize. I find that conclusion is supported in section 4.2 and table 1, too.  

The paper will be improved if such motivation will be clarified, with an experiment to demonstrate this use-case. 

**W2:** Isn’t the method hindered by whatever the text encoder does not represent well, or what the diffusion process did not accurately learn? Seeing as these are inherent components, this seems like a significant drawback, which effectively nullifies possible advantages of this method. You touch on this matter in lines 312–317 and section 5. I would appreciate a clarification here.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) on line 238, "denoising with either y1 and y2" is a little confusing in that "either" seems to imply an exclusive OR while "and", Algo 1 and the definition d_ours(y1,y2) show the need for both?  I could be mistaken, but I think the later is the case, in which case you should change "either" to be "both"

2) The line starting at 257 was a little unclear to me.  Specifically "we note that they do not usually correspond exactly" ?

3) nit: 313 "is" -> "are"
soundness: 3
strengths: The work proposes a novel method for assessing semantic similarity between two text sequences by comparing the distance between image distributions generated over time by diffusion models conditioned on them. 

The authors show the method does well on human annotation semantic similarity baselines.

The method offers a new possible avenue for the evaluation of text-conditioned generative models which allows some interpretability of representation similarities for text conditioned image generation models.   While its not the most performative method, it seems like one that could be expanded/built upon both to improve its performance, but also possibly as a way of guiding/improving diffusion model training.
summary: The authors propose to use the distance between image distributions generated by a diffusion model conditioned on two text sequences as a novel measure of semantic similarity.   They specifically propose using the Jensen-Shannon divergence between the reverse-time diffusion stochastic differential equations (SDEs) induced by two text sequences which can be directly computed via Monte-Carlo sampling. 

Empirically, while not performing as well as embedding models trained specifically for semantic comparison tasks  ( CLIP, SimCSE-BERT, etc ), the author's approach outperforms zero-short encoder models and aligns well with human-annotated scores from the Semantic Textual Similarity and Sentences involving Compositional Knowledge benchmarks.

The authors provide some findings from ablations over the choice of prior distributions (uniform vs dirac), number of monte carlo steps and the choice of diffusion model as well.
weaknesses: While the authors cited weaknesses with the approach, the authors could do a better job motivating possible applications opened up by their method.  

The paper qualitative experiment is a little shallow. Expanding it and doing error analysis ( where does the method perform well/poorly ) on results would help add more clarity into the method.

The interpretability angle of the paper is also a little lacking and under explained.  Outside of the few examples given (one in the paper and two in the appendix ), is there anyway to automate interpretability results or use them for error analysis in a useful way.

The paper could have done an ablation compared their symmetric JSD approach to just using KL-Divergence to show the boost obtained.

The last line of the 2nd paragraph ( about pixel values not depending on distant knowledge or cultural background ) seems debatable/unnecessary since a text passage (about democracy, celebration, a feast, etc ) could be portrayed visually in different ways depending very much on knowledge/cultural background while still referring to the same semantic concept

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1.	Is there a specific set of STS that you found the proposed method works?
soundness: 2
strengths: 1.	This paper proposes an interesting perspective linking text and diffusion models. The idea of contextual and concept-level similarity is natural and convincing.
2.	This paper is well-written and easy to follow.
summary: The paper discusses the measurement of semantic similarity at the concept level. The authors propose an interesting method to measure semantic similarity via the distribution of the image with guidance from paired different textual expressions. On various STS tasks, the authors show comparable performance between the proposed methods and various textual or multi-modal semantic similarity measurements.
weaknesses: 1.	The empirical experiments are too weak. On the one hand, the performance is not as good as the previous methods. The baselines listed are slightly outdated. It would be interesting if OpenAI ada embeddings, Llama 3.1, and Gemma can be compared. Surpassing BERT-CLS is not very convincing. On the other hand, STS may not be the best arena for the novel method the authors proposed, since there are a lot of examples related to paraphrasing, instead of conceptual relevance. It would be great if the authors study a specific slice that is more relevant to the idea, e.g., examples with a not-complex scene and the difference mainly comes from the concepts. Knowledge graphs (e.g., COMET, Atomic) or WordNet depth distance may provide better comparative performance or qualitative examples.
2.	This paper can be linked with more recent advances in this area in NLP, e.g., C-STS (https://arxiv.org/abs/2305.15093) discusses the conditional semantic textual similarity in addition to STS; Instructor (https://github.com/xlang-ai/instructor-embedding) and PIR (https://arxiv.org/abs/2405.02714) discuss the changes in semantic embeddings under different instructions. The related work and experiment suite can help further improve the draft in discussing the semantic similarity.
3.	It would be good if further content is added to the paper especially since there are 7.5 pages in the current draft. For example, the authors can include experiments on potential performance improvement over various downstream tasks: Z-LaVI (https://arxiv.org/abs/2210.12261), for example, explores how visual imagination helps improve model performance on tasks such as word sense disambiguation in a zero-shot manner. Similar tasks can be included in the draft.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: * The paper is well-written and easy to read.
* The idea of a visually grounded similarity metric between sentences is very interesting and could be a useful addition to the community’s toolbox for measuring different aspects of semantic similarity.
* The authors provide a theoretical understanding and justification for the introduced distance metric.
summary: This paper proposes a novel method for calculating the similarity between two pieces of text that is visually grounded. For each sentence, the authors use the sentence to guide a text-conditioned diffusion model to remove the noise and generate a new image given some initial noisy image.
Each sentence guides the diffusion model to generate intermediate denoised images with a distinct distribution. To compare two sentences, the authors define the distance between the two sentences as the distance between the corresponding distributions of the intermediate denoised images.
weaknesses: My main concern with this paper is the lack of a clear contribution that is well-motivated and also well-supported by the presented empirical evidence. In the last paragraph of section 1, the authors claim three separate contributions. Here are my concerns about each claimed contribution:

First contribution (Lines 82-83): “we propose an approach for evaluating semantic similarity between text expressions that is grounded in the space of visual images.”

Although I agree that a novel notion of similarity metric is interesting, the paper lacks a solid argument to support why it is needed or useful. Some of the unanswered questions are:
* Why do we need a new similarity metric? In paragraph 2, the authors provide some arguments about why comparing images is easier for humans than comparing text due to language and knowledge differences. But, that argument does not hold for models since the similarity between texts is often calculated by a single model. So, there is no difference in knowledge or judgment between models. Moreover, even if this limitation is actually valid for text similarity measures based on dense vectors, it also applies to the proposed similarity metric. After all, just like there is model that is creating the dense vector, there is a model that is doing the denoising and image generation. So, there is still a model involved with all the specific and often unknown limitations and biases that come with any model.
* As the authors mentioned in Line 312, these generative models rely on some encoder to encode the text. So, all the limitations of the encoder model (i.e., cosine similarity between dense vectors) also apply to the proposed similarity metric. So, why should we add the extra level of complexity?
* Even beyond the motivation, the experimental results do not suggest that the new similarity metric is consistently better than the vector-based methods. The most interesting comparison is the performance of the proposed similarity metric compared to the performance of the text encoder that is used in the stable diffusion generative model, which is clip-vit-14. Comparing these two, it seems like in just three out of the seven datasets, the new similarity metric performs better, which is less than half the times.
* In general, to show the superior performance of the proposed similarity metric, more datasets that cover more diverse domains are needed.

Second contribution (Lines 84-85): “Our method has a unique advantage over traditional language-based methods that, in addition to providing a numerical score, it also provides a visual expression, or ‘explanation’, for comparison, enabling better interpretability of the learnt representations”:

I completely agree with this advantage of the proposed method, and I think it might be the exact case where their method shines. But, other than three figures that show qualitatively what this interpretation might look like, there is no other discussion or experiment on this.
To claim this as the main contribution, the authors should provide extensive experiments and discussion on the types of explanation that their method provides in different use cases, discuss how these explanations can be useful for the community, and how their method compares to existing interpretability methods.


Third contribution (Lines 86-87): “our method is the first to enable quantifying the alignment of semantic representations learnt by diffusion models compared to that of humans, which can open up new avenues for the evaluation text-conditioned diffusion models”

Again, I completely agree that this can be a strength of the proposed method. But, similar to my previous point, to claim this as the main contribution, the authors should evaluate several diffusion models, discuss helpful insights that their metric provides, compare their evaluation results with previous evaluation metrics, and explain their differences, advantages, and weaknesses.

---

I also have an issue with the lack of a baseline to measure the merits of the proposed similarity metric. This paper proposes two different concepts. One is the notion of text similarity through images, and the second is the proposed similarity metric to accomplish this. Assuming that the authors provide adequate motivation and support for the importance of the notion of text similarity through evoked images, they should also prove that the complexity of their similarity metric is justified. For example, what if I just generate the final fully denoised images using each sentence and then measure the similarity of the generated images using something like CLIP or DINO embeddings?

Finally, I think the limitations that the authors mention in Section 5 deserve more than a one-sentence acknowledgment. For example, the authors mention that their method is computationally expensive. It would be helpful for the community to know the exact computational resources used for the experiments.  
The authors also mention the limitations that are caused by the use of text encoders such as CLIP by diffusion models. It is important to explore and analyze these limitations and also show the merits of the proposed method compared to just using the outputs of the text encoder with cosine similarity in the first place.

Meta-review: The paper proposes a novel measurement of semantic similarity based on the imagery of the texts. The authors propose using Jensen-Shannon divergence (JSD) between the diffusion  process conditioned on the input  text, via Monte-Carlo sampling. 

The reviewers agreed the proposed approach is novel and highlight the paper is well-written and easy to follow. The idea of a visually grounded similarity metric between sentences is interesting and has its potential to be expanded. 

The major concerns from the reviewers are:
1. Lack of the practical usage of the proposed approach, it is not better than existing textural similar models. And it could be slower with the diffusion process.  
2. Deeper analysis / more evidence for better understand the proposed approach including incorporating with other text based approaches, the interpretability, and limitation. 

After discussion, the reviewers agree this is an interesting submission but can be improved significantly.

Decision: Reject