Conjuring Semantic Similarity

Authors: Tian Yu Liu, Stefano Soatto
Primary area: interpretability and explainable AI
Keywords: Semantic Similarity, Interpretability, Diffusion Models

Abstract:
The semantic similarity between sample expressions measures the distance between their latent 'meaning'. Such meanings are themselves typically represented by textual expressions, often insufficient to differentiate concepts at fine granularity. We propose a novel approach whereby the semantic similarity among textual expressions is based {\em not} on other expressions they can be rephrased as, but rather based on the imagery they evoke. While this is not possible with humans, generative models allow us to easily visualize and compare generated images, or their distribution, evoked by a textual prompt. Therefore, we characterize  the semantic similarity between two textual expressions simply as the distance between image distributions they induce, or 'conjure.'  We show that by choosing  the Jensen-Shannon divergence between the reverse-time diffusion stochastic differential equations (SDEs) induced by each textual expression, this can be directly computed via Monte-Carlo sampling. Our method contributes a novel perspective on semantic similarity that not only aligns with human-annotated scores, but also opens up new avenues for the evaluation of text-conditioned generative models while offering better interpretability of their learnt representations.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730488335780

thread_type:
official_review

<thread_object>
creator: reviewer_wvfy

<content>
summary:
This paper proposes a novel method for calculating the similarity between two pieces of text that is visually grounded. For each sentence, the authors use the sentence to guide a text-conditioned diffusion model to remove the noise and generate a new image given some initial noisy image.
Each sentence guides the diffusion model to generate intermediate denoised images with a distinct distribution. To compare two sentences, the authors define the distance between the two sentences as the distance between the corresponding distributions of the intermediate denoised images.

strengths:
* The paper is well-written and easy to read.
* The idea of a visually grounded similarity metric between sentences is very interesting and could be a useful addition to the community’s toolbox for measuring different aspects of semantic similarity.
* The authors provide a theoretical understanding and justification for the introduced distance metric.

weaknesses:
My main concern with this paper is the lack of a clear contribution that is well-motivated and also well-supported by the presented empirical evidence. In the last paragraph of section 1, the authors claim three separate contributions. Here are my concerns about each claimed contribution:

First contribution (Lines 82-83): “we propose an approach for evaluating semantic similarity between text expressions that is grounded in the space of visual images.”

Although I agree that a novel notion of similarity metric is interesting, the paper lacks a solid argument to support why it is needed or useful. Some of the unanswered questions are:
* Why do we need a new similarity metric? In paragraph 2, the authors provide some arguments about why comparing images is easier for humans than comparing text due to language and knowledge differences. But, that argument does not hold for models since the similarity between texts is often calculated by a single model. So, there is no difference in knowledge or judgment between models. Moreover, even if this limitation is actually valid for text similarity measures based on dense vectors, it also applies to the proposed similarity metric. After all, just like there is model that is creating the dense vector, there is a model that is doing the denoising and image generation. So, there is still a model involved with all the specific and often unknown limitations and biases that come with any model.
* As the authors mentioned in Line 312, these generative models rely on some encoder to encode the text. So, all the limitations of the encoder model (i.e., cosine similarity between dense vectors) also apply to the proposed similarity metric. So, why should we add the extra level of complexity?
* Even beyond the motivation, the experimental results do not suggest that the new similarity metric is consistently better than the vector-based methods. The most interesting comparison is the performance of the proposed similarity metric compared to the performance of the text encoder that is used in the stable diffusion generative model, which is clip-vit-14. Comparing these two, it seems like in just three out of the seven datasets, the new similarity metric performs better, which is less than half the times.
* In general, to show the superior performance of the proposed similarity metric, more datasets that cover more diverse domains are needed.

Second contribution (Lines 84-85): “Our method has a unique advantage over traditional language-based methods that, in addition to providing a numerical score, it also provides a visual expression, or ‘explanation’, for comparison, enabling better interpretability of the learnt representations”:

I completely agree with this advantage of the proposed method, and I think it might be the exact case where their method shines. But, other than three figures that show qualitatively what this interpretation might look like, there is no other discussion or experiment on this.
To claim this as the main contribution, the authors should provide extensive experiments and discussion on the types of explanation that their method provides in different use cases, discuss how these explanations can be useful for the community, and how their method compares to existing interpretability methods.


Third contribution (Lines 86-87): “our method is the first to enable quantifying the alignment of semantic representations learnt by diffusion models compared to that of humans, which can open up new avenues for the evaluation text-conditioned diffusion models”

Again, I completely agree that this can be a strength of the proposed method. But, similar to my previous point, to claim this as the main contribution, the authors should evaluate several diffusion models, discuss helpful insights that their metric provides, compare their evaluation results with previous evaluation metrics, and explain their differences, advantages, and weaknesses.

---

I also have an issue with the lack of a baseline to measure the merits of the proposed similarity metric. This paper proposes two different concepts. One is the notion of text similarity through images, and the second is the proposed similarity metric to accomplish this. Assuming that the authors provide adequate motivation and support for the importance of the notion of text similarity through evoked images, they should also prove that the complexity of their similarity metric is justified. For example, what if I just generate the final fully denoised images using each sentence and then measure the similarity of the generated images using something like CLIP or DINO embeddings?

Finally, I think the limitations that the authors mention in Section 5 deserve more than a one-sentence acknowledgment. For example, the authors mention that their method is computationally expensive. It would be helpful for the community to know the exact computational resources used for the experiments.  
The authors also mention the limitations that are caused by the use of text encoders such as CLIP by diffusion models. It is important to explore and analyze these limitations and also show the merits of the proposed method compared to just using the outputs of the text encoder with cosine similarity in the first place.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730509818673

thread_type:
official_review

<thread_object>
creator: reviewer_txae

<content>
summary:
The paper discusses the measurement of semantic similarity at the concept level. The authors propose an interesting method to measure semantic similarity via the distribution of the image with guidance from paired different textual expressions. On various STS tasks, the authors show comparable performance between the proposed methods and various textual or multi-modal semantic similarity measurements.

strengths:
1.	This paper proposes an interesting perspective linking text and diffusion models. The idea of contextual and concept-level similarity is natural and convincing.
2.	This paper is well-written and easy to follow.

weaknesses:
1.	The empirical experiments are too weak. On the one hand, the performance is not as good as the previous methods. The baselines listed are slightly outdated. It would be interesting if OpenAI ada embeddings, Llama 3.1, and Gemma can be compared. Surpassing BERT-CLS is not very convincing. On the other hand, STS may not be the best arena for the novel method the authors proposed, since there are a lot of examples related to paraphrasing, instead of conceptual relevance. It would be great if the authors study a specific slice that is more relevant to the idea, e.g., examples with a not-complex scene and the difference mainly comes from the concepts. Knowledge graphs (e.g., COMET, Atomic) or WordNet depth distance may provide better comparative performance or qualitative examples.
2.	This paper can be linked with more recent advances in this area in NLP, e.g., C-STS (https://arxiv.org/abs/2305.15093) discusses the conditional semantic textual similarity in addition to STS; Instructor (https://github.com/xlang-ai/instructor-embedding) and PIR (https://arxiv.org/abs/2405.02714) discuss the changes in semantic embeddings under different instructions. The related work and experiment suite can help further improve the draft in discussing the semantic similarity.
3.	It would be good if further content is added to the paper especially since there are 7.5 pages in the current draft. For example, the authors can include experiments on potential performance improvement over various downstream tasks: Z-LaVI (https://arxiv.org/abs/2210.12261), for example, explores how visual imagination helps improve model performance on tasks such as word sense disambiguation in a zero-shot manner. Similar tasks can be included in the draft.

questions:
1.	Is there a specific set of STS that you found the proposed method works?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730569404685

thread_type:
official_review

<thread_object>
creator: reviewer_yfjv

<content>
summary:
The authors propose to use the distance between image distributions generated by a diffusion model conditioned on two text sequences as a novel measure of semantic similarity.   They specifically propose using the Jensen-Shannon divergence between the reverse-time diffusion stochastic differential equations (SDEs) induced by two text sequences which can be directly computed via Monte-Carlo sampling. 

Empirically, while not performing as well as embedding models trained specifically for semantic comparison tasks  ( CLIP, SimCSE-BERT, etc ), the author's approach outperforms zero-short encoder models and aligns well with human-annotated scores from the Semantic Textual Similarity and Sentences involving Compositional Knowledge benchmarks.

The authors provide some findings from ablations over the choice of prior distributions (uniform vs dirac), number of monte carlo steps and the choice of diffusion model as well.

strengths:
The work proposes a novel method for assessing semantic similarity between two text sequences by comparing the distance between image distributions generated over time by diffusion models conditioned on them. 

The authors show the method does well on human annotation semantic similarity baselines.

The method offers a new possible avenue for the evaluation of text-conditioned generative models which allows some interpretability of representation similarities for text conditioned image generation models.   While its not the most performative method, it seems like one that could be expanded/built upon both to improve its performance, but also possibly as a way of guiding/improving diffusion model training.

weaknesses:
While the authors cited weaknesses with the approach, the authors could do a better job motivating possible applications opened up by their method.  

The paper qualitative experiment is a little shallow. Expanding it and doing error analysis ( where does the method perform well/poorly ) on results would help add more clarity into the method.

The interpretability angle of the paper is also a little lacking and under explained.  Outside of the few examples given (one in the paper and two in the appendix ), is there anyway to automate interpretability results or use them for error analysis in a useful way.

The paper could have done an ablation compared their symmetric JSD approach to just using KL-Divergence to show the boost obtained.

The last line of the 2nd paragraph ( about pixel values not depending on distant knowledge or cultural background ) seems debatable/unnecessary since a text passage (about democracy, celebration, a feast, etc ) could be portrayed visually in different ways depending very much on knowledge/cultural background while still referring to the same semantic concept

questions:
1) on line 238, "denoising with either y1 and y2" is a little confusing in that "either" seems to imply an exclusive OR while "and", Algo 1 and the definition d_ours(y1,y2) show the need for both?  I could be mistaken, but I think the later is the case, in which case you should change "either" to be "both"

2) The line starting at 257 was a little unclear to me.  Specifically "we note that they do not usually correspond exactly" ?

3) nit: 313 "is" -> "are"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712172046

thread_type:
official_review

<thread_object>
creator: reviewer_rcyr

<content>
summary:
The paper proposes a novel similarity measurement–textual similarity based on the imagery the texts evoke. They propose learning this similarity by computing the Jensen-Shannon divergence between diffusion processes conditioned on the two compared prompts, using Monte-Carlo sampling. The method is then compared to existing similarity measurements on the STS benchmarks and is ablated.

strengths:
**S1:** The idea of learning semantic textual similarity through the images such expressions evoke is creative, original, and intriguing.

**S2:** Related Work provides a detailed and relevant account of existing work in the subject.

weaknesses:
**W1:** The premise of the paper–obtaining a similarity measurement based on the imagery texts evoke–is unique and interesting, but I don’t understand what use-case is it tailored to address. If the use-case is no different than measuring similarities in text-only environments, I don’t see why it is preferable over methods that are inherently text-only, are easier to scale, and are equipped to represent abstract notions, which are difficult to visualize. I find that conclusion is supported in section 4.2 and table 1, too.  

The paper will be improved if such motivation will be clarified, with an experiment to demonstrate this use-case. 

**W2:** Isn’t the method hindered by whatever the text encoder does not represent well, or what the diffusion process did not accurately learn? Seeing as these are inherent components, this seems like a significant drawback, which effectively nullifies possible advantages of this method. You touch on this matter in lines 312–317 and section 5. I would appreciate a clarification here.

questions:
**Questions:**
Is it possible to detect poor representations automatically in the underlying text encoder using this similarity measure?


**Other:**
I had a hard time understanding Figure 1, consider redesigning it or breaking it down to two figures
</content>
</thread_object>

</thread>
<thread>
creation date:
1731808231696

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Submission Updated

comment:
We are happy to hear that our work is well-received by the reviewers with regards to its “creativity”, “originality”, “novelty” (rCYr, yfJv), and that our idea of “visually grounded semantic similarity” being an “interesting perspective linking text and diffusion models” is “natural and convincing” while backed up by “theoretical understanding and justification” (tXAe, WVfY). We are also heartened to see reviewers actively proposing novel directions of research that our work has opened up, including “a way of guiding/improving diffusion model training” (yfJv), “exploring how visual imagination helps improve model performance” (tXAe), and as a “useful addition to the community’s toolbox for measuring different aspects of semantic similarity” (WVfY).

We greatly appreciate the reviewers' valuable feedback, and have incorporated several of their suggestions in our paper revision. Our paper is the first work to define a (theoretically grounded) notion of semantic similarity for text-conditioned diffusion models, which we show can be used for evaluating semantic alignment of such models with human annotators. Nonetheless, we have included several comparable baselines in our new revision, where we show that our approach indeed outperforms all other methods used for quantifying semantic alignment in diffusion models.

We have also expanded our evaluation to include word similarity datasets, stratified based on the parts of speech from which the words originate. This allows for finer-grained analysis and interpretation of the semantic relations captured by diffusion models, and we refer to our revised draft for the full results and discussion.

To summarize, our revision includes, but is not limited to, the following changes:

- Added comparisons to four new baselines in Table 1 - Initial Timestep Prediction, Final Timestep Prediction, Direct Output Comparison, and a version of our method based on KL-Divergence, where we show that our method outperforms all baselines across all datasets.
- Added experiments on RG65 and SimLex-999 datasets in Table 2.
- Added an analysis on failure modes of diffusion models, with respect to preserving semantic relations learnt by their text-encoders, in Sec 4.4.
- Added Sec 4.2 to motivate baselines.
- Expanded contributions of our method in Sec 5.
- Modified Figure 1 
- Other changes discussed in individual responses to reviewers
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel measurement of semantic similarity based on the imagery of the texts. The authors propose using Jensen-Shannon divergence (JSD) between the diffusion  process conditioned on the input  text, via Monte-Carlo sampling. 

The reviewers agreed the proposed approach is novel and highlight the paper is well-written and easy to follow. The idea of a visually grounded similarity metric between sentences is interesting and has its potential to be expanded. 

The major concerns from the reviewers are:
1. Lack of the practical usage of the proposed approach, it is not better than existing textural similar models. And it could be slower with the diffusion process.  
2. Deeper analysis / more evidence for better understand the proposed approach including incorporating with other text based approaches, the interpretability, and limitation. 

After discussion, the reviewers agree this is an interesting submission but can be improved significantly.