On the Cycle Consistency of Image-Text Mappings

Authors: Caroline Chan, Hyojin Bahng, Fredo Durand, Phillip Isola
Primary area: foundation or frontier models, including LLMs
Keywords: cycle consistency, multimodal learning, vision-language modeling, text-to-image generation, synthetic data

Abstract:
The increasing exchange of image and text in large multimodal models leads us to ask: to what degree are mappings from text to image, and back, cycle-consistent? First, we find that current image-to-text models paired with text-to-image models do achieve a degree of perceptual cycle consistency, even when these models are not trained to have this effect. However, these mappings are far from perfect, motivating us to analyze in what ways they fail. First, we observe a strong correlation between cycle consistency and downstream performance in both image captioning and text-to-image generation. Next, we investigate how divergent are text-to-image mappings as a function of the number of objects described by the text, and how it affects achieving cycle consistency. Surprisingly, we find that more descriptive text leads to a a broader distribution of generated images, but also results in overall better reconstructions. Finally, we show possible challenges of training cycle consistent models due to the sensitivity of text-to-image models.

Decision: reject
Reviewer scores: 5, 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730234820781

thread_type:
official_review

<thread_object>
creator: reviewer_bhgf

<content>
summary:
The paper (Cycle Consistency of Image-Text Mappings) investigates the degree to which the image-text mappings have a cyclical consistency. Although existing models do not train for this consistency explicitly, for a subset of models this cyclic consistency is enforced. In terms of application, the authors find that the measure of cycle-consistency correlates relatively well with downstream accuracy — which can help perform quick tests on the capabilities of the model without requiring a curated benchmark.  Overall, I believe that the paper is insightful, but lacks a strong application using those insights except for an approximate performance check for downstream tasks.

strengths:
Below I state the strengths and weaknesses of the model: 

Strengths:

- The paper is a thorough empirical study on the cycle-consistency of image-text representations across most of the popular T2I and I2I models. The results though not super surprising (as they are often trained / fine-tuned on the similar training sets) are well documented and can be crucial for the community. 
- The observation regarding the correlation between image-text cyclical consistency and downstream task accuracy can be useful to quickly check the effectiveness of the model.  One question regarding this: In Fig. 4, SDXL-Turbo is used as a decoder for the image reconstruction case and LLaVa-1.5-13B for text generation. How does this design choice affect the correlation between cycle consistency and downstream performance? The authors should ideally provide some ablation on this design choice.

weaknesses:
Weaknesses:
- The application of using cycle-consistency as an approximate measure for downstream task accuracy is an interesting use-case; However, I believe they are proxy for only two tasks (Image captioning and T2I generation performance). To be useful in practice, I suggest the authors to add in more tasks concerning these models (e.g., VQA amongst others) and check if cycle-consistency can still be an approximate measure of task accuracy. 
- I find the Sec.(5) to be intriguing, but the authors should highlight how some of the takeaways be used towards training / fine-tuning models with better downstream capabilities. 
- The authors select CIDEr score for captioning performance measurement; Have the authors considered using a strong MLLM for measuring captioning performance and using it to measure the correlation with?
- (This is not a weakness - but a discussion point) — Based on the insights, what do the authors think about building unified image-to-text and text-to-image models while enforcing cyclical consistency? Will it lead to better downstream performance than training these models independently.

questions:
Overall, this paper is a nice empirical study on cyclical consistency of image-text mappings, but I would urge the authors to respond to the Weaknesses during the rebuttal. I am open to improving the score based on the rebuttal discussion. Looking forward to the discussion. 

See Weaknesses for additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730284671472

thread_type:
official_review

<thread_object>
creator: reviewer_op89

<content>
summary:
In this work, the authors explored about the multi-modal cycle consistency of current text-to-image(T2I) and image-to-text(I2T) generation models. 

They paired various off-the-shelf T2I and I2T models to build a cycle model and measured the input-output difference.  They found that current state-of-the-art models possess a certain level of perceptual cycle consistency, even when they're not explicitly trained with cycle consistency objectives.
Then, they argued that as the performance of the individual T2I/I2T module increases, the cycle consistency improves.

To further analyze and find possible factors that can affect to achievement of cycle consistency, the authors suggested the concept of 'divergence' in T2I mappings. And they claimed that more detailed and informed text prompts showed more divergent output space, yet improved cycle consistency.
Finally, the authors demonstrated that a slight perturbation of text input sometimes results in higher variation in the T2I model output, which could be a challenge to achieve better cycle consistency.

strengths:
-Considering the current generative models become more challenging to inject cycle consistency because of their iterative sampling process, their behavior on cycle consistency is an interesting question.

-The script is well-written and clearly presents its claim.

weaknesses:
-Some experiments are not well-designed, which makes the corresponding findings seem to lack contributions or doubtful.

1. Section 3 stated to demonstrate "more cycle-consistent models have better T2I/I2T downstream performance",  but its content only shows that "better T2I/I2T models are more cycle-consistent", which are not the same.
It seems too natural that combining better T2I&I2T models improves cycle-consistency of the pair, since they provide high-quality data that contains major information of the input. On the other hand, it's still questionable that satisfying cycle consistency guarantees better T2I&I2T performance.
(e.g. A perfect Image->Text->Image reconstruction can be achieved if the I2T model writes down all pixel values in one long string. A perfect Text->Image->Text reconstruction can be achieved if the T2I model produces the image that contains the entire input text visually.)

2. In Figure 6, synthesized input captions with fewer tags don't seem to actually contain less information. In the first row, the input caption for 1 Tag is very long and specific, more detailed than 2~5 Tag captions. In the second row, the 1 Tag caption already contains the info of the second tag "reflects". This could be the reason that the divergence decreased with fewer tags, since better cycle consistency (more tags) coming with more divergence seems counter-intuitive.

questions:
-In Table 1\~2, the presented values alone are not enough to tell if each I2T+T2I model pair has a good cycle consistency since there's no baseline performance or threshold was suggested. Although the authors showed several cases in Figure 2\~3, could the authors provide any kind of baseline scenario for comparison?

-Since the sampling process image-to-text models can be also stochastic, could you also provide the analysis on the divergence of I2T models?

-What does the analysis of the divergence and sensitivity of I2T models suggest for creating more cycle-consistent models? It would be better if there's a clearer statement about how the results on divergence and sensitivity imply about cycle consistency.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590281336

thread_type:
official_review

<thread_object>
creator: reviewer_31tl

<content>
summary:
This paper presents several intriguing phenomena regarding the cycle consistency of image-text mappings with text-to-image models and image-to-text ones. It demonstrates (1) that more advanced models achieve better cycle consistency; (2) a strong correlation between cycle consistency and tasks such as image captioning and text-to-image generation; (3) that the number of objects described by the text affects text-to-image mappings; and (4) that text-to-image models are sensitive to prompt variations.

strengths:
- The paper is well presented and easy to read.
- The paper demonstrates extensive experiments incorporating multiple combinations of T2I and I2T models.

weaknesses:
Major issues:
- Regarding Table 1 and 2, the analysis of why image-to-text models have greater impact than text-to-image models is hand-wavy. There should be more discussion on this.
- Causality Direction: While the paper demonstrates correlation between cycle consistency and model performance (captioning/generation), it fails to address the causality direction. The improved cycle consistency is likely a consequence of better model capabilities rather than a contributing factor, which diminishes the practical utility of cycle consistency as a metric.
- The claim that "more descriptive text leads to a a broader distribution of generated images" is not convincing. The experiments does not properly control the caption length. Figure 6 shows a case where the 1-tag caption exceeds the 5-tag caption in length.
- The abstract makes several claims that aren't supported by the paper's content:
  - "analyze in what ways they (cycle consistency) fail": there are no such discussions in the paper.
  - "how it affects achieving cycle consistency": there are no such discussions in the paper.
  - "we show possible challenges of training cycle consistent models due to the sensitivity of text-to-image models": there are no explorations of training cycle-consistent models in the paper.

Minor issues:
- "more descriptive text leads to a a broader distribution of generated images" has double "a".
- On the 4th line in page 4, the sentence "Therefore, examine how a text-to-image mapping can diverge from one fixed text prompt into many different images." is incomplete.
- At the end of page 8, "Table 5" should be "Table 6".

questions:
- Is image-text cycle consistency a meaningful metric for model development? Should improving cycle consistency be a priority for model designers? What are the concrete applications or benefits of enhanced cycle consistency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701546512

thread_type:
official_review

<thread_object>
creator: reviewer_qccd

<content>
summary:
This paper analyzes the cycle consistency of current image/text generative models, which represents how well the original input is preserved when it consecutively passes through two generative models. To quantify the cycle consistency of images and text, the authors use metrics that measure perceptual similarity and present results for various combinations of image and text generative models. Using several benchmarks, the authors suggest that cycle consistency alone can imply the performance of models on downstream tasks by showing a high correlation between the two, thereby eliminating the need for creating additional test sets.

strengths:
- The paper presents various quantitative analysis results on the proposed claim while also visualizing various qualitative results.
- The authors analyze a less explored aspect of generative models and provide insights into its significance.

weaknesses:
- There is little analysis on the differences between the models used to measure cycle consistency. The paper simply mentions that recent models perform better, without analyzing whether the differences stem from their training data, objective functions, specific architectures, etc. Authors could have provided a table summarizing these differences and discussed how these factors may contribute to the observed performance differences in cycle consistency.
- In sections 4 and 5, it is unclear what message the authors are trying to convey. It is ambiguous how these sections relate to the cycle consistency discussed in sections 2 and 3. Authors could have better linked these sections to the overall narrative, such as explicitly stating how the divergence in text-to-image mappings (Section 4) and sensitivity in image-to-text mappings (Section 5) impact or relate to cycle consistency.

questions:
- When calculating cycle consistency for each modality, one of two generative models is fixed. (SDXL Turbo for image cycle consistency / LLaVA 1.5-13b for text cycle consistency) Would results show the same trend if the fixed models were changed?
- If richer and more detailed data improves cycle consistency, can we say that recent models show better performance because they use quality data? It could lead to valuable insights if authors examined the training data characteristics of the better-performing models to see if there's a correlation with data quality, and discussed how this relates to cycle consistency performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708350286

thread_type:
official_review

<thread_object>
creator: reviewer_go1x

<content>
summary:
The paper analyzes the cycle consistency of image-to-text and text-to-image models. The study shows that while current models exhibit a level of cycle consistency, there is room for improvement, especially T2I models are sensitive to slight changes in prompts.

strengths:
1. The paper focuses on interesting topics about the cycle consistency of by analyzing the cycle consistency of T2I and I2T models.
2. It provides a comprehensive analysis of cycle consistency in image-to-text and text-to-image mappings, highlighting the correlation between cycle consistency and downstream performance in tasks such as image captioning and text-to-image generation.

weaknesses:
1. Although the paper presents that T2I models are more sensitive to small changes in input prompts, it lacks an in-depth analysis of why different combinations of T2I and I2T models yield varying performance. For example, how does the training dataset affect the cycle consistency? How does the pre-trained model in T2I or I2T affect the cycle consistency? 
2. The paper does not sufficiently analyze why specific combinations of I2T and T2I models perform differently in terms of image and text cycle consistency. For example, BLIP2 underperforms compared to LLaVA1.6 in image cycle consistency while surpassing it in text cycle consistency.
3. The analysis in the paper highlights that text-to-image models are highly sensitive to slight changes in prompt structure (word choice, order, and length), which can lead to inconsistencies. However, the paper stops short of proposing concrete solutions or mitigation strategies for this issue.
4. The evaluation conducted solely on 1k MS COCO data is limited, especially since MS COCO captions often lack detailed descriptions of the images

questions:
1. Recent research shows the hallucination problems in Multimodal LLM and compositional problems in T2I. How can the proposed method avoid this issue? For example, an input prompt could result in the generation of an incorrect image, which might then lead to an MLLM producing captions that are incorrect but resemble the original prompt. In this case, the cycle consistency might be high, but the actual performance should be low.
2. What is the cycle consistency on long captions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732792798001

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response to Reviewers

comment:
We thank the reviewers for their helpful feedback and thoughtful insights. We are pleased that the reviewers found our topic **interesting** (go1x, oP89, QcCd), our analysis **comprehensive and thorough** (go1x, 31TL, BHgf), and **well presented** (31TL, oP89).

Firstly, we would like to clarify the goal of our paper is to present an **empirical study of cycle consistency in image-text mappings**. We observe growing cycle consistency across a wide range of image-to-text and text-to-image models, i.e., images and text are becoming increasingly interchangeable in their representations. We analyze 1) what factors are driving this trend, 2) what kinds of images and texts are exchangeable (i.e., cycle-consistent), 3) and sources of variance in cycle consistency.

We summarize our main updates to the paper as follows:

### Paper Reorganization:
**Section 3: Analysis on factors contributing to cycle consistency**.
* As suggested by go1x, QcCd, 31TL, we have included an analysis of factors contributing to cycle consistency. 
* We find increasing cycle consistency with LLM scale and with high-quality dataset re-captioning.

**Section 4: Properties of cycle-consistent texts and images.**
* As mentioned by go1x, we include analysis on object hallucination in text (Section 4.3).
* As suggested by oP89, 31TL, QcCd, we study cycle consistency as a function of caption length (Section 4.4). 

**Section 5: Variance in cycle consistency** (formerly called divergence/sensitivity). 
* As suggested by QcCD, oP89, we relate Sections 4 and 5 to cycle consistency by analyzing how random seed selection, temperature sampling, and prompt and caption style cause variance in cycle consistency.
* Unlike the previous experiment, we address the effect of caption length separately in Section 4.4, and focus on sources of variance in this section. 

### Dataset Updates:
* As suggested by go1x, we substitute COCO with the **Densely Captioned Images** (DCI) [1] dataset and update cycle consistency results accordingly. The DCI dataset features **higher resolution images** annotated with more **detailed captions** compared to COCO, improving the quality of our analysis.

### Model Updates:
* To analyze factors driving cycle consistency, we select image-to-text models with **disclosed** architecture, scale, and dataset details and weights.

[1] A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions, Urbanek et al., CVPR 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received ratings of 5, 5, 5, 3, 3, and was unanimously recommended for rejection by all reviewers.

The paper study the cycle consistency of image-text mappings using combinations of text-to-image (T2I) and image-to-text (I2T) models. The authors observe a positive correlation between cycle consistency and certain model attributes such as descriptiveness, prompt-following ability, and reduced hallucination. They further evaluate factors affecting cycle consistency, including language model scale, dataset quality, and prompt style.

While this work provides an empirical exploration of cycle consistency in multimodal mappings, the primary contribution is based on observations rather than novel methods. It analyzes the correlation between cycle consistency and downstream task performance but does not focus much on methodologies or conclusive insights for improving model performance beyond existing literature.

Strengths:
- This study tackles an less-explored area, offering a detailed empirical analysis of cycle consistency in multimodal systems.
- The analysis spans multiple off-the-shelf models and considers factors like language model scale, dataset quality, and variance in prompts.
- The experiments are clearly presented with adequate visualizations and metrics.

Area for improvements:
- Limited novelty - the paper primarily presents observations without introducing a novel methodology or theoretical contribution. Many conclusions reiterate well-understood dependencies (e.g., high-quality datasets improve model performance).
- Although cycle consistency is analyzed, the paper lacks clear evidence or arguments for its practical significance in real-world applications.
- Major revisions during the rebuttal phase introduced significant changes to the experiments, conclusions, and structure of the paper. This makes it challenging to assess the original contribution versus the revised content.
- Insufficient depth in causal analysis. Thsi paper does not convincingly establish whether cycle consistency is a cause of improved performance or merely a byproduct of better models.

Despite its merits, the submission falls short in terms of methodological novelty, practical utility, and conclusive contributions to the field. It does not meet the bar for acceptance at ICLR. We encourage the authors to address the feedback provided in the reviews and consider submitting a revised version in the future.