TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models

Authors: Leigang Qu, Haochuan Li, Tan Wang, Wenjie Wang, Yongqi Li, Liqiang Nie, Tat-Seng Chua
Primary area: generative models
Keywords: Multimodal Large Language Models, Text-to-Image Generation, Cross-Modal Retrieval

Abstract:
How humans can effectively and efficiently acquire images has always been a perennial question. A classic solution is *text-to-image retrieval* from an existing database; however, the limited database typically lacks creativity. By contrast, recent breakthroughs in *text-to-image generation* have made it possible to produce attractive and counterfactual visual content, but it faces challenges in synthesizing knowledge-intensive images. In this work, we rethink the relationship between text-to-image generation and retrieval, proposing a *unified* framework for both tasks with one single Large Multimodal Model (LMM). Specifically, we first explore the intrinsic discriminative abilities of LMMs and introduce an efficient generative retrieval method for text-to-image retrieval in a training-free manner. Subsequently, we unify generation and retrieval autoregressively and propose an autonomous decision mechanism to choose the best-matched one between generated and retrieved images as the response to the text prompt. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both creative and knowledge-intensive domains. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, *i.e.*, Flickr30K and MS-COCO, demonstrate the superiority of our proposed framework.

Decision: accept (poster)
Reviewer scores: 5, 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730381200385

thread_type:
official_review

<thread_object>
creator: reviewer_wzfq

<content>
summary:
This paper rethinks the relationship between text-to-image generation and retrieval, presenting a novel and pragmatic task, i.e., TIGeR, to meet the challenging information need of humans in real-world scenarios. The authors identify limitations in traditional retrieval methods, which rely on existing image databases, and in generation approaches that struggle with knowledge-intensive content. Motivated by these limitations, the authors proposed a novel unified framework called TIGeR-ONE that combines both tasks using a single LMM. To endow LMMs with the retrieval abilities, they introduce a training-free generative retrieval method that leverages LMM's discriminative abilities, allowing for efficient text-to-image retrieval. Additionally, an autonomous decision mechanism selects either generated or retrieved images as the most suitable response to a given text prompt. The authors develop TIGeR-Bench, a benchmark covering creative and knowledge-intensive image domains, to standardize evaluation. Their approach demonstrates superior performance on the unified benchmark, text-to-image and chat-to-image retrieval and generation benchmarks.

strengths:
1.	The authors introduce an innovative task that combines retrieval and generation, addressing a gap in traditional approaches that often consider these processes separately. This integrated task is highly practical, as it reflects real-world information needs where users may require either a retrieved image from a database or a newly generated one, depending on the context.
2.	The authors propose a novel unified framework that unifies text-to-image generation and retrieval within a single LMM, streamlining both tasks into a cohesive process. By using a single LMM, the authors simplify the model architecture, allowing the system to handle both retrieval of existing images and generation of new ones without additional model switching or training steps.
3.	The authors introduce a training-free generative retrieval method that enhances both the effectiveness and efficiency of text-to-image retrieval. The proposed method leverages the pre-existing discriminative abilities of Large Multimodal Models (LMMs) to perform retrieval without the need for extensive training on large datasets. 
4.	They build a unified benchmark, i.e., TIGeR-Bench, to standardize evaluation across creative and knowledge-intensive tasks, serving as a strong foundation to attract more researchers to explore complex multimodal information acquisition.

weaknesses:
1.	Considering there has been prior work (i.e., GILL) addressing text-to-image retrieval and generation, the difference and advantages of the proposed method over GILL should be further highlighted clearly.  
2.	The paper lacks analysis on how the unified model’s performance varies with prompts that have similar intentions but are expressed differently. In practical use, different users may convey the same idea with varied wording, prompts, and questions. How sensitive is the proposed method to these variations? 
3.	Given that the distribution over creative domains and knowledge domains may not be uniform in real-world scenarios, an analysis of the model’s performance and decision-making behavior under unbalanced distribution conditions would be beneficial.

questions:
1.	Is there any experiment demonstrating the decision behavior of the proposed model? 
2.	Compared to dense retrieval methods, such as CLIP, what advantages do MLLMs and the proposed generative retrieval framework offer?

The authors have solved my concerns.
I stand by my positive score, after reading other reviewers' comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730450912915

thread_type:
official_review

<thread_object>
creator: reviewer_xrkp

<content>
summary:
1. Explore the retrieval capabilities of Large Language Models (LMMs) to enable text-to-image (T2I) generation retrieval.
2. Unify generation and retrieval in an autoregressive manner.
3. Propose an autonomous decision mechanism to select the best-matched image from generated and retrieved images.
4. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both 5. creative and knowledge-intensive domains.
6. Conduct extensive experiments on TIGeR-Bench and two retrieval benchmarks, namely Flickr30K and MS-COCO.

strengths:
1. Propose a unified framework for text-to-image generation and retrieval.
2. Propose an autonomous decision mechanism to automatically select the most suitable image from the retrieved images.
3. To validate the capability of unified generation and retrieval, introduce a new benchmark, TIGeR-Bench.
4. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the pipeline's capabilities.
5. The article is well-written, with clear and precise explanations.

weaknesses:
1. The challenges faced by generation models in knowledge-intensive tasks are mentioned, but the proposed approach’s effectiveness in consistently addressing these challenges without hallucination remains unclear. Explicit quantitative analysis on knowledge domains might be helpful.
2. Can TIGeR-ONE adapt to user-specific preferences in the decision-making process, especially in creative tasks? If not, how could this be incorporated to enhance user experience?
3. I noticed the multi-round generation in Figure 7. Could you please explain how this process maintains consistency with the user-provided image?

questions:
As shown in Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730489116515

thread_type:
official_review

<thread_object>
creator: reviewer_kruz

<content>
summary:
This paper introduces a unified framework for text-to-image generation and retrieval using LMM. It proposes an efficient method leveraging LMMs' discriminative abilities. It also presents an autonomous decision mechanism for selecting between generated and retrieved images. The authors also developed TIGeR-Bench to evaluate generation and retrieval abilities across creative and knowledge-intensive domains simultaneously.

strengths:
- This paper is well-written and easy to follow. 
- Efficient retrieval and accurate decision-making are crucial in unifying generation and retrieval, balancing creativity and knowledge. The proposed TIGER-One solves these two problems within one LMM is elegant.
- The proposed TIGER-One seems effective and shows superior generation results in experiments.

weaknesses:
- A  preliminary introduction of beam search is recommended to add.
- The authors indicate that the similarity score is calculated by one of the three proxies. More details and discussion of the proxies chosen in decision-making are recommended. 
- With the help of TIGER-One, the LMM demonstrates improved results (Tab. 2).  It’s interesting to explore the addition of dream images based on knowledge (RAG) as a third option alongside generation and retrieval in future work

questions:
see the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611239485

thread_type:
official_review

<thread_object>
creator: reviewer_vq4r

<content>
summary:
This paper studies how to effectively combine the T2I generation and retrieval into a combined framework. For this task, this paper makes two folds of contribution: 1) This paper proposes a new T2I generation and retrieval pipeline, where the generation and retrieval branches run in parallel, followed by a final image ranking/selection process to decide the final output. 2) This paper proposes a benchmark to evaluate the joint T2I generation and retrieval. The benchmark contains half creative and half knowledge based image-text pairs.

strengths:
- The overall presentation of this paper is good. This paper contains a lot of content which is hard to make the presentation organized and easy to follow. The authors did a good job in my opinion.
- Joint T2I generation and retrieval is a important task. The proposed benchmark set is key to make different methods devoted for the task comparable.
- The experimental results show that the proposed method has good performance than other generation or retrieval-based methods.

weaknesses:
- In my opinion, the biggest drawback of this paper is that it lacks technical novelty. This work looks more like a prototype of an application than a research paper. All modules used by this paper has been used by previous papers and the combination of them contributes very limited new knowledge or insight. I understand that this work belongs to a type of papers that is more engineer than research. I put this point inside the weakness section but I am open to any discussion about the fit of this paper to iclr with authors, other reviewers, and AC if possible.
- I am confused about the comparison results in Table 2. What is the difference between the SEED-LLaMA vs Ours (SEED-LLaMA)? Does the latter one used both retrieval and generation? If so, this comparison seems to be somewhat unfair. If not, what is the components of the proposed method that lead to the performance improvement over the baseline. More discussion is welcomed.

questions:
Please see the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730797913866

thread_type:
official_review

<thread_object>
creator: reviewer_x2jm

<content>
summary:
This paper presents an approach to address the limitations of current text-to-image generation (T2I-G) and retrieval (T2I-R) systems. Particularly, it propose to unify generation and retrieval auto-regressively, and use a decision making model to decide whether to use generated or retrieved image as the response to a text prompt.

strengths:
- Unification of Generation and Retrieval: The core contribution of unifying T2I-G and T2I-R within a single framework centered with LMMs. This framework leverages the strengths of both paradigms, mitigating the limitations of relying on either alone for offering visual content per user query. In an ideal world, the users can obtain a factual image generation when its query is centered at knowledge-intensive factual entity, and see a creative image when queried for imaginary scene.

- New benchmark: This paper creates a dedicated benchmark, TIGER-Bench, which accesses image generation in both creative and knowledge-intensive domains, and offers a more comprehensive evaluation platform.

weaknesses:
- Lots of details related to presentation needs improvement: 

1. Figure  1 is referring to many things that has not be introduced before, it is hard for people to understand forward beam-search & reverse re-ranking at the first glimpse. Many terms are not really explained in caption, or the introduction and people needs to leap to methodology section to really understand the meaning. 

2. Section 3.2 is not very clear to me, I couldn't find a connection one why those three metrics are used, and also why the metric it has to be training-free. Meanwhile, I also think that comparing to those proposed proxy metrics, a more straightforward way is to directly ask the LMM a visual question, i.e. "<image> Is the presented image is aligned with the following caption? <caption>. Answer yes or no.", and then measure the likelihood on yes and no. 

3. Given that the authors also agree that "It is inefficient due to |G| times of forward propagation" for computing the proposed cross-modal similarity, why not considering leveraging a contrastively trained text-to-image similarity function? The forward beam search + reverse re-ranking seems unnecessarily complicated.

- Lack of simple baseline method: It would make a lot sense to compare a simple zero-shot / fine-tuned LMM that makes a decision between whether to use the SDXL's generated image or CLIP model's retrieved image, according to a user's input query. IMO, this method would be very competitive to the proposed approach.

- The core experimental result is based on author's proposed benchmark, which is not very convincing given that it is really hard to judge the quality and validity of this newly introduced benchmark. It appears to me that the authors are being the player and judge at the same time, which is quite tricky to assess the significance of this work. 

- Missing relevant works: 

The following works have discussed how to leverage image retrieval for image generation and should be discussed and cited. 

1. Re-Imagen: Retrieval-Augmented Text-to-Image Generator
2. KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities

questions:
Please see weakness for my questions

--- Post-rebuttal comments:

I improved my scores based on authors' response.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732398978958

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Key differences between TIGeR-One and dense retrieval methods based on contrastive learning

comment:
Dear reviewers, 

Thank you for the time and effort you put into providing detailed and constructive feedback. Your insights are invaluable in helping us improve the quality of our work, and we are fully committed to incorporating your suggestions into our revision process. 
To address the common concern regarding the **key differences between the proposed TIGeR-One framework and dense retrieval methods** (e.g., CLIP and BLIP) based on contrastive learning, we would like to re-emphasize the following four points: 

a) **Elegant Unification of Generation and Retrieval**: As illustrated in Fig. 2, our forward beam search unifies generation and retrieval within an autoregressive framework. Both tasks can be performed in parallel, yielding generated and retrieved visual tokens with only one time of auto-regression. 

b) **Training-Free and LLM Compatibility**: Our training-free proxies harness intrinsic cross-modal alignment from generative pre-training with next-token prediction. Inheriting this generative manner, the beam search and ranking process does not require additional training, avoiding potential conflicts between multiple learning objectives and improving training efficiency for future fine-tuning.

c) **Inference Efficiency**: Contrastive learning methods often require significant computational overhead. Two-tower frameworks (e.g., CLIP) scale with $|\mathcal{G}|$, where $|\mathcal{G}|$ denotes the database size, while one-tower frameworks (e.g., BLIP) require  $|\mathcal{G}| \times V \times T$  operations (where V and T are visual and textual tokens). In contrast, our method scales with $N (N << |\mathcal{G}|)$, where $N$ denotes the number of visual tokens representing an image, ensuring greater efficiency.

d) **Superior Retrieval Performance**: Beyond Tab. 3, we compare retrieval performance on Flickr30K and MS-COCO between the existing dense retrieval LMM, i.e., GILL [c], and our method (see the table below). Fig. 5 further highlights our efficiency advantage. These results confirm the effectiveness and superiority of our approach in both retrieval performance and efficiency.

|                 | MSCOCO |        |        | Flickr30k |        |        |
|---|:----:|:----:|:-----------:|:------:|:------:|:------:|
|                   | R@1    | R@5    | R@10   | R@1       | R@5    | R@10   |
| GILL              | 29.10  | 53.84  | 65.26  | 58.40     | 84.48  | 90.56  |
| Ours (LaVIT)      | 44.81  | 62.61  | 68.28  | 68.84     | 82.92  | 86.44  |
| Ours (SEED-LLaMA) | **46.11**  | **69.02**  | **76.13**  | **71.70**     | **91.82**  | **95.44**  |

[c] Jing Yu Koh et al. Generating images with multimodal language models. NeurIPS’23
</content>
</thread_object>

</thread>
<thread>
creation date:
1732534274525

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of PDF Revisions

comment:
Dear Reviewers,

Thank you for your thoughtful engagement in reviewing our work and for providing valuable comments and constructive suggestions. We have carefully revised the manuscript to address your feedback. In the updated PDF, revisions are highlighted in blue for your convenience. The main updates are as follows:

1. **Figure 1** has been polished to enhance intuitiveness by removing confusing details [Reviewer x2Jm].
2. We have cited and discussed Re-Imagen and KITTEN in the **Related Work** section (lines 119–121) [Reviewer x2Jm].
3. The differences between the base models and the proposed model are now explicitly highlighted to ensure **a fair performance comparison** (lines 369–371) [Reviewer VQ4r].
4. **A preliminary introduction to beam search** has been added (lines 251–257) [Reviewer KRUZ].
5. Highlight how our model maintains **identity consistency** with the user-provided image (lines 485 - 530)[Reviewer xRkP]
6. Add the **agent baselines** in Tab 2 and corresponding discussions in lines 371–374 [Reviewer x2Jm]

 We believe these revisions address your concerns and help clarify any ambiguities or misunderstandings. As the rebuttal deadline (Nov 26, AOE) approaches, we sincerely welcome your feedback and look forward to further discussions.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a framework to unify text-to-image generation and retrieval by running the generation and retrieval branches, followed by a image selection process for the final output. 

As reviewers pointed out, the major strengths of this paper are:
1) The idea of unifying text-to-image generation and retrieval is practically quite important, the proposed framework of using forward beam search is solid.
2) A benchmark to evaluate the joint T2I generation and retrieval is also provided. 
3) The proposed TIGER-One shows superior results in experiments

The major weaknesses are:
1) The author's evaluation is specific to the method they proposed.  The evaluation can be further improved by including evaluation under more realistic scenarios.
2) As shown in authors' rebuttal, a zero-shot prompting LLM-based decision model (Qwen2-VL) that decides whether or not using generated or retrieved result seems to get only slightly worse results than the much more complicated proposed system.
3) Multiple reviewers pointed out some technical parts can be presented more clearly. The current presentation is not very easy to understand.

Overall, I think the direction of unifying text-to-image generation and retrieval is very practical and the proposed method is overall solid. So I would encourage this work and recommend "Accept as a poster".