TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models

## Abstract

Abstract How humans can efficiently and effectively acquire images has always been a perennial question.
A typical solution is text-to-image retrieval from an existing database given the text query; however, the limited database typically lacks creativity.
By contrast, recent breakthroughs in text-to-image generation have made it possible to produce fancy and diverse visual content, but it faces challenges in synthesizing knowledge-intensive images.
In this work, we rethink the relationship between text-to-image generation and retrieval and propose a unified framework in the context of Multimodal Large Language Models (MLLMs).
Specifically, we first explore the intrinsic discriminative abilities of MLLMs and introduce a generative retrieval method to perform retrieval in a training-free manner.
Subsequently, we unify generation and retrieval in an autoregressive generation way and propose an autonomous decision module to choose the best-matched one between generated and retrieved images as the response to the text query.
Additionally, we construct a benchmark called TIGeR-Bench, including creative and knowledge-intensive domains, to standardize the evaluation of unified text-to-image generation and retrieval.
Extensive experimental results on TIGeR-Bench and two retrieval benchmarks, i.e. , Flickr30K and MS-COCO, demonstrate the superiority and effectiveness of our proposed method.

## 1 Introduction

The explosion of visual information on the Web significantly challenges human information access.
Text-to-Image Retrieval (T2I-R) [, , ] is one of the main channels to efficiently obtain visual information given a text query.
However, T2I-R is limited to retrieving existing images in the database, lacking flexibility and creativity.
Recent years have witnessed thrilling progress in Text-to-Image Generation (T2I-G) [, , ],
which directly generates new images to satisfy human’s visual information needs.
However, T2I-G falls short of handling knowledge-intensive concepts, *e.g.*, landmarks and natural species, leading to the hallucination problem [, ].
In this light, a single T2I-R or T2I-G model may not satisfy diverse and evolving human information needs.
It is pivotal to unify both T2I-R and T2I-G within a framework for visual information delivery.

Figure: Figure 1: We consider both creative domains and knowledge-intensive domains for general image acquisition towards unified Text-to-Image Generation and Retrieval (TIGeR), and build the TIGeR-Bench including 8 domains for the unified comprehensive evaluation.
Refer to caption: x1.png

To this end, a straightforward solution is to empower discriminative models with the generation ability.
However, the early-stage trial (*e.g.*, JEM []) requires extra generative training and may compromise the original discriminative power.
Another recent attempt [, ] is to adapt generative diffusion models [] to achieve discriminative tasks.
Despite the significance, these methods are limited to diffusion models and inevitably suffer from the notorious inefficiency problem caused by iterative denoising [].
Moreover, diffusion models are usually tailored for simple discriminative tasks such as image classification [, ] and it is difficult for diffusion models to process complex human intentions for large-scale retrieval in practical scenarios [].

Different from diffusion models,
Multimodal Large Language Models (MLLMs) offer another form of generative paradigm to solve broad vision-language problems, garnering significant attention for their powerful intention understanding and instruction following abilities [, ].
Recently, notable efforts [, , , , ] incorporate MLLMs with external T2I-G models [, ] for image synthesis.
Nevertheless, these works only consider T2I-G without T2I-R.
Even though GILL [] fuses an LLM and the image encoder & decoder to achieve the unified ability of generation and retrieval, it highly depends on an external retriever (*i.e.*, CLIP []) for the discriminative ranking in the dedicated embedding space.
As such, the inefficiency problem [, ] of the dense retrieval paradigm still hinders large-scale image retrieval of GILL.

In this work, we propose to elegantly unify Text-to-Image Generation and Retrieval (TIGeR) within MLLMs, a promising task to achieve more flexible image acquisition, as shown in Fig. 1.
To this end, we first delve into the intrinsic bidirectional (*i.e.*, text-to-image and image-to-text) discriminative abilities of MLLMs in Sec. 3.2.
Specifically, we study the potential of bidirectional conditional likelihoods as the proxies of semantic similarity by assessing the performance in image-text matching.
Based on the discriminative abilities, we present an efficient generative retrieval method with forward beam search and reverse re-ranking as in Sec. 3.3, unifying both T2I-R and T2I-G in an autoregressive generation manner.
Besides, we propose an autonomous decision mechanism to adaptively choose one from retrieved and generated images according to user prompts.
To comprehensively evaluate the performance of MLLMs on TIGeR, we build a benchmark (TIGeR-Bench) in Sec. 4 from two aspects, one for creative images from the counterfactual world or users’ imaginative prompts [], and the other is derived from six knowledge-intensive domains (*e.g.*, news, landmark, and natural species).
Additionally, we carry out extensive experiments to assess the TIGeR performance of representative MLLMs, verifying the effectiveness of the proposed method.
Overall, we summarize the contributions into three points.

- •
Driven by diverse and complex practical scenarios for visual information access, we propose to unify text-to-image generation and retrieval in the context of MLLMs.
- •
We comprehensively inspect the intrinsic cross-modal discriminative abilities of MLLMs and propose a training-free generative retrieval approach based on bidirectional discrimination, thus unifying text-to-image generation and retrieval in an autoregressive generation manner.
And we present an autonomous decision mechanism to adaptively select retrieved or generated images.
- •
We construct a comprehensive image acquisition benchmark, including creative and knowledge-intensive domains, to evaluate the TIGeR performance of existing MLLMs. Extensive experiments on TIGeR-Bench and two T2I-G benchmarks including Flickr30K [] and MS-COCO [] verify the effectiveness of the proposed framework.

## 2 Related Work

Text-to-Image Generation.
T2I-G has aroused wide attention in both academia and industry over the past decade, with advancements ranging from Generative Adversarial Networks [] to Auto-regression Models [, ] and Diffusion Probabilistic Models (DPMs) [, ]. Recent breakthroughs in DPMs, guided by the scaling law [, ], have propelled T2I-G to new heights, *e.g.*, models like DALL-E 2 [] and DALL-E 3 [], the Imagen series [], and the Stable Diffusion (SD) series [, , ].
Concurrently, efforts have been made to enhance composed text-image alignment [, , , ] and cater to human preference [, , ]. However, prior works predominantly focus on creative domains for image generation, neglecting the imperative of knowledge-intensive image acquisition tailored to human needs.

Multimodal Information Retrieval.
Early studies on multimodal information retrieval focused on feature representation [, ] and modality interaction [, ] for precise cross-modal similarity estimation. Recent advancements, propelled by large-scale pre-training, have led to improved retrieval performance and generalization in vision-language models [, , ]. More recently, researchers have explored more challenging scenarios, such as fine-grained interaction [], equivariant similarity [], multimodal instruction following [], and chat-based IR []. Despite thrilling progress, retrieval systems are inherently limited by database size, and incapable of creating new visual content.

Multimodal Large Language Models.
Empowered by the versatility and flexibility of LLMs, pioneering works on MLLMs have shown impressive understanding capabilities [, ].
Recent research explores image generation through two categories: 1) Continuous Visual Representation methods aim to align visual representations from LLMs with condition embeddings of SD through regression [, , , , ] or score distillation [] objectives. 2) Discrete Visual Tokenization methods [, , , ] first encode an image into a sequence of discrete codes [, , , ], and then employ next-token prediction to train MLLMs. To synthesize images, the discrete codes are decoded into the pixel space via VQ-GAN or SD. In this work, we resort to the discrete paradigm to be consistent with the inherent discreteness of language.
Compared with the existing work, our TIGeR method achieves comprehensive image acquisition, encompassing both content creation and knowledge retrieval within a unified framework.

## 3 Methodology

In this section, we first formulate the unified problem in Sec. 3.1, and then explore the intrinsic cross-model discriminative ability of MLLMs for semantic similarity measurement in Sec. 3.2. Based on the ability, we propose a unified framework, as shown in Fig. 2, including generative retrieval in Sec. 3.3, synchronous generation and retrieval, and decision-making in Sec. 3.4.

### 3.1 Problem Formulation

The goal of this work is to satisfy complex human visual information needs by acquiring images from diverse domains in a unified MLLM framework. We formulate this problem autoregressively:

$$ $p(Y|X)=\prod_{i=1}^{N}p(y_{i}|Y_{<i},X),$ (1) $$

where $X$ denotes a textual prompt provided by humans, tokenized into a sequence $X=[x_{1},...,x_{M}]$.
$Y=[y_{1},...,y_{N}]$ denotes the sequence of visual tokens that can be decoded into an image, with $Y_{<i}$ referring to the tokens before $i$-step. By sampling from the conditional distribution, we obtain a sequence instance, *i.e.*, $Y^{*}\sim p(Y|X)$, where $Y^{*}\in\mathbb{V}^{N}$ and $\mathbb{V}$ denotes the visual token space defined by a visual vocabulary, and $|\mathbb{V}|=V$ is the vocabulary size. $\mathbb{V}^{N}$ denotes the Cartesian product of $N$ token spaces, *i.e.*, the whole discrete visual space.
To achieve unified generation and retrieval, an MLLM is required to possess the following three capabilities:
$\bullet$ Creativity to generate novel photorealistic images by sampling from $p(Y|X)$,
$\bullet$ Discrimination to match vision and language and retrieve the relevant images from an image database or gallery $\mathcal{G}$,
$\bullet$ Decision to automatically determine the superior option between generation and retrieval.

Considering the proficiency of recent MLLMs [, , ] in creative text-to-image generation, we will focus on exploring the discriminative ability for text-to-image retrieval and unified generation and retrieval with decision-making in the following.

### 3.2 Intrinsic Discriminative Ability of MLLMs

For effective text-to-image retrieval, LLMs should possess a strong discriminative ability which is fundamental for accurate semantic similarity estimation between the prompt and images in the database. To probe such intrinsic capability, we will introduce three training-free proxies as follows.

**Table 1: Text-to-image ranking performance of three similarity (Sim.) proxies for SEED-LLaMA [] and LaVIT [] on MS-COCO [].**
| Sim. Proxy | SEED-LLaMA | LaVIT |  |  |
| --- | --- | --- | --- | --- |
| R@1 | R@5 | R@1 | R@5 |  |
| Random | 0.02 | 0.10 | 0.02 | 0.10 |
| $\log p(Y|X)$ | 3.50 | 8.79 | 0.02 | 0.16 |
| $\log\frac{p(Y|X)}{p(Y)}$ | 26.25 | 54.57 | 23.43 | 48.52 |
| $\log p(X|Y)$ | 49.34 | 75.45 | 50.35 | 70.87 |

To estimate the semantic similarity $s(X,Y)$ between a given text prompt/query $X$ and an image $Y$, a straightforward approach is to employ the conditional likelihood based on autoregressive factorization as the proxy:

$$ $s(X,Y)=\log p(Y|X)=\sum_{i=1}^{N}\log p(y_{i}|Y_{<i},X),$ (2) $$

where $p(Y|X)$ denotes the likelihood of an image $Y\in\mathcal{G}$ from the database conditioned on the given prompt $X$. In practice, we can attain it by computing the cross entropy between the predicted logits and the image tokens.
However, we observed this proxy performs weakly in practice, as shown in Tab. 1. Similar phenomena also happened in recent studies [, ] for diffusion and captioning models.
We attribute this problem to the modality bias, including the visual bias for T2I-G models and the linguistic bias for image captioning models, respectively.
In this study, the visual bias largely stems from the unbalanced distribution, *i.e.*, $p(Y)$, and thus we introduce an alternative proxy as,

$$ $s(X,Y)=\log\frac{p(Y|X)}{p(Y)^{\eta}}=\sum_{i=1}^{N}\log p(y_{i}|Y_{<i},X)- \eta\sum_{i=1}^{N}\log p(y_{i}|Y_{<i}),$ (3) $$

where we use the visual prior $p(Y)$ to help debiasing with a strength factor $\eta$, which can be estimated by $p(Y|\bar{X})$. $\bar{X}$ refers to a special prompt without any descriptive content, *e.g.*, a null character or “Can you give me an image?”.

In addition to the debiasing strategy, we propose another option to circumvent the unbalanced prior distribution. Different from the generation process without the image $Y$, the retrieval task allows the model to access all images in the database, which means we can estimate the semantic similarity in a reverse way by predicting the conditional likelihood of another modality, *i.e.*, $X$, with:

$$ $s(X,Y)=\log p(X|Y)=\sum_{i=1}^{M}\log p(x_{i}|X_{<i},Y).$ (4) $$

To perform this reverse calculation, we view MLLMs as image captioners.

The above three training-free proxies of semantic similarities could be calculated in parallel because all tokens have been accessible beforehand.
After calculating the similarities between the given textual prompt and all images from the database, we could sort them to attain the ranking list.
Compared with the prior work GILL [], this method is seamlessly integrated with generative pre-training, eliminating the need for additional discriminative training, such as contrastive learning, and benefits from fine-grained cross-modal interaction within MLLMs.

Figure: Figure 2: Overview of the framework to unify text-to-image generation and retrieval. Images from the database are first tokenized into discrete codes and a lookup table is maintained for the correspondence between discrete codes and images. The given prompt $X$ is first fed into a MLLM and Forward Beam Search is performed to retrieve and generate images in parallel. The prompt and obtained images are then fed into the same MLLM for Reverse Re-Ranking and Decision-making.
Refer to caption: x2.png

### 3.3 MLLMs-based Generative Retrieval

The above proxies make it possible to calculate cross-modal similarities and carry out ranking based on which.
However, it is inefficient since the requirement of $|\mathcal{G}|$ times of forward propagation of MLLMs, each time with the extensive parameterization and the intricate attention interaction.
To tackle this issue and achieve an optimal balance between efficiency and recall, we introduce forward beam search and reverse re-ranking, as shown in Fig. 2.

Forward Beam Search.
Inspired by the advancement of generative retrieval [, ], we adopt constraint generation via autoregressive token decoding and beam search [] with the beam size $B$ to recall a fixed number of images.
Specifically, we compress all images in the database into discrete tokens and store them in a Trie structure. This Trie structure constrains the sampling space and ensures that the generated prefix at any timestep corresponds to at least one image in the database. Once the beam search is finished, we could obtain a ranking list of $B$ sequences of visual tokens, each of which corresponds to an image in the database.
This process aims to obtain a list of images given the prompt, thus the direction has to be $X\rightarrow Y$, which means we can only adopt the two forward proxies illustrated in Eqn. 2 or Eqn. 3. However, the forward beam search method could significantly improve the efficiency since it only requires $N(N\ll|\mathcal{G}|)$ times of forward propagation of MLLMs. $N$ denotes the length of the visual token sequence for an image.

Reverse Re-Ranking.
Despite the improved efficiency, the semantic matching accuracy of the two forward proxies is obviously weaker than that of the reverse one in Eqn. 4, as shown in Tab. 1. Given the ranking list $\mathcal{R}=[Y_{1},...,Y_{B}]$ obtained by forward beam search, we resort to the reverse proxy in Eqn. 4 for re-ranking and attain the final ranking list $\mathcal{R}^{*}$.

### 3.4 Unifying Generation and Retrieval

The discrete visual tokenization strategy enables MLLMs to generate both language and visual content in an autoregressive generation manner. The proposed forward beam search in Sec. 3.3 performs retrieval by autoregressively generating visual tokens one by one. Naturally, we can unify generation and retrieval based on MLLMs under the same autoregression framework and make a decision between the generated and retrieved images to meet complex real-world requirements.

Synchronous Generation and Retrieval.
In this unified framework, MLLMs can synchronously conduct unconstrained and constrained token decoding for image generation and retrieval, respectively.
As shown in Fig. 2, these two tasks can be performed in parallel by maintaining respective search paths, which also requires $N$ times of forward propagation.
Each path corresponds to a sequence of discrete visual tokens. We can generate a new image $Y^{G}$ by a diffusion decoder conditioned on the sequence, and meanwhile, immediately find the retrieved top-1(^1^11This work only considers one generated image and the top-1 retrieved image, but the proposed framework can also acquire more than one images and choose the best one.) image $Y^{R}$ via the lookup table.

Decision Making.
Given the generated image $Y^{G}$ and the retrieved image $Y^{R}$, we choose the better one based on the intrinsic discriminative abilities of MLLMs, as discussed in Sec. 3.3. Specifically, we calculate two semantic similarities $s(X,Y^{G})$ and $s(X,Y^{R})$ using the three proxies mentioned above and choose the image with the higher similarity. Different from the retrieval process, which performs semantic matching within the database, this decision-making module may require stronger discriminative power from MLLMs due to the potential gap between generated and retrieved images.

## 4 TIGeR-Bench

To evaluate the performance of the unified problem, we build a comprehensive benchmark (TIGeR-Bench), as shown in Fig. 1.
It covers creative and knowledge-intensive image acquisition domains.

Creative Domains.
Human creative image requirements emphasize obtaining intricate visual content that is challenging to capture in the real world. This includes unusual and counterfactual compositions of concepts (*e.g.*, “A steamed train bellows rainbow-hued smoke”) and imaginary scenes that align with real users’ genuine creative preference.
To meet the two aspects, we collect prompt-image pairs from the well-designed WHOOPS! [] dataset and a large-scale open dataset named Pick-a-Pic [] which stems from a web platform collecting real users’ creation intention, respectively.

Knowledge Domains.
To acquire knowledge-intensive images, a model is required to have sufficient knowledge from specific vertical domains and disciplines. We focus on 6 knowledge domains including logo [], history and news [], landmark [], food [], nature species [], Wiki miscellaneous [], and collect text-image pairs from six high-quality datasets. Different from previous content-oriented image-text data [, ] including common things in daily life, the collected data focuses on the knowledge alignment across modalities, *e.g.*, the texts may only convey concept names without any descriptions in appearance.

Based on the two aspects of domains, we collect pairwise image-text data from 8 domains, constructing the evaluation benchmark containing 6,000 data samples, with 3,000 for creative domains and 3,000 for knowledge domains.
See Appendix. A for more details of data sources, construction pipeline, and extension to long prompts and chat scenarios.

**Table 2: Performance comparison on TIGeR-Bench. “Token” refers to visual tokenization during image synthesis, including continuous (Cont.) and discrete (Dist.) approaches. Entries by gray are expert models for T2I retrieval or generation, and those with a blue background denote that an image query is first generated and then used to perform image-to-image retrieval. Entries with a gray background denote our methods.**
| Method | Size | LLM | Token | CLIP-T $\uparrow$ | CLIP-I $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Text-to-Image Generation |  |  |  |  |  |
| SDXL [] | 2.6B | - | Cont. | 26.79 | 46.71 |
| GILL [] | 8B | OPT-6.7B | Cont. | 14.16 | 13.72 |
| Emu [] | 14B | LLaMA-13B | Cont. | 22.26 | 40.78 |
| Emu 2 [] | 37B | LLaMA-33B | Cont. | 24.25 | 44.24 |
| DreamLLM [] | 8B | Vicuna-7B | Cont. | 24.34 | 42.77 |
| SEED-LLaMA [] | 8B | LLaMA-7B | Dist. | 22.00 | 43.02 |
| LaVIT [] | 11B | LLaMA-7B | Dist. | 27.07 | 48.75 |
| Text-to-Image Retrieval |  |  |  |  |  |
| CLIP (ViT-B/32) [] | 151M | - | Cont. | 25.22 | 53.95 |
| SDXL [] | 2.6B | - | Cont. | 15.41 | 35.96 |
| Emu [] | 14B | LLaMA-13B | Cont. | 14.44 | 34.46 |
| Emu 2 [] | 37B | LLaMA-33B | Cont. | 14.69 | 36.38 |
| DreamLLM [] | 8B | Vicuna-7B | Cont. | 15.41 | 37.18 |
| SEED-LLaMA [] | 8B | LLaMA-7B | Dist. | 14.78 | 36.93 |
| LaVIT [] | 11B | LLaMA-7B | Dist. | 16.34 | 39.25 |
| GILL [] | 8B | OPT-6.7B | Cont. | 10.96 | 16.30 |
| Ours (SEED-LLaMA) | 8B | LLaMA-7B | Dist. | 16.95 | 40.30 |
| Ours (LaVIT) | 11B | LLaMA-7B | Dist. | 21.30 | 50.03 |
| Unified Text-to-Image Generation and Retrieval |  |  |  |  |  |
| GILL [] | 8B | OPT-6.7B | Cont. | 12.12 | 15.25 |
| Ours (SEED-LLaMA) | 8B | LLaMA-7B | Dist. | 23.98 | 50.52 |
| Ours (LaVIT) | 11B | LLaMA-7B | Dist. | 28.45 | 61.37 |

## 5 Experiments

### 5.1 Datasets, Baselines, and Evaluation Metrics

We use the constructed TIGeR-Bench to evaluate the unified performance, and the two widely-used benchmark datasets, *i.e.*, Flickr30K [] and MS-COCO [], to assess the text-to-image retrieval performance.
The compared baselines mainly include recent MLLMs [, , , , , ] which can generate images, as well as generation and retrieval expert models [, ].
Following T2I-G [, , ], the unified performance is measured by the CLIP score [] including CLIP-T for text-image alignment and CLIP-I for the alignment between the predicted image and the ground-truth image.
As for T2I-R [, ], we adopt the standard metric Recall at K, R@K (K=1, 5, and 10) for short.
Details on datasets, baselines, and evaluation metrics are provided in Appendix. C.

**Table 3: Text-to-image retrieval performance comparison on Flickr30K and MS-COCO. Entries by gray denote dense retrieval methods and others are generative retrieval methods.**
| Method | Flickr30K (1K) | MS-COCO (5K) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |
| CLIP (ViT-B/32) [] | 68.70 | 90.60 | 95.20 | 37.80 | 62.40 | 72.20 |
| GRACE (Structured ID) [] | 37.40 | 59.50 | 66.20 | 16.70 | 39.20 | 50.30 |
| IRGen [] | 49.00 | 68.90 | 72.50 | 29.60 | 50.70 | 56.30 |
| Ours (LaVIT) | 68.84 | 82.92 | 86.44 | 44.81 | 62.61 | 68.28 |
| Ours (SEED-LLaMA) | 71.70 | 91.82 | 95.44 | 46.11 | 69.02 | 76.13 |

### 5.2 Performance Comparison

Unified Performance on TIGeR-Bench.
We compare MLLM baselines and our method and report the comprehensive results on TIGeR-Bench in Tab. 2, including separate and unified tasks.
The results show that our proposed method achieves the best performance on the unified benchmark, compared with expert generation [] or retrieval models [], the state-of-the-art MLLMs, and GILL [].
Due to half of the data being sourced from knowledge domains, current generation models, *e.g.*, SDXL [], and MLLMs could not handle the unified problem well.
Moreover, the proposed method could achieve impressive retrieval results, especially compared with other MLLMs or SDXL. Compared with the vanilla SEED-LLaMA and LaVIT, our method significantly improves the retrieval performance and enables them to deal with the unified problem.

Text-to-Image Retrieval Performance on Flickr30K [] and MS-COCO [].
As shown in Tab. 3, we compare the proposed method with the representative dense retrieval model CLIP [] and two generative retrieval baselines [, ] which have been specially trained on the two datasets. In contrast, the proposed method is training-free but achieves the best performance across all baselines. It verifies the effectiveness of the proposed generative retrieval method and demonstrates that MLLMs are capable of retrieval despite the sole optimization objective of next-token prediction.

**Table 4: Ablation study on TIGeR-Bench investigating Reverse Re-Ranking (RRR) and two decision-making strategies, *i.e.*, Forward with Eqn. 3 and Reverse with Eqn. 4. %Retr. denotes the percentage of retrieved images selected as results.**
| RRR | Decision | Ours (SEED-LLaMA) | Ours (LaVIT) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-T $\uparrow$ | CLIP-I $\uparrow$ | R@1 $\uparrow$ | %Retr. | CLIP-T $\uparrow$ | CLIP-I $\uparrow$ | R@1 $\uparrow$ | %Retr. |  |  |
|  | Forward | 22.63 | 49.71 | 26.80 | 42.10 | 27.19 | 49.59 | 28.13 | 4.38 |
| ✓ | Forward | 23.72 | 48.86 | 29.23 | 12.72 | 27.28 | 49.62 | 49.37 | 1.40 |
|  | Reverse | 23.89 | 50.52 | 26.80 | 25.60 | 28.23 | 56.51 | 28.13 | 30.47 |
| ✓ | Reverse | 22.84 | 49.54 | 29.23 | 61.47 | 28.45 | 61.37 | 49.37 | 56.25 |

### 5.3 In-depth Analysis

Ablation Study on Reverse Re-Ranking (RRR) and Decision-Making for TIGeR.
We evaluate the proposed method based on SEED-LLaMA and LaVIT on TIGeR by different RRR and decision settings, and report the unified and retrieval performance as well as the retrieval percentage in Tab. 4.
Based on the results, we have the following discussions:
1) RRR could consistently improve the retrieval performance for SEED-LLaMA and LaVIT, but may not help in unified performance for SEED-LLaMA, because unified performance is also influenced by decision strategies.
2) Compared with the forward decision with Eqn. 3, the reverse decision with Eqn 4 could enhance the unified performance in most cases for both models, which reflects the reverse decision may have stronger discriminative power across more domains.
3) Intuitively, we expect the most correctly retrieved images can be selected and the left wrong ones can be remedied by generation.
However, we find that the two MLLMs may suffer from a generation preference problem. Especially, LaVIT always prefers to choose generated images even though the retrieved images are correct, as shown by the low %Retr. in the first two settings. One of the reasons may be that there is significant gaps between the pre-trained and fine-tuned image generation data and the TIGeR-Bench.
In all, in addition to the modality bias discussed in Tab. 1, the difference between the two directional ranking and decision may be attributed to the unbalance between captioning (image-to-text) and text-to-image data during the training phase of MLLMs.

Visual Modality Debiasing for Discriminative Power.
In Sec. 3.3, we discussed the visual modality bias problem with the forward $\log p(Y|X)$ in Eqn. 2 as the similarity proxy, and adopt a debiasing strategy $\log\frac{p(Y|X)}{p(Y)}$ by considering the unconditional likelihood.
To explore the influence of the debiasing strength, we set different values of the factor $\eta$ in Eqn. 3 and a series of results are shown in Fig. 5.3.
They show that the ranking performance is sensitive to the debiasing strength and reaches the highest point around $\eta=1$, verifying the effectiveness of the unconditional debiasing strategy.

Forward Beam Search (FBS) and Reverse Re-Ranking for Retrieval.
Considering the trade-off of retrieval efficiency and recall, we present FBS and RRR, respectively. As shown in Fig. 5.3, we compare the ranking (dotted lines) and retrieval performance (solid lines) and explore the impact of beam size and RRR. In the ranking experiments, we adopt the proxies in Sec. 3.3 to calculate similarities, and then rank the whole database based on which. The comparison results demonstrate that ranking with the forward $\log\frac{p(Y|X)}{p(Y)}$ seems the upper bound of FBS since FBS may miss the target image with the limited beam size.
Benefiting from the reverse matching proxy $\log p(X|Y)$, RRR could help FBS break through the ceiling and significantly improve recall.
Additionally, regardless of similarity proxies or base MLLMs employed, increasing the beam size can reduce the recall gap between retrieval and ranking.

Efficiency of Generative Retrieval.
We analyze the efficiency of the proposed generative retrieval method for T2I-R and compare it with that of the representative dense retrieval method CLIP [] in Fig. 5.3. The efficiency of dense retrieval gets worse with the increase in the database size due to more matching in the common feature space. In contrast, the proposed method keeps constant efficiency regardless of the database size. We relegate more efficiency analysis to Appendix. C.2.

### 5.4 Qualitative Analysis

In Fig. 3, we compare our methods with SDXL on TIGeR-Bench, covering both creative and knowledge domains. SDXL, despite its strong generative abilities, struggles in knowledge domains. In contrast, our method excavates the intrinsic discriminative power of MLLMs and endows them with the retrieval ability, enabling more effective handling for knowledge-intensive image acquisition.
Besides, we explore more complicated multi-turn chat scenarios with multimodal context and both image retrieval and generation or editing requirements in Fig. 4.
Seamlessly integrated into MLLMs without training and benefiting from multimodal instruction tuning, the proposed model inherits the basic multimodal understanding, generation, and user intention capturing capabilities of MLLMs and meanwhile shows potential for knowledge-intensive tasks in vertical domains.

Figure: Figure 3: Qualitative results on TIGeR-Bench. The prefix prompt "Give me an image of" is omitted here. Green ticks and red crosses highlight correct and wrong retrieval results.
Refer to caption: x6.png

Figure: Figure 4: Example of multi-turn chat based on SEED-LLaMA with unified generation and retrieval.
Refer to caption: x7.png

## 6 Conclusion

In this work, we start with the practical requirements for image acquisition, analyze the weaknesses of single generation and retrieval, and propose to unify these two tasks within MLLMs. Toward this end, we first delve into the intrinsic discriminative abilities of MLLMs for semantic matching and propose a generative retrieval method to perform text-to-image retrieval in an auto-regressive manner. Besides, under the same auto-regressive framework, we unify generation and retrieval synchronously and present an autonomous decision strategy to select the best image. The proposed framework exhibited effectiveness and versatility across the constructed TIGeR-Bench and two retrieval benchmarks.

## Appendix A TIGeR-Bench Details

### A.1 Data Collection

Data Source.
To comprehensively evaluate unified text-to-image generation and retrieval, we build a benchmark called TIGeR-Bench, encompassing both creative and knowledge-intensive domains.
For the creative domains, the data is derived from authentic user prompts that reflect real-world needs, requiring high levels of novelty and creativity.
We collect the data from the WHOOPS! [] and Pick-a-Pic [] datasets:

- •
WHOOPS! []: The WHOOPS! dataset consists of 500 commonsense-defying prompt-image pairs created by designers. First, the designers think of counterfactual prompts by combining two elements or concepts that violate commonsense, *e.g.*, “Albert Einstein holding a smartphone”. Next, they are guided to use text-to-image generation tools (*e.g.*, Midjourney, DALL-E [], and Stable Diffusion []) to synthesize images using these counterfactual prompts. Finally, the designers verify the ‘weirdness’ of generated images to guarantee the data quality.
- •
Pick-a-Pic []: The Pick-a-Pic dataset consists of real-world user prompts and corresponding generated images, annotated with user preference, gathered from the Pick-a-Pic web application. In detail, we collect our data from Pick-a-Pic v2 (^2^22[https://huggingface.co/datasets/yuvalkirstain/pickapic_v2](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2)).

For knowledge-intensive domains, we collect data encompassing a wide range of categories to fulfill users’ needs for visual knowledge, including Logo-2K+ [], Visual News [], Google Landmark v2 [], Food2k [], iNaturalist [], WIT [].

- •
Logo-2K+ []: Logo-2K+ is a large-scale real-world logo dataset, containing 167,140 images with 2,341 categories and 10 root categories, *e.g.*, food, clothes, and institution.
- •
Visual News []: Visual News is a large-scale dataset comprising over one million news images along with associated news articles, image captions, author information, and additional metadata. Distinguished from other image captioning datasets, this dataset prioritizes factual contexts, including individuals, locations, and events, sourced from prominent news outlets such as The Guardian, BBC, USA Today, and The Washington Post.
- •
Google Landmark v2 []: Google Landmark v2 includes approximately 5M images annotated with 200k distinct instance labels representing human-made and natural landmarks. It is collected from Wikimedia Commons.
- •
Food2K []: Food2K is a food recognition dataset with 2,000 categories and more than 1 million images, covering cereal products, vegetables, bread, snack, soup and porridge, barbecue, egg products, dessert, beam products, seafood, fried food, and meat.
- •
iNaturalist []: The iNaturalist dataset is constructed to reflect the diversity of the natural world, featuring an unbalanced distribution of species. It encompasses a total of 5,000 species of plants and animals, accompanied by 859,000 images.
- •
WIT []: Wikipedia-based Image Text (WIT) is a large multimodal multilingual dataset, comprising 37.6 million image-text pairs representing real-world entities. It encompasses 11.5 million unique images across 108 Wikipedia languages. The texts are sourced from 3 primary channels: reference descriptions, attribution descriptions, and alt-text descriptions.

Prompts.
The WHOOPS! and Pick-a-Pic datasets contain prompts, while Visual News also provides natural language descriptions, serving as user prompts or queries. For the remaining five datasets, only category names or concepts represented by single words or phrases are available. To address this, we utilize a template to formulate them into complete prompt sentences, *i.e.*, “Give me an image of [concept]”.

These datasets, originally designed for different purposes, are effectively repurposed as the creative domain and knowledge-intensive domain candidates within the TIGeR-Bench.

**Table 5: The statistics of TIGeR-Bench. We keep the ratio of $1:1$ for creative and knowledge domains and collect 6,000 high-quality text-image pairs in total.**
| Domain | Data Source | #Text-Image Pairs |
| --- | --- | --- |
| Creative | WHOOPS! [] | 500 |
| Pick-a-Pic [] | 2500 |  |
| Knowledge-intensive | Logo-2K+ [] | 500 |
| Visual News [] | 500 |  |
| Google Landmark v2 [] | 500 |  |
| Food2K [] | 500 |  |
| iNaturalist [] | 500 |  |
| WIT [] | 500 |  |

### A.2 Automatic Data Filtration

Data Split.
To evaluate text-to-image generation and retrieval, we prioritize selecting the original test split of each dataset to construct TIGeR-Bench. In cases where only a validation set is provided, we default to utilizing the validation set.

Filtration Pipeline
Given that all 8 datasets have undergone individual single-modality quality assessments during their construction, our emphasis now lies on cross-modal relevance and generation challenge properties. We proceed with the following three steps for data filtration.

- 1)
To ensure a strong alignment between the positive text and image pairs for both generation and retrieval, we employ a filtering process to remove weakly relevant text-image pairs (*e.g.*, outliers or noisy pairs) across 7 datasets except for WHOOPS! due to its limited scale. Specifically, we calculate the CLIP-T scores ($S_{gt}$) between the ground-truth images and texts, and remove pairs with CLIP-T scores lower than 30.0. Considering the large scale of Pick-a-Pic, we then randomly sample 7,500 pairs as candidates for the following human quality validation phase.
- 2)
As discussed in Sec. 1, T2I-G models may struggle with synthesizing knowledge-intensive images. To identify challenging concepts in the above six knowledge-intensive datasets, which pose difficulties for current state-of-the-art T2I-G models, we first employ open-sourced models including the SD series [, ] to generate images by feeding the textual prompts in candidates as conditional input. Subsequently, we calculate the average CLIP-T scores ($S_{gen}$) over images generated by multiple models for each prompt. We then calculate the difference between the scores of the ground-truth pair and the generated pair for each prompt, *i.e.*, $\Delta=S_{gt}-S_{gen}$.
- 3)
Finally, we select the top 1,000 unique instance pairs – comprising 1,000 different prompts and 1,000 different images – with the highest values of $\Delta$ for each knowledge dataset. The remaining examples form a new candidate set with 500 WHOOPS! instances, 7,500 Pick-a-Pic instances, and 1,000 instances for the six knowledge datasets.

### A.3 Human Annotator Filtration

To further improve the data quality of TIGeR-Bench, human annotators were employed to mark evaluate each text-image pair across three aspects: text, image, and pair. Specifically, as for each text-image pair, considerations include the conciseness and unambiguity of the text, the clarity and usefulness of the image, and the relevance of the text-image pair. Annotators assigned a score of 0 (not satisfied) or 1 (satisfied) for each aspect. Finally, only text-image pairs meeting satisfaction across all three aspects were retained.

### A.4 Data Sampling

To strike a balance between adequacy and efficiency in evaluation, we retain all 500 samples in WHOOPS!, and further randomly sample 2,500 data instances from Pick-a-Pic, along with 500 instances from each knowledge-intensive dataset. The statistics of TIGeR-Bench are presented in Fig. 5. Maintaining a balanced ratio of $1:1$ between creative and knowledge domains, we finally obtain a total of 6,000 high-quality text-image pairs.

## Appendix B Model Details

In this work, we introduce and implement our approach for unified text-to-image generation and retrieval, based on two foundation MLLMs: SEED-LLaMA [] and LaVIT []. The details of these two models are as follows.

SEED-LLaMA produces 32 discrete visual codes for each image via the SEED tokenizer. This tokenizer is composed of a Causal Q-Former, a learnable codebook, and an MLP (only for training), and is trained with contrastive learning and reconstruction objectives. SEED-LLaMA takes discrete visual codes as input for multimodal understanding tasks such as image captioning and VQA, and outputs discrete visual codes. The output codes are then fed into the unCLIP-SD model [, ] to generate images.

LaVIT obtains discrete visual codes with variable lengths using a meticulously designed dynamic visual tokenizer, which comprises a token selector, a token merger, and a reconstruction decoder (used solely for training).
This tokenizer is trained with a reconstruction objective.
During tokenization, LaVIT samples a binary decision mask from a Gumbel distribution to select visual patches and quantize them into discrete tokens.
To ensure reproducibility and stability in tokenization, we depart from LaVIT and employ a deterministic selection method, where a patch is selected if its importance score exceeds a threshold of 0.5; otherwise, it is discarded.
With this discriminative tokenization strategy, we pre-tokenize the 6 knowledge-intensive datasets of TIGeR-Bench, resulting in average, maximum, and minimum lengths of discrete tokens at 88, 130, and 37, respectively.
During image generation, LaVIT first autoregressively produces a sequence of discrete visual tokens and then decodes them into an image using a diffusion model initialized with SD-v1.5 [] or SDXL [].
In contrast to SEED-LLaMA, which utilizes discrete visual tokens as input for multimodal understanding and generation, LaVIT takes continuous visual features from the token merger as input.

## Appendix C Experimental Details

### C.1 Baselines

As shown in Tab. 2, we compare the proposed method with several baselines on TIGeR-Bench across three aspects, *i.e.*, text-to-image generation, text-to-image retrieval, and unified generation and retrieval. We introduce these baselines in the following.

- •
Text-to-Image Generation Baselines: There include the expert model SDXL [] and recent MLLMs with image generation abilities. The MLLMs in this category are GILL [], Emu [], Emu 2 [], DreamLLM [], SEED-LLaMA [], and LaVIT [].
- •
Text-to-Image Retrieval Baselines: These include the expert model CLIP (ViT-B/32) [] and recent MLLMs. Currently, GILL [] is the only MLLM with retrieval ability, which maps the embeddings of special visual tokens into the CLIP feature space. Although other MLLMs [, , , , ] do not directly support text-to-image retrieval, we evaluate them through a two-step process: 1) generating an image query conditioned on the text prompt, and 2) performing nearest neighbor search for image-to-image retrieval using the CLIP (ViT-B/32) image encoder as the feature extractor and cosine similarity as the metric.
- •
Unified Text-to-Image Generation and Retrieval: GILL [] is the only baseline capable of performing both text-to-image generation and retrieval. It incorporates and trains a binary classifier to decide between generation and retrieval tasks.

### C.2 Implementation Details

The proposed method is training-free and based on SEED-LLaMA [] and LaVIT [].
We utilize the 8B version of SEED-LLaMA and load the parameters of supervised fine-tuning. For LaVIT, we employ the 11B model with SDXL as the pixel decoder.
We combine all images in the 6 knowledge-intensive datasets and tokenize them into discrete tokens. Subsequently, we build the mapping between images and tokens. Based on these discrete tokens, we construct a Trie for efficient storage and constrained generation.
The beam size for retrieval is set to 800, and the timestep for generation is 25.

## Appendix D Additional Experiments

In this section, we carry out extensive experiments and obtain quantitative and qualitative results to explore the unified text-to-image generation and retrieval problem and the proposed MLLMs-based method.

### D.1 Additional Quantitative Results

#### D.1.1 Unified Performance Comparison

To broadly compare the performance of baselines and our method for unified text-to-image generation and retrieval, we report the results with the CLIP-T score and the CLIP-I score as the evaluation metrics across 8 domains in TIGeR-Bench, in Tab. 6 and Tab. 7, respective.

**Table 6: Unified performance comparison on the CLIP-T score across 8 domains in TIGeR-Bench.**
| Method | Creative Domains | Knowledge Domains | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Counterfactual | Preference | Logo | News | Landmark | Nature | Food | Wiki |  |  |
| SDXL [] | 36.90 | 30.09 | 16.36 | 24.96 | 21.51 | 26.00 | 24.25 | 20.99 | 26.79 |
| CLIP (ViT-B/32) [] | 16.61 | 15.86 | 36.17 | 34.95 | 32.51 | 30.56 | 29.44 | 31.00 | 24.21 |
| GILL [] | 10.80 | 11.22 | 14.65 | 9.49 | 14.31 | 12.92 | 13.64 | 13.55 | 12.12 |
| Emu [] | 23.61 | 23.98 | 17.24 | 19.21 | 21.54 | 23.43 | 21.25 | 21.00 | 22.26 |
| Emu 2 [] | 29.49 | 26.21 | 23.67 | 20.85 | 19.56 | 26.09 | 20.53 | 19.81 | 24.25 |
| DreamLLM [] | 27.16 | 23.47 | 25.57 | 25.13 | 24.27 | 23.31 | 20.78 | 24.20 | 23.98 |
| SEED-LLaMA [] | 27.18 | 23.97 | 16.73 | 19.66 | 19.03 | 22.66 | 19.63 | 19.29 | 22.00 |
| LaVIT [] | 34.60 | 29.07 | 16.70 | 25.17 | 24.14 | 29.59 | 25.07 | 24.26 | 27.07 |
| SEED-LLaMA (Ours) | 27.16 | 23.47 | 25.57 | 25.13 | 24.27 | 23.31 | 20.78 | 24.20 | 23.98 |
| LaVIT (Ours) | 32.05 | 25.39 | 35.87 | 24.30 | 32.38 | 31.28 | 27.59 | 31.01 | 28.45 |

**Table 7: Unified performance comparison on the CLIP-I score across 8 domains in TIGeR-Bench.**
| Method | Creative Domains | Knowledge Domains | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Counterfactual | Preference | Logo | News | Landmark | Nature | Food | Wiki |  |  |
| SDXL [] | 65.91 | 55.38 | 14.21 | 42.09 | 35.43 | 44.94 | 45.60 | 35.40 | 46.71 |
| CLIP (ViT-B/32) [] | 31.33 | 26.68 | 93.55 | 91.04 | 71.54 | 75.38 | 71.39 | 75.59 | 53.60 |
| GILL [] | 15.93 | 13.61 | 20.38 | 15.96 | 14.34 | 18.25 | 15.49 | 14.52 | 15.25 |
| Emu [] | 43.17 | 43.95 | 22.23 | 34.25 | 38.53 | 46.97 | 48.56 | 35.97 | 40.78 |
| Emu 2 [] | 59.07 | 49.17 | 32.27 | 36.27 | 33.44 | 49.77 | 40.72 | 33.51 | 44.24 |
| DreamLLM [] | 53.93 | 46.22 | 33.16 | 32.09 | 37.87 | 46.06 | 43.82 | 35.13 | 42.77 |
| SEED-LLaMA [] | 52.76 | 49.37 | 18.50 | 37.89 | 33.78 | 45.46 | 46.55 | 34.50 | 43.02 |
| LaVIT [] | 65.79 | 53.64 | 20.87 | 42.20 | 39.77 | 53.66 | 52.48 | 42.05 | 48.75 |
| SEED-LLaMA (Ours) | 52.67 | 47.81 | 51.94 | 56.86 | 48.76 | 51.99 | 52.53 | 52.52 | 50.52 |
| LaVIT (Ours) | 60.80 | 46.56 | 92.68 | 57.32 | 72.88 | 74.70 | 69.79 | 75.44 | 61.37 |

#### D.1.2 Prompt/Query Extension

In this section, we mainly study the influence of prompts on the retrieval and generation performance in knowledge-intensive scenarios. Toward this end, we first let SEED-LLaMA, Gemini Pro, and GPT-4o to explain the raw query with knowledge concepts from 6 domains. Subsequently, we concatenate the raw prompt and expanded ones to form new long text prompts and feed them to SEED-LLaMA. The results for text-to-image generation and retrieval are listed in Tab. 8 and Tab. 9, respectively. We can see that prompt/query expansion with strong LLMs could promote both generation and retrieval performance. Meanwhile, weak LLMs may introduce false explanations and do harm to the generation and retrieval performance.

Besides, we also investigate the generation and retrieval performance separately in multi-round chat scenarios. To construct such scenarios, we use the APIs of powerful Gemini Pro and GPT-4o to imagine chat contexts and then feed the context into SEED-LLaMA. Experimental results on these two tasks are reported in Tab. 10.

We also carry out experiments in multimodal chat scenarios to explore the interplay between generation and retrieval, as shown in 11. Specifically, we concatenate the retrieved top-1 images behind the chat context and then evaluate the generation performance. Similarly, we concatenate the generated images behind the chat context and then evaluate the retrieval performance.

**Table 8: Text-to-image generation performance on TIGeR-Bench (Knowledge) in long text scenarios, with CLIP-T and CLIP-I scores as evaluation metrics across various prompt/query expansion methods including Self-Expansion, Gemini Pro, and GPT-4o. For expansion, we guide LLMs to explain the appearance characteristics in detail with their expert language knowledge for given raw queries by giving them detailed instructions. After that, the queries can be expanded into longer texts and are combined with raw queries as input for text-to-image generation. We perform generative retrieval with 200 beams.**
| Expansion Method | Ours (SEED-LLaMA) | Ours (LaVIT) |  |  |
| --- | --- | --- | --- | --- |
| CLIP-T | CLIP-I | CLIP-T | CLIP-I |  |
| Raw Query | 19.50 | 36.11 | 24.16 | 41.84 |
| Self-Expansion | 18.07 | 35.12 | - | - |
| Gemini-Pro | 17.48 | 34.74 | 22.38 | 39.28 |
| GPT-4o | 20.36 | 38.95 | 23.91 | 42.59 |

**Table 9: Text-to-image retrieval performance on TIGeR-Bench (Knowledge) in long text scenarios, with recall as the evaluation metric across various prompt/query expansion methods including Self-Expansion, Gemini Pro, and GPT-4o. For expansion, we guide LLMs to explain the appearance characteristics in detail with their expert language knowledge for given raw queries by giving them detailed instructions. After that, the queries can be expanded into longer texts and are combined with raw queries as input for text-to-image retrieval. We perform generative retrieval with 200 beams.**
| Expansion Method | Ours (SEED-LLaMA) | Ours (LaVIT) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |  |
| Raw Query | 22.57 | 36.80 | 43.23 | 25.63 | 43.63 | 49.40 |
| Self-Expansion | 17.20 | 30.10 | 36.77 | - | - | - |
| Gemini-Pro | 18.57 | 34.27 | 40.30 | 19.07 | 36.80 | 43.10 |
| GPT-4o | 25.00 | 42.50 | 48.90 | 25.20 | 46.03 | 52.17 |

**Table 10: Text-to-image generation and retrieval performance on TIGeR-Bench (Knowledge) in chat scenarios, across various chat generation methods including Gemini Pro and GPT-4o. We guide LLMs to imagine a scenario where a user intends to know a concept and asks for an image with their expert language knowledge for given raw queries by giving them detailed instructions and in-context examples. After that, the queries can be expanded into multi-round chat contexts as input for text-to-image generation and retrieval. Considering that LaVIT was not fine-tuned by chat instructions, we only carry out experiments based on SEED-LLaMA. We perform generative retrieval with 200 beams.**
| Expansion Method | Text-to-Image Generation | Text-to-Image Retrieval |  |  |  |
| --- | --- | --- | --- | --- | --- |
| CLIP-T | CLIP-I | R@1 | R@5 | R@10 |  |
| Raw Query | 19.50 | 36.11 | 22.57 | 36.80 | 43.23 |
| Gemini-Pro | 17.71 | 34.58 | 17.83 | 31.70 | 36.77 |
| GPT-4o | 19.40 | 38.17 | 24.03 | 40.73 | 47.83 |

**Table 11: Text-to-image generation and retrieval performance on TIGeR-Bench (Knowledge) in multimodal chat scenarios. Based on the chat contexts with pure text generated by GPT-4o, we can perform generation and retrieval. Afterwards, we concatenate the generated or retrieved top-1 images with the chat contexts and form the multimodal context, to explore the influence on retrieval and generation, respectively. Considering that LaVIT was not fine-tuned by chat instructions, we only carry out experiments based on SEED-LLaMA. We perform generative retrieval with 200 beams.**
| Expansion Method | Image Context | Text-to-Image Generation | Text-to-Image Retrieval |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| CLIP-T | CLIP-I | R@1 | R@5 | R@10 |  |  |
| Raw Query | - | 19.50 | 36.11 | 22.57 | 36.80 | 43.23 |
| GPT-4o | Retrieved | 18.62 | 38.20 | - | - | - |
| GPT-4o | Generated | - | - | 15.87 | 29.13 | 35.60 |

#### D.1.3 Ablation Study

In this part, we conduct comprehensive ablation studies on the components of the proposed method to study the effectiveness.

First, we compare the alignment performance between separate generation, retrieval, and unified variants across all 8 domains. The results are listed in Tab. 12 and Tab. 13 with the CLIP-T and CLIP-I score as the evaluation protocols, respectively. Besides, although Flickr30K and MS-COCO are the general datasets describing daily common scenes, we also investigate the three variants on them, as shown in Tab. 14

Second, we further study the effects of the directions of re-ranking and decision-making, across 8 domains in Tab. 15, and 2 general datasets, *i.e.*, Flickr30K and MS-COCO, in Tab. 16.

In addition, we delve into the discriminative abilities by forward and reverse ranking methods, as well as forward beam search and reverse re-ranking in Tab. 17 on 6 knowledge-intensive domains.

**Table 12: Comparison of CLIP-T score between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on 8 domains of TIGeR.**
| Method | Creative Domains | Knowledge Domains | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Counterfactual | Preference | Logo | News | Landmark | Nature | Food | Wiki |  |  |
| Ours (SEED-LLaMA) |  |  |  |  |  |  |  |  |  |
| Generation | 27.18 | 23.97 | 16.73 | 19.66 | 19.03 | 22.66 | 19.63 | 19.29 | 22.00 |
| Retrieval | 11.04 | 10.36 | 27.87 | 26.21 | 23.73 | 20.25 | 19.56 | 22.99 | 16.95 |
| Unified | 27.16 | 23.47 | 25.57 | 25.13 | 24.27 | 23.31 | 20.78 | 24.20 | 23.98 |
| Ours (LaVIT) |  |  |  |  |  |  |  |  |  |
| Generation | 34.60 | 29.07 | 16.70 | 25.17 | 24.14 | 29.59 | 25.07 | 24.26 | 27.07 |
| Retrieval | 13.21 | 12.17 | 35.84 | 23.64 | 32.58 | 31.08 | 27.61 | 30.76 | 21.30 |
| Unified | 32.05 | 25.39 | 35.87 | 24.30 | 32.38 | 31.28 | 27.59 | 31.01 | 28.45 |

**Table 13: Comparison of CLIP-I score between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on 8 domains of TIGeR-Bench.**
| Method | Creative Domains | Knowledge Domains | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Counterfactual | Preference | Logo | News | Landmark | Nature | Food | Wiki |  |  |
| Ours (SEED-LLaMA) |  |  |  |  |  |  |  |  |  |
| Generation | 27.18 | 23.97 | 16.73 | 19.66 | 19.03 | 22.66 | 19.63 | 19.29 | 22.00 |
| Retrieval | 11.04 | 10.36 | 27.87 | 26.21 | 23.73 | 20.25 | 19.56 | 22.99 | 16.95 |
| Unified | 27.16 | 23.47 | 25.57 | 25.13 | 24.27 | 23.31 | 20.78 | 24.20 | 23.98 |
| Ours (LaVIT) |  |  |  |  |  |  |  |  |  |
| Generation | 34.60 | 29.07 | 16.70 | 25.17 | 24.14 | 29.59 | 25.07 | 24.26 | 27.07 |
| Retrieval | 13.21 | 12.17 | 35.84 | 23.64 | 32.58 | 31.08 | 27.61 | 30.76 | 21.30 |
| Unified | 32.05 | 25.39 | 35.87 | 24.30 | 32.38 | 31.28 | 27.59 | 31.01 | 28.45 |

**Table 14: Comparison between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on the Flickr30K and MS-COCO datasets. Performance is evaluated by the CLIP-T score.**
| Method | Flickr30K | MS-COCO |
| --- | --- | --- |
| Ours (SEED-LLaMA) |  |  |
| Generation | 28.65 | 27.74 |
| Retrieval | 29.86 | 28.73 |
| Unified | 30.01 | 29.09 |
| Ours (LaVIT) |  |  |
| Generation | 37.05 | 35.59 |
| Retrieval | 27.54 | 27.13 |
| Unified | 33.69 | 32.24 |

**Table 15: Ablation study on 8 domains of TIGeR-Bench investigating Reverse Re-Ranking (RRR) and two decision-making strategies, *i.e.*, Forward with Eqn. 3 and Reverse with Eqn. 4. Performance is evaluated by the CLIP-T score.**
| RRR | Decision | Creative Domains | Knowledge Domains | All |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Counterfactual | Preference | Logo | News | Landmark | Nature | Food | Wiki |  |  |  |
| Ours (SEED-LLaMA) |  |  |  |  |  |  |  |  |  |  |
|  | Forward | 26.64 | 22.41 | 24.02 | 23.64 | 22.85 | 20.34 | 19.63 | 22.40 | 22.63 |
| ✓ | Forward | 27.14 | 23.95 | 23.57 | 23.18 | 22.82 | 24.57 | 20.62 | 22.93 | 23.72 |
|  | Reverse | 27.16 | 23.47 | 25.57 | 25.13 | 24.27 | 23.31 | 20.78 | 24.20 | 23.98 |
| ✓ | Reverse | 26.81 | 20.19 | 27.29 | 23.22 | 24.16 | 26.04 | 21.79 | 23.84 | 22.84 |
| Ours (LaVIT) |  |  |  |  |  |  |  |  |  |  |
|  | Forward | 34.59 | 29.00 | 16.84 | 25.53 | 24.74 | 29.56 | 24.93 | 25.05 | 27.19 |
| ✓ | Forward | 32.05 | 29.07 | 17.15 | 25.57 | 24.76 | 29.60 | 25.17 | 25.20 | 27.28 |
|  | Reverse | 33.83 | 28.17 | 24.67 | 28.88 | 28.06 | 29.02 | 25.65 | 27.84 | 28.23 |
| ✓ | Reverse | 32.05 | 25.39 | 35.87 | 24.30 | 32.38 | 31.28 | 27.59 | 31.01 | 28.45 |

**Table 16: Ablation study on Flickr30K and MSCOCO investigating Reverse Re-Ranking (RRR) and two decision-making strategies, *i.e.*, Forward with Eqn. 3 and Reverse with Eqn. 4. %Retr. denotes the percentage of retrieved images selected as results.**
| RRR | Decision | Flickr30K | MS-COCO |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-T $\uparrow$ | R@1 $\uparrow$ | %Retr. | CLIP-T $\uparrow$ | R@1 $\uparrow$ | %Retr. |  |  |
| Ours (SEED-LLaMA) |  |  |  |  |  |  |  |
|  | Forward | 28.89 | 58.50 | 39.52 | 25.95 | 26.17 | 67.61 |
| ✓ | Forward | 29.68 | 71.70 | 26.92 | 28.71 | 46.11 | 33.91 |
|  | Reverse | 30.01 | 58.50 | 35.98 | 28.61 | 26.17 | 26.23 |
| ✓ | Reverse | 30.02 | 71.70 | 51.88 | 29.09 | 46.11 | 60.69 |
| Ours (LaVIT) |  |  |  |  |  |  |  |
|  | Forward | 37.03 | 47.86 | 0.20 | 35.34 | 23.20 | 3.44 |
| ✓ | Forward | 37.04 | 68.84 | 0.10 | 35.58 | 44.81 | 0.23 |
|  | Reverse | 36.18 | 47.86 | 24.34 | 34.67 | 23.20 | 20.60 |
| ✓ | Reverse | 33.69 | 68.84 | 41.84 | 32.24 | 44.81 | 49.11 |

**Table 17: Recall@1 performance comparison of Forward Ranking, Reverse Ranking, Forward Beam Search (FBS) with different beam sizes, and BFS + Reverse Re-Ranking (RRR). Experiments are conducted based on SEED-LLaMA and LaVIT on 6 knowledge-intensive domains of TIGeR-Bench.**
| Method | Logo | News | Landmark | Nature | Food | Wiki | ALL |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Ours (SEED-LLaMA) |  |  |  |  |  |  |  |
| Forward Ranking | 56.00 | 45.60 | 22.60 | 2.60 | 4.40 | 30.40 | 26.93 |
| Reverse Ranking | 61.80 | 40.40 | 26.60 | 15.00 | 7.00 | 32.40 | 30.53 |
| FBS (#Beam=100) | 39.40 | 29.60 | 10.40 | 8.80 | 4.60 | 19.40 | 18.70 |
| FBS (#Beam=800) | 56.00 | 46.80 | 20.20 | 3.60 | 4.60 | 29.60 | 26.80 |
| FBS (#Beam=100) + RRR | 37.40 | 25.80 | 10.00 | 10.20 | 6.00 | 18.60 | 18.00 |
| FBS (#Beam=800) + RRR | 61.20 | 39.60 | 22.60 | 15.40 | 6.80 | 29.80 | 29.23 |
| Ours (LaVIT) |  |  |  |  |  |  |  |
| Forward Ranking | 37.80 | 52.20 | 25.00 | 10.00 | 11.40 | 33.20 | 28.27 |
| Reverse Ranking | 92.20 | 41.00 | 56.00 | 36.40 | 23.60 | 63.80 | 52.17 |
| FBS (#Beam=100) | 16.60 | 46.00 | 20.20 | 9.40 | 10.00 | 28.40 | 21.77 |
| FBS (#Beam=800) | 37.00 | 53.40 | 24.40 | 10.80 | 11.40 | 31.80 | 28.13 |
| FBS (#Beam=100) + RRR | 27.00 | 30.80 | 30.60 | 16.80 | 15.40 | 41.00 | 26.93 |
| FBS (#Beam=800) + RRR | 87.20 | 40.40 | 51.60 | 34.60 | 23.00 | 59.40 | 49.37 |

### D.2 Additional Qualitative Results

We showcase more examples of our SEED-LLaMA and LaVIT in both creative and knowledge-intensive domains in Fig. 5 and Fig. 6.

In the creative domain, the CLIP model, limited to retrieving images from the database, shows significant discrepancies when compared to the ground truth images. Our SEED-LLaMA and LaVIT, capable of both generation and retrieval, tend to favor image generation in the creative domain. However, our models also exhibit decision errors. For instance, as demonstrated in the last two rows of Fig. 5, the models incorrectly selected misleading retrieved images.

As shown in Fig. 5, our models has the advantages over SDXL in the knowledge-intensive domain, accurately retrieving the correct results. However, decision errors still occur. We leave further exploration of the decision strategy for future work.

In Fig. 7, we compare our models with current Text-to-Image baseline models such as Emu2, DreamLLM, and GILL, which can autonomously decide between retrieval and generation. Our models are consistently retrieving the correct images in the knowledge-intensive domain. In this domain, Emu2, DreamLLM, and GILL fail to generate closely matching images, highlighting the limitations of current MLLMs.

We further explored two scenarios: Augmented Generation for Better Retrieval and Augmented Retrieval for Better Generation. In the Augmented Generation for Better Retrieval scenario, we first use the MLLM’s capability to generate an image before performing image retrieval. The generated image, along with the retrieval prompt, is then used as input for the retrieval process. As shown in Fig. 8, generating an image beforehand improves the model’s retrieval performance.

In the Augmented Retrieval for Better Generation scenario, we leverage our model’s generative retrieval capabilities to perform an image retrieval before generating an image. The retrieved image, along with the generation prompt, is then used as input for the generation process, similar to Retrieval-Augmented Generation (RAG). As shown in Fig. 9, performing image retrieval beforehand improves the stability and quality of the generated images.

Figure: Figure 5: Qualitative results in TIGeR-Bench creative domain. We use ticks or crosses to highlight the selected results from generation or retrieval. Green ticks indicate the correct generated images and red crosses indicate the wrong retrieved images. Black ticks refer to the correct retrieved images despite the creative domain.
Refer to caption: x8.png

Figure: Figure 6: Qualitative results in TIGeR-Bench knowledge domain. We use ticks or crosses to highlight the selected results from generation or retrieval. Green ticks indicate the correct retrieved images and red crosses indicate the wrong generated images.
Refer to caption: x9.png

Figure: Figure 7: Qualitative results in TIGeR-Bench knowledge domain compared to Text-to-Image generation baselines. Green tick indicates the final output by our model is correct and Red cross indicates the final output by our model is incorrect.
Refer to caption: x10.png

Figure: Figure 8: Augmented generation for better retrieval. Green box is the additional generation step. Green tick indicates the final output by our model is correct and Red cross indicates the fina3l output by our model is incorrect.
Refer to caption: x11.png

Figure: Figure 9: Augmented retrieval for better generation. Green box is the additional retrieval step. Green tick indicates the final output by our model is consistent and Red cross indicates the final output by our model is inconsistent.
Refer to caption: x12.png

Figure: Figure 10: Short prompt and long prompt retrieval comparison on TIGeR-Bench knowledge domain. Red box is the retrieve result of short prompt. Green box is the retrieve result of long prompt. Green tick indicates the final output by our model is correct and Red cross indicates the final output by our model is incorrect.
Refer to caption: x13.png

One major limitation of the CLIP model for retrieval is its limited context length. Our model leverages the advantage of the LLM’s long context length, making retrieve with longer prompts possible. As shown in Fig. 10, extending the prompt further enhances the retrieval performance of our model.

## Appendix E Future Work

In the future, we plan to investigate the root causes of modality biases from various perspectives, including data distribution, model architecture, and optimization objectives. We will also examine the potential impacts of these biases on generative and discriminative tasks. Additionally, we aim to study more complex contexts involving interleaved multimodal content to advance comprehensive unified generation and retrieval tasks. Finally, it would be valuable to explore the deeper relationships and possible interactions between generation and retrieval (e.g., retrieval-augmented generation and generation-augmented retrieval) within the TIGeR framework.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see weakness for my questions

--- Post-rebuttal comments:

I improved my scores based on authors' response.
soundness: 3
strengths: - Unification of Generation and Retrieval: The core contribution of unifying T2I-G and T2I-R within a single framework centered with LMMs. This framework leverages the strengths of both paradigms, mitigating the limitations of relying on either alone for offering visual content per user query. In an ideal world, the users can obtain a factual image generation when its query is centered at knowledge-intensive factual entity, and see a creative image when queried for imaginary scene.

- New benchmark: This paper creates a dedicated benchmark, TIGER-Bench, which accesses image generation in both creative and knowledge-intensive domains, and offers a more comprehensive evaluation platform.
summary: This paper presents an approach to address the limitations of current text-to-image generation (T2I-G) and retrieval (T2I-R) systems. Particularly, it propose to unify generation and retrieval auto-regressively, and use a decision making model to decide whether to use generated or retrieved image as the response to a text prompt.
weaknesses: - Lots of details related to presentation needs improvement: 

1. Figure  1 is referring to many things that has not be introduced before, it is hard for people to understand forward beam-search & reverse re-ranking at the first glimpse. Many terms are not really explained in caption, or the introduction and people needs to leap to methodology section to really understand the meaning. 

2. Section 3.2 is not very clear to me, I couldn't find a connection one why those three metrics are used, and also why the metric it has to be training-free. Meanwhile, I also think that comparing to those proposed proxy metrics, a more straightforward way is to directly ask the LMM a visual question, i.e. "<image> Is the presented image is aligned with the following caption? <caption>. Answer yes or no.", and then measure the likelihood on yes and no. 

3. Given that the authors also agree that "It is inefficient due to |G| times of forward propagation" for computing the proposed cross-modal similarity, why not considering leveraging a contrastively trained text-to-image similarity function? The forward beam search + reverse re-ranking seems unnecessarily complicated.

- Lack of simple baseline method: It would make a lot sense to compare a simple zero-shot / fine-tuned LMM that makes a decision between whether to use the SDXL's generated image or CLIP model's retrieved image, according to a user's input query. IMO, this method would be very competitive to the proposed approach.

- The core experimental result is based on author's proposed benchmark, which is not very convincing given that it is really hard to judge the quality and validity of this newly introduced benchmark. It appears to me that the authors are being the player and judge at the same time, which is quite tricky to assess the significance of this work. 

- Missing relevant works: 

The following works have discussed how to leverage image retrieval for image generation and should be discussed and cited. 

1. Re-Imagen: Retrieval-Augmented Text-to-Image Generator
2. KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness section
soundness: 2
strengths: - The overall presentation of this paper is good. This paper contains a lot of content which is hard to make the presentation organized and easy to follow. The authors did a good job in my opinion.
- Joint T2I generation and retrieval is a important task. The proposed benchmark set is key to make different methods devoted for the task comparable.
- The experimental results show that the proposed method has good performance than other generation or retrieval-based methods.
summary: This paper studies how to effectively combine the T2I generation and retrieval into a combined framework. For this task, this paper makes two folds of contribution: 1) This paper proposes a new T2I generation and retrieval pipeline, where the generation and retrieval branches run in parallel, followed by a final image ranking/selection process to decide the final output. 2) This paper proposes a benchmark to evaluate the joint T2I generation and retrieval. The benchmark contains half creative and half knowledge based image-text pairs.
weaknesses: - In my opinion, the biggest drawback of this paper is that it lacks technical novelty. This work looks more like a prototype of an application than a research paper. All modules used by this paper has been used by previous papers and the combination of them contributes very limited new knowledge or insight. I understand that this work belongs to a type of papers that is more engineer than research. I put this point inside the weakness section but I am open to any discussion about the fit of this paper to iclr with authors, other reviewers, and AC if possible.
- I am confused about the comparison results in Table 2. What is the difference between the SEED-LLaMA vs Ours (SEED-LLaMA)? Does the latter one used both retrieval and generation? If so, this comparison seems to be somewhat unfair. If not, what is the components of the proposed method that lead to the performance improvement over the baseline. More discussion is welcomed.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: see the weakness part
soundness: 4
strengths: - This paper is well-written and easy to follow. 
- Efficient retrieval and accurate decision-making are crucial in unifying generation and retrieval, balancing creativity and knowledge. The proposed TIGER-One solves these two problems within one LMM is elegant.
- The proposed TIGER-One seems effective and shows superior generation results in experiments.
summary: This paper introduces a unified framework for text-to-image generation and retrieval using LMM. It proposes an efficient method leveraging LMMs' discriminative abilities. It also presents an autonomous decision mechanism for selecting between generated and retrieved images. The authors also developed TIGeR-Bench to evaluate generation and retrieval abilities across creative and knowledge-intensive domains simultaneously.
weaknesses: - A  preliminary introduction of beam search is recommended to add.
- The authors indicate that the similarity score is calculated by one of the three proxies. More details and discussion of the proxies chosen in decision-making are recommended. 
- With the help of TIGER-One, the LMM demonstrates improved results (Tab. 2).  It’s interesting to explore the addition of dream images based on knowledge (RAG) as a third option alongside generation and retrieval in future work

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: As shown in Weaknesses
soundness: 3
strengths: 1. Propose a unified framework for text-to-image generation and retrieval.
2. Propose an autonomous decision mechanism to automatically select the most suitable image from the retrieved images.
3. To validate the capability of unified generation and retrieval, introduce a new benchmark, TIGeR-Bench.
4. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the pipeline's capabilities.
5. The article is well-written, with clear and precise explanations.
summary: 1. Explore the retrieval capabilities of Large Language Models (LMMs) to enable text-to-image (T2I) generation retrieval.
2. Unify generation and retrieval in an autoregressive manner.
3. Propose an autonomous decision mechanism to select the best-matched image from generated and retrieved images.
4. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both 5. creative and knowledge-intensive domains.
6. Conduct extensive experiments on TIGeR-Bench and two retrieval benchmarks, namely Flickr30K and MS-COCO.
weaknesses: 1. The challenges faced by generation models in knowledge-intensive tasks are mentioned, but the proposed approach’s effectiveness in consistently addressing these challenges without hallucination remains unclear. Explicit quantitative analysis on knowledge domains might be helpful.
2. Can TIGeR-ONE adapt to user-specific preferences in the decision-making process, especially in creative tasks? If not, how could this be incorporated to enhance user experience?
3. I noticed the multi-round generation in Figure 7. Could you please explain how this process maintains consistency with the user-provided image?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1.	Is there any experiment demonstrating the decision behavior of the proposed model? 
2.	Compared to dense retrieval methods, such as CLIP, what advantages do MLLMs and the proposed generative retrieval framework offer?

The authors have solved my concerns.
I stand by my positive score, after reading other reviewers' comments.
soundness: 4
strengths: 1.	The authors introduce an innovative task that combines retrieval and generation, addressing a gap in traditional approaches that often consider these processes separately. This integrated task is highly practical, as it reflects real-world information needs where users may require either a retrieved image from a database or a newly generated one, depending on the context.
2.	The authors propose a novel unified framework that unifies text-to-image generation and retrieval within a single LMM, streamlining both tasks into a cohesive process. By using a single LMM, the authors simplify the model architecture, allowing the system to handle both retrieval of existing images and generation of new ones without additional model switching or training steps.
3.	The authors introduce a training-free generative retrieval method that enhances both the effectiveness and efficiency of text-to-image retrieval. The proposed method leverages the pre-existing discriminative abilities of Large Multimodal Models (LMMs) to perform retrieval without the need for extensive training on large datasets. 
4.	They build a unified benchmark, i.e., TIGeR-Bench, to standardize evaluation across creative and knowledge-intensive tasks, serving as a strong foundation to attract more researchers to explore complex multimodal information acquisition.
summary: This paper rethinks the relationship between text-to-image generation and retrieval, presenting a novel and pragmatic task, i.e., TIGeR, to meet the challenging information need of humans in real-world scenarios. The authors identify limitations in traditional retrieval methods, which rely on existing image databases, and in generation approaches that struggle with knowledge-intensive content. Motivated by these limitations, the authors proposed a novel unified framework called TIGeR-ONE that combines both tasks using a single LMM. To endow LMMs with the retrieval abilities, they introduce a training-free generative retrieval method that leverages LMM's discriminative abilities, allowing for efficient text-to-image retrieval. Additionally, an autonomous decision mechanism selects either generated or retrieved images as the most suitable response to a given text prompt. The authors develop TIGeR-Bench, a benchmark covering creative and knowledge-intensive image domains, to standardize evaluation. Their approach demonstrates superior performance on the unified benchmark, text-to-image and chat-to-image retrieval and generation benchmarks.
weaknesses: 1.	Considering there has been prior work (i.e., GILL) addressing text-to-image retrieval and generation, the difference and advantages of the proposed method over GILL should be further highlighted clearly.  
2.	The paper lacks analysis on how the unified model’s performance varies with prompts that have similar intentions but are expressed differently. In practical use, different users may convey the same idea with varied wording, prompts, and questions. How sensitive is the proposed method to these variations? 
3.	Given that the distribution over creative domains and knowledge domains may not be uniform in real-world scenarios, an analysis of the model’s performance and decision-making behavior under unbalanced distribution conditions would be beneficial.

Meta-review: This paper presents a framework to unify text-to-image generation and retrieval by running the generation and retrieval branches, followed by a image selection process for the final output. 

As reviewers pointed out, the major strengths of this paper are:
1) The idea of unifying text-to-image generation and retrieval is practically quite important, the proposed framework of using forward beam search is solid.
2) A benchmark to evaluate the joint T2I generation and retrieval is also provided. 
3) The proposed TIGER-One shows superior results in experiments

The major weaknesses are:
1) The author's evaluation is specific to the method they proposed.  The evaluation can be further improved by including evaluation under more realistic scenarios.
2) As shown in authors' rebuttal, a zero-shot prompting LLM-based decision model (Qwen2-VL) that decides whether or not using generated or retrieved result seems to get only slightly worse results than the much more complicated proposed system.
3) Multiple reviewers pointed out some technical parts can be presented more clearly. The current presentation is not very easy to understand.

Overall, I think the direction of unifying text-to-image generation and retrieval is very practical and the proposed method is overall solid. So I would encourage this work and recommend "Accept as a poster".

Decision: Accept (Poster)