TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
Authors: Leigang Qu
Haochuan Li
Tan Wang
Wenjie Wang
Yongqi Li
Liqiang Nie
Tat-Seng Chua
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Text-to-Image Generation
Cross-Modal Retrieval
Primary area: generative models
TL;DR: This paper explores the intrinsic discriminative ability of multimodal foundation models, and proposes a unified framework combining text-to-image generation and retrieval.

Abstract: How humans can effectively and efficiently acquire images has always been a perennial question. A classic solution is *text-to-image retrieval* from an existing database; however, the limited database typically lacks creativity. By contrast, recent breakthroughs in *text-to-image generation* have made it possible to produce attractive and counterfactual visual content, but it faces challenges in synthesizing knowledge-intensive images. In this work, we rethink the relationship between text-to-image generation and retrieval, proposing a *unified* framework for both tasks with one single Large Multimodal Model (LMM). Specifically, we first explore the intrinsic discriminative abilities of LMMs and introduce an efficient generative retrieval method for text-to-image retrieval in a training-free manner. Subsequently, we unify generation and retrieval autoregressively and propose an autonomous decision mechanism to choose the best-matched one between generated and retrieved images as the response to the text prompt. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both creative and knowledge-intensive domains. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, *i.e.*, Flickr30K and MS-COCO, demonstrate the superiority of our proposed framework.

Published as a conference paper at ICLR 2025
TIGER: UNIFYING TEXT-TO-IMAGE GENERATION AND RETRIEVAL WITH LARGE MULTIMODAL MODELS
Leigang Qu
Haochuan Li Tat-Seng Chua!
Tan Wang
Wenjie Wang 3*
Yongqi Li*
Liqiang Nie
National University of Singapore Nanyang Technological University
University of Science and Technology of China "Hong Kong Polytechnic University
"Harbin Institute of Technology (Shenzhen)
(leigangqu jenjiewang96, nieliqiang, liyongqi JJ @ggmail, com
hacchuanCu.c www.uuseeeee edu tan317@ntu. edu. sg
descts suuss edd..s ag
ABSTRACT
How humans can effectively and efficiently acquire images has always been a perennial question. A classic solution is text-to-image retrieval from an existing database; however, the limited database typically lacks creativity. By contrast, recent breakthroughs in lexi-to-image generation have made it possible produce attractive and counterfactual visual content, but it faces challenges in synthesizing knowledge-intenan intensive images. In this work, rethink the relationship between text-to-image generation and retrieval, proposing a unified framework for both tasks with one single Large Multimodal Model (LMM). Specifically, we first explore the intrinsic discriminative abilities of LMMs and introduce an efficient generative retrieval method for text-to-imane retrieval in training- ffree manner. Subsequently, we unify generation and retrieval autoregressively and propose an autonomous de- cision mechanism choose the best-matched one between generated and retrieved images as the response to the text prompt. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bencee -Bnch,, a benchmark spanning both creative and knowledge-intensivee domains. Extensive experiments on TIGeR- Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the superiority of our proposed framework. The code, models, and benchmark are available at http::::///tttt t2i giw.ggittubb...
INTRODUCTION
The explosion of visual information the Web significantly challenges human information access. Text-to-Image Retrieval (T21-R) (Radford et al., 2021 Yu et al. 2022 Li et al. 2023b) is one of the main channels obtain visual information given text prompt. However, T2I-R is limited to retrieving existing images in the database, lacking flexibility and creativity. Furthermore, as the database expands, retrieval costs increase significantly. Recent years have witnessed thrilling progress
in Text-to-Image Generation (T21-G) (Ramesh et al., 2022 Rombach et al. 2022 Podell et al. (023). which directly generates new images to meet human visual information needs. However, T21-G struggles with knowledge-nttensive intensive concepts such as landmarks and natural species (see the right part of Fig. []. often resulting hallucination issues (Kim et al. 20244: Huang et al.. 2024b). In this light, a single T21-R or T21-G model may not satisfy the diverse and evolving human information needs. It
is pivotal to unify both T2I-R and T2I-G within a framework for visual information delivery.
To this end, a straightforward solution is to empower discriminative models with the generation ability. However, the etrly-stage trial (e.g., JEM (Grathwohl et al. 2019)) requires extra generative training and may compromise the original discriminative power. Another solution adapts generative models such as diffusion models (Podell et al.) (023) to achieve the discriminative tasks (Li et al. 2023a| Clark& & Jaimi 2023). Despite the significance, these methods are limited to diffusion models and inevitably suffer from the notorious inefficiency problem caused by iterative denoising (Ho et al.
"Corresponding Authors: Tan Wang and Wenjie Wang

---

Published as a conference paper at ICLR 2025
Counterfactual Composition
Logo
L0G0
News History
NEWS
Victoria London
X
Disney
Poxor
Prompts
Word Embedding
Knowledge- intensive Prompts
DAGNE
sullion show
Landmark
Nature Species
A lamb peacefully next to cdultt wolf.
lying
A pile. fruit has a blue apple.
Large Mulimodal Models (LM
Two holochio nis standin on pale
Victeria
Memori
Real-User Preference
London
Decision
Lookup
Food
Wiki Misc.
Diffusion
Y
T-bone
Grroeén DS 20
beautiful
broken heart with sticking out
Generazed
Retrieved Images
steok
dancing unicomss
bullat
Figure I: TIGeR-ONE unifies T21-G and T2I-R through single LMM in training- free au- toregressive way, with a decision mechanism to adaptively select between generated and retrieved images based prompts. Besides, construct T1GeR-Bench, encompassing eight creative and knowledge-intene intensive domains in total to facilitate a comprehensive evaluation of TIGeR.
2020). Moreover, diffusion models usually tailored for simple discriminative tasks such image classification (Li et al.] [203a He et al. (023) and making them less suitable for processing complex human prompts for lcrge-scale retrivval practical scenarios (Schuhmann et al. 2022
Unlike diffusion models, Large Multimodal Models (LMMs) offer another form of generative paradigm solve broad vision-language problems, garnering significant attention for their powerful language understanding and instruction following abilities (Ouyang et al. [022 Touvron et al.] (202))
Recently, notable efforts LMMs (Sun et al., 2023a; Ge al. 2023 Dong | 2023) integrate
Large Language Models (LLMs) with external T21-G models (Rombach et al. 2022) for image
synthesis. However, most studies focus solely on T2I-G, neglecting T2I-R. Even though GILL (Koh) et al. 2023) fuses an LLM and the image encoder & decoder to enable both generation and retrieval an ensemble strategy, essentially incorporates an external retriever (i.e., CLIP (Radford et al., 2021)) for dense retrieval in dedicated embedding space. As such, GILL brings extra alignment costs and still suffers from the inefficiency problem (discussed in Sec. 2 and Sec. 5.3) of the dense retrieval paradigm, particularly in large-scale image retrieval scenarios (Tay et al. 2022).
To this end, we propose to unify Text-to- Image Generation and Retrieval (TIGeR) in this work, and present model-agnostic framework named TIGeR-ON ONE that achieves this unification within one single LMM, enabling flexible and efficient image acquisition as shown in Fig. We first delve into the intrinsic bidirectional (i.e.. text-to- soke-to-imaee and image-to-textt discriminative abilities of LMMs in Sec. 3.2 Specifically, we investigate three likelihood- proxies to estimate cross-modal semantic Based these proxies, TIGeR- -OE adopts efficient generative retrieval method with forward beam search and reverse re-ranking as in Sec. 3.3 unifying both T2I-R and T2I-G in an autoregressive generation manner. Moreover, TIGeR- -OE presents autonomous decision mechanism to adaptively select between retrieved and generated images based on user prompts.
Existing benchmarks (Saharia and retrieval separately, with limited domain coverage. To comprehensively evaluate the performance of LMMSs on TIGeR, we build a benchmark called TIGeR-Benche (Sec. It encompasses creative images from the counterfactual world and imaginative scenarios (Kirstain et al. 2024). and knowledge- intensive images from six diverse domains (e.g., logo, landmark, and natural species). We carry out extensive experiments to assess the TIGeR performance of representative LMMs TIGeR-Bench and two T2I-R benchmarks, validating the effectiveness and efficiency of TIGeR-ONE. Overall, we summarize the contributions into three points.
2022 Huang et al. 2023 Weyand et al. 2020) assess generation
Driven by the complementary roles of text-to- eoxt-to-image generation and retrieval in visual informa- tion access, we propose unifying both tasks to meet complex human information needs.
2

---

Published as a conference paper at ICLR 2025
We comprehensively inspect the intrinsic cross-modal discriminative abilities of LMMs and propose TIGeR-ONE, a model-agnostic framework for the TIGeR task. TIGeR-ONE performs text-to-image generation and retrieval in a training-free frreinnggeeeeeeeee autoregressive manner, selecting the best-matched result autonomously and efficiently.
We construct a comprehensive image acquisition benchmark, TIGeR-Bence to evaluate the performance of TIGeR LMMs in creative and hhowledge-intenstin domains. Extensive experiments on JiGeR-Bench and two T2I-G benchmarks including Flickr30K (Young et al.) 2014) and MS-COCO (Lin et al. 2014) verify the effectiveness of TIGeR-ONE.
RELATED WORK
Text-to-Image Generation. T2I-G has aroused wide attention in both academia and industry over
the past decade, with advancements ranging from Generative Adversarial Networks (Reed et al.
2016) to Auto-regression Models (Ding et al, 202T) and Diffusion Probabilistic Models (DPMs) (Ho
ct al.
2020). Recent breakthroughs in DPMs, guided by the scaling law (Kaplan et al. [020]] Li
ct al.
2024a), have propelled T2I-G to new heights, e.g., models like DALL-E 2 (Ramesh ct al.
2022) and DALL-E (Betker et al. 2023), the Imagen series (Saharia et al. 2022 and the Stable
Diffusion (SD) series (Rombach et (0022 Podell et al.] | Esser et al. (2024). Concurrently,
efforts have been made to enhance the composed text-image alignment (Feng al. [022 Chefer ctt al. [203]] Qu et al. 2024] Yang et al. 2024) and cater to human preference (Lee et al. 2023
Xu et al. 2024). Some studies recognize the importance of knowledge-intensee
image acquisition.
employing RAG (Chen et al. 2022) or constructing benchmark (Huang et al. (2024)). However, they focus solely on generation and overlook the potential for unifying generation and retrieval.
Text-to-Image Retrieval. Early studies on multimodal information retrieval focused on feature representation (Faghri et al. 2017 et al. (2099) and modality interaction (Lee et al.) 2018 Qu et al. (02T) for precise cross-modal similarity estimation. Recent advancements, propelled by large- scale pre-training, have led to improved retrieval performance and generalization in vision-language models (Radford et al.] 202 Li et al. 1 Dou et al., (202)). More recently, researchers have explored more challenging scenarios, such as fine-graines interaction (Lin et al., (024a)) equivariant similarity (Wang al. 2023), multimodal instruction following (Wei et al. (2023). and chat- bast- IR (Levy et al. 2024). Despite thrilling progress, retrieval systems are inherently limited by database size, and incapable of creating new visual content.
Large Multimodal Models. Empowered by the versatility of LLMs, pioneering works on LMMs
have shown impressive understanding capabilities (Liu al. 2023 Zhu et al. 2023a). Recent
research explores image generation through two categories: 1) Continuous Visual Representation
methods aim to align visual representations from LLMs with condition embeddings of SD through
regression (Koh et al. [2023] | [Sun et al. 2023a Wu et al. [023] Dong et al. [2023] Zhu et al.) 2023b)
or score distillation (Dong et al. (2023) objectives. GILL (Koh et al. (023) combines an external
dense retriever, i.e., CLIP, and a diffusion decoder with an LLM to achieve retrieval and generation,
respectively. However, such an ensemble approach brings extra alignment costs and may encounter an alignment gap (Zhao et al.) 2024). Moreover, the dense retriever suffers from inefficiency (Li cct al. 2024b), as requires extensive similarity comparisons between the query and all items in the database. 2) Discrete Visual Tokenization methods Yu et al.) 2023 Lu et al. 2023 Ge al. 2023)
first encode an image into a sequence of discrete codes (Esser et al. 2021 Van Den Oord et al. 2017
Ge et al. 2023 Jin et al. 2023). and then employ next-token prediction to train LMMs. To synthesize images, the discrete codes are decoded into the pixel space via VQ-GAN or SD. In this work, we resort to the discrete paradigm to be consistent with the inherent discreteness of language. Compared with existing work, TIGeR-ONY achieves comprehensive image acquisition, encompassing both content creation and knowledge retrieval within single framework.
METHODOLOGY
We first formulate the task of unifying T2I-G and T21-R through LMMs in Sec. 3.1 the intrinsic cross-model discriminative ability of LMMs in Sec. 3.2 Based the discriminative ability, we propose the TIGeR-ON framework, as shown in Fig. including generative retrieval in Sec. 3.3 synchronous generation and retrieval, and decision-making in Sec. 3.4
We then explore
3

---

Published as a conference paper at ICLR 2025
3.1 TASK FORMULATION
TIGeR aims to satisfy complex human visual information needs by unifying T2I-G and T21-R in a unified LMM framework. We formulate this problem in autoregressive generation manner:
p(| II
(I)
where denotes a textual prompt provided by humans, tokenized into a sequence X [2], T TM] and Y [y...... yv| denotes the sequence of visual tokens (Ge et al. 2023) that can be decoded into an image, with Y referring to the tokens before step 2. By sampling from the conditional distribution, obtain sequence instance, Le., Y* ~ p( ( (X),,, where y* E V and V denotes the visual token space defined by a visual vocabulary, and [V] = V is the vocabulary size. V denotes the Cartesian product of N token spaces, i.e., the whole discrete visual space.
To achieve unified T2I-G and T2I-R, an LMM is required to possess the following three capabilities: 1) creativity to generate a novel photorealistic image Y based on the visual tokens sampled from p(Y|)) 2) discrimination to measure semantic similarity between prompt X and each image Y. and then retrieve the relevant image Y from a database G, formally, Y arg maxyes (Y| and 3) decision to automatically determine the superior option by comparing p( p(Y)) (X) and p(Y (X) for generation and retrieval, respectively, ultimately yielding the optimal result Y*
Considering the proficiency of recent LMMs (Dong el al. 2023 Zheng et al. 2023 Ge et al. 2023 in creative T2I-G, we shed more light on exploring the discriminative ability for T2I-R and the potential of unifying generation and retrieval with decision- decision-making in the remaining of this section.
3.2 INTRINSIC DISCRIMINATIVE ABILITY OF LMMS
For effective T2I-R, we first probe the discriminative capability of LLMs and present three training- free proxies to estimate the semantic similarity between the prompt and images in the database.
Proxy 1: Conditional Likelihood, To estimate the se- mantic similarity s(X, Y") between given text prompt X and an image Y E G. a straightforward approach to employ the conditional likelihood based on autoregressive factorization as the proxy:
Table 1: Text-to-image ranking perfor-
mance of three similarity (Sim.) prox-
ies for SEED-LLaMA (Ge et al. 2023)
and LaVIT (Jin al. 2023) on MS-
COCO Lin et al. 2014
s(x, Y) log P X Σ log logp(uilYes, X),
SERD-LL
L&VIT
Sim. Proxy
R@1
R@5
RØ1
R@5
where xX) denotes the likelihood of autoregressively generating Y conditioned on the given X. In practice, we attain it by computing the cross entropy between the predicted logits and the image tokens. However, as shown in Tab. [ observed that this proxy performs poorly. We
Random
0.02
0.10
0.02
0.10
3.50
8.79
0.02
0.16
log
26.25
54.57
23.43
48.52
log pl (X()
49.34
75.45
50.35
70.87
attribute this issue to visual bias, caused by the interference of the visual prior p(Y). Although similar phenomena have been noted in recent studies (Krojer et al., 2023 Lin et al. (024b) on diffusion and captioning models, the visual bias problem in LMMs has yet to receive adequate research attention.
Proxy Debiasing Pointwise Mutual Information. In this study, the visual bias largely stems from the unbalanced distribution p(Y), and thus we introduce an alternative proxy based on Pointwise Mutual Information (PMI) (Role & Nadif 2011 Li al. 2015
Lin et 2024b) as,
p(|I)) p(Y)"
s(X, Y) log
logp(y, X)
Σ log p(y: Yes).
where we use the visual prior pYY) to help debiasing with a strength factor 77. p(Y) can be approxi- mately estimated by p(Y]X), where X refers to a special prompt without any descriptive content, e.g., null character or "Can you give me an image?". The results in Tab, demonstratee that this proxy could significantly alleviate the visual bias issue.
4

---

Published as a conference paper at ICLR 2025
Discrete Tokenization
Forward Beam Search
Ind Rank Lest
Reverse Re-Ranking & Decision (Y)
log
MY (X)
Victoria Memonial
por
ponstriitt
Embedcing
Quantrre
Repcot
&
Encoder
Unssed
LMMs
Y-X
Model +
Enypl (X,
Large Multimodal Models (LMMs)
X --Y
Re-Rtanknn List
Word
Embedding
<amg> <img
SD
<j/mg>
X: Give me image of Victoria Memorial.
Figure 2: Overview of the TIGeR-ONE framework to unify text-to-image generation and retrieval. Images from the database are first tokenized into discrete codes and lookup table is maintained for the correspondence between discrete codes and images. The given prompt X is first fed into an LMM and Forward Beam Search is performed retrieve and generate images in parallel. The prompt and obtained images are then fed into the same LLM for Reverse Re-Ranking and Decision-making.
Proxy Reverse Conditional Likelihood. addition to the debiasing strategy, we propose another option circumvent the unbalanced prior distribution. Different from the generation process without the image Y. the retrieval task allows the model to access all images in the database. This means we can estimate the semantic similarity in reverse way by predicting the conditional likelihood of X given to alleviate the visual bias of T2I-G models. Formally,
M
s (X, Y) = (ogg(X())
Σ
(4)
where the LMMs work image captioners estimate the semantic similarity. As shown in Tab. D this reverse proxy outperforms others, effectively revealing the discriminative abilities of LMMs.
Now we could estimate similarities between given prompt and all |G] images in the database by traversing each image and calculating a proxy. Afterward, we could sort them to attain a ranking list, enabling T2I-R. Compared with the prior work GILL (Koh et al./ (202),, any of the three proxies can be seamlessly integrated with next-token prediction, the mainstream paradigm of generative pre-training. This proxy-based approach eliminates the need for additional discriminative training, such as contrastive learning, and benefits from fine-grained cross-modal interaction within LMMs.
3.3 LMMS-BASED GENERATIVE RETRIEVAL
The above proxies make it possible to calculate cross-modal similarities for T2I-R. However, is inefficient due [و] times of forward propagation, each time with the extensive parameterization and the heavy internal attention interaction. To tackle this issue and achieve an optimal balance between efficiency and recall, we introduce forward beam search and reverse re-ranking, as shown in Fig.
Forward Beam Search (FBS). Inspired by the advancement of generative retrieval (Tay et al. 2022 Li al., 2024b), we adopt constraint generation autoregressive token decoding and beam search (Freitag & Al-Onaizan 2017) with the beam size B to recall B images. Specifically, we compress images the database into discrete tokens and store them in a Trie structure. This Trie structure constrains the sampling space and ensures that the generated prefix at any timestep corresponds to at least one image in the database. Once the beam search finished, we could obtain ranking list of B sequences of visual tokens, each of which corresponds to image. This process aims to obtain list of images given the prompt, thus the direction has to be X + Y. which means we can only adopt the two forward proxies illustrated in Eqn. 2 or Eqn. 3 However, FBS method could significantly improve the efficiency since it only requires [NN < |G] times of forward propagation of LMMs, where N denotes the length of the visual token sequence for image.
5

---

Published as a conference paper at ICLR 2025
Reverse Re-Ranking (RRR). Despite the improved efficiency, the semantic matching ability of the two forward proxies is noticeably weaker than that of the reverse proxy in Eqn. I as shown in Tab. [ Given the ranking list R [Yi..... Yn) obtained by forward beam search, we resort to the reverse proxy in Eqn. g.n..for for re-ranking and attain the final ranking list R
3.4 TIGER-ONE UNIFYING GENERATION AND RETRIEVAL WITH ONE LMM
The discrete visual tokenization strategy enables LMMs to generate both language and visual content in an autoregressive generation manner. The proposed forward beam search in Sec. 3.3 performs retrieval in the same autoregressive manner to generate visual tokens (which have been tokenized and saved in a database). Naturally, we can unify generation and retrieval with one LMM under the autoregression framework and make a decision between the generated and retrieved images.
Synchronous Generation and Retrieval. In TIGeR-ONE, an LMM can synchronously conduct un- constrained and constrained token decoding processes for image generation and retrieval, respectively, As shown in Fig. 2 these two tasks can be performed in parallel by maintaining respective search paths, which only requires .N forward propagations, ensuring efficiency. Each path corresponds to sequence of discrete visual tokens. We can generate new image Y by diffusion decoder conditioned on the sequence, and meanwhile, immediately find the retrieved top- ! image y
Decision Making. Given the generated Y and the retrieved Y we choose the better one based on the discriminative ability of LMMs, as discussed in Sec. 3.3 Specifically, we calculate two similarities s sXX, YO) and (XX, Y using one of the three proxies and choose the image with the higher similarity. Since we have computed similarities between the given prompt and shortlisted images through forward beam search and reverse re-ranking using the debiasing and reverse proxies, the decision process incurs no additional computational cost and can be performed efficiently.
4
To evaluate the performance of our method on TIGeR, we build a comprehensive benchmark (TIGeR- Bench), as shown in Fig. [ It covers creative and knowledge-intensive image acquisition domains.
Creative Domains. Creative images emphasize the intricate visual content that is challenging to capture in the real world. It includes unusual and counterfactual compositions of concepts (e.g., "A steamed train bellows rainbow-hued smoke") and imaginary scenes aligning with real users' preference. To meet the two aspects, we collect prompt-image pairs from the well-designed WHOOPS! (Bitton-Guetta et al.) 2023) dataset and a large-scale open dataset named Pick-a- Pic (Kirstain et al. 2024) which stems from web platform collecting real users' creative intention.
Knowledge-intensiee Domains. Acquiring knowledge-intensive images requires models with extensive world knowledge and the ability to align such knowledge with visual objects or concepts. We focus on six knowledge domains including logo, history and news, landmark, food (Min et al. 2023), nature species, Wiki miscellaneous, and collect text-image pairs from six high-quality datasets. Different from previous content-orient -rrinted data (Lin et al., 2014) covering common objects in daily life, the collected data requires stronger cross-modal knowledge alignment, as the texts often consist solely of concept names without any descriptions in appearance. We collect pairwise image-text data from eight domains, constructing the evaluation benchmark containing 6k data samples, with 3k for creative domains and 3k for knowledge domains. See Appendix A more details.
5 EXPERIMENTS
5.1 DATASETS, BASELINES, AND EVALUATION METRICS
We WiGeR-Benchs to evaluate the unified performance, and the two widely-used benchmark datasets, i.e., Flickr30K (Young et al.) 2014) and MS-COCO (Lin et al., 22144), to assess the text-to-image retrieval performance. The compared baselines mainly include recent LMMs (Koh et al., 2023 Sun
al. 2023b Ge et al. 2023 Jin et al. 2023) which generate images, as well generation and
'This work only considers one generated image and the top-1 retrieved image, but the proposed framework also acquire more than images batch-wise and choose the hest one.

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison TIGcR-Bench. "Token" refers to visual tokenization during image synthesis, including continuous (Cont.) and discrete (Dist.) approaches. Entries by gray are expert models for T2I retrieval or generation, and those with a cyan background denote that an image query is first generated and then used for image-to-imagg retrieval. Entries with gray background denote our methods. Methods with "+" denote agents, where models with are decision models.
Method
Size
LLM
Token
CLIP-T
CLIP-1
Text-t--maage
SDXL
Podelleea al
MI24
2.68
Cont.
26.79
46,71
GILL
Koh
SB
OPT-6.7B
Coal.
14.16
13.72
LLaMA-13B
Ema
37B
LLaMA-33B
Coat.
24.25
44.24
Dreami (1xmn Bi M23
8B
Vicuna-7I8
Cont.
24.34
42,77
LLaMA-7B
LaVIT
ct al. 2025
11B
LLaMA-7B
Dist.
27.07
48.75
Test-to- Eon--o--mooge Retrieval
SDXL Podell dle bIzA
2.6B
Coot.
15.41
38.96
Emn
Ser EAPAD
148
LLaMA-138
Cont.
14,44
34,46
LLaMA-33B
DreamL
2123
8B
Vicuna-7B
Coat.
15.41
37.18
2025
88
L1 1LaMA-28
Dist.
14.78
36.93
LaVIT
11B
LLaMA-7B
Dist.
16.34
39.25
a A123
OPT-6.7B
Coot.
10.96
16.30
Ours (S.E0
SB
LLaMA-7B
Dist.
16.95
10.30
Ours (LaVIT)
11B
LLaMA-7B
Dist.
21.30
50.03
Unified Tesi-lo-lmage Generation and Reirievof
GILL (Kolh 2023 SOXI, SEED-LLaMA
8B
OPT-6,7B
Cont.
12.12
15.25
118
47.51
SDXL + CLIP Owen2-VL (Wagg et 2024
IIB
Qwen2-7B
Coal.
27.91
60:65
Ours (SEED LLaMA)
SB
LLaMA-7B
Dist.
23.98
50.52
(l
118
LLaMA-7B
28.45
61.37
Table 3:
Text-to-imee image retrieval performance comparison on Flickr30K and MS MS-COCO. Entries by gray denote dense retrieval methods and others are generative retrieval methods. Entries with gray background denote our methods.
Flickr30K (1K)
MS-COCO (5K)
Method
R@81
R@85
R@ 10
R&1
R@5
R 10
CLIP (VIT-B/22) RRdiford el 2021
68.70
90.60
95.20
37.80
62.40
72.20
GRACE GRACE (Structurcu 101
20246
37.40
59.50
66.20
16.70
39.20
$0.30
Zhang 2023
72.50
Our (LavIT)
68.84
82.92
86.44
44.81
62.61
68.28
Ours (SEED-LLaMA)
71.70
91.82
95.44
46.11
76.13
retrieval expert models (Podell et al.) 2023
Radford al. 2021). Following T21-G (Podell al.
2023 Esser et al., 2024 Saharia et al. 2022), the unified performance is measured by the CLIP score Hessel al. 2021) including CLIP-T for text-image alignment and CLIP-I for the alignment between the predicted image and the ground-trum truth image. As for T2I-R (Hessel et al., [221]] Faghri jet al. (017), we adopt the standard metric Recall at K, R@K K (K=1, 5, and 10) for short. Details on datasets, baselines, and evaluation metrics are provided in Appendix A
5.2 PERFORMANCE COMPARISON
Unified Performance on TTGeR-Benche We compare LMM baselines and our method, reporting results on TIGeR-Bench in Tab. 2 including separate and unified tasks. The base models (i.e., SEED- LLaMA and LaVIT) and Ours share the same architectures and parameters. The key differences lie in the integration of FBS. RRR. and decision decision-making mechanisms, which are training- free and non non-parametric. The results demonstrate our method outperforms expert generation (Podell et al. 2023) or retrieval models (Radford et al. (021). the state-of-the offtthe-rtt L.MMs (Koh al. 2003), and even agent methods with expert models for generation and retrieval and the strong understanding LMM, i.e., Qwen2-VL Wang et al.) (2024). as the decision model. Due to half of the data being sourced from knowledge domains, current generation models, e.g., SDXL (Podell et al. 2023), and LMMs could not handle the unified problem well. Moreover, the proposed method could achieve impressive retrieval results, especially compared with other LMMs or SDXL. Compared with the vanilla SEED-LLaME and LIVIT, our method significantly improves retrieval performance by dealing with the unified problem.
7

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study on TIGeR-Bench investigating Reverse Re-Ranking (RRR) and two decision- making strategies, i.., Forward with Eqn. of retrieved images selected results.
3 and Reverse with Eqn. Retr.. denotes the percentage
Ours (SEED-LLaMA)
Ours (LaVIT)
RRR
Decision
CLIP-T
CLIP-1
R@1 f
Ret..
CLIP-T
CLIP-1
R@1 f
&Ret.
Forward
22.63
49.71
27.19
49.59
4.38
Forward
48.86
49.62
X
50.52
28.23
56.51
Reverse
22.84
49.54
29.23
61.47
28.45
61.37
49.37
56.25
Text-to-Image Retrieval Performance Flickr30K (Young al., 2014) and MS-COCO LLi et al. 2014). As shown in Tab. 3 we compare the proposed method with the representative dense retrieval model CLIP (Radford al. 2021) and two generative retrieval baselines (Zhang al. 2023 Li et al. 2024b) which have been specially trained on the two datasets. In contrast, the proposed method is training-free but achieves the best performance across all baselines. It verifies the effectiveness of the proposed generative retrieval method and demonstrates that LMMs are capable of retrieval despite the sole optimization objective of next-token prediction.
Chat-to-Image Acquisition Performance on VisDial (Das ef al. 2017). Following GILL (Khh
al.
2023), we conducted experiments on VisDial to evaluate TIGeR-One based on LaVIT (Jin 2023) and SEED-LLaMA (Ge et al. 2023) for image acquisition in multi-turn chat scenarios.
et al.
We compared our method with GLIDE (Nichol et al. (021), SD v1.5 (Rombach et al.) 2022), GILL, LaVIT and SEED-LLaMA, as shown in Tab. 4 The results demonstrate: The original SEED-LLaMA ex- hibits the strongest multi-turn chat- to-image generation abilities, achiev- ing the highest CLIP-I scores. 2) Our proposed unified framework signifi- cantly improves the performance of both LaVIT and SEED-LLaMA. 3) The improvement of our method be- comes more pronounced with more rounds. For example, it achieves the most significant improvement in 10 rounds for SEED-LLLMA, which ifies the effectiveness of our method on complex multi-turn chat scenarios.
Table 4: Chat-to-image acquisition performance on Vis- Dial (Das et al.) (017). The CLIP-I score is used as the evaluation metric. Unlike previous approaches limited to image generation (Gen.), our method autonomously selects between generation and retrieval (Retr.).
Method
Gen. Retr. round rounds 10 rounds
Chat to Image Generation
GLIDE Nichol
2021
X
58.7
Rombach
2022
X
62.9
62.2
GILL Koh al
X
52.8
62.1
64.5
I-VIT
(Jin et all
2023
X
46.6
52.3
59.3
(Ge ct al.
2023
57.0
64.4
67.9
Unified Char-to-Image Generation and Retrieval
Ours (LaVIT)
V
51.7
60.0
65.8
+5.1
+7.7
+6.5
Ours (SEED-LLaMA)
57.0
65.4
70.9
+0.0
+1.0
+3.0
5.3 IN-DEPTH ANALYSIS
Ablation Study on Reverse Re-Ranking (RRR) and Decision-Making for TIGeR. We evaluate TIGeR-One based on SEED-LLaMA and LaVIT by different RRR and decision settings, and report the unified and retrieval performance well as the retrieval percentage in Tab. We have the following discussions: 1) RRR could consistently improve the retrieval performance for SEED- LLaMA and LaVIT, but may help in unified performance for SEED-LLaMA, because unified performance is also influenced by decision strategies. 2) Compared with the forward decision with Eqn. 3 the reverse decision with Eqn 4 could enhance the unified performance in most cases for both models, which reflects the reverse decision may have stronger discriminative power across more domains. 3) Intuitively, we expect the most correctly retrieved images can be selected and the left wrong ones can be remedied by generation. However, we find that the two LMMs may suffer from generation preference problem. Especially, LaVIT always prefers to choose generated images even though the retrieved ones are correct, shown by the low &Retr. in the first two settings. One of the reasons may be that significant gaps exist between the pre-ted and fine-tur tenetundd image generation data and TIGeR-Bench. In all, besides the modality bias discussed in Tab. [ the difference between the two directional ranking and decision may be attributed to the unbalance between captioning (image-to-text) and text-to-image data the training phase of L.MMs.
8

---

Published as a conference paper at ICLR 2025
25
30
SEEDALLNA
50
43
FBS RAR (5)
FAS [S]
Rebrieval Effciency
Ours (SFED-LLMMA)
CLIP (VIT-B/3Z)
GILL
Recall@11
20-
1
.25
40-
FIS 0.)
FBS + +RRR
1.5
30
10-
25-
0.5
20
15-
022 04 066
06 1 1.2 14
16
50
200
400
800
95
11
Dabiasing Factorn
Beam Size
Database Size (10")
Figure 3: The influence of the debiasing factor in Eqn. 3 on the forward ranking per- formance of SEED-LLaMA and LaVIT on the MS-COCO dataset. The best performance is achieved around 1) =
Figure 4: Retrieval perfor- mance on MS-COCO with different beam sizes and re- ranking strategies. Light and dark dash lines denote the for- ward and reverse ranking per- formance, respectively. S: SEED-LLaMA, L: LaVIT.
Figure 5: Comparison of trieval efficiency quantified by the number of processed prompts per second among CLIP (ViT-B/32), GILL, and the proposed generative re- trieval method based on SEED- LLaMA.
Visual Modality Debiasing for Diserininative Power. In Sec. 3.3 we discussed the visual modality bias problem with the forward log p( (X) in Eqn. 2 as the similarity proxy, and adopt debiasing x proxy log "(Y by considering the unconditional likelihood. To explore the influence of the debiasing strength, set different values of the factor 7) Eqn. and a series of results are shown in Fig. They show that the ranking performance is sensitive to the debiasing strength and reaches the highest point around r) 1, verifying the effectiveness of the unconditional debiasing strategy.
Forward Beam Search (FBS) and Reverse Re-Ranking for Retrieval. Considering the trade-off of retrieval efficiency and recall, we present FBS and RRR, respectively. As shown Fig- we compare the ranking (dotted lines) and retrieval (solid lines) performance and explore the impact of beam size and RRR. In the ranking experiments, we adopt the proxies in Sec. 3.3 to calculate similarities, and then rank the whole database. The comparison indicates that ranking with the debiasing proxy x(Y1X) seems the upper bound of FBS since FBS may miss the target image with the limited (log p(Y)""
beam size. Benefiting from the reverse proxy (log (())))) RRR could help FBS break through the ceiling and significantly improve recall. Additionally, regardless of similarity proxies or base LMMs employed, increasing the beam size can reduce the recall gap between retrieval and ranking.
Efficiency of Generative Retrieval. We analyze the efficiency of the proposed generative retrieval method for T2I-R and compare it with two dense retrieval methods, i.e. GILL Koh et al. (2023) and CLIP (Radford et al.. 2021) in Fig. S The efficiency of dense retrieval gets worse with the increase in the database size due to more matching in the common feature space. In contrast, the proposed method keeps almost constant efficiency regardless of the database size.
Prompt Expansion in Chat Scenarios. We guide Gemini-Pro (Reid et al. 2024) and GPT-4o (Ope- nAI 2024) to imagine scenario where user intends to know a concept and seeks an image. We provide them with detailed instructions and in- context examples, leveraging their expert language knowledge to process raw prompts. The prompts are then expanded into multi-round chat contexts, serving input for T2I generation and retrieval. Results in Tab. 6 indicate that unifying them could utilize the abundant knowledge within LMMs to improve knowledge-intene intensivee image acquisition.
Table 6: Performance TIGeR- BennhBBBehhh-- (Knowl- edge) in chat scenarios, across various chat gen- eration methods.
Expansion Method
Generation
T21 Retrieval
CLIP-TT
CL.IP-1
R@1
R@5
R@ 10
Raw Prompt
19.50
36.11
22.57
36.80
43.23
Gemini-Pro
17.71
34.58
17.83
31.70
36.77
GPT-4o
19.40
38.17
24.03
40.73
47.83
5.4
QUALITATIVE ANALYSIS
In Fig. we compare our methods with SDXL on TIGeR-Bench, covering both creative and knowledge domains. Besides, explore multi-turn chat scenarios with multimodal context and both image retrieval and generation or editing requirements in Fig. 7 Owing to its training- nature, our model-agnostic framework fully inherits the interfeaved capabilities of the base model, facilitating

---

Published as a conference paper at ICLR 2025
History and News
Counterfactual
P Pefference
Logo
Food
S pecies
Miscellaneous
SOHL
(SEEDLLaNE) LaMA)
Seats
Hear
Neat Year
Cu
A steamed train bellows rainbow- snoke
Running leaping lion
of Heart
the turkey
Eigin Cathedral
Fried crab curry
Four bandedd Stink Buy Wosp
Kamadhanu
Figure 6: Qualitative results on HIGeR-Bench The prefix prompt "Give me an image of"" is omitted here. Green ticks and red crosses highlight correct and wrong retrieval results.
User Prompt
Generated
Retrieved
Con recommend me place to go during my holiday trip Poris?
pity that can't along trip. my kitty
Certainly! I highly recommend the Eiffel Tower. It's s must-see iconic symbol of Ports and the recagni zoble landmarks was and stands at meters tall. You can enjoy the stunning views of the city from the top, and there also various exhibitions and held
you generate an image sitting in front of the Eiffle Tower?
throughout the addition, the Eiffel Tower is surrouncled beautiful park called the Champ de Mars, where nelox and enjoy the atmossperee.
In
Cool. Can you photo of the Eiffel Tower? just con't wait see
Awesomel Can you makeless wear sunglssses?
Figure 7: Example of multi-turn chat based SEED-LLaMA with unified generation and retrieval.
accurate interleaved image generation with identity preservation. Additionally, compared to the base model, our method can proactively retrieve knowledge-intensee sendge-intensiveeeee images (e.g., the Eiffel Tower) and maintain their key characteristics throughout the interaction process.
CONCLUSION
In this work, we start with the practical requirements for image acquisition, analyze the weaknesses of single generation and retrieval, and propose to unify these two tasks within MLLMs. Toward this end, we first delve into the intrinsic discriminative abilities of MILMs for semantic matching and propose a generative retrieval method to perform sext-to-image retrieval in an auto-regressive manner. Besides, under the same rego-segreesive framework, unify generation and retrieval synchronously and present an autonomous decision strategy to select the best image. The proposed framework exhibited effectiveness and versatility across the constructed
and two retrieval benchmarks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dalll--5 pdf. 2(3):8, 2023.
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: Whoops! lisionaadd-laguugee bench- mark of synthetic and compositional images. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2616-2627, 2023.
Nicolas Boulanger- Lewanwwwkk,, Yoshua Bengio, and Pascal Vincent. Audio chord recognition with recurrent neural networks. In ISMIR, pp. 335-340. Curitiba, 2013.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42(4))1-10, 2023.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval -aagmeeed text-to-image generator. arXiv preprint arXiv:2209.144: 14491, 2022.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. arXiv preprint arXiv:2303.15233, 2023.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 326-335, 2017.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 19822-19885,, 2021.
Rumpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. /499, 2023.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-lang and-aauuggee transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 18166-18176, 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- nigh-resslution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03300. 2024.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv: 15 2017.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint orXiv:2212.05032. 2022.
Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. arXiv preprint urXiv:1702.000 000.00000.... 806, 2017.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama and draw with seed tokenizer. arXiv preprint arxiv:2310.0//// 2023.
Will Grathwohl, Kuan-Chieh Wang. Jömn Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv=19/2.03265 2019.

---

Published as a conference paper at ICLR 2025
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint 2012.
Xuchai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. Discriminative diffusion models as few-shot vision and language learners, arXiv preprint arXiv:2305. 1tt.::32...1..,,,, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2104.085 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Hsin-Ping Huang, Xinyi Wang, Yonatan Bitton, Hagai Taitelbaum, Gaurav Singh Tomar, Ming-Wei Wei Chang, Xuhui Jia, Kelvin CK Chan, Hexiang Hu, Yu-Chuan Su, et al. Kitten: A knowledge- intensive evaluation of image generation on visual entities. arXiv preprint arX1v:2410.11824, 2024a.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2icoompeenhh: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747, 2023.
Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. Visual hallucinations of multi- modal large language models. arXiv preprint arXiv:2402.14668 2024b.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, et al. Unified language-vision pretraining in Ilm with dynamic discrete visual tokenization. arXiv e-prints, pp. arXiv-2309, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar "xXv:2001.0886// 2020.
Scunghoi Kim, Chen Jin, Tom Diethe, Matteo Figini, Henry FJ Tregidgo, Asher Mullokandov, Philip Teare, and Daniel Alexander. Tackling structural hallucination in image translation with local diffusion. arXiv preprint arXiv:2404.0m 055900 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhuudinov. Generating images with multimodal language models. arXiv preprint arXiv:2305. 17216 2023.
Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, and Siva Reddy. Are diffusion models lasion-and-language reasoners? In Thhirty-svennth Conference on Neural Information Processing Systems, 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:23002. /2192, 2023.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference computer vision (ECCV), pp. 201-216, 2018
Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Chatting makes perfect: Chat-based image retrieval. Advances in Neural Information Processing Systems, 36, 2024.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. arXiv preprint arXiv:2303. 2...:2333..6233,. 2023a.
Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R Manmatha. Ashwin Swaminathan, Zhuowen Tu, Stefamo Ernon, and Stefano Soatto. On the scalability of diffusion- based text-to- image generation. Xiv preprint arXiv:2404.02883 2024a.
12

---

Published as a conference paper at ICLR 2025
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity- promoting objective function for neural conversation models. arXiv preprint arXiv:1510.0305,, 2015.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint 1rX15 2023b.
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image- text matching. Proceedings the IEEE/CVE international conference on computer vision, pp. 4654 4662, 2019.
Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua. Generative cross-modal retrieval: Memorizing images in multimodal language models for retrieval and beyond. arXiv preprint 10X0:2240..00805, 2024b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne Preftmr: Scaling up fine-grained late- interaction multi-modal retrievers. arXiv preprini arXiv:2402.08322 2024a.
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. In International Conference on Machine Learning, 2024b.
Fuxiao Liu, Yinghan Wang, Tianlu Wang. and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint arXiv:2010.03743, 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint drXiv:2312.17172 2023.
Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. Large scale visual food recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12Xiv:2112.107444 2021.
OpenAl. Introducing gpt-4o: our fastest and most affordable flagship model. httpss:/pplt form viisio 2024. Accessed: 2024-05-26.
openai htt/://////ooo/i/////////c/////e
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955 2023.
Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction modeling for image-text text retrieval. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 1104-1113, 2021.
13

---

Published as a conference paper at ICLR 2025
Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, and Tat-Seng Chua. Dis- criminative probing and tuning for text-to-image generation. arXiv preprint arXiv:2403.04321, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
François Role and Mohamed Nadif. Handling the impact of low frequency events on co-occurrence based measures of word similarity-a case study of pointwise mutual information. In International Conference on Knowledge Discovery and Information Retrieval, volume 2, pp. 218-223, Seitepress, 2011.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10688 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 08-25278-25294 2022.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.13288 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.0522 00.2222..22.... 2023b.
I Sutskever. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3211 2014.
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory differentiable search index. Advances in Neural 2022.
Information Processing Systems,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 092888 2023.
14

---

Published as a conference paper at ICLR 2025
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Jing Wang, Weiqing Min, Sujuan Hou, Shengnan Ma, Yuanjie Zheng, Haishuai Wang, and Shuqiang Jiang. Logo-2k+: A large-scale logo dataset for scalable logo classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6194-6201, 2020.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/91, 2024.
Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint arXiv:2311.17136, 2023.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2575-2584, 2020.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. arXiv preprint arX1v:2309.0553 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Ling Yang, Zhaochen Yu, Chenlin Meng. Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal 1lms. arXiv preprint ar arXiv:2401.11708, 2024.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.0259/ 2(3), 2023.
Yidan Zhang, Ting Zhang, Dong Chen, Yujing Wang, Qi Chen, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, Fan Yang, et al. Irgen: Generative modeling for image retrieval. arXiv preprint 10X1::2303.00126 2023.
Fei Zhao, Taotian Pang, Chunhui Li. Zhen Wu, Junjie Guo, Shangyu Xing, and Xinyu Dai. Aligngpt: Multi-modal large language models with adaptive alignment capability, arXiv preprint wrXiv:2405.14129, 2024.
Kaizhi Zheng, Xuchai Hc, and Xin Eric Wang. Minigpt-5: Interleaved vision-and sasion-and-langee -an--ang generation via generative vokens. arXiv preprint 10X.v:23100002293 2023.
15

---

Published as a conference paper at ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing languae understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023a.
Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. VI-gpt: A generative pre-trained transformer for vision and language understanding and generation. arXiv preprint wrXXi:2312.09255: 2023b.
16

---

Published as a conference paper at ICLR 2025
A TIGER-BENCH DETAILS
A. DATA COLLECTION
Data Source. To comprehensively evaluate unified text-to-image generation and retrieval, we build a benchmark called T'GeR-Bench, encompassing both creative and knowledge-nntenivee domains. For the creative domains, the data derived from authentic user prompts that reflect real-world needs, requiring high levels of novelty and creativity. We collect the data from the WHOOPS! BiitoonGGuttaa al. 2023) and Pick-a-Pe Kirstain et al. (2024) datasets:
WHOOPS! [Bitton-Guee et al. (023): The WHOOPS! dataset consists of 500 commonsense-dllying prompt-image pairs created by designers. First, the designers think of counterfactual prompts by combining two elements or concepts that violate commonsense, e.g.. "Albert Einstein holding a smartphone". Next, they are guided to use text-to-image generation tools (e.g., Midjourney, DALL-E Ramesh et al. (202T). and Stable Diffusion[Rom: bbac et al. 2022)) to synthesize images using these counterfactual prompts. Finally, the designers verify the 'weirdness" of generated images to guarantee the data quality.
Pick-a-Pic Bitton- Genttn et al. (2023): The Pick-a-Pic dataset consists of rerl-world user prompts and corresponding generated images, annotated with user preference, gathered from the Pick-a-Pic web application. In detail, we collect our data from Pick-a-Pic v2
For knowledge-intensii domains, we collect data encompassing wide range of categories to fulfill users' needs for visual knowledge, including Logo-2K+ Wang ct al. (2020), Visual News Liu et al. (0020), Google Landmark v2 Weyand et al. 2020), Food2k Min al. (2023). iNaturalist Van Horn
et al. 2018), W et al. (2024).
Logo-2K+ Wang al. 2020 Logo-2K+ is a large-scale real-world logo dataset, containing 167,140 images with 2.34I categories and 10 root categories, e.g., food, clothes, and institution.
Visual News Liu et al. (2000): Visual News is a large-scale dataset comprising over one million news images along with associated news articles, image captions, author information, and additional metadata. Distinguished from other image captioning datasets, this dataset prioritizes factual contexts, including individuals, locations, and events, sourced from prominent news outlets such as The Guardian, BBC, USA Today, and The Washington Post.
Google Landmark v2 Liu et al. (202)): Google Landmark v2 includes approximately SM images annotated with 200k distinct instance labels representing human-made and natural landmarks. It is collected from Wikimedia Commons.
Food2K Min et )023): Food2K is food recognition dataset with 2,000 categories and more than I million images, covering cereal products, vegetables, bread, snack, soup and porridge, barbecue, egg products, dessert, beam products, seafood, fried food, and meat.
iNaturalist Van Horn et al. (2018): The iNaturalist dataset is constructed to reflect the diversity of the natural world, featuring an unbalanced distribution of species. It encompasses total of 5,000 species of plants and animals, accompanied by 859,000 images.
WIT Kirstain et al. 2024h: Wikipedia-based Image Text (WIT) is a large multimodal multilingual dataset, comprising 37.6 million image-text pairs representing real-world entities. It encompasses 11.5 million unique images across 108 Wikipedia languages. The texts are sourced from primary channels: reference descriptions, attribution descriptions. and alt-text descriptions.
Prompts. The WHOOPS! and Pick-a-Pic datasets contain prompts, while Visual News also provides natural language descriptions. serving user prompts or queries. For the remaining five datasets, only category names or concepts represented by single words or phrases are available. To address this, we utilize template to formulate them into complete prompt sentences, i.e. "Give me an image of [concept]".
These datasets, originally designed for different purposes, are effectively repurposed as the creative domain and knowledge-intensinn domain candidates within the HIGeR-Bendc
17

---

Published as a conference paper at ICLR 2025
Table 7: The statistics of TIGeR-Bench. We keep the ratio of 1 for creative and knowledge domains and collect 6,000 high-quality text-image pairs in total.
Domain
Data Source
#Text-Image Pairs
WHOOPS
et al.
2023
500
Creative
Pick-a-Pic Kirslain el at
2024
2500
Logo-2K: Wang et al.
2020
500
Visual News Liu ct
2020
500
Google Landmark v2 Weyand
2020
500
Knowlelgg-intensivee
Food2K Min al. 2023
500
iNaturalis/ Van
2018
500
WIT Kirstun
500
A.2 AUTOMATIC DATA FILTRATION
Data Split. To evaluate text-to-image generation and retrieval, we prioritize selecting the original test split of each dataset to construct TIGeR-Bence In cases where only validation set is provided, we default to utilizing the validation set.
Filtration Pipeline Given that all 8 datasets have undergone individual single-modale quality assessments during their construction, our emphasis now lies on cross-modal relevance and generation challenge properties. proceed with the following three steps for data filtration.
1) ensure a strong alignment between the positive text and image pairs for both generation and retrieval, employ filtering process to remove weakly relevant text-image pairs (e.g., outliers or noisy pairs) across datasets except for WHOOPS! due to its limited scale. Specifically, we calculate the CLIP-T scores (Sgr) between the ground-truth images and texts, and remove pairs with CLIP-T scores lower than 30.0. Considering the large scale of Pick-a-Pic, then randomly sample 7,500 pairs as candidates for the following human quality validation phase.
2) As discussed in Sec. [] T2I-G models may struggle with synthesizing knowledge-intensive images. To identify challenging concepts in the above six knowledge-intensive datasets, which pose difficulties for current stateoff-the-art T2I-G models, first employ open- sourced models including the SD series Rombach et al. (2022); PPodell et al. (2023) to generate images by feeding the textual prompts candidates as conditional input. Subse- quently, we calculate the average CLIP-T scores (Syen) over images generated by multiple models for each prompt. We then calculate the difference between the scores of the ground- truth pair and the generated pair for each prompt, i.e., A Syt
Sgen-
3) Finally, we select the top 1,000 unique instance pairs comprising 1. 1,000 different prompts and 1,000 different images with the highest values of A for each knowledge dataset. The remaining examples form a new candidate set with 500 WHOOPS! instances, 7,500 Pick-a-Pic instances, and 1,000 instances for the six knowledge datasets.
A.3 HUMAN ANNOTATOR FILTRATION
To further improve the data quality of VTGeR-Benncl human annotators were employed to mark evaluate each text-image pair across three aspects: text, image, and pair. Specifically, for each text-image pair. considerations include the conciseness and unambiguity of the text, the clarity and usefulness of the image, and the relevance of the text-image pair. Annotators assigned score of 0 (not satisfied) or I (satisfied) for each aspect. Finally, only text-image pairs meeting satisfaction across all three aspects were retained.
A.4 DATA SAMPLING
To strike balance between adequacy and efficiency in evaluation, retain all 500 samples in WHOOPS!, and further randomly sample 2,500 data instances from Pick-a-Pic, along with 500 instances from each knowledge-intensive dataset. The statistics of TTGeR-Bench presented in
18

---

Published as a conference paper at ICLR 2025
Fig. Maintaining balanced ratio of 1 between creative and knowledge domains, finally obtain a total 6,000 high-quality text-image pairs.
B
METHOD DETAILS
B.1
BEAM SEARCH
Beam search Graves 20122; Boulanger- Lewandowski et al. (2013) was originally proposed for decod- ing tokens in sequence-to- sequuencce models and widely used in neural machine translation Sutskever (2014). Based on breadth-firg search (BFS), explores search tree from the root to the leaves. At each level, beam search generates all possible child nodes based the current prefixes, then sorts and selects the top-B paths by their conditional likelihood. Unlike BFS, which considers all paths, beam search maintains only B valid paths at each level and prunes others. As result, it produces B ranked sequences.
Boulanger-Lewandowskr
B.2 BASE MODELS
In this work, we introduce and implement our approach for unified text-to- image generation and retrieval, based on two foundation MLLMs: SEED-LLaMA Ge al. (2023) and LaVIT Jin al. (2023). The details of these two models are as follows.
SEED-LLaMA produces 32 discrete visual codes for each image via the SEED tokenizer. This tokenizer is composed of a Causal Q-Former, a learnable codebook, and an MLP (only for training), and is trained with contrastive learning and reconstruction objectives. SEED-LLaMA takes discrete visual codes input for multimodal understanding tasks such as image captioning and VQA, and outputs discrete visual codes. The output codes are then fed into the unCLIP-SD model Rombach et al. (2022); Ramesh et al. (2022) to generate images.
LaVIT obtains discrete visual codes with variable lengths using a meticulously designed dynamic visual tokenizer, which comprises token selector, token merger, and reconstruction decoder (used solely for training). This tokenizer trained with a reconstruction objective. During tokenization, LaVIT samples binary decision mask from Gumbel distribution to select visual patches and quantize them into discrete tokens. To ensure reproducibility and stability in tokenization, we depart from VIT and employ deterministic selection method, where patch is selected if its importance score exceeds threshold of 0.5; otherwise, it is discarded. With this discriminative tokenization strategy, we pre-tokenize the 6 knowledge-intensie maximum, and minimum lengths of discrete tokens at 88, 130, and 37, respectively. During image generation, LaVIT first autoregressively produces sequence of discrete visual tokens and then decodes them into an image using diffusion model initialized with SD-v1.5 Rombach et al. (2022) or SDXL Podell et al. 2023). In contrast to SEED-LLaMA, which utilizes discrete visual tokens as input for multimodal understanding and generation, LaVIT takes continuous visual features from the token merger as input.
datasets of TIGeR-Beneee resulting in average.
C EXPERIMENTAL DETAILS
C.I BASELINES
As shown in Tab. 2 we compare the proposed method with several basclines on TIGeR-Bench across three aspects, i.c., text-to-image generation, text-to-image retrieval, and unified generation and retrieval. We introduce these baselines in the following.
Text-to- Image Generation Baselines: There include the expert model PDXLlodelll et al. (2023) and recent MLLMs with image generation abilities. The MLLMs in this category are GILL Koh et al. (2023). Emu Sun et al. (2023b), Emu Sun et al. (023a). DreamL Dong
ct al. (023). SEED-LLaMA Ge al. (202)). and La Jin al. (2023)
Text-to-Imaen Immgt Retrieval Baselines: These include the expert model CLIP (ViT-B/32) Rad ford et al.] (2021) and recent MLLMs. Currently, GILL Koh et al. (2023) is the only MLLM with retrieval ability, which maps the embeddings of special visual tokens into the CLIP
19

---

Published as a conference paper at ICLR 2025
Table 8: Unified performance comparison on the CLIP-T score across 8 domains in TIGeR-Bench.
Method
Creative Domains
Knowledge Landmark
Domains Nature
Counierfactual
Preference
Logo
News
Food
Wiki
SDXL
Podell
2123
36,90
30.09
16.36
24-96
21.51
26.00
24.25
20.99
26,79
NIT-B5 LONTNTO et
2021
36.17
32.51
Glll1Kohe Kohee al MI21
10.80
11.22
14.65
9.49
14.31
12.92
13.64
13.55
12.12
Emu
23.61
23,98
17.24
19.21
21.54
23.43
21.25
21.00
22.26
Dreani Sun 24
2023a
27.16
23:17
25.57
25.13
24.27
23.31
20.78
24.20
23.98
SEED:LLaMrt Labzztie 01
2025
27.18
23,97
16.73
19.66
19.03
22,66
19.63
19.29
22.00
LaVIl ot
(OUS)
23.31
LaVIT (Ours)
32.05
23.39
35.87
24.30
32.38
31.28
27.59
31,01
28.45
feature space. Although other MLLMs Sun et al. (202bb a); Dong et al. (2023 x Ge et al. (202)) Jin al. (2023) do not directly support text text-to-iman to-mage retrieval, evaluate them through a two-step process: 1) generating an image query conditioned on the text prompt, and 2) performing nearest neighbor search for image-to-image retrieval using the CLIP (ViT-B/32) image encoder as the feature extractor and cosine similarity as the metric.
Unified Text-fo-Image Generation and Retrieval: GILL Koh et al. (2023) is the only baseline capable performing both text-to-image generation and retrieval. incorporates and trains a binary classifier to decide between generation and retrieval tasks.
C.2 IMPLEMENTATION DETAILS
The proposed method is training-free and based on SEED-LLaMA Ge et al. (2023) and La Jin (ct al. (2023). We utilize the 8B version of SEED-LLaMA and load the parameters of supervised fine-tuning, For LaV,T, employ the 11B model with SDXL the pixel decoder. We combine all images the mnowledge-intensive datasets and tokenize them into discrete tokens. Subsequently, build the mapping between images and tokens. Based on these discrete tokens, construct Trie for efficient storage and constrained generation. The beam size for retrieval set to 800, and the timestep for generation is 25.
D ADDITIONAL EXPERIMENTS
In this section, we carry out extensive experiments and obtain quantitative and qualitative results to explore the unified text-to-image generation and retrieval problem and the proposed MLLMs-based method.
D.1 ADDITIONAL QUANTITATIVE RESULTS
D.1.1 UNIFIED PERFORMANCE COMPARISON
To broadly compare the performance of baselines and method for unified text-to-image generation and retrieval, we report the results with the CLIP-T score and the CLIP-I score as the evaluation metrics across 8 domains in TIGeR-BER TIGeR-Bench, in Tab. & and Tab. 9 respective. In addition, show the retrieval percentage of our method on 8 domains in TIGeR- Bench to understand the automatic decision in Tab. 10
1.1.2 PROMPT/QUERY EXTENSION
In this section, we mainly study the influence of prompts on the retrieval and generation performance in inowledge-intensive scenarios. Toward this end, we first let SEED-LLaMA, Gemini Pro, and GPT- 4o to explain the raw query with knowledge concepts from domains. Subsequently, we concatenate the raw prompt and expanded ones to form new long text prompts and feed them to SEED-LLaMA. The results for text-to-inme to-iiage generation and retrieval are listed in Tab. IT and Tab. I2 respectively. We can that prompt/query expansion with strong LLMs could promote both generation and retrieval performance. Mcanwhile, weak LLMs may introduce false explanations and do harm to the generation and retrieval performance.
20

---

Published as a conference paper at ICLR 2025
Table 9: Unified performance comparison on the CLIP-I score across 8 domains in TIGeR-Benche
Method
Creative Domains
Knowledge Landmark
Domains Nature
Counierfactual
Preference
Logo
News
Food
Wiki
SDXL
Podell
2123
65.91
55.38
14.21
42,09
35.43
44,94
45.60
35,40
46,71
NIT-B5 LON et
2021
75.38
[0l1KKhheta al MI21
15.93
13.61
20.38
15.96
14.34
18.25
15.49
14.52
15.25
Emu
43.17
43,95
22.23
34.25
38.53
46.97
48.56
35,97
-00.78
Dreani
Sun 2e
2023a
53,93
46.22
33.16
32.09
37.87
46.06
43.82
35.13
42.77
SEED-LLaMAtio Labztiee 11
2025
52.76
49.37
18.50
37.89
33.78
45,46
46.55
34,50
43.02
Laviiiinn
48.75
(OUS)
51.99
LaVIT (Ours)
60.80
46.56
92.68
57.32
72.88
74,70
69.79
75.44
61.37
Table 10: Retrieval percentage of our method based on SEED-LLaMA and LaVIT on domains in TIGeR-Bench.
Method
Counterfactual
Creative Domains
Preference
Logo
News
Knowledge Landmark
Demains
Nature
Food
Wiki
SHED&LLMM (Ours)
1.0%%
9.6%
44.2%
44.2%
53,6%
26 2%
40.2%
49.8%
25.6%
LaVIT (Ours)
15.0%
27.5%
99.8%
82.0%
88.6%
80.6%
87.6%
84.0%
56.3%
Table 11: Text-to-image generation performance on TIGeR- Bench (Knowledge) in long text scenarios, with CLIP-T and CLIP-I scores as evaluation metrics across various prompt/query expansion methods including Exta-sonnnno,n, Gemini Pro, and GPT-4o. For expansion, we guide LLMs to explain the appearance characteristics in detail with their expert language knowledge for given queries by giving them detailed instructions. After that, the queries can be expanded into longer texts and are combined with raw queries input for text-to-image generation. We perform generative retrieval with 200 beams.
Ours (SEED-LLaMA)
Ours (LaVIT)
Expansion Method
CLIP-T
CLIP-I
CLIP-T
CLIP-I
Raw Query Self- Seff-Expansion
19.50
36.11
24.16
41,84
18.07
35.12
Gemini-Pro
17.48
34.74
22.38
39.28
GPT-4o
20.36
38.95
23.91
42.59
Table 12: Text-to-image retrieval performance TIGeR-Bench (Knowledge) in long text scenarios, with recall as the evaluation metric across various prompt/query expansion methods including Self- Expansion, Gemini Pro, and GPT-4o. For expansion, we guide LLMs explain the appearance characteristics in detail with their expert language knowledge for given raw queries by giving them detailed instructions. After that, the queries can be expanded into longer texts and are combined with raw queries input for text-to-image retrieval. We perform generative retrieval with 200 beams.
Ours (SEAD-LLaMA)
Ours (LaVIT) R@5
Expansion Method
R@5
R@10
R@1
R@ 10
Raw Query
22.57
36.80
43.23
43.63
49.40
Self-Expunsion
17.20
30.10
36.77
Gemini-Pro
18.57
34.27
40.30
19.07
36.80
43,10
GPT-4o
25.00
42.50
48.90
25.20
46.03
52.17
We also carry experiments in multimodal chat scenarios explore the interplay between genera- tion and retrieval, as shown in [33 Specifically, we concatenate the retrieved top-1 images behind the chat context and then evaluate the generation performance. Similarly, we concatenate the generated images behind the chat context and then evaluate the retrieval performance.
1.3.3 ABLATION STUDY
In this part, we conduct comprchensive ablation studies on the components of the proposed method study the effectiveness.
21

---

Published as a conference paper at ICLR 2025
Table 13: Text-to-image generation and retrieval performance on TIGeR-Bench (Knowledge) in multimodal chat scenarios. Based on the chat contexts with pure text generated by GPT-4o, we can perform generation and retrieval. Afterwards, we concatenate the generated or retrieved top-1 images with the chat contexts and form the multimodal context, to explore the influence on retrieval and generation, respectively. Considering that LaVIT was not fine-tuned by chat instructions, we only carry out experiments based on SEED-LLaMA, We perform generative retrieval with 200 beams.
Text-to-Imian Image Generation
Text-to- Image Retrieval
Expansion Method
Image Context
CLIP-T
CLIP-I
R@
R@5
R@
Raw Query
19,50
36.11
22.57
36.80
43,23
GPT-4o
Retrieved Generated
18.62
38.20
GPT-4o
15.87
29.13
35.60
Table 14: Comparison CLIP-T score between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT 8 domains of TIGeR.
Creative Domains
Snowledge Domains Landmark
Method
All
Counterfactual
Preference
Logo
News
Nature
Food
Wiki
Ours (SEED-LLaMA)
Generation
27.18
23.97
16.73
19.66
19.03
22.66
19.63
19.29
22.00
Retrieval
11.04
10.36
27.87
26.21
23.73
20.25
19.56
22,99
16.95
Unified
27.16
23.47
25.57
25.13
24.27
23.31
20.78
24.20
23.98
Ourr (LaVrT)
Gencration
34.60
29.07
16.70
25.17
24.14
29.59
25.07
24.26
27.07
Retrieval
13.21
12.17
35.84
23.64
32.58
31.08
27.61
30.76
21.30
Unified
32.05
25.39
35.87
24.30
32,38
31.28
27.59
31.01
28.45
Table 15: Comparison CLIP-I score between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT 8 domains of TIGeR-Benche
Creative Domains
Knowledge Domains Landmark
Method
All
Counterfactual
Preference
Logo
News
Nature
Food
Wiki
Ours (SEED- LLaMA)
Generution
27.18
23.97
16.73
19.66
19.03
22.66
19.63
19.29
22.00
Retrieval
11.04
10.36
27.87
26.21
23.73
20.25
19.56
22,99
16.95
Unified
27.16
25.57
25.13
24.27
23.31
24.20
Ours (LaVIT)
Generation
34,60
29.07
16.70
25.17
24.14
29.59
25.07
24.26
27.07
Retrieval
13.21
12.17
35.84 25.84
23.64
32.58
31.08
27.61
30.76
21.30
Unified
32.05
25.39
35.87
24.30
32.38
31.28
27.59
31.01
28.45
First, we compare the alignment performance between separate generation, retrieval, and unified variants across all 8 domains. The results are listed in Tab. and and Tab. [15 with the CLIP-T and CLIP-I score as the evaluation protocols, respectively. Besides, although Flickr30K and MS-COCO the general datasets describing daily common scenes, also investigate the three variants them, as shown in Tab. 16
Second, further study the effects of the directions of re-ranking and decision-making, across domains in Tab. 17 and 2 general datasets, i.e., Flickr30K and MS-COCO, in Tab. [8
In addition, delve into the diseriminative abilities by forward and reverse ranking methods, as well as forward beam search and reverse re-ranking in Tab. 19 on 6 knowledge-intensive domains.
22

---

Published as a conference paper at ICLR 2025
Table 16: Comparison between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on the Flickr30K and MS-COCO datasets. Performance is evaluated by the CLIP-T score.
Method
Flickr30K
MS-COCO
Ours (SEED-LLaMA)
Generation
28.65
27.74
Retrieval
29.86
28.73
Unified
30.01
29.09
Ours (LaVIT)
Generation 37.05
35.59
Retrieval
27.54
27.13
Unified 33.69
32.24
Table 17: Ablation study on 8 domains of TIGeR Bench investigating Reverse Re-Ranking (RRR) and two decision-make making strategies, i.e., Forward with Equ. and Reverse with Equ. Performance is evaluated by the CLIP-T score.
BeGch-Bench
Creative Domains
Knowledge Domains Landmark
RRR
Decision
Counterfactual
Preference
Logo
News
Nature
Food
Wiki
All
Ours
(SEED-LLaMA)
Forward
26.64
22.41
24.02
23.64
22.85
20.34
19.63
22.40
Forward
14
23.95
23.57
23.18
22.82
24.57
20.62
22.93
23.72
Reverse
27.16
23.47
25.57
25.13
24.27
23.31
20.78
24.20
23.98
Reverse
26.81
20.19
23.22
24.16
26.04
21.79
23.84
22.84
(LTVIT)
Forward
34.59
29.00
16.84
25.53
24.74
29.56
24.93
25.05
27.19
Forward 32.05
29.07
17.15
25.57
24.76
29.60
25.17
25.20
27.28
Reverse
33.83
28.17
24.67
28.88
28.06
29.02
25.65
27.84
28.23
Reverse
32.05
25.39
35.87
24.30
32.38
31.28
27.59
31.01
28.45
Table 18: Ablation study Flickr30K and MSCOCO investigating Reverse Re- (RRR) and two decision decision-making strategies, i.e., Forward with Equ. and Reverse with Eqn. %Retr. denotes the percentage of retrieved images selected as results.
Flickr30K R@1
MS-COCO R@ t
RRR
Decision
C
Retr..
CLIP-T
Retr.
Ours -SaMA)
Forward
28.89
58.50
39.52
25.95
26.17
67.61
Forward
29.68
71.70
26.92
28.71
46.11
33.91
Reverse
30.01
58.50
35.98
28.61
26.17
26.23
Reverse
30.02
71.70
51.88
29.09
46.11
60.69
Ours (LaVIT)
Forward
37.03
47.86
0.20
35.34
23.20
3.44
Forward
37.04
68.84
35.58
44.81
Reverse
36.18
47.86
24.34
34.67
23.20
20.60
Reverse
33.69
68.84
41.84
32.24
44.81
49.11
D.2 ADDITIONAL QUALITATIVE RESULTS
We showcase examples of our SEED-LLaMA and LaVIT in both creative and knowledge- intensive domains Fig. and Fig. 9
In the creative domain, the CLIP model, limited to retrieving images from the database, shows significant discrepancies when compared to the ground truth images. Our SEED-LLaMA and LaVIT, capable of both generation and retrieval, tend to favor image generation in the creative domain. However, our models also exhibit decision emrors. For instance, as demonstrated in the last two rows of Fig. & the models incorrectly selected misleading retrieved images.
23

---

Published as a conference paper at ICLR 2025
Table 19: Recall@ @ performance comparison of Forward Ranking, Reverse Ranking, Forward Beam Search (FBS) with different beam sizes, and BFS + Reverse Re-Ranking (RRR). Experiments are conducted based on SEED-LLaMA and LaVIT on 6 knowledge-intensing domains of JIGeR-Bence
Method
Logo
News
Landmark
Nature
Food
Wiki
ALL
Ours (SEED-LLaMA)
Forward Ranking
56,00
45.60
22.60
2.60
4.40
30.40
26.93
Reverse Ranking
61.80
40.40
26.60
15.00
7.00
32.40
30.53
FBS (#Bcam=100)
39.40
29.60
10.40
8.80
4.60
19.40
18.70
FBS (#Bcam=800)
56.00
46.80
20.20
3.60
4.60
29.60
26.80
FBS (#Beam=100) RRR
37.40
25. 80
10.00
10.20
6.00
18.60
18.00
FBS (#Beam=800) RRR
61.20
39.60
22.60
15.40
6.80
29.80
29.23
Ours (LaVIT)
Forward Ranking
37.80
52.20
25,00
10.00
11.40
33.20
Reverse Ranking
92.20
41.00
56.00
36.40
23.60
63.80
52.17
FBS (#Bcam=100)
16.60
46.00
20.20
9.40
10.00
28.40
21.77
FBS (MBeam=800)
37,00
53,40
24.40
10.80
11.40
31.80
28.13
FBS (#Beam=100)) RRR
27.00
30.80
30.60
16.80
15.40
41.00
26.93
FBS (#Ream=800)- RRR
87.20
40.40
51.60
34.60
23.00
59.40
49.37
As shown in Fig. 8 our models has the advantages over SDXL in the knowledge-intensive domain, accurately retrieving the correct results. However, decision errors still occur. We leave further exploration of the decision strategy for future work.
In Fig. 10 we compare our models with current Text-to-Image baseline models such as Emu2, LL,, and GILL, which can autonomously decide between retrieval and generation. Our models are consistently retrieving the correct images in the knowledge-intensive domain. In this domain, Emu2, DreamLLM, and GILL fail generate closely matching images, highlighting the limitations of current MLLMs.
We further explored scenarios: Augmented Generation for Better Retrieval and Augmented Retrieval for Better Generation. In the Augmented Generation for Better Retrieval scenario, we first use the MLLM's capability to generate an image before performing image retrieval. The generated image, along with the retrieval prompt, is then used as input for the retrieval process. As shown in Fig. generating an image beforehand improves the model's retrieval performance.
In the Augmented Retrieval for Better Generation scenario, we leverage our model's generative retrieval capabilities to perform an image retrieval before generating an image. The retrieved image, along with the generation prompt. is then used input for the generation process, similar Retrieval- Augmented Generation (RAG). As shown in Fig. performing image retrieval beforchand improves the stability and quality of the generated images.
One major limitation of the CLIP model for retrieval is its limited context length. Our model leverages the advantage of the LLM's 's long context length, making retrieve with longer prompts possible. As extending the prompt further enhances the retrieval performance of our model.
shown in Fig. 13
E FUTURE WORK
In the future, we plan to investigate the root causes of modality biases from various perspectives, including data distribution, model architecture, and optimization objectives. We will also examine the potential impacts of these biases on generative and discriminative tasks. Additionally, we aim to study more complex contexts involving interleaved multimodal content advance comprehensive unified generation and retrieval tasks. Finally, it would be valuable to explore the deeper relationships and possible interactions between generation and retrieval (e.g., retrieval-aus augmented generation and generation-augnet augmented retrieval) within the TIGeR framework.
24

---

Published as a conference paper at ICLR 2025
LaVIT Gen
LaVIT Ret
SEED-LLaMA
SEED-LLaMA
Ground Truth
CLIP-B
Gen
Ret
Baby eating a hot chili pepper
PICKFORD
Mary Poppins flying with balloons
Alpha -Bits
A pile of fruit has blue apple
CAMAY
a pink pistol
digital painting of european dragon with wings
lacT
Movie keyart of mario surrounded by goombas zoombies, guns, bullet tracing vfx, hazing environment, drone view from the sky, cinematic light, by kyoto animation and peter mohrbacher, high radiosity, splatter on ground, color dodge on lights, simon stälenhag background, green pallete
pregnant lady
inside of the black hole, super realistic, string theory, space neurons and neural networks, parallel universes, radiated radar view, twisted loophole
Figure 8: Qualitative results in TIGeR-Bende creative domain. We use ticks or crosses highlight the selected results from generation or retrieval. Green ticks indicate the correct generated images and red crosses indicate wrong retrieved images. Black ticks refer to the correct retrieved images despite the creative domain.
25

---

Published as a conference paper at ICLR 2025
LaVIT Gen
LaVIT Ret
SEED-LLaMA Gen
SEED-LLaMA Ret
Ground Truth
SDXL
ANITE NOLN DAYT HAS ON
i O preem
f preem
F preem
c'tsss
The logo of Preem
Prinets
the worst princess by anna kemp and sara ogilvie
Sir Thomas Brisbane Planetarium
Chicken heart string
Moonflower
TAME POOUS
Dangerous Love (1920 film)
Shiogama-jinja
international space stationa bright orange zinnia has blossomed aboard the international space station marking first in space us astronaut scott kelly who took the photograph tweeted first flower ever grown in space makes its debut zinnias grow easily earth and blossom in abundance in summer but on the iss they have had trouble adapting microgravity
Figure Qualitative results in TIGeR-Bende knowledge domain. We use ticks or to highlight the selected results from generation or retrieval. Green ticks indicate the correct retrieved images and red crosses indicate the wrong generated images.
26

---

Published as a conference paper at ICLR 2025
Emu2
DreamLLM
GILL
SEED-LLaMA
LaVIT
distanndTT
r
MSTMolllll
MIDE MOTE MIDY EILL PUP HE ERTASR MIUNN HOUR
Microsoft
Microsof
The logo Microsoft
LIFAN Store No: 612232
BENEFITS STREE
BENEFITS STREE
benefits street prompted debate about socalled poverty pom
BEL BELINDA
DO TO
J0S
JE0 DOION
DOLON
MAGORIC
some 700000 federal workers have been sent home unpaid leave
MED MEIE WITTE 9EA
Old Swedes, Wilmington
SABm DIPISTS CRTS
Anterior cingulate cortex
CANKT (AIPTH)
GAIGA STER NUR MIBPU
Cape Sugarbird
Mixed agaric with corlander
AB BA
Alora turbinata
Figure Qualitative results in TIGeR-Bench knowledge domain compared to Text-to-Image generation baselines. Green tick indicates the final output by our model is correct and Red cross indicates the final output by our model is incorrect.
27

---

Published as a conference paper at ICLR 2025
User Prompt
Generated
Retrieved
f3 Augmented Generation for Better Retrieval
recently saw an image taken with the NASA/ESA Hubble Space Telescope featuring something that seems to be smiling. Can you tell about
Hi, heard about the HP TouchSmart bx2z.
Can you tell me what it is?
The HP TouchSmart bx2 convertible laptop that released 2009.
It features a touchscreen display that can be folded, allowing function a laptop and tablet. was one of the early models to popularize touch Input is known for innovative design.
the center of this image is the galaxy cluster SDSS J1038+4849. The 'smila' effect gravitational lenaing. where the massive gravity of the cluster bends light galaxies creating curved arcs that look like smiley face.
That's fascinating! Can you show me an image galaxy cluster?
Please give image the HP Laptop.
Cool. wander what the TouchSmart 1x2z looks like. Can you show me?
Can you show me the image of the galaxy SDSS J1038+4849 be smiling?
Microsoft
Figure 1I:
Augmented generation for better retrieval. Green box the additional generation step. Green tick indicates the final output by our model is correct and Red cross indicates the fina3l output by our model incorrect.
28

---

Published as a conference paper at ICLR 2025
User Prompt
Generated
Retrieved
Augmented Retrieval for Better Generation
Hi, heard about Rose cake. Can you tell me what is?
Hi. I plan visit Sweden and have heard about Degeberga kyrka. Can you tell me what it is?
Rose cake is traditional Chinese pastry that incorporates rose petals and sometimes rose essence, giving distinct Typically enjoyed during special occasions, known delicate, sweet flavor and soft,
Degeberga kyrka traditional Swedish church featuring whitewashed exterior with tall, pointed red-tlled roof and a square tower lopped spire, siluated picturesque rural setting. Generate an image of Degeberga kyrka.
Sounds delicious! Can you show an image rose cake?
Can you show image of Degeberga kyrka?
I want to design new building like it. Can you generate an image?
My brother's birthday is coming. Can you give me design drawing rose cake?
Figure 12: Augmented retrieval for better generation. Green box is the additional retrieval step. Green tick indicates the final output by our model is consistent and Red cross indicates the final output by model inconsistent.
29

---

Published as a conference paper at ICLR 2025
Prompt Retrivval
Prompt Retrivval
Generate image of Picklid sweat garlic
Generate image of dezrkk dekdra drayton 32
Pickied sweet gafie typioally appears small, whole garlic doves submerged siigtly amber celor-d brine, ottan glossy, sightly look sometimas jar. image hint garlie.
Deonka Deldra Draylon, 32-year old indildual, was one th victims of nishtclu shooting Orlando, Flonce, which image drayton
LavIT
LaviT
SEED-LLaMA
SEED LLD-
Ganarate image bloams flamonce
Generate of the Mire
Twotone blooms Linaria reliculsta 'Flamence' vibrant, delicate
mlRC styliced combination lowercase appears lowercase uppercase yellow, playtul design.
bright yellow colors, combiration visual appeal. Generate image adding blooms Inaria reticuiata flamanco
Generate an Image of the logo at Mire.
LeViT
LaVIT
Maroplane
BEED-LLaMA
SEED-LLaMA LLaMA
Figure 13: Short prompt and long prompt retrieval comparison on TIGeR-Bench knowledge domain. Red box is the retrieve result of short prompt. Green box is the retrieve result tick indicates the final output by our model correct and Red cross indicates the final output by our model is incorrect.
of long prompt. Green
30


Reviews:
[{"review_id": "wgjcT85SAW", "reviewer": "Reviewer_x2Jm", "summary": "This paper presents an approach to address the limitations of current text-to-image generation (T2I-G) and retrieval (T2I-R) systems. Particularly, it propose to unify generation and retrieval auto-regressively, and use a decision making model to decide whether to use generated or retrieved image as the response to a text prompt.", "questions": "Please see weakness for my questions\n\n--- Post-rebuttal comments:\n\nI improved my scores based on authors' response.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Unification of Generation and Retrieval: The core contribution of unifying T2I-G and T2I-R within a single framework centered with LMMs. This framework leverages the strengths of both paradigms, mitigating the limitations of relying on either alone for offering visual content per user query. In an ideal world, the users can obtain a factual image generation when its query is centered at knowledge-intensive factual entity, and see a creative image when queried for imaginary scene.\n\n- New benchmark: This paper creates a dedicated benchmark, TIGER-Bench, which accesses image generation in both creative and knowledge-intensive domains, and offers a more comprehensive evaluation platform.", "weaknesses": "- Lots of details related to presentation needs improvement: \n\n1. Figure  1 is referring to many things that has not be introduced before, it is hard for people to understand forward beam-search & reverse re-ranking at the first glimpse. Many terms are not really explained in caption, or the introduction and people needs to leap to methodology section to really understand the meaning. \n\n2. Section 3.2 is not very clear to me, I couldn't find a connection one why those three metrics are used, and also why the metric it has to be training-free. Meanwhile, I also think that comparing to those proposed proxy metrics, a more straightforward way is to directly ask the LMM a visual question, i.e. \"<image> Is the presented image is aligned with the following caption? <caption>. Answer yes or no.\", and then measure the likelihood on yes and no. \n\n3. Given that the authors also agree that \"It is inefficient due to |G| times of forward propagation\" for computing the proposed cross-modal similarity, why not considering leveraging a contrastively trained text-to-image similarity function? The forward beam search + reverse re-ranking seems unnecessarily complicated.\n\n- Lack of simple baseline method: It would make a lot sense to compare a simple zero-shot / fine-tuned LMM that makes a decision between whether to use the SDXL's generated image or CLIP model's retrieved image, according to a user's input query. IMO, this method would be very competitive to the proposed approach.\n\n- The core experimental result is based on author's proposed benchmark, which is not very convincing given that it is really hard to judge the quality and validity of this newly introduced benchmark. It appears to me that the authors are being the player and judge at the same time, which is quite tricky to assess the significance of this work. \n\n- Missing relevant works: \n\nThe following works have discussed how to leverage image retrieval for image generation and should be discussed and cited. \n\n1. Re-Imagen: Retrieval-Augmented Text-to-Image Generator\n2. KITTEN: A Knowledge-Intensive Evaluation of Image Generation on Visual Entities", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ReFa5pJNaX", "reviewer": "Reviewer_VQ4r", "summary": "This paper studies how to effectively combine the T2I generation and retrieval into a combined framework. For this task, this paper makes two folds of contribution: 1) This paper proposes a new T2I generation and retrieval pipeline, where the generation and retrieval branches run in parallel, followed by a final image ranking/selection process to decide the final output. 2) This paper proposes a benchmark to evaluate the joint T2I generation and retrieval. The benchmark contains half creative and half knowledge based image-text pairs.", "questions": "Please see the weakness section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The overall presentation of this paper is good. This paper contains a lot of content which is hard to make the presentation organized and easy to follow. The authors did a good job in my opinion.\n- Joint T2I generation and retrieval is a important task. The proposed benchmark set is key to make different methods devoted for the task comparable.\n- The experimental results show that the proposed method has good performance than other generation or retrieval-based methods.", "weaknesses": "- In my opinion, the biggest drawback of this paper is that it lacks technical novelty. This work looks more like a prototype of an application than a research paper. All modules used by this paper has been used by previous papers and the combination of them contributes very limited new knowledge or insight. I understand that this work belongs to a type of papers that is more engineer than research. I put this point inside the weakness section but I am open to any discussion about the fit of this paper to iclr with authors, other reviewers, and AC if possible.\n- I am confused about the comparison results in Table 2. What is the difference between the SEED-LLaMA vs Ours (SEED-LLaMA)? Does the latter one used both retrieval and generation? If so, this comparison seems to be somewhat unfair. If not, what is the components of the proposed method that lead to the performance improvement over the baseline. More discussion is welcomed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5xU3t9i9k4", "reviewer": "Reviewer_KRUZ", "summary": "This paper introduces a unified framework for text-to-image generation and retrieval using LMM. It proposes an efficient method leveraging LMMs' discriminative abilities. It also presents an autonomous decision mechanism for selecting between generated and retrieved images. The authors also developed TIGeR-Bench to evaluate generation and retrieval abilities across creative and knowledge-intensive domains simultaneously.", "questions": "see the weakness part", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to follow. \n- Efficient retrieval and accurate decision-making are crucial in unifying generation and retrieval, balancing creativity and knowledge. The proposed TIGER-One solves these two problems within one LMM is elegant.\n- The proposed TIGER-One seems effective and shows superior generation results in experiments.", "weaknesses": "- A  preliminary introduction of beam search is recommended to add.\n- The authors indicate that the similarity score is calculated by one of the three proxies. More details and discussion of the proxies chosen in decision-making are recommended. \n- With the help of TIGER-One, the LMM demonstrates improved results (Tab. 2).  It’s interesting to explore the addition of dream images based on knowledge (RAG) as a third option alongside generation and retrieval in future work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zerDjhiKm3", "reviewer": "Reviewer_xRkP", "summary": "1. Explore the retrieval capabilities of Large Language Models (LMMs) to enable text-to-image (T2I) generation retrieval.\n2. Unify generation and retrieval in an autoregressive manner.\n3. Propose an autonomous decision mechanism to select the best-matched image from generated and retrieved images.\n4. To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both 5. creative and knowledge-intensive domains.\n6. Conduct extensive experiments on TIGeR-Bench and two retrieval benchmarks, namely Flickr30K and MS-COCO.", "questions": "As shown in Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Propose a unified framework for text-to-image generation and retrieval.\n2. Propose an autonomous decision mechanism to automatically select the most suitable image from the retrieved images.\n3. To validate the capability of unified generation and retrieval, introduce a new benchmark, TIGeR-Bench.\n4. Extensive experiments on TIGeR-Bench and two retrieval benchmarks, i.e., Flickr30K and MS-COCO, demonstrate the pipeline's capabilities.\n5. The article is well-written, with clear and precise explanations.", "weaknesses": "1. The challenges faced by generation models in knowledge-intensive tasks are mentioned, but the proposed approach’s effectiveness in consistently addressing these challenges without hallucination remains unclear. Explicit quantitative analysis on knowledge domains might be helpful.\n2. Can TIGeR-ONE adapt to user-specific preferences in the decision-making process, especially in creative tasks? If not, how could this be incorporated to enhance user experience?\n3. I noticed the multi-round generation in Figure 7. Could you please explain how this process maintains consistency with the user-provided image?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7dl2mH542k", "reviewer": "Reviewer_wZFq", "summary": "This paper rethinks the relationship between text-to-image generation and retrieval, presenting a novel and pragmatic task, i.e., TIGeR, to meet the challenging information need of humans in real-world scenarios. The authors identify limitations in traditional retrieval methods, which rely on existing image databases, and in generation approaches that struggle with knowledge-intensive content. Motivated by these limitations, the authors proposed a novel unified framework called TIGeR-ONE that combines both tasks using a single LMM. To endow LMMs with the retrieval abilities, they introduce a training-free generative retrieval method that leverages LMM's discriminative abilities, allowing for efficient text-to-image retrieval. Additionally, an autonomous decision mechanism selects either generated or retrieved images as the most suitable response to a given text prompt. The authors develop TIGeR-Bench, a benchmark covering creative and knowledge-intensive image domains, to standardize evaluation. Their approach demonstrates superior performance on the unified benchmark, text-to-image and chat-to-image retrieval and generation benchmarks.", "questions": "1.\tIs there any experiment demonstrating the decision behavior of the proposed model? \n2.\tCompared to dense retrieval methods, such as CLIP, what advantages do MLLMs and the proposed generative retrieval framework offer?\n\nThe authors have solved my concerns.\nI stand by my positive score, after reading other reviewers' comments.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe authors introduce an innovative task that combines retrieval and generation, addressing a gap in traditional approaches that often consider these processes separately. This integrated task is highly practical, as it reflects real-world information needs where users may require either a retrieved image from a database or a newly generated one, depending on the context.\n2.\tThe authors propose a novel unified framework that unifies text-to-image generation and retrieval within a single LMM, streamlining both tasks into a cohesive process. By using a single LMM, the authors simplify the model architecture, allowing the system to handle both retrieval of existing images and generation of new ones without additional model switching or training steps.\n3.\tThe authors introduce a training-free generative retrieval method that enhances both the effectiveness and efficiency of text-to-image retrieval. The proposed method leverages the pre-existing discriminative abilities of Large Multimodal Models (LMMs) to perform retrieval without the need for extensive training on large datasets. \n4.\tThey build a unified benchmark, i.e., TIGeR-Bench, to standardize evaluation across creative and knowledge-intensive tasks, serving as a strong foundation to attract more researchers to explore complex multimodal information acquisition.", "weaknesses": "1.\tConsidering there has been prior work (i.e., GILL) addressing text-to-image retrieval and generation, the difference and advantages of the proposed method over GILL should be further highlighted clearly.  \n2.\tThe paper lacks analysis on how the unified model’s performance varies with prompts that have similar intentions but are expressed differently. In practical use, different users may convey the same idea with varied wording, prompts, and questions. How sensitive is the proposed method to these variations? \n3.\tGiven that the distribution over creative domains and knowledge domains may not be uniform in real-world scenarios, an analysis of the model’s performance and decision-making behavior under unbalanced distribution conditions would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)