Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond (2024)

Abstract: The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to “recall” the relevant image from its parameters as the response. Achieving this target presents notable challenges, including inbuilt visual memory and visual recall schemes within MLLMs. To address these challenges, we introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images and involves two training steps: learning to memorize and learning to retrieve. The first step focuses on training the MLLM to memorize the association between images and their respective identifiers. The latter step teaches the MLLM to generate the corresponding identifier of the target image, given the textual query input. By memorizing images in MLLMs, we introduce a new paradigm to crossmodal retrieval, distinct from previous discriminative approaches. The experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image candidate sets. The code is released at https://github.com/liyongqi67/GRACE.

heading: 1 Introduction
text: Recently, we have witnessed the explosive development of generative large language models (LLMs), such as GPT series (Radford et al., 2019; Brown et al., 2020) and LLaMA (Touvron et al., 2023a,b). Undergone extensive pretraining on document corpora and instruction tuning, these language models have demonstrated an impressive ability to memorize a lot of knowledge in their parameters and
*Corresponding authors.
effectively recall them to answer users’ instructions and queries. As shown in Figure 1, GPT41 could directly respond to the user’s question, “Who is Sheldon Cooper?”, without any external document or database. Building upon the advancements of LLMs, multimodal LLMs (MLLMs) (Alayrac et al., 2022; Li et al., 2023a; Liu et al., 2023; Zhu et al., 2023; Huang et al., 2023) have been developed to expand the capabilities beyond text and allow users to express their needs using visual input.
Despite the impressive capabilities of LLMs and MLLMs, their responses are limited to textual outputs. For instance, a user might ask, “What does Sheldon Cooper look like?” as shown in Figure 1. While the MLLM tries to describe the person’s appearance, it is often said that “an image is worth a thousand words.” It would greatly enhance the response capabilities of MLLMs if they could give visual outputs, like a photograph in this case.
A straightforward solution is to enhance MLLMs with external image synthesis tools, like diffusion models (Dhariwal and Nichol, 2021; Ho et al., 2020) and Generative Adversarial Networks (Goodfellow et al., 2020), for visual output capabilities.
1https://openai.com/gpt-4.
11851
However, a significant challenge with these modules is their propensity to produce unrealistic or hallucinatory images, which cannot accurately describe real-world images, such as a photograph of “Sheldon Cooper”. The integration of an image retrieval module (Radford et al., 2021) seems a more viable solution. Nonetheless, such a combination often encounters a transition gap between two independent modules (Lewis et al., 2020). Considering the massive benefits of LLMs in memorizing textual knowledge, a bold and innovative idea emerges: Is it possible to equip MLLMs with the ability to memorize visual information within their parameters for retrieval and beyond? In this light, we formulate a generative cross-modal retrieval task: given a user query for visual content, MLLMs are expected to recall desired images from their parameters directly as the response.
Accomplishing this task poses a significant challenge, necessitating the presence of two essential abilities of MLLMs: 1) Visual memory. As the prerequisite requirement, the MLLM model must possess the capability to memorize visual information within its parameters. This goes beyond simply encoding images into dense vectors within a vector database. It necessitates a distinct, differentiable, and integrated visual memory scheme within MLLMs’ parameters. 2) Visual recall. Given a textual query, the MLLM should be able to recall the relevant visual information from the complicated visual memory bank. Above this, for user comprehension, the activated visual information must be grounded to the complete and original images rather than mere patches or fragmented visuals.
In this work, we propose a novel GeneRAtive Cross-modal rEtrieval framework, GRACE, to overcome the above issues. GRACE assigns images unique identifiers, where each identifier is a distinct string representing an image. Based on the identifiers, GRACE comprises two training steps, as illustrated in Figure 2. 1) Learning to memorize. Given an image, the MLLM is trained to generate the corresponding identifier string via the standard text generation loss. The goal of this phase is for the MLLM to effectively learn and memorize the associations between the visual content of images and their respective identifiers. 2) Learning to retrieve. The MLLM is trained to generate the identifier string of the relevant image while given a textual query. In this way, the MLLM learns to associate user queries with visual memory. After the two training steps above, GRACE enables genera-
tive cross-modal retrieval: given a textual query, the MLLM generates an identifier string corresponding to a real image.
We delve into GRACE from various perspectives, including different identifier types, effectiveness, and efficiency of the generative paradigm. We evaluate GRACE on text-image matching datasets to verify the feasibility of generative cross-modal retrieval. Without any image’s visual information during inference, GRACE performs comparably to the advanced two-tower approaches (e.g., CLIP (Radford et al., 2021)) and demonstrates higher efficiency with large-scale image sizes. It is acknowledged that as a new retrieval paradigm, GRACE still lags behind one-tower approaches. One-tower approaches are only applicable to ranking stage due to their low efficiency, while GRACE and CLIP are specifically designed for the retrieval stage. By comprehensive analysis, we hope to comprehensively understand its capabilities and limitations.
We believe exploring generative cross-modal retrieval holds great significance.
• Benefiting from inbuilt visual memory within MLLMs, GRACE introduces a new paradigm to cross-modal retrieval. GRACE transforms the original matching problem into a generation problem, eliminating the need for negative samples during training and retrieval index during inference. No matter the size of the image set, the retrieval efficiency remains constant. This new cross-modal retrieval paradigm leaves much room for investigation.
• Inbuilt visual memory serves for retrieval, yet its utility extends beyond mere retrieval. In Section 4.5, we demonstrate that the MLLM could describe the memorized image and even answer questions about the memorized images, just like humans do. This opens up the possibility of injecting personalized visual experiences of humans into MLLMs for them to memorize and understand an individual’s journey, and accomplish more visual tasks.

heading: 2 Related Work

heading: 2.1 Cross-modal Retrieval
text: The current cross-modal retrieval (text-image matching) approaches can be categorized into the two frameworks and the one-tower framework
based on how modality interaction is handled. Onetower framework (Chen et al., 2020; Diao et al., 2021; Lee et al., 2018; Qu et al., 2021) embraces fine-grained cross-modal interactions to achieve matching between fragments (e.g., objects and words). As for the two-tower framework (Chen et al., 2021; Faghri et al., 2017; Zheng et al., 2020; Qu et al., 2020), images and texts are independently mapped into a joint feature space in which the semantic similarities are calculated via cosine function or Euclidean distance. Both the one-tower framework and the two-tower framework formulate the cross-modal retrieval as a discriminative problem, which relies on discriminative loss and negative samples to learn an embedding space. In this work, we explore a new generative paradigm for cross-modal retrieval.

heading: 2.2 Generative Retrieval
text: Generative retrieval is an emerging new retrieval paradigm in text retrieval, which generates identifier strings of passages as the retrieval target. Instead of generating entire passages, this approach uses identifiers to reduce the amount of useless information and make it easier for the model to memorize and learn (Li et al., 2024). Different types of identifiers have been explored in various search scenarios, including passage titles (Web URLs), numeric IDs, and substrings of passages, as shown in previous studies (De Cao et al., 2020; Tay et al., 2022; Li et al., 2023c,b,d; Sun et al., 2024). Generative retrieval gains a lot of attention in text retrieval, as it could take advantage of the powerful generative language models. Zhang et al. proposed a generative image-to-image retrieval framework. However, facilitating cross-modal retrieval (textto-image retrieval) in a generative way is still an untapped problem.

heading: 2.3 Multimodal Language Model
text: We have witnessed the explosive development of generative language models, such as GPT (Radford et al., 2019; Brown et al., 2020) and LLaMA (Touvron et al., 2023a), that demonstrate remarkable capabilities in instruction following and in-context learning. Building upon the advancements of LLMs, MLLMs (Alayrac et al., 2022; Li et al., 2023a; Liu et al., 2023; Zhu et al., 2023; Huang et al., 2023) have been developed to enable LLMs to process images as input. Despite the success of MLLMs in various vision-language tasks, they currently lack the ability to unify cross-modal retrieval
into their application. In this work, we propose a generative cross-modal retrieval framework that empowers MLLMs to retrieve relevant images from their parameters given textual queries.

heading: 3 Method

heading: 3.1 Preliminary
text: Task definition. Generative cross-modal retrieval defines new requirements, i.e., removing visual input during inference, for cross-modal retrieval, but could be evaluated with original cross-modal tasks. Text-to-image retrieval aims to retrieve relevant images from a database DI when given a textual query q.
Multimodal language model. As our method is conducted based on multimodal language models, it is essential to give relevant background of multimodal language models. Multimodal language models could be regarded as generative language models that incorporate image inputs, including GPT4V2, BILP (Li et al., 2023a), flamingo (Alayrac et al., 2022), and Kosmos (Huang et al., 2023). Considering factors including convenience and model sizes, we have chosen Flamingo as the backbone for our method and took the open-flamingo implementation (Awadalla et al., 2023).
Flamingo consists of three main components: a generative language model, a visual encoder, and cross-attention layers. The visual encoder is responsible for extracting patch features from the input images. The generative language model receives text input that includes a special token, “<image>”, which indicates the presence of an image. Through the cross-attention layers, the “<image>” token could attend to the patch features extracted by the visual encoder. This allows Flamingo to predict the next text token based on all previous text tokens and the most recent image. For more detailed information, please refer to the original paper on Flamingo.

heading: 3.2 Overview
text: In this work, we present GRACE, a novel generative cross-modal retrieval framework, as illustrated in Figure 2. As previously discussed, addressing the challenges of visual memory and visual recall is essential for generative cross-modal retrieval. Towards this objective, GRACE assigns unique iden-
2https://openai.com/research/ gpt-4v-system-card.
MLLM
id: 443 id: 4396 id: 12138
MLLM
id: 443 id: 4396 id: 12138 Query: A person in a blue robe is walking on the street. Query: A girl hops across the river, from rock to rock. Query: Two men are playing guitar in a band. MLLM id: 13768 Query: Three dogs play together in a field (a) Learning to memorize (b) Learning to retrieve (c) Inference
Visual Encoder
Language Model
MLLM
id: 12138
Predict the identifier for the <image>
Visual Encoder
Language Model
MLLM
id: 12138
Predict the image identifier corresponding to the given query: A girl hops across the river, from rock to rock.
Visual Encoder
Language Model
MLLM
id: 13768
Predict the image identifier corresponding to the given query: Three dogs play together in a field.
(a) Learning to memorize (b) Learning to retrieve (c) Inference
Figure 2: Illustration of our proposed generative cross-modal framework, GRACE, which involves two training steps. (a) Learning to memorize: GRACE trains an MLLM model to memorize images into its parameters. (b) Learning to retrieve: GRACE trains the model to generate the target image’s identifiers given queries. (c) Inference: The MLLM directly generates identifiers as the retrieval results.
tifiers to images in the dataset DI . This strategy allows the model to learn mappings from images to their respective identifiers, facilitating visual memory. Moreover, the model could generate identifiers as retrieval results rather than generate real images. Representing images as identifiers underpins our training scheme, which is divided into two core steps: “learning to memorize” and “learning to retrieve”. The two training steps are designed to enable the model to effectively memorize images in parameters and subsequently learn to recall them in response to textual queries.

heading: 3.3 Image Identifiers
text: Image identifiers are crucial for the whole framework, and we explore the following different types of identifiers:
String identifier. We randomly shuffle the images in DI , and assign them digital numbers ranging from 1 to |DI |. It is noted that the digital numbers are represented as strings in MLLMs and may be tokenized into multiple tokens determined by the tokenizer. For instance, an image may be assigned the identifier “13768” and tokenized into two tokens: “13” and “768”.
Numeric identifier. Similar to the string iden-
tifier, the numeric identifier ranges from 1 to |DI |. However, we include spaces in the numeric identifier, resulting in the tokenization into individual digits. For example, an image with the identifier “1 3 7 6 8” will be tokenized into the sequence of tokens “1”, “3”, “7”, “6”, and “8”. It is worth noting that the numeric identifier only utilizes ten tokens from the vocabulary to represent images, but the sequence length is typically longer than that of the string identifier.
Semantic identifier. Since the identifiers are utilized to represent images, image captions that describe the content of images can be considered as identifiers. These image captions are naturally token sequences that can be learned by multimodal language models. Some images in |DI | belong to the test set, and their captions should not be utilized. To avoid data leaks, we train an image caption model based on the training set and generate captions for the images in the test set as their identifiers.
Structured identifier. We assign structure identifiers to images using an unsupervised clustering approach. We utilize the image encoder in CLIP to obtain the embeddings of images. Subsequently, we apply the k-means algorithm (Ahmed et al., 2020) to cluster these embeddings, resulting in all images being grouped into k clusters. Each document is then assigned an identifier based on the number of their cluster IDs. For clusters that contain more than a certain number of documents (denoted as c), we recursively apply the algorithm (Tay et al., 2022). In this process, the identifier of the next level is appended to the existing identifier, forming a hierarchical structure. We represent each cluster using special tokens, such as "C_1_3",
which indicates the third cluster in the first level. These special tokens are added to the token vocabulary of the multimodal language model. Similar images tend to have similar structured identifiers, meaning they have similar paths in the cluster tree.
Atomic identifier. We assign a dedicated token as its identifier to identify each image uniquely. We expand the token vocabulary by introducing new tokens to ensure compatibility with the existing tokens. Each image is then assigned a special token, such as "I_13768", which is a complete token in the vocabulary and will not be further tokenized into sub-tokens. This approach allows us to avoid any conflicts with the original tokens while providing a distinct identifier for each image.
We present the various types of identifiers for the same image in Figure 3, highlighting their distinct characteristics. It is evident that different identifier types possess different attributes. String, numeric, and atomic identifiers do not provide any prior knowledge about the image content, whereas semantic and structured identifiers do. Furthermore, the use of structured and atomic identifiers necessitates the inclusion of new tokens in the vocabulary, whereas the other identifier types do not require such modifications.

heading: 3.4 Learning to Memorize
text: We have represented images in the dataset DI using unique identifiers, that is, as a sequence of tokens. Then we train a multimodal language model, denoted as MLLM, to encapsulate these images within its parameters. Specifically, for an image i ∈ DI , we train the model to associate this image with its corresponding identifier, denoted as I. This process is formulated as follows:
I = MLLM(i; inst-m), (1)
where inst-m is a textual instruction given as “Predict the identifier for the <image>”. Here, “<image>” is a placeholder token in Flamingo, designed to focus on the visual features of the input. This learning to memorize step allows the model to learn the mappings from visual inputs to their corresponding identifiers, to effectively encode imagelevel visual memories within its parameters.

heading: 3.5 Learning to Retrieve
text: Merely memorizing images within its parameters is insufficient for the MLLM. The model must be capable of recalling the corresponding images in
response to users’ queries. To achieve this, we train the MLLM to predict the appropriate identifier when given a specific query q. This process is outlined as follows:
I = MLLM(q; inst-r), (2)
where inst-r is a textual instruction, “Predict the image identifier corresponding to the given query”.

heading: 3.6 Inference
text: Post-training, the MLLM model could retrieve images akin to text generation. The process involves inputting a query into the MLLM, and then the model predicts several identifier strings through beam search. Since each identifier uniquely corresponds to an image, the generation results are the retrieval results.
Constrained generation. To confine the generation to within-corpus results and ensure they fall within the test set, we implement constrained beam search in the MLLM. This approach leverages a Trie, a form of k-ary search tree, for efficient key location within a set. Specifically, we store all image identifiers into the Trie. The Trie structure, upon receiving a prefix string, suggests potential tokens found in the identifiers. This mechanism ensures that every generated identifier accurately matches an existing image’s identifier. Furthermore, we employ beam search (Sutskever et al., 2014), a widelyused technique, for generating multiple identifiers concurrently. These identifiers are each assigned a language model score, facilitating the creation of a ranked list based on these scores. Consequently, the ranked identifiers correspond to a ranked list of images.

heading: 4 Experiments

heading: 4.1 Datasets and Baselines
text: We evaluated our proposed generative cross-modal retrieval framework, GRACE, on two commonlyused datasets: Flickr30K (Young et al., 2014) and MS-COCO (Lin et al., 2014). Flickr30K contains 31,783 images sourced from Flickr. Each image is associated with five human-annotated sentences. We adopted the data split used by Li et al., comprising 29,783 images for training, 1,000 for validation, and 1,000 for testing. MS-COCO comprises 123,287 images, and each MS-COCO image comes with five sentences of annotations. We followed the dataset split proposed in (Lee et al., 2018), utilizing 113,287 images for training, 5,000 for
validation, and 5,000 for testing. Consistent with prior studies (Young et al., 2014; Chen et al., 2021), we evaluated our method using the standard recall metric R@K where K is set to 1, 5, and 10.
Considering the efficiency and applicability, we compared GRACE with two-tower approaches, including VSE++ (Faghri et al., 2017), Dualpath (Zheng et al., 2020), CAMERA (Qu et al., 2020), and CLIP (Radford et al., 2021), as our baseline models. One-tower approaches usually have heavy computational overhead, focusing on the ranking stage rather than the retrieval stage. Therefore, we did not include them as baselines.
Implement Details are detailed in Appendix A.

heading: 4.2 Overall Results
text: The summarized comparisons are presented in Table 1. Analysis of this table led to the following observations: 1) GRACE demonstrated the capability to recall relevant images in response to textual queries without input of image content. This underscores the feasibility of generative cross-modal retrieval. 2) We also noticed variability in performance among GRACE with different identifiers. Specifically, numeric and string identifiers yielded very low performance on the MS-COCO dataset. This poor performance can be attributed to the lack of pre-knowledge provided by these identifiers to the MLLM. The inconsistent correlation between similar images and their identifiers makes it challenging for the MLLM to memorize and establish accurate relationships, especially as the dataset size increases. Furthermore, numeric identifiers underperform string identifiers, likely due to their
requirement for more generation steps, which increases the chance of errors. 3) In contrast, semantic identifiers, which are based on the image’s content, showed better results than numeric and string identifiers. However, their effectiveness was somewhat limited due to the minimal differentiation among semantic identifiers for different images. This was particularly problematic in cases where images shared the same captions, causing the model to generate semantically correct but contextually incorrect identifiers. 4) Structured identifiers achieved good performance by effectively utilizing the image’s embedding information through a clustering approach. This hierarchical structure significantly enhanced the MLLM’s ability to memorize all images in the dataset. 5) Finally, atomic identifiers were found to be the most effective, even outperforming the CLIP model. This approach assigns a unique token in the vocabulary for each image, ensuring distinct identification. However, this method also has its challenges, as increasing the number of images directly enlarges the vocabulary size of the MLLM, potentially impacting scalability.
These findings highlight the importance of identifier types in generative cross-modal retrieval and shed light on the trade-offs involved in different approaches.

heading: 4.3 Ablation Study
text: Our approach integrates two key training steps: learning to memorize and learning to retrieve. Does the “learning to memorize” phase significantly enhance retrieval performance? During the inference stage, we employed constrained generation to en-
sure the prediction of valid identifiers. How crucial is constrained generation to the overall retrieval process? To address these questions, we performed experiments by selectively omitting the “learning to memorize” step and the constrained generation process. The outcomes of these experiments are detailed in Table 2.
In our experiments, we observed a slight decrease in performance when the “learning to memorize” training step was removed. This suggests that while important, this step is not the sole contributor to effective retrieval. Intriguingly, the “learning to retrieve” phase can be considered another form of memorization, where the model focuses on the image’s description rather than its visual content. As a result, the model retains some capability to recall correct images even without the “learning to memorize” step. However, a significant decline in performance was noted upon removing the constrained generation step. This can be attributed to two primary factors. (1) Generation of out-ofcorpus identifiers: without constrained generation, the model tends to predict identifiers that do not correspond to any image in the corpus. This issue is especially pronounced with semantic identifiers, where the model may generate any textual description, leading to inaccurate retrieval. (2) Prediction of identifiers belonging to the training set. For other types of identifiers, while the model still predicts special tokens corresponding to these identifiers, it often predicts images in the training set. The vast number of images in the training set could also be relevant to the given textual query, significantly increasing the difficulty of recalling the correct image
in the test set.

heading: 4.4 Efficiency Analysis
text: In large-scale cross-modal retrieval, efficiency emerges as a crucial factor. This is why the onetower framework, effective for small-scale ranking, falls short in the retrieval stage. To address this, we conducted experiments comparing the efficiency of CLIP and GRACE. CLIP can pre-encode all images into vectors, incurring most of its inference cost from text encoding and calculating the similarity between text embeddings and image embeddings. In contrast, the generative framework necessitates generating identifiers. We assessed the query latency of both CLIP and GRACE to varying image sizes, with detailed results presented in Figure 4.
Our findings are insightful. Firstly, CLIP’s inference speed decreases progressively as image size increases, owing to the escalating number of similarity calculations required. Secondly, the inference speed of our generative framework remains nearly constant, a result of encoding all images directly into its parameters. Thirdly, when image sizes exceed a certain threshold (about 150,000 images), our generative framework surpasses CLIP in terms of inference speed, and this advantage grows as image sizes continue to increase. Lastly, these findings underscore that the generative framework is not only capable of large-scale image retrieval but can also perform comparably to two-tower approaches.

heading: 4.5 Beyond Cross-modal Retrieval
text: We enable the MLLM to memorize images within its parameters using unique identifiers. Once the images are adequately memorized, the MLLM can produce the corresponding images (identifiers) to respond to users’ queries, as illustrated in Figure 5 (a).
While the visual memory in the MLLM facilitates image retrieval, its applications are not restricted to retrieval alone after other instruction tunings. We present two examples in Figure 5 (b) and Figure 5 (c), respectively.
• Describing memorized images: As the MLLM has successfully memorized certain images, it is capable of providing a description of the image’s content when prompted. As depicted in the examples shown in Figure 5, when given an instruction such as “please describe the context of the image 3,756”, the model is able to provide a description of the image, albeit not in great detail.
• QA over memorized images: Similarly, the model is capable of answering some questions over the memorized images. Given an instruction consisting of the image identifier and question, the model can answer based solely on memorization without any image input.

heading: 4.6 Beam Size Analysis
text: We conducted experiments to analyze the beam size of GRACE, as detailed in Appendix B.

heading: 5 Conclusion and Future Work
text: In this paper, we delved into a novel memorization mechanism for the MLLM to memorize images
within its parameters. Building upon inbuilt visual memory within MLLM, we proposed a generative cross-modal retrieval framework, which introduces a fresh paradigm in cross-modal retrieval. This paradigm transforms the original matching problem into a generation problem, eliminating the need for negative samples during training and image indexing during inference. Our experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image sizes. Furthermore, we showcased the MLLM’s ability to interact (e.g., describe and QA) with memorized images, following specific instructions.
Moving forward, we aim to further develop this topic from the following perspectives. On the one hand, although our generative framework achieves comparable performance to previous cross-modal retrieval approaches, there are still challenges to address, such as the limitations of current identifiers. Exploring more effective identifiers, like “visual tokens (Van Den Oord et al., 2017)”, would help to enhance generative cross-modal retrieval further. On the other hand, since we have enabled MLLMs to memorize and interact with images, it opens up the possibility of injecting personalized visual experiences of humans into MLLMs for them to understand an individual’s visual journey and accomplish more visual tasks.

heading: Acknowledgments
text: The work described in this paper was supported by National Natural Science Foundation of China (62076212), Research Grants Council of Hong Kong (PolyU/5210919, PolyU/15207821, and PolyU/15207122), and PolyU internal grants (ZVQ0).
Limitations
This work introduces a new paradigm in text-image retrieval, but it also has some limitations to be addressed. 1) The evaluation of GRACE’s image retrieval ability on Flickr30K and MS-COCO was compared with two-tower baselines. However, it is important to note that Flickr30K and MS-COCO are also used as benchmarks for text-image ranking approaches, where one-tower frameworks have dominated. This may confuse newcomers to the field, as they may perceive GRACE and two-tower approaches as lagging behind the one-tower framework. However, it should be noted that GRACE and two-tower approaches focus on image retrieval, placing high demands on retrieval efficiency, while one-tower approaches are primarily suitable for the ranking stage, allowing for more time-consuming calculations to improve performance. 2) The identifiers currently used by GRACE are not as satisfactory as expected, only yielding results comparable to previous methods. However, as a pioneering work, the main significance of this work lies in validating the feasibility of generative cross-model retrieval. Further research is expected to enhance this paradigm.
Ethics Statement
The datasets used in our experiment are publicly released and labeled through interaction with humans in English. In this process, user privacy is protected, and no personal information is contained in the dataset. The scientific artifacts that we used are available for research with permissive licenses. And the use of these artifacts in this paper is consistent with their intended use. Therefore, we believe that our research work meets the ethics of ACL.

heading: B Beam Size Analysis
text: GRACE relies on beam search to obtain top-k retrieval results. We conducted detailed experiments to understand the impact of varying beam sizes, and the findings are illustrated in Figure 6. The atomic identifier is excluded from this experiment as it only requires one generation step, and beam size will not affect its performance.
Increasing the beam size exhibits marginal benefits. This observation aligns with our expectations, as candidates with larger beam sizes generally score lower, diminishing their likelihood of being the top result. In terms of Recall@10, we observed a notable improvement in performance with the expansion of the beam size. This enhancement is attributed to the inclusion of candidates that would otherwise be missed in scenarios with a more constrained beam size.
C Image Tokenization
As a part of future work, we also assess the effectiveness of using visual tokens as identifiers. Specifically, we follow the IRGen method (Zhang et al., 2023) to discretize each image into 4 visual tokens. The results of this evaluation are presented in Table 3. It is evident from the results that visual tokens as identifiers outperform most other types of identifiers. This is attributed to the fact that visual tokens are derived from the content of the image, demonstrating the potential of image tokenization in cross-modal retrieval.

heading: Future Work
text: C Image Tokenization
As a part of future work, we also assess the effectiveness of using visual tokens as identifiers. Specifically, we follow the IRGen method (Zhang et al. , 2023) to discretize each image into 4 visual tokens. The results of this evaluation are presented in Table 3. It is evident from the results that visual tokens as identifiers outperform most other types of identifiers. This is attributed to the fact that visual tokens are derived from the content of the image, demonstrating the potential of image tokenization in cross-modal retrieval