Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
Authors: Mathilde Caron
Alireza Fathi
Cordelia Schmid
Ahmet Iscen
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual entity recognition
generative models
multimodal LLM
Primary area: machine_vision
TL;DR: The paper introduces a method to create a better dataset for visual entity recognition using a multimodal language model for label verification and metadata generation.

Abstract: Web-scale visual entity recognition, the task of associating images with their corresponding entities within vast knowledge bases like Wikipedia, presents significant challenges due to the lack of clean, large-scale training data. In this paper, we propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Instead of relying on the multimodal LLM to directly annotate data, which we found to be suboptimal, we prompt it to reason about potential candidate entity labels by accessing additional contextually relevant information (such as Wikipedia), resulting in more accurate annotations. We further use the multimodal LLM to enrich the dataset by generating question-answer pairs and a grounded fine-grained textual description (referred to as "rationale") that explains the connection between images and their assigned entities. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks (e.g. +6.9% improvement in OVEN entity task), underscoring the importance of high-quality training data in this domain.

Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
Mathilde Caron
Alireza Fathi
Cordelia Schmid
Ahmet Iscen
Google DeepMind
Abstract
Web-scale visual entity recognition, the task of associating images with their corre- sponding entities within vast knowledge bases like Wikipedia, presents significant challenges due to the lack of clean, large-scale training data. In this paper, we pro- pose novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Instead of relying on the multimodal LLM to directly annotate data, which we found be suboptimal, we prompt it to reason about potential candidate entity labels by accessing additional contextually relevant information (such as Wikipedia), resulting in more accurate annotations. We further use the multimodal LLM to enrich the dataset by generating question-answer pairs and grounded fine- grained textual description (referred to as "rationale") that explains the connection between images and their assigned entities. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-ee performance on web-scale visual entity recognition tasks (e.g. +6.9% improvement in OVEN entity task), underscoring the importance of high-quality training data in this domain.
Introduction
Entities are at the core of how we represent and organize knowledge, as seen in prominent encyclo- pedias like Wikipedia, where each article is dedicated to a specific entity. In the field of computer vision, the task of visual entity recognition aims to identify entities within query images. This capability is not only a fundamental building block for various entity- visual understanding tasks. including "info-seeking" Visual Question Answering (VQA) [10 22 31 and News Content 24 54], but also has numerous commercial applications. Despite the progress made in recent years, current models often struggle with web-scale visual entity recognition. These models, typically trained on free-form image captions 15 [56), tend to hallucinate entities or output overly generic ones, leading to suboptimal performance. We hypothesize that the root of this issue lies in the lack of clean, large-scale training data specifically designed for visual entity recognition.
Understanding
Recent efforts have been made to address this problem by transforming existing captioning datasets into entity recognition datasets [7] (25 For example, Caron et al. 77] propose to match each Wikipedia entity name to most similar captions a large image-caption database [9] 46) and use the corresponding images as visual examples for the considered entity. Although this method has resulted in state-of-the- the-the-rt art performance, it still has significant limitations: the resulting datasets often noisy, with poor matching between the image content and the candidate entity. First, source of mistake is due to the ambiguity of language. For example the entity bishop of Ilandaff may refer both to a person and flower species [18 35). Second, there is some noise inherent to the used image-caption dataset that affects the results. For example in Fig. aa), an image of a building is incorrectly linked to the Wikipedia entity Negative equity. This mismatch occurs because of the caption "Worst areas for negative equity", which is likely to have been extracted from an investment-related website. The irrelevant association of the caption with the building image results in the inaccurate connection to the Wikipedia entity. Third, the text embedding matching between
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Corrected entity:
Corrected entity:
Residential building
Nemutobrycoom con pulmeri
Rationale
Multimodal LLM
+
Rationale
Caption: areas for negative equity"
resiceential builsing with pink exterior and white trim.
The
Caption: "iemakeerccon
The tal,
Multimodal
espece nemalobrycon palmon"
Its side.
text matching &
Diverse QAs
text matching
Diverse QAs
Candidate entity:
Q1] What style of building?
Candidate entity:
color the
N Negative equity
A1] GSerrann
Nematocampa resisteria
Purple
Negsine cppity is defict of
(2) What the type of
Nemoteeampp redebroi or horned speroonn nooh, is uooh of Funily Geometridtee L...]
What habiist Rih?
ownerss squity, occan lng when
windows the building?
salue of asser used Io ..d (+
ABI Sash whocow
Freerwater
(a) Irrelevant caption
(b) Incorrect caption-entity text matching
Figure 1: Two failure cases of the visual entity recognition dataset of [7]. Our proposed method overcomes these limitations by prompting a multimodal LLM to correct candidate entities. The LLM has access to relevant context such as the candidate entity Wikipedia page and the input image-caption pair. We also enrich the dataset with rationales and question/answer pairs covering diverse entities.
entity name and caption is not always accurate. For instance, in Fig. lbb,, the caption "nematobrycon espèce nematobrycon palmeri" is incorrectly matched with the entity name nematocampa resistaria, which two distinct animal species (a fish and a moth). Moreover, another limitation of these datasets is that they typically focus on a single entity per image, which is a restrictive scenario, as most images contain multiple entities.
In this paper, we overcome these different limitations by proposing a novel methodology to curate high-quality, large-scale dataset for web-scale visual entity recognition leveraging the capabilities of modern multimodal Large Language Models (LLMs) available through public APIS 2 15 37).
Our approach is unique in that we do not rely on the multimodal LLM for direct annotation, which found be suboptimal. Instead, we prompt the LLM to reason about candidate entity labels by accessing additional contextual relevant information, such as the original image captions and external knowledge sources like Wikipedia. This approach significantly improves the annotation quality of the resulting dataset, as evidenced by our thorough ablation studies. Moreover, we employ the multimodal LLM augment our dataset with rationales that explain the relationship between images and their corresponding entities. We observe in our experiments that training on this additional metadata improves the performance and visual entity understanding of the models. Finally, we address the previous limitation of focusing on a single entity per image by prompting the multimodal pairs that cover a diverse range of entities in the image.
L.M to generate several question-anse
We conduct extensive experiments evaluate the effectiveness of our approach. The results demon- strate that models trained on our automatically curated data achieve state-of- -hhe-or-the-art performance on Visual Entity
web-scale visual entity recognition tasks, notably on the challenging Open- recognitioN (OVEN) benchmark [17 (e.g- +6.9% on the OVEN entity split and +3.8% on the OVEN query split). Remarkably, we obtain these results with moderate-size models, which are orders of magnitude smaller than competing approaches, highlighting the importance of high-quality training data this domain. We further demonstrate the validity and effectiveness of our dataset when utilized as memory base for approaches such as visual matching, showcasing its versability and potential for various applications. In summary, our contributions are threefold:
We introduce a novel methodology to curate large-scale dataset for web-scale visual entity recognition, using a multimodal LLM as verification and annotation tool.
We enrich the dataset with additional metadata, including question-anse -anseer pairs and ratio- nales, generated by the multimodal LLM.
We demonstrate the effectiveness of our approach with thorough ablation study and by achieving state-of-the-an performance web-scale visual entity recognition tasks.
2 Related work
Visual entity recognition. Entities are key to knowledge representation and organization, as seen in prominent encyclopedias like Wikipedia. Visual entity recognition is the task of identifying entities based on visual queries [13 43 44, and a critical component of many complex applications. One such application is info-seeking VQA, which involves providing answers to questions about Another application is entity-aware captioning [28] 34 61], technique frequently employed in tasks such news content comprehen-
the detailed properties of finegrained entities [10 22 31

---

Text-to-text k-NN search
Retrieved
Tiger
database
image-caption pairs
Entity name
The ijee the largos livitg ca specics and menbee the peius Purthea native to Asa, his powerffl full musouke boiy wis lang had and paws, long rail. enang for back. modl vertical emrees (.:]
Original caption:
My little
Original caption: Tiger in the wild
loaclty park
Candidate entity:
Tiger
Candidate entity: Tiger
Prompts: this image of <ligers? Your either "YSS! or "NO"
definition
cTiger?: [-]- If your answer ia "No", use the capt ion of
Multimodal LLM
the image se deacribe the inage. Explain your anawer by
Page content
describing the image visual attributes. [-]]
Output: NO, [...]]
Output: YES, [...]
WIKIPEDIA
Corrected entity:
Cat
Corrected entity:
Tiger
Rationalc & GA
Rationale QAA
Figure LLM-Refined Entity-WebLL (REW) dataset. We propose a method refine the Entity-WebLI dataset of Caron et al. 77 by prompting a multimodal LLM to verify and correct Wikipedia entities. We also prompt the multimodal L.LM to output visually grounded rationales and question/answer pairs about diverse attributes of the image. Complete prompts are in Appendix A
sion 1 14 24 54]. Recent research has expanded the scope of visual entity recognition to include web-scale and open-domain entities [17 21 41] Of particular interest, Hu et al. [17] introduce the Open-domain Visual Entity recognitioN (OVEN) benchmark, which includes over million entities from English Wikipedia, covering broad range of concepts such animals, buildings, organizations, landmarks, etc. Caron et al. 7 recently achieved tfate-of-the-arr results on the OVEN benchmark by re- purposing auto-regressive generative models for this task. A key component of their approach is pretraining on a new entity-basee dataset instead of captions. In this paper, we build on the work of (7) by improving their pretraining dataset using external multimodal LLMs.
Using LLMs annotation tools. Since their remarkable success and widespread accessibility,
LLMs [36 40 (3 have been used in many different ways to obtain better supervision for training 62 For example, L.MSs have been used generate question-answer pairs from Wikipedia pages 8 or from transcribed video narrations [59], while other works use LLMs to rephrase questions into sentences [60]. Of particular interest, Hsich al. [16] show that prompting LLMs to output rationales as additional supervision for training small models in multi-task framework is effective strategy. A recent related trend consists in prompting LLMs to generate high-quality instruction-following training samples [ 50], an approach which has succesfully been extended multimodal tasks (26). Unlike our work, these approaches prompt LLMs with text input only while we feed both image and text to multimodal LLMs. Another key difference is that we use the multimodal LLM verification and correction tool based on candidate annotations rather than using its raw output as supervision, which we find to be suboptimal for the task of web-scale finegrained entity recognition.
various tasks 23
32
32
45
59
60,
3 Method
In this section, we describe how to leverage the capabilities of modern multimodal LLMs 15] in conjunction with external knowledge repositories such Wikipedia create a clean, large-scale training dataset specifically designed for web-scale visual entity recognition. We refer to the resulting dataset "LLM-Refined Entity- -ebbII (REW) and an overview of our method is in Fig. 2
3.1 Preliminaries
Wikipedia-scale visual entity recognition. Following recent research in web-scale visual entity recognition [7]. our goal to train models capable of accurately matching any given image-text query (Iv, 2x) to an entity among vast finegrained set E of possible entities. In this work, unless specified otherwise, the set of entities & consists of the 6 million entities from English Wikipedia. Each entity E & comes with an entity name l. corresponding to the entity Wikipedia page title.
Easety-based pretraining dataset. Previous works have observed that auto-regressive image cap- tioning models like GIT [56] or PaI.I 9 have suboptimal results when transferred to visual entity recognition (see Tab. D. due to the differences between captioning and entity recognition tasks 17 To address this, Caron et al. (7] propose training such models on entity-based data, not captions, and automatically create new dataset called "Entity-WebLy for this purpose. In short, for each
3

---

Wikipedia entity, the authors find the most relevant image-caption pairs in WebLI through nearest neighbor search in the CLIP text embedding space [4T] between encoded entity names and captions. They then replace the captions of the corresponding retrieved images with the considered entity name. In this work, we refer the entities obtained in this manner as "candidate entities". Further details on the Entity-WebL.J dataset are in 77. We describe in the following how to improve over such a dataset by leveraging the capabilities of modern multimodal LLMs.
3.2 Generating finegrained entities and descriptions with multimodal LLMs
Entity verification and correction. Our goal is to overcome the limitations of existing entity-based based pretraining dataset discussed in the introduction. To improve the correspondence quality between images and entities, we propose to use existing multimodal LLMs to verify the assignment of an entity to an image. In particular, given an image T, and corresponding entity e from the Entity- WebLI dataset, we prompt a multimodal L.LM with the task of verifying if xy is an image representing entity Interestingly, we make two important findings in our experiments (see Tab. F.. First, directly predicting an entity e is a more challenging problem than verifying the validity of the proposed entity. Intuitively, this is because multimodal LLMs either output too generic entities or hallucinate when they do not know the correct entity. This is expected since LLMs haven') I specifically been trained to identify entities at very high levels of granularity. This results in sub-optimal performance for the finegrained entity recognition tasks, where the goal is to recognize non non-generic finegrained entities.
Second, we find that the verification by a multimodal LLM is more precise when has access to external metadata such as the page content of the candidate Wikipedia entity and the original caption. Intuitively, the Wikipedia content allows the multimodal LLM to know which visual attributes to look for in the image z.. while the original caption gives hints about the corrected entity when the multimodal LLM detects that the candidate e is not correct. Note that the corrected entity might not always coincide with actual Wikipedia entity since we do not provide the list of the 6M Wikipedia entities to the L.M.M We still include these corrected entities in our training dataset and simply constrain decoding to the 6M entities at inference time (see details in Sec. 4.1) Similar techniques have also been used into entity generation in multimodal contexts by works like GMEL (48) and Our prompt for entity verification and correction is available in Appendix A.3.
AutoVER 588)
Generating rationales. At the same time as asking the multimodal LLMs to verify the entities assigned to training images, we also prompt it to provide a visually grounded rationale for its proposed entity. This allows explain the connection between the image and the assigned entity as can be seen in the examples in Fig. 3 and Fig. 4 We observe in our experiments that it improves the performance of the models for entity recogmition (see Tab. 4.. When training our model (see details in See. 1.3). we follow multi-task learning strategy where we prepend task prefixes to the input examples so that the model can output differently based on whether it is asked to predict an entity name or rationale.
Generating question-answer pairs (QAs). The Wikipedia entity recognition OVEN benchmark [17] consists of two splits: an entity split and query split. In the query split, images typically contains multiple entities, and the input question xe determines which entity should be recognized by the visual entity recognition models. We observe in our early experiments and in Tab. [ that training on the Entity-WebLI dataset (7] consistently results poor performance in the query split, while achieving state-of-the-are performance for the OVEN entity split. We hypothesize that this is due to how EintityWeblle dataset constructed (see details in Sec. 3.T.. The k-NN search the text embedding space favors short captions containing entity names. As result, a single, unambiguous, entity is assigned to each image, a scenario quite different to the examples in the query split.
To overcome this problem, we prompt the multimodal LLM to generate question- answer pairs for several, diverse entities in the input image (see prompt in Appendix Appendix(A.3.I) A.3.I]. The multimodal LLM has access to the input image, the verified/corrected entity as well as the rationale it previously generated. We empirically evaluate in the ablation study in Tab. 5 the impact of these on the question/answeer pairs and resulting trained models.
3.3 Model training
Auto-regressive generative models. Previous works have shown the effectiveness of generative approaches for entity recognition both in the pure NLP domain 12 recently, in visual entity recognition benchmarks [7]. Motivated by their success, we perform entity recognition by generating Wikipedia entity names (i.e. page titles t.) in an auto-regressive
30 39
42
49 51 and, more

---

fashion. This is akin to multimodal version of GENRE 12 or to the GER-CAPTION variant of 77. Formally, we transform an input image-text query pair I (z,, 2x) into set of N d-dimensional embeddings X E Rxxd formed by concatenating the visual encoder output of Iy and the text tokenizer output of Z₁. We use an auto-regressive text decoder g(-) to generate the target text y. As detailed in the following paragraph, the target text can either be an entity name or a rationale. Specifically, the decoder predicts each text token y& from the target text (total length is K) given both the set of preceding token embeddings Y <k and the input image-text embeddings X. We train with a language modeling objective:
K
1 K
L
Σ
eyv, [X: ((XXxe<))) Y<l))
(1)
where [:] corresponds to the concatenation operation in the first dimension and é is the softmax cross-entropy loss with label-smoothing [33]. We average this loss over minibatches of examples and update the weights of the visual encoder and text decoder with back-propagation.
Multi-task learning. We train our models jointly with three distinct tasks: (i) predicting the verified/corrected visual entity, (ii) generating the rationale and (iii) answering the questions generated by the multimodal L.LM. These three tasks are text generation tasks and follow the same framework introduced in the previous paragraph. They differ in the nature of the input text zt and target text y. For entity recognition and question answering. the target text y corresponds to a possible entity name and the input text Ie corresponds to question. For rationale generation, the target text y corresponds to the rationale generated by multimodal LLM and the input text De consists simply of a prefix specifying to the model that this task is distinct from entity generation. Our final multi-task loss objective is:
Crinal Leaut + Rattinnae LoA
where LEmity Liaaiooalle and LQa correspond to entity, rationale and answer generation respectively.
4 Experiments
4.1 Experimental setting
We detail here the most salient experimental details. Full experimental setting is in Appendix A.3
REW training dataset. Our training dataset builds upon the Entity- Entity-WebLI WebLI dataset 77 (see Sec. 3.1 which itself is based on WebLI [9] a dataset already deduplicated against the train, val, and test splits of 68 common vision/vision-Lanee languuge datasets [9]. The Entity-Weble dataset is further aggressively filtered against OVEN by removing any image which has a CLIP-score higher than 0.95 with an OVEN image 77 This ensures that there is no downstream data leakage in our REW dataset. We build two versions of REW dataset: REW-5M (4.5M images) and REW-47M (47M images). Each training image attached to verified/corrected entity, a rationale and question/answer pairs. In the main results section (Sec. 4.2 we train on REW-47M dataset for 600k steps while for analysis and ablation study (Sec. 4.3 train REW-5M for shorter schedule (200k steps). We also validate our dataset refining methodology using the LAION dataset [66] as the image-caption base dataset.
Downstream task: OVEN. We consider the entity and query splits of the OVEN benchmark (17). For both splits, the goal is to output a Wikipedia entity given an input image and question. OVEN validation and test splits are divided into seen and unseen entities. The seen examples correspond to entities present in the OVEN training set while unseen entities are a subset of entities not present in the training set. We report the harmonic mean (HM) of top-1 accuracy scores between "seen" and "unseen" entities [17]. We specify in our results if we report results before or after additional finetuning on the training set of OVEN ("+ seen finetune").
Downstream task: finegrained datasets. We also report results on Oxford Flowers [55] Sun397 [57], Food101 [5]. FGVC-Aircraft (29 and Sports100 [I7] finegrained datasets in the zero-shot mode. We choose these datasets since there is direct mapping between their class vocab- ularies and the Wikipedia entities that our model is trained to output. The resulting class label to Wikipedia entity mappings are given in Appendix A.5
Inference. We perform decoding with beam search. When evaluating on OVEN, we discard all decoded texts that are not one of the 6M Wikipedia entities. This akin to constraining the beam search only at the last decoding step. Constraining from the first decoding step is too costly in our
5

---

Table 1: Comparison with the state of the art on OVEN entity and query test splits. We report the harmonic mean (HM) of the seen and unseen sets (top-1 accuracy) before and after finetuning on OVEN training seen categories ("+ seen finetune"). We indicate model architectures and their total number of parameters ("# par.") in billions well as the training dataset details. Query split
Entity split
seen finetune
+ seen finetune
Model
#par (B)
Dataset
HM
seen unseen
HM
seen unseen
HM
seen unseen
HM
seen unseen
Dual encoders
CLIPfusion 17
0.9
OpenAl
41
5.2
5.6
4.9
8.4
33.6
4.8
1.6
1.3
2.0
2.7
25.8
1.4
CLIP2CLIP
0.9
OpenAl
1
5.2
4.9
11.5
12.6
10.5
1.6
1.3
2.0
3.5
3.8
Generative approaches
PaLI-3B 9
3
WebLI-1B 9
19.1
6.0
16.7
27.4
12.0
PaI.I-17B 9
17
WehL.I1BB 9
1.8
3.3
1.2
16.0
28.3
11.2
9.2
14.1
6.8
27.1
36.2
21.7
GiT-Large [56]
0.4
WebLI-100M
9
2.1
4.7
1.4
6.5
13.7
4.2
3.9
5.1
3.2
15.6
28.9
10.7
GER-ALD 7
0.4
Entity-WebLI 7
17.7
18.3
17.2
22.7
31.5
17.7
6.3
6.0
6.7
5.8
14.1
3.6
GiT-Large 556
0.4
Entity-WebLI
7
19.1
19.8
18.5
20.1
25.9
16.4
10.4
9.8
11.0
10.1
17.7
7.1
Gar-Large [56]
0.4
REW-47M (Ours)
23.6
25.7
21.7
29.6
36.0
25.1
30.0
31.2
28.9
30.9
39.2 25.5
Table 2: Zero-shot transfer of generative models to finegrained image classification. We report top-1 accuracies. All models are run by and are based on the same architecture.
Model
Training dataset
Flowers 35
Sun397 [S7
Food S
Aircraft 29
Sports100 17
GaT-Large 56
WebLI- 100M 9
39.1
45,8
55.7
7.4
57.9
GiT-Large
[56
Entity- I
79.8
45.1
66.5
27.7
77.2
GER-ALD
Entity-WebLI
86.7
45.9
78.0
37.4
74.6
GiT-Large
56
REW-47M (Ours)
88.2
50.2
80.4
50.3
78.0
implementation with such large million-scale label space. For finegrained datasets, we perform constrained beam search decoding at all decoding steps since the label spaces are smaller.
Model training implementation details. We use GiT--Large [56]: it consists of a visual encoder (CLIP-L/14 (41)) and 6-layer text decoder with internal dimension visual encoder is first pre-trained jointly on WebLI-100M [9] and Conceptual Captions-12MM 12M [47] while the decoder is randomly initialized. We batch size of 4096, learning rate of le-5 for the visual encoder and le for the decoder, label smoothing of 0.2 and no weight decay. We use standard inception crop data augmentation. For the multimodal LLM, we use Gemini Pro [5). The public API is available at aai.ooglle.dv
768. Following 771, the
4.2 Main results
Comparison with the state of the art OVEN. In Tab. [ observe that PaLI and GiT-Large models pretrained on captioning datasets have suboptimal performance, especially in the entity split. Intuitively this is because the entity split tackles finegrained entity recognition while query split is more reflective of generic VQA task, which leverages the language understanding abilities learned from captioning. Hence, the query split task more aligned with the captioning pretraining than the entity split task. In Tab. we see that the model trained on our REW dataset instead of captioning data results in state-of-the-ard performance the OVEN benchmark, both before and after further OVEN finetuning on the "seen" classes. Notably, our model outperforms the captioning PaL.- 17B model by large margins: +13.6 top1 HM test accuracy on the entity split and +3.8 on the query split, while using x2x less parameters.
Finally, we report the zero-shot performance of the multimodal LLM on OVEN: it reaches 13.3 HM top-1 the entity split and 29.5 HM top-1 the query split. These numbers suggest that we are not merely distilling from the considered multimodal LLM as outperform its performance on this benchmark by +10.3 on entity and 1.4 on query sets while using orders of magnitude less parameters. The analysis in Tab. will further demonstrate the importance of using the multimodal

---

Table 3: Visual matching. We report top-1 accuracies of visual matching with CLIP or DINOv2 ViT-L/14 visual backbones. We compare two types of annotations for the visual matching memory database: either the candidate entities or our multimodal LLM corrected entities. We report the absolute improvements of using the latter compared to the former between parentheses as well as the relative improvement averaged across the six datasets in the last column.
Memory dataset
OVEN-Ent Flowers Sun397
Food101 Aircraft Sports Avg.relative A
CLIP-L/14 buckbone 11
Candidate entities
16.3
81.1
37.7
80.9
42.1
70.4
+ multimodal LLM correction (9.8(+3.5)
(1.1() (9.7(+12.0)
(9.1(-18) 44.6(+2.5) 74.8(-44)
+10.5%
DINOv2-L/14 backbone
33
Candidate entities
19.1
91.7
37.9
75.7
34.6
76.8
+ multimodal LLM correction 24.8(+
90.5 1.2) 52.3(+14.4) 0.8) 38.3 (2.4(-5.6)
+13.9%
LLM as a verification and correction tool rather than teacher since directly using its predictions as targets result in poor performance.
Zero-shot transfer finegrained datasets. In Tab. 2 we observe that the model trained on our proposed training dataset REW-47M transfers effectively to several finegrained datasets. The model trained on REW demonstrates superior transferability compared the same model trained on captions or Entity- WebLI. This result shows the higher quality of the REW dataset.
Using REW dataset as memory base. We explore the potential of our REW dataset when utilized as a memory base for visual matching in Tab. [3] and for retrieval-enhancee contrastive (RECO) training [18] in Appendix A.I.I Each image in the memory is either associated with the candidate entity from text k-NN matching (as Entity- I 77) or with the multimodal LLM corrected entity (our method for REW). In Tab. 3 we report the results of visual matching with two popular visual backbones G 41 and across six different finegrained visual entity recognition datasets for which we have the mapping from class label to Wikipedia entity. We see in Tab. 3 that corrected entities lead to better visual matching performance across the board which suggest that they are better annotations, describing more accurately the content of the images. In fact, using our corrected entities boosts the performance in average by +10.5% relative improvement when considering CLIP 41 and by +13.9% relative improvement with DINOv2 38
4.3 Analysis and ablation study
Unless specified otherwise, models in this section are trained on the REW-5M dataset.
Importance of entity verification and correction. In Tab. 4 we train models with different source of annotations for the target entities. First, we observe that directly using the multimodal LLM raw output as target (which is akin to a distillation scenario) results in suboptimal performance (row I). By inspecting qualitative results, hypothesis is that this can be attributed to the LLM's tendency to produce hallucinations, overly generic answers, or outputs in an incorrect format (long descriptive captions instead of entities). Second, we validate in Tab. 4 the importance of the multimodal LLM correction step compared to using the candidate entities from text k-NN matching as in Entity- WebLI 7 this strategy improves the performance of the resulting model by more than 6 points (row3 3 versus row 2). Finally, providing additional contextual information to the multimodal LLM results in a substantial boost of 1.7 points (row 4 versus row 3), benefit that further illustrate through our qualitative analysis in Fig. 3 We note that the multimodal LLM refines the candidate entities 76% of the time, i.e. validates the candidate entity 24% of the time (see Fig. Bc).
Qualitative analysis on the importance of the multimodal LLM correction. In Fig. 3 we identify two typical failure cases of the multimodal LLM correction step when it lacks access to Wikipedia and original caption metadata. The first case (sce examples in Fig. 3aj involves the LLM making incorrect corrections by providing generic or hallucinated outputs. For example, it identifies an image of the Bronte baths as an outdoor swimming pool or it makes mistakes by recognizing incorrect plant or animal species. In contrast, the multimodal LLM with access to external metadata can get a hint about the correct entities by reading from the original caption. As a matter of fact, note that simply using the original captions as targets leads to suboptimal performance as shown by Caron et al. (7). Intuitively, original captions are usually not in the form of an entity and can noisy and not reflective of the visual entity of the image, as illustrated in the examples in Fig. 3b
7

---

Table 4: Importance of entity verification and correction. We report HM top-1 accuracy on OVEN validation entity split. To isolate the effect of the entity target source, we train only with entity targets (Le only with loss L- We do not perform seen finetuning and evaluate the models directly after pretraining. All models are trained with the same pretraining images and use the same architecture.
Entity target source
Entity split (HM)
Multimodal LLM raw output
3.2
2 Candidate entity from text k-NN matching (as in Entity-WebLI WebLI
(7)
6.7
3 + multimodal LLM correction without access to Wikipedia & original caption
12.7
4 multimodal LLM correction with access to Wikipedia & original caption
14.1
(a) Incorrect candidate entity due to incorrect matching between original caption and Wikipedia entity names
Original caption:
Original caption: "nematobrycon espsce nematobyyon palmen"
Original captione
"teran's nomalmene rubescens"
"brente bamhs"
Candidate entity::
Candidate entity:
Candidate entity:
Tetraclita rubescens
Nemalocampa resistaria
Guildford Crescent Beths
LLM access Wikipadia& caption
LLM with access to Wikipedia & caption
LLM
Wikipadia caption
Corrected entity: Homalomena rubescens
Corrected entity: Nematobrycon palmeri
Corrected entity: Bronto baths
Rotiondte The main objest in the irago
heart
Rationale: hos apupl body, yellow
black
Rationale: image
an oupdior oooan pool with wavds
green reddish
rumning to
LLM without access to Wikipedia & caption
LLM without access to Wikipedia caption
LLM without access to Wilkppedia caption
Corrected entity: Alocasia cuprea
Corrected entity: Freshwater hatchetfish
Corrected entity: Outdoor swimming pool
Rationale: The imaga shoust a zbnn with on coal lesw no thot srw
Rationale: The in heewwterr ratchettih, hype of tit with
Rationale An sundoer eoemmiing poct iitee with water and sumordde conents. The pool the adga calt
dark redd hndpatp in co with prominent vairs
long then body and aitrary sheen. The tad foried.
(b) Incorrect candidate entity due to irrelevant original caption
Original caption:
Original caption:
Original caption:
'west forever'
boyfriend"
"my sang"
Candidate entity:
Candidate entity:
Candidate entity:
West
Boyfriend
Good song
with access to Wikipedia& caption
LLM with access Wikipedia & caption
with access Wikipediaa caption
Corrected entity: Motorcycle
Corrected entity: T-shirt
Corrected entity:
Rationale: heee ana these motongyc ss one road the deswet.
Rationals: The mage of i-atet with white taot thet saya What Boyised?".
RationatacA lege blakk und wtise sattoal with a red stripw un the uad, raeding stabbardwwhh begu crew dack.
LLM without access to Wikipedia & caption
LLM without, access Wikipedie caption
LLM without access to Wikipedia & caption
Corrected entityy Motorcycle
Corrected entity: T-shirt
Corrected entity: Sailing yacht
Rationate: image showe a group people riding
Rationale: mage 1-chirt
Rationale: lagg black white solloont moving quickt
motorcyckes
dacapp.
through
(c) Incorrect correction by multimodal LLM without access to Wikipedia external knowledge
Original caplion:
Original caplion:
Original caplion:
"arosgrain"
'broad billed hummingbird 2" Candidate entity:
hhalcd's rufescens"
Candidate entity:
Candidate entity:
Grosgrain
Broad-billed hummingbird
Haliotis
LLM with access Wikipedia caption
LLM with access to Wikipedia& caption
LLM with access to Wikipedia& & caption
Corrected entity: Grosgrain
Corrected entity: Broad- billed hummingbird
Corrected entity: Haliolis rufescens
lattnaelecautttle spoe's prezbon with prominent trirovesse
Radionatec The aid has long poeted bi and brihtt nd
Radoonaecc h sang, ava-stpped shall with motteeiggen
ris, whac a dertingg cranactetetc el greagras.
settwrr whac are daraciantic at traat-t aed humminghed.
and brown rnfor seres emai hares oreg the edge.
LLM without access to Willpedia & caption
LLM without access to Wikipadia & caption
LLM without access to Wilppedia caption Northen abalone
Corrected entity: Woven fabric
Corrected ontity: Green-crowwee woadnymph Radionale: The
Corrected
Ralionalo:T The shows dffeort typos offibric.
oma with then book
Rationale: The image
and while oval shaped onel with rough toxtue and hole near the sip.
and grean body. The bid hrveeing in frost of rod flwerr.
Figure 3: Qualitative analysis of the importance of the entity verification and correction step.
The second failure case of the multimodal LLM lacking access to metadata occurs when the model corrects information that it should not, likely due to lack of knowledge about the entities involved (see examples in Fig. [cp] This issue is reflected in the LLM rationale, indicating need for further external information about the candidate entities. By contrast, the model with access to Wikipedia can read about the visual attributes which are characteristic of the candidate entity and look for these in the image. This process is reflected in the LLM rationale: for example, the model with Wikipedia access the grosgrain entity page validates this entity with the rationale: "Mulriple spools of ribbon with prominent transverse ribs, which is a defining characteristic of grosgrain" while the model without Wikipedia seems to lack specific knowledge about grosgrain.
Generating rationales and QA pairs with multimodal LLM. In Tab. 5 (left), we probe the importance of providing external metadata to the LLM when generating rationales and question answer pairs. First, we observe in Tab. (left) that the model without access to any metadata (row 1) has suboptimal performance. Incorporating external input, such as Wikipedia content and original captions, during the rationale generation process enhances the quality and pertinence of the rationale. Consequently, this leads to an improvement in performance (row 2). Importantly, this performance boost is even higher if the multimodal L.M can leverage the improved rationale when generating the QAs (row 3).
8

---

Table 5: Ablation study: (left): Generating rationales and QA pairs with multimodal LLM. (right): Multi-task training with generated rationales and QAs We validate the robustness to the base image-caption dataset used by performing the latter ablation with both WebLI 9 and LAION [4]. We report HM top-1 accuracy on OVEN validation splits directly after REW training.
Metadata available to multimodal LLM
LLM-Refined Entity-
at rationale gen.
at QAs gen.
Entity Query
WebLI
LAAINN
Ø Wiki & caption
Ø
14.2
26.7
Eaiity LRationae Con Entity Query Entity Query
14.7
27.6
1
14.1
5,4
10.7
5,6
Wiki & caption
rationale
15.5
29.0
2
V
/
14.6
6.7
11.4
4 Wiki & caption
entity
16.4
27.2
3
v
/
15.9
25.1
13.2
25.3
5 Wiki & caption
entity + rationale
16.0
28.2
4
v
v
16.0
28.2
13.4
28.2
In row we notice that while is important for the LLM have access to the entity name during QA generation to improve entity split performance, it negatively impacts query split performance, as seen the comparison between row and row This observation can be intuitively explained by the fact that when the L.MM has access to the entity name, it is more likely to generate question/answer pairs focused on the main entity, rather than considering other attributes present the image. However, these diverse attributes are often mentioned in the rationale, as demonstrated in the qualitative examples of Fig. 4 in Appendix. Therefore, having access to the rationale increases the variety of the generated QA pairs. Taking into account these observations, our default model (row 5) is configured to generate QAs by providing the model with access to both the entity and rationale. This approach aims to achieve balance between strong performance in entity and query splits.
Multi-task training with generated rationales and QAs. In Tab. B (right), confirm the impor- tance of multi-task training with the various output types obtained from the multimodal LLM. We conduct this experiment using two different base image-caption datasets: WebLI [9] (our default) and the publicly available LAION dataset [46]. Our results demonstrate that both rationale and QA objectives contribute the improved performance of our model. Intuitively, rationales help clarify the connection between entities and visual attributes, while QA training encourages the model to focus multiple entities within the image. Moreover, both objectives enhance the model's language understanding, which we found to be important especially for the query split.
Robustness to the base image-caption dataset. Lastly, we observe that the results presented in Tab 5 (right) are consistent across the different image-caption datasets used in our experiments. This suggests that our findings are robust and not specific to the WebLI dataset.
4.4 Results with open source models
Finally, we run additional set of experiments with the open source PaliGemma 3 and Gemma 27B models [52]. We validate that the results are inline with the results when using the Gemini- Pro model [15 Since Gemma lacks visual input processing, we replace direct image input with automatically generated captions. Specifically, we employ the open-source PaliGemma model to generate descriptive captions for each image using the prompt: "Describe the visual attributes of this image.". We then integrate these captions into the existing prompts outlined in Sec. A.3. by prepending the text: "Here are the visual attributes of the image:".
We evaluate this approach on the 5M subset of WebLI and LAION. We compare the Gemma and Gemini-Pro variants of our method in Tab, with the SOTA methods trained on the 5M subset of EebiLL We see in Tab. that in all cases our method gives substantial improvements over the prior work GER-ALD 7 and GiT-Large trained on Entity-Webla In Tab. g of the Appendix, similarly to Tab. (right) we evaluate the impact of the different losses when using the version of our method with open source models. We verify that conclusions are similar with private and open source models. As seen Tab. 8 the main difference of performance between Gemini-Pro and Gemma variants comes from the QA loss. While Gemini- Pro has access to input images when generating QA pairs, Gemma generates QA pairs based on the PaliGemma caption (and rationale). This limits the variety of the generated QA pairs, resulting in a lower final accuracy.

---

Table 6: Results with open source PaliGemma and Gemma models. We report HM top-1 accuracy on OVEN validation entity split directly after REW training. We use a 5M subset for all the pretraining datasets.
Pretraining dataset
Entity split (HIM) Query split (HMM
GiT-Large
Entity-WebLI
9.1
5.6
GER-ALD
Entity- I
10.2
GiT-Large
LLM-Refined Entity-LAION with Gemma
11.6
23.4
GiT-Large
LLM-Refined Entity-LATON with Gemini Pro
13.4
28.2
GiT-Larre
L.LM-Refined Entity-Wehh with Gemma
14.2
24.3
GiT-Large
LLM-Refined Entity- WebbLI with Gemini Pro
16.0
28.2
Discussion
Limitations. Our work relies on the use of multimodal LLMs with external knowledge bases, such as Wikipedia, for dataset creation and annotation. This dependence on using external data to prompt LLMs may limit the applicability of the approach in scenarios where such external knowledge is searce or unavailable. Also, the proposed approach involves prompting the LLM to reason about candidate entity labels and generate additional metadata. This process is time-consuming and computationally expensive when considering multimodal LLMs with billions of parameters [[5], which may limit the scalability of the approach to even larger datasets. A direction for future work could be to overcome some of the limitations of OVEN we find our study (Appendix A.2) and create additional benchmarks for web-scale visual entity recognition.
Conclusion. We propose a novel methodology to curate high-quality large-scale dataset, REW, for web-scale visual entity recognition using multimodal LLMs. Our approach significantly improves the quality of the dataset and bypasses the need for manual annotation. We also enrich the dataset with additional metadata, including question-answer pairs and rationales, generated by the L.LM, which further improves the performance of the models. We achieve state-of-the art performance on web-scale visual entity recognition tasks, highlighting the critical role of high-qualit training data in this challenging domain. While our methodology for multimodal LLM-based data curation shows promise, we recognize significant opportunities for further enhancement. Integrating additional tools and external knowledge sources holds potential to improve its effectiveness. Furthermore, we anticipate that our approach can be broadly applicable to other visual tasks requiring extensive training data. Broader impact discussion is in Appendix A.4
References
Hammad Ayyubi, Tianqi Liu, Arsha Nagrani, Xudong Lin, Mingda Zhang, Anurag Arnab, Feng Han, Yukun Zhu, Jialu Liu, and Shih-Fu Chang. Video summarization: towards entity-awan captions. arXiv preprint arXiv:2312.0226 2023.
[2] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Komblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 390,, 2023.
[3] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Paligemma: A versatile 3b vlm for transfer. arXiv preprins arXiv:240.007726 2024.
[4] Ali Furkan Biten, Lluis Gomez, Marçal Rusinol, and Dimosthenis Karatzas. Gond news, everyone! context driven entity-aware captioning for news images. In CVPR, 2019.
[5] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- food-101-ma 100-10001mmmggggg discriminative components with random forests. In ECCV, 2014.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, ICCV, 2021.
[7] Mathilde Caron, Ahmet Iscen, Alireza Fathi, and Cordelia Schmid, A generative approach for wikipedia- scale visual entity recognition. In CVPR, 2024.
10

---

[8] Soravit Changpinyo, Doron Kukliansky. Idan Szpektor. Xi Chen, Nan Ding, and Radu Soricut, All you may need for vqa image captions. arXiv preprini SKKi::2205.08883 2022.
[9] Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointl scaled multilingual language- image model. ICLR, 2023.
[10] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wer Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302.117733 2023.
[11] Wei Chiang. Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu. Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. 2023.
[12] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. arXiv preprint Xiv::0100.0000 00904, 2020.
[13] Mark Everingham, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88, 2010.
[14] Xiyan Fu, Jun Wang, and Zhenglu Yang. Mm-avs: A full-scale dataset for multi-modal summarization. In NAACL, 2021.
[15] Gemini Team Google. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312. 1105, 2023.
[16] Cheng-Yo Yu Hsieh, Chun-Liang Li, Chih- Kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.023: 02300, 2023.
[17] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal. Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wai Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. ICCV, 2023.
[18] Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Retrieval-enhancae contrastive vision-text models. ICLR, 2024.
[19] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Nevel dataset for fine- fine-grained image categorization. In First Workshop on Fime-Grained Visual Categorization, CVPR, 2.
[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
[21] Alina Kuznetsova, Hassan Rom. Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Puset, Shahab Kamali, Stefan Popov, Mattco Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[22] Paul Lemner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, dataset for knowledge-based visual question answering about named entities. In ACM SIGIR, 2022.
[23] Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In CVPR, 2022.
[24] Fuxiao Liu, Yinghan Wang. Tianlu Wang, and Vicente Ordonez. Visual news; Benchmark and challenges in news image captioning. arXiv preprint arXiv:2010.037443 2020.
[25] Haotian Liu, Kilho Son, Jianwei Yang, Cc Liu, Jianfeng Gao, Yong Jac Lec, and Chunyuan Li. Learning customized visual models with retrieval-augmenter knowledge. In CVPR, 2023.
[26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NewrIPS, 2024.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
[28] Di Lu, Spencer Whitehead, Lifu Huang, Heng Ji, and Shih-Fu Chang. Entity-aware image caption generation. arXiv preprint 180i 07889, 2018.

---

[29] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[30] Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler. Dsi++: Updating transformer memory with new documents. arXiv preprint arXiv:2212.0977 09744, 2022.
[31] Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine- aen-iga categories. In /CCV, 2023.
[32] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In ICCV, 2023.
[33] Rafael Müller, Simon Korblithh and Geoffrey E Hinton. When does label smoothing help? NeurIPS, 2019.
[34] Khanh Nguyen, Ali Furkan Biten, Andres Mafla, Lluis Gomez, and Dimosthenis Karatzas. Show, interpret and tell: entity-aware contextualised image captioning in wikipedia. In AAAI, 2023.
[35] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, 2008.
[36] OpenAL. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[37] OpenAL. Gpt-év(ision) system card. System Card, 2023. Version 1.0.
[38] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023.
[39] Ronak Pradcep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q Tran. How does generative retrieval scale to millions of passages? arXiv preprint 2 18441, 2023.
[40] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[42] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan H Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q Tran, Jonah Samost, al. Recommender systems with generative retrieval. arXiv preprint arXiv:2305.08: 055655 2023.
[43] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Manor. Imagenet-211 pretraining for the masses. arXiv preprint arXiv: 104. 10972, 2021.
[44] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[45] Shibani Santurkar, Yann Dubois, Rohan Taori, Percy Liang, and Tatsunori Hashimoto. Is a caption worth a thousand images? a controlled study for representation learning. arXiv preprint arXiv:2207.00 07633, 2022.
[46] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Sb: An open large-scale dataset for training next generation image-text models. arXiv preprint mxXiv:2210.0844) 2022.
[47] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmmed, image alt-text dataset for automatic image captioning. In ACL, 2018
[48] Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. Generative multimodal entity linking. arXiv preprint 230: 12725, 2023.
[49] Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten Rijke, and Zhaochun Ren. Learning to tokenize for generative retrieval. NeurIPS, 2023.
12

---

[50] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. Stanford alpaca: An insllwciionfoolloiingg Ilama model. https://gittub 2023.
[51] Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Gupta, et al. Transformer memory as a differentiable search index. NeurIPS, 2022.
[52] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
[53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922. 2023.
[54] Alasdair Tran, Alexander Mathews, and Lexing Xie. Transform and tell: aware news image captioning. In CVPR, 2020.
[55] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[56] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin. Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint 14000 2022.
[57] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey 20o. In CVPR, 2010.
[58] Zilin Xiao, Ming Gong, Paola Cascante-Bonilla, Xingyao Zhang. Jie Wu, and Vicente Ordonez. Grounding language models for visual entity recognition. arXiv preprint arXiv:2402. 22222422..899 18695, 2024.
[59] Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In ICCV, 2021.
[60] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Multimodal neural script knowledge through vision and language and sound. CVPR, 2022.
[61] Wentian Zhao and Xinxiao Wu. Boosting entity-aware image captioning with multi-modal knowledge graph. IEEE Transactions on Multimedia, 2023.
[62] Yue Zhao, Ishan Misra, Philipp Krähenbuhl, and Rohit Girdhar. Learning video representations from large language models. arXiv preprint arXiv:2212.04501, 2022.
[63] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 2017.
13

---

Eridg: Parc des Du Chaumort
Exily: Vilonblled
mmagraa hurge
kimple levsn kenpie
Buyeeaa 11 Tourond inniaan
urdfer kdAk
parcis very barg
C1)) Whhul Bee name bhe partin image?
Oi] Whasknddl blinds owitn the image?
phou the nome
top offife
(o)) Warriem ooch lmngee
laka mmage??
UBI in vothoe image??
dealts CosChaaammt
look and flowrra
Easty:
reperttinng
wage
Rattonalo The feg vou browe dkin sas bense spons. stting grouns
F Fandogant Resiisonn Sow!
seeedao--rnnee paine:
Guestarn-naee paire:
Ard Sieve fdevads pelice
(haa bnardfite shurch? Farnooyattin
C Nhat sh na Yrokum oftit www.eeoofwweeeeoeeeeoooooooo (2urkh
AX (fronn with de tecenssph
CX) Whasla
A2] glasa
Figure 4: Qualitative examples of entities, rationales and question-answer pairs obtained with the multi-modal LLM. Our prompt encourage asking questions about diverse entities in the image.
Table 7: Retrieval-enhanced contrastive training (RECO). We compare three types of annotations for memory database: original caption, candidate entities our multimodal LLM corrected entities. We report the absolute improvements over the CLIP baseline between parentheses as well as the relative improvement averaged across the datasets in the last column. For reference, also include the results from Iscen et al. [88 but note that they use a memory base 20x bigger (WebLI-1B)
Memory dataset
Cars 20
CUB
ImNet 44
Flowers
33
Places 63
Dogs
99
Avg.relative A
CLIP-L/14
75.6
61.7
75.6
75.5
42.0
72.7
CLIP-L/14 + RECO
Captions WebLI-1B IX
82.8
73.4
76.1
79.5
43.6
73.9
+6.7%
Captions WcbLI-47M
74.6(-10)
73.8(+1211
75.8(-02))
77.8(+2.3)
43.7(+1.7)
73.4(-0.7)
+4.4%
Entity-WebLI
76.0++04)
69.3(+7.6)
75.8(-02)
77.7(+2.2)
43.5(+1.5)
72.8(+0.1)
+3.3%
REW-47M (Ours)
76.2
72.6(+10.9 76.0(-0.)
81.6(+6.1)
43.7(+1.7)
73.8
45.4%
A Appendix and supplemental material
A.1 Additional results
A.1.1 A different application of our dataset: memory base for RECO.
We explore the potential of our dataset when utilized as a memory base for retrieval-enhanced contrastive training (RECO) [8] in Tab. 7 Each image in the memory either associated with the original WebLI caption, with the candidate entity from text k-NN matching (as in Entity- Entity-WebLI ( or with the multimodal LLM corrected entity (our method REW-47M). In Tab. 7 we show that using the multimodal LLM to correct entity entities results in a relative improvement compared to CLIP of +5.4% on average across the datasets considered. Using the original dataset captions instead results in smaller relative improvement of +4.4%. Lastly, we see in Tab. 7 that are able to achieve performance comparable to or even better than RECO's results on some datasets, despite using memory base that is 20 times smaller. This finding suggests that smaller amount of high-quality annotated data can be just as effective, not more so, than a larger amount data with lower annotation quality.
A.1.2 Opensource models PaliGemma and Gemma
see Tab. that the main difference of performance between Gemini-Pro and Gemma variants comes from the QA loss. While Gemini-Pro has access to input images when generating QA pairs, Gemma generates QA pairs based on the PaliGemma caption (and rationale). This limits the variety of the generated QA pairs, resulting in lower final accuracy.
14

---

Table 8: Analysis of the impact of multi-task training with generated rationales and QAs with private versus opensource models. We perform this experiment with both WebLI [ and LAION 46]]. We report HM top-1 accuracy on OVEN validation splits directly after REW training.
LLM-Refined Entity-
WebLI
-LAION
Lrany
Caationae
SQA
Entity
Query
Entity
Query
with private Gemini Pro model A55
14.1
5.4
10.7
5.6
/
14.6
6.7
11.4
6.9
16.0
28.2
13.4
28.2
with opensource PaliGemma 3 and Gemma (52 models
11.9
5.8
13.3
6.3
10.6
9.7
14.2
24.3
11.6
23.4
Question: What the mach object?
Question: What twn prwsarted! In the image?
Question: Which type ofanima diepctedd image Ground truth entily: Hay
Ground truth entiyy: Barrel caclus
Ground truth entily: Telescopic sight
Top-5 predictions:
Top-5 predictions:
Top-6 predictions:
1: Echinorachia
Spotting scope
Hay
Echinocactia grawoni
Monociaar
Telescopic sight
Cactus
Binoculars
Agricultural machinery
& Echinocactus platyacanthus
Vartex
Figure 5: Qualitative examples of suboptimal annotations in OVEN benchmark. We show the input question, input image, OVEN ground truth entity as well as the top-5 predictions our model.
A.2 Limitations of the OVEN benchmark
The goal of this paper is to develop models capable of matching any given image-text query to an entity with a high precision. While the results in this work show that improve upon the state of the art on the challenging web-scale visual entity recogmition OVEN benchmark, we note that our best model still makes mistakes more than 70% of the time (see Tab. which is far from optimal. While this shows that there still lot of room for improvement for further research in that direction, provide short analysis in this section of some limitations of the OVEN benchmark itself.
In Fig, show the top-5 predictions of model for various examples from the OVEN validation set. We observe that, in some cases, our model's 's predictions are more accurate than the expected ground truth entities provided by the OVEN benchmark. This is because some of the positive entities in the OVEN validation or test set may have a better match within the negative entities. For instance, model correctly predicts Echinocactus grusonii before the expected Barrel cactus entity, as the former is a more specific and accurate description of the cactus in the image. Similarly, our model predicts spotting scope instead of the expected telescopic sight, as the former more precise description of the visual entity in question.
To address these limitations, we suggest considering the top-k predictions when comparing the performance of different models. For example, our best-performing model's HM score increases dramatically from 29,6 to 57.0 when we considered the top-10 10 predictions instead of just the top-1 prediction.
Finally, the right side of Fig 5 shows an example where the input question, Which type of animal is depicted in the image?. is not relevant to the input image, as there is no animal present. In this case, our model disregards the input question and outputs entities that are present the image but are not animals such as hay or tractor for example. This raises the question of whether the model should prioritize the input question and provide a more relevant answer, even it means potentially sacrificing accuracy. For instance, the model could output response such as "There is no animal in the image." which would more relevant the user's '' query, but would not be strictly accurate in terms of entity recognition.
15

---

Table Statistical significance. We report the harmonic mean (HM) of the seen and unseen splits (top-1 accuracy) on OVEN training seen categories for different seeds. We use the small version of REW for this experiment. We report the mean and the standard deviation the last row.
Entity split
Query split
Model
Pretraining data
HM
seen
unseen
seen
unseen
Seed
REW (4.5M)
16.2
19.6
13.7
29.1
32.6
26.3
Seed
REW (4.5M)
15.6
19.0
13.3
27.9
32.3
24.5
Seed
REW (4.5M)
16.3
19.6
13.9
28.0
32.7
24.5
Seed
REW (4.5M)
16.1
19.7
13.6
27.5
32.4
23.8
Seed
REW (4.5M)
15.9
19.3
13.5
28.7
32.6
25.5
Mean and standard deviation
16.0+a.2
19.4
13.6
28.2
32.6 =0.1
24.9
A.3
Experimental Setting and Details
A.3.1 Prompts
We input the image to the multimodal LLM as well as the following prompts.
Entity correction and rationale generation prompt. We include below our full prompt template for the entity correction and rationale generation process. We insert the original caption (original caption), the candidate entity as (candidate entity) and the Wikipedia page content summary of the candidate entity as (Wikipedia summary).
You are working on an entity recognition task. Is this image of candidate entity? Your answer must either 'YES' or 'NO'.
Here is the definition of candidate entity: Wikipedia summary.
your answer is 'YES', you must use the definition of candidate entity answer whether this is an image of candidate entity.
If your answer is 'NO', you must use caption of the image original caption describe the main object in the image with the most specific English Wikipedia article title, where the response follows the format @@ @response@.. " You must then explain your answer by describing the visual attributes of the image.
If you answer 'YES', your explanation MUST be based on the definition of candidate entity. If you answer is 'NO", your explanation MUST ONLY be based the visual cues of the image, and should NOT contain candidate entity.
Your explanation must be concise.
Your explanation MUST NOT exceed two sentences.
Question/answer pair generation prompt. We include below our full prompt template for the question/answer pairs generation process. We insert the previously verified/corrected entity as (entity) and the previously generated rationale (rationale).
You are working on a visual question answering task.
This is an image of entity.
Your rationale the following: rationale.
Your task is to generate 3 question/answer pairs describing the visual attributes of this image. The questions MUST be diverse and cover several entities of the image, including the main object in the image or image itself. The answers MUST be specific English Wikipedia article titles. The answers MUST based on the visual content of the image and the provided rationale.
The format for the question/answer pairs is Q: Qqqusttion>> A:canswer>. The questions MUST NOT contain What the main object the image?.
16

---

A.3.2 Statistical significance of the experiments
We report the performance variance on our model trained on the small version of our dataset in order to assess the statistical significance of our results. We run this experiment five times with different random seeds. We perform the evaluation for each model separately and report in Tab. the accuracy for each run in the different OVEN splits. We also report the mean over these five runs and the standard deviation.
A.3.3 Compute resources for the experiments
Our models are trained on 256 TPUv3. The short schedule 200k steps training lasts for 15 hours while the longer 600k steps training lasts for 44 hours. The full scope of our research project involved more runs than what is reported in the paper. This is because we conducted preliminary explorations and experimented with various design choices that ultimately did not yield successful results. While these failed experiments are not presented in the tables of the paper, we believe that they are an important part of any research project and contributed to the development of our final approach.
A.3.4 Implementation details
Training and inference on REW dataset. We use GiT-Large [56): it consists of a visual encoder
(CLIP-L/14 [4T) and 6-layer text decoder with internal dimension d 768. Following [7]. the visual encoder is first pre-trained jointly on WebLI-100M [9] and Conceptual Captions- 12M [47] while the decoder is randomly initialized. We use batch size of 4096, learning rate of 1e for the visual encoder and le for the decoder, label smoothing of 0.2 and no weight decay. We use AdamW optimizer [27] and cosine learning rate schedule with final learning rate of 0. We use standard inception crop data augmentation for the images. We set the maximum decoding length to 32 tokens and the maximum number of context tokens 32 tokens well. We find that 32 tokens is enough to comprehensively tokenize all the OVEN input questions as well as all the Wikipedia entity names. The decoding beam size set to 30.
Finetuning on OVEN train set. We finetune models on OVEN training set for 10,000 steps with a batch size of 256. Note that we equally balance the query and entity split contribution during finetuning. We choose the learning rate out of three values (1e 3e le 6) based on the HM performance on the OVEN validation set. Label smoothing is set at 0.1. Note that the finetuning schedule relatively short (10,000 steps) because we observe that long finetuning (or equivalently. using a large learning rate) causes the model to forget about the unseen categories.
A.4 Broader impacts discussion
Potential positive societal impacts. Our proposed methodology for curating high-quality, large- scale dataset for web-scale visual entity recognition using multimodal LLMs has the potential to significantly impact the field of computer vision. By improving the quality of training data, can enhance the performance of models for various entity-aware visual understanding tasks, including info-seeking VQA News content understanding, leading to more accurate and reliable results. Furthermore, our approach has the potential to democratize the process of dataset creation, reducing the time and resources required for manual annotation. This can enable researchers and practitioners with limited resources to create high-quality datasets.
Potential negative societal impacts. However, it is essential to consider the potential risks and ethical implications of our work. The use of LLMs for dataset creation and annotation raises concerns about bias and fairness, as the models may reflect and perpetuate the biases present in the data they were trained on. Therefore, is important to carefully evaluate and mitigate any potential biases in the datasets created using our approach. Additionally, the use of large-scale
such as WebLI 9 or LAION [46] databases for entity recognition raises concerns about privacy and data protection, as the images and captions used for training may contain sensitive information. Therefore, essential to ensure that the datasets created using our approach are compliant with relevant privacy and data protection regulations and that appropriate measures are taken to protect the privacy and security of the data.
In summary, our work has the potential to significantly impact the field of computer vision and enable the development of more accurate and reliable models for various entity-aware visual understanding
17

---

tasks. However, it is essential to carefully consider and address the potential risks and ethical implications of our work ensure that it is used for the benefit of all.
A.5 Class label to Wikipedia entity mappings
We selected five finegrained datasets for evaluation because their class categories align with a subset of the Wikipedia entities that our model is trained to recognize. We obtained a preliminary mapping of class labels to Wikipedia entities from the authors of OVEN and then improved it through a careful manual review.
Oxford Flowers 35 We use the following class name to Wikipedia entity name mapping:
[pink primrose: oenothera speciosa: hard-leaved pocket orchid: paphiopedilum micranthum; can- terbury bells: campanula medium; sweet pea: sweet pea: english marigold: calendula officinalis; tiger lily: lilium lancifolium; moon orchid: phalaenopsis amabilis; bird of paradise: strelitzia;
monkshood: aconitum; globe thistle: echinops; snapdragon: antirrhinum; colt's foot: tussilago;
king protea: protea cynaroides; spear thistle: cirsium vulgare; yellow iris: iris pseudacorus; globe-
flower: trollius europaeus; purple coneflower: echinacea purpurea: peruvian lily: alstroemeria:
balloon flower: platycodon; giant white arum lily: zantedeschia; fire lily: lilium bulbiferum; pincushion flower: scabiosa; fritillary: fritillaria; red ginger: alpinia purpurata; grape hyacinth:
muscari; corn poppy: papaver rhoeas; prince of wales feathers: amaranthus hypochondriacus;
stemless gentian: stemless gentian; artichoke: artichoke; sweet william: dianthus barbatus; carna-
tion: dianthus caryophyllus; garden phlox: phlox panieulata; love in the mist: nigella damascena; mexican aster: cosmos bipinnatus; alpine sea holly: eryngium alpinum; ruby-lipped cattleya: cattleya labiata; cape flower: nerine bowdenii; great masterwort: astrantia major; siam tulip:
curcuma alismatifolia; lenten rose: hellebore; barbeton daisy: gerbera jamesonii; daffodil: narcis- sus (plant); sword lily: gladiolus; poinsettia: poinsettia; bolero deep blue: eustoma russellianum;
wallflower: erysimum; marigold: tagetes; buttercup: ranunculus; oxeye daisy: leucanthemum vulgare; common dandelion: taraxacum officinale; petunia: petunia; wild pansy: viola tricolor; primula: primula: sunflower: common sunflower; pelargonium: pelargonium: bishop of Ilandaff: dahlia "bishop of Ilandaff'; gaura: gaura; geranium: geranium: orange dahlia: tithonia rotundifolia; pink-yellow dahlia?: dahlia; cautleya spicata: cautleya spicata; japanese anemone: eriocapitella japonica; black-eyed susan: rudbeckia hirta; silverbush: convolvulus eneorum; californian poppy:
eschscholzia californica; osteospermum: osteospermum; spring crocus: crocus vernus; bearded
iris: iris (plant); windflower: anemonoides blanda; tree poppy: romneya; gazania: gazania; azalea:
azalea; water lily: nymphaeaceae; rose: rose; thorn apple: datura; morning glory: morning
glory; passion flower: passiflora; lotus: nelumbo nucifera; toad lily: tricyrtis hirta; anthurium:
anthurium; frangipani: plumeria; clematis: clematis; hibiscus: hibiscus; columbine: aquilegia; desert-rose: adenium obesum; tree mallow: malva arborea; magnolia: magnolia; cyclamen: cy- clamen; watercress: watercress; canna lily: canna (plant); hippeastrum: hippeastrum: bee balm: monarda; ball moss: wallisia; foxglove: digitalis; bougainvillea: bougainvillea; camellia: camellia; mallow: althaea officinalis; mexican petunia: ruellia simplex; bromelia: bromelia; blanket flower: gaillardia; trumpet creeper: campsis radicans; blackberry lily: iris domestica].
Sun397 [57].
We use the following class name to Wikipedia entity name mapping:
(abbey: abbey; airplane cabin: aireraft cabin; airport terminal: airport terminal; alley: alley; am- phitheater: amphitheatre; amusement arcade: amusement arcade; amusement park: amusement park; anechoic chamber: anechoic chamber; apartment building/outdoor: apartment; apse/indoor:
apse; aquarium: aquarium; aqueduct: aqueduct (bridge); arch: arch; archive: archive; arrival gate/outdoor: airport apron; art gallery: gallery; art school: art school; art studio: stu- dio; assembly line: assembly line; athletic field/outdoor: pitch (sports field); atrium/public:
atrium (architecture); attic: attic; auditorium: auditorium; auto factory: automotive industry; badlands: badlands; badminton court/indoor: badminton; baggage claim: baggage reclaim; bakery/shop: bakery; balcony/exterior: balcony; balcony/interior: mezzanine; ball pit: ball pit;
ballroom: ballroom; bamboo forest: bamboo; banquet hall: banquet hall; bar: bar (establishment);
barn: barn; barndoor: barnyard; baseball field: baseball field; basement: basement; basílica:
basilica; basketball court/outdoor: basketball court; bathroom: bathroom; batters box: batting
(baseball); bayou: bayou; bazaar/indoor: bazaar; bazaar/outdoor: marketplace; beach: beach;
beauty salon: beauty salon; bedroom: bedroom; berth: bunk bed; biology laboratory: laboratory:
bistro/indoor: bistro; boardwalk: boardwalk; boat deck: deck (ship); boathouse: boathouse;
bookstore: bookselling; booth/indoor: convention (meeting); botanical garden: botanical garden;
18

---

bow window/indoor: window; bow window/outdoor: bow window: bowling alley: bowling alley;
boxing ring: boxing ring: brewery/indoor: brewery; bridge: bridge; building facade: façade;
bullring: bullring; hurial chamber: chamber tomb; bus interior: bus; butchers shop: butcher;
butte: butte; cabin/outdoor: log cabin; cafeteria: cafeteria; campsite: campsite; campus: cam- pus; canal/natural: canal (garden history); canal/urban: canal; candy store: confectionery store; canyon: canyon; car interior/backseat:: car seat; car interior/trontseat: bucket seat; carrousel:
carousel; casino/indoor: casino; castle: castle; catacomb: catacombs; cathedral/indoor: sanc-
tuary; cathedral/outdoor: cathedral; cavern/indoor: cave; cemetery: cemetery; chalet: chalet;
cheese factory: creamery; chemistry lab: chemistry; chicken coop/indoor: chicken; chicken coop/outdoor: poultry farming; childs room: child care; church/indoor: nave; church/outdoor:
church (building); classroom: classroom; clean room: cleanroom; cliff: cliff; cloister/indoor:
cloister; closet: closet; clothing store: clothes shop; coast: coast; cockpit: cockpit; coffee shop:
coffeehouse; computer room: computer lab; conference center: convention center; conference
room: conference hall; construction site: construction; control room: control room; control
tower/outdoor: air traffic control; corn field: harvest; corral: pen (enclosure); corridor: hallway;
cottage garden: cottage garden; courthouse: courthouse; courtroom: courtroom; courtyard:
courtyard; covered bridge/exterior: covered bridge; creek: stream; crevasse: crevasse; crosswalk:
pedestrian crossing; cubicle/office: cubicle; dam: dam; delicatessen: delicatessen; dentists office:
dentistry; desert/sand: desert; desert/vegetation: deserts and xeric shrublands; diner/indoor: din-
ner; diner/outdoor: diner; dinette/home: table (furniture); dinette/vehicle: recreational vehicle;
dining car: dining car; dining room: dining room: discotheque: nightelub: dock: dock: door- way/outdoor: door; dorm room: dormitory: driveway: driveway; driving range/outdoor: driving range; drugstore: pharmacy (shop); electrical substation: electrical substation; elevator/door: automatic door; elevator/interio:: elevator; elevator shaft: shaft (mechanical engineering); engine room: engine room; escalator/indoor, escalator; excavation: excavator; factory/indoor: fac-
tory; fairway: lawn; fastfood restaurant: fast-food restaurant; field/cultivated: field (agriculture);
field/wild: wild field; fire escape: fire escape; fire station: fire station; firing range/indoor: shooting range; fishpond: fishpond; florist shop/indoor: floristry; food court: food court; forest/broadleaf:
forest; forest/needleleaf:: conifer; forest path: forest track; forest road: agricultural road; formal
garden: formal garden; fountain: fountain; galley: galley (kitchen); game room: game room; garage/indoor: garage (residential); garbage dump: waste container; gas station: filling station; gazebo/exterior: gazeho; general store/indoor: retail; general store/outdoor: general store; gift
shop: gift shop: golf course: golf course; greenhouse/indoor: garden; greenhouse/outdoor: green- house; gymnasium/indoor: gymnasium (school); hangar/indoor: airship hangar; hangar/outdoor:
hangar; harbor: harbor; hayfield: hay; heliport: heliport; herb garden: kitchen garden; highway:
highway; hill: hill; home office: small office/home office; hospital: hospital; hospital room: room
service; hot spring: hot spring: hot tub/outdoor: hot tub; hotel/outdoor: hotel; hotel room: suite
(hotel); house: house; hunting lodge/outdoor: hunting and shooting in the united kingdom; ice
cream parlor: ice cream parlor; ice floe: ice floe; ice shelf: ice shelf: ice skating rink/indoor:
ice skating; ice skating rink/outdoor: ice rink; iceberg: iceberg; igloo: igloo; industrial area:
industrial district; inn/outdoor: inn: islet: islet; jacuzzi/indoor: jacuzzi; jail/indoor: prison; jail
cell: prison cell; jewelry shop: jewellery store; kasbah: kasbah; kennel/indoor: kennel; ken-
nel/outdoor: dog crate; kindergarden classroom: kindergarten; kitchen: kitchen; kitchenette:
kitchenette; labyrinth/outdoo:: labyrinth; lake/natural: lake; landfill: landfill; landing deck:
flight deck; laundromat: self-service laundry; lecture room: lecture room; library/indoor: library;
library/outdoor: public library; lido deck/outdoor: lido; lift bridge: vertical-lift bridge; lighthouse:
lighthouse; limousine interior: limousine; living room: living room; lobby: lobby (room); lock
chamber: lock (water navigation); locker room: changing room; mansion: mansion; manufactured
home: manufactured housing: market/indoor: grocery store; market/outdoor: farmers' market;
marsh: marsh; martial arts gym: martial arts; mausoleum: mausolcum; medina: medina quarter; moat/water: moat; monastery/outdood monastery; mosque/indoor: islam; mosque/outdoor: mosque; motel: motel: mountain: mountain; mountain snowy: glacier; movie theater/indoor: movie theater; museum/indoor: museum; music store: music store; music studio: recording studio;
nuclear power plant/outdoor: nuclear power plant; nursery: nursery (room); oast house: oast
house; observatory/outdoor: observatory; ocean: ocean; office: desk; office building: office; oil
refinery/outdoor: oil refinery; oilrig: oil rig: operating room: operating theater; orchard: orchard;
outhouse/outdoorr outhouse; pagoda: pagoda; palace: palace; pantry: pantry; park: park; parking
garage/indoor: parking space; parking garage/outdoor: multistorey car park; parking lot: parking
lot; parlor: parlour: pasture: pasture; patio: patio; pavilion: pavilion: pharmacy: pharmacy; phone
19

---

booth: telephone booth; physics laboratory: physics; picnic area: picnic; pilothouse/indor:
bridge (nautical); planetarium/outtoorr planetarium: playground: playground: playroom: fam-
ily room; plaza: town square; podium/indoor: lectern; podium/outdoor: podium; pond: pond;
poolroom/esablishment billiard room; poolroom/home: billiard table; power plant/outdoor:
power station; promenade deck: promenade deck; pub/indoor: pub; pulpit: pulpit; putting green:
greenskeeper; racecourse: horse racing; raceway: race track; raft: raft; railroad track: railway
track; rainforest: rainforest; reception: front office; recreation room: recreation room; residential neighborhood: residential area: restaurant: restaurant; restaurant kitchen: chef; restaurant patio: outdoor dining; rice paddy: paddy field; riding arena: riding hall; river: river; rock arch:
natural arch; rope bridge: simple suspension bridge; ruin: ruins; runway: runway; sandbar:
shoal; sandbox: sandpit; sauna: sauna; schoolhouse: school; sea cliff: cliffed coast; server room:
server room; shed: shed; shoe shop: shoemaking; shopfront: storefront; shopping mall/indoor:
shopping mall; shower: shower; skatepark: skatepark; ski lodge: ski lodge; ski resort: ski resort; ski slope: alpine skiing; sky: sky; skyscraper: skyscraper; slum: slum; snowfield: snow field;
squash court: squash (sport); stable: stable; stadium/baseball: ballpark; stadiun/foothall: stadium; stage/indoor: stage (theatre); staircase: stairs; street: street; subway interior: public transport; sub- way ptation/platform: metro station; supermarket: supermarket: sushi bar: sushi; swamp: swamp; swimming pool/indoor: swimming pool; swimming pool/outdoor: lido; synagogue/indoor: bema; synagogue/outdoors synagogue; television studio: television studio; temple/east asia: temple; temple/south asia: hindu temple; tennis court/indoor: carpet court; tennis court/outdoor: tennis court: tent/outdoor: tent; theater/indoor procenium: proscenium; theater/indoor seats: theatre; thriftshop: charity shop; throne room: throne room; ticket booth: box office; toll plaza: toll road; topiary garden: topiary: tower: tower; toyshop: toy store; track/outdoor: running track; train railway: rail transport; train station/platform: train station; tree farm: tree farm; tree house:
tree house: trench: trench; underwater/coral reef: coral reef; utility room: utility room; valley:
valley; van interior: van; vegetable garden: vegetable farming; veranda: veranda; veterinarians
office: veterinarian; viaduct: viaduct; videostore: video rental shop; village: village; vineyard:
vineyard; volcano: volcano; volleyball court/indoo:: volleyball; volleyball court/outdoor: beach
volleyball; waiting room: waiting room; warchouse/indoor: warehouse; water tower: water tower; waterfall/block: rapids; waterfallfann waterfall; waterfall/plunge: plunge pool; watering hole: depression (geology); wave: wave; wet bar: wet bar; wheat field: wheat fields; wind farm: wind farm; windmill: windmill; wine cellar/barrel storage: barrel; wine cellar/bottle storage: wine cellar; wrestling ring/indoor: wrestling ring; yard: yard (land); youth hostel: hostel].
F0od101 (5). We use the following class name to Wikipedia entity name mapping:
(apple pie: apple pie; baby back ribs: pork ribs: baklava: baklava; beef carpaccio: carpaccio; beef tartare: steak tartare; beet salad: vinegret; beignets: beignet; bibimbap: bibimbap; bread pudding:
bread pudding; breakfast burrito: breakfast burrito; bruschetta: bruschetta; caesar salad: caesar
salad; cannoli: cannoli; caprese salad: caprese salad; carrot cake: carrot cake; ceviche: ceviche;
cheesecake: cheesecake; cheese plate: cheese; chicken curry: chicken curry; chicken quesadilla: quesadilla; chicken wings: chicken as food; chocolate cake: chocolate cake; chocolate mousse:
mousse: churros: churro; clam chowder: clam chowder; club sandwich: club sandwich; crab cakes:
crab cake; creme brulee: crème brûlée; croque madame: croque monsicur; cup cakes: cupcake;
deviled eggs: deviled egg; donuts: doughnut; dumplings: dumpling; edamame: edamame; eggs benedict: eggs benedict; escargots: snails as food; falafel: falafel; filet mignon: filet mignon; fish
and chips: fish and chips; foie gras: foie gras; french fries: french fries; french onion soup: french
onion soup: french toast: french toast; fried calamari: squid as food; fried rice: fried rice; frozen yogurt: frozen yogurt; garlic bread: garlic bread; gnoechi: gnocchi; greek salad: greek salad; grilled cheese sandwich: grilled cheese; grilled salmon: list of potato chip brands; guacamole:
guacamole; gyoza: jiaozi; hamburger: hamburger; hot and sour soup: hot and sour soup: hot dog: hot dog: huevos rancheros: huevos rancheros; hummus: hummus; iee cream: ice cream;
lasagna: lasagna; lobster bisque: bisque (food); lobster roll sandwich: lobster roll; macaroni and cheese: macaroni and cheese; macarons: macaron; miso soup: miso soup; mussels: mussel; nachos: nachos; omelette: omelette; onion rings: onion ring; oysters: oyster; pad thai: pad thai;
paella: paella; pancakes: pancake; panna cotta: panna cotta; peking duck: peking duck; pho: pho; pizza: pizza; pork chop: pork chop; poutine: poutine; prime rib: standing rib roast; pulled pork sandwich: pulled pork; ramen: ramen; ravioli: ravioli; red velvet cake: red velvet cake; risotto: risotto; samosa: samosa; sashimi: sashimi; scallops: scallop; seaweed salad: wakame; shrimp and grits: shrimp and grits; spaghetti bolognese: bolognese sauce; spaghetti carbonara: carbonara;
20

---

spring rolls: spring roll; steak: steak; strawberry shortcake:
shortcake; sushi: sushi; tacos: taco;
takoyaki: takoyaki; tiramisu: tiramisu; tuna tartare: tuna; waffles: waffle].
FGVC-Aireraft [29]. We use the following class name to Wikipedia entity name mapping:
[Boeing 707: boeing 707: Boeing 727: boeing 727; Boeing 737: boeing 737; Boeing 747: boeing
747: Boeing 757: boeing 757: Boeing 767: boeing 767: Boeing 777: boeing 777; A300: airbus
a300; A310: airbus a3 10; A320: airbus a320 family; A330: airbus a330: A340: airbus a340; A380:
airbus a380; -TR-42: atr 42; ATR-72: atr 72; An-12: antonov an- 12: BAE 146: british aerospace
146; BAE-125: british aerospace 125; Beechcraft 1900: beechcraft 1900; Bocing 717: boeing 717:
C-130: lockheed c-130 hercules; C-47: douglas c-47 skytrain; CRJ-200: bombardier crj crj100/200;
CRJ-700: bombardier crj700 series; Cessna 172: cessna 172; Cessna 208: cessna 208 caravan;
Cessna Citation: cessma citation family; Challenger 600: bombardier challenger 600 series; DC-10:
medonnell douglas de- 10; DC-3: douglas dc-3; DC-6: douglas dc-6; DC-8: douglas dc-8; DC-9:
medonnell douglas dc-9; DH-82: de havilland tiger moth: DHC-1: de havilland canada dhe-1
chipmunk; DHC-6: de havilland canada dhc-6 twin otter: Dash 8: de havilland canada dash 8;
DR-400: robin dr400; Dornier 328: dornier 328; Embraer E-Jet: embraer e-jet family; EMB-120
embraer emb 120 brasilia: Embraer ERJ 145: embraer erj family; Embraer Legacy 600: embraer
legacy 600; Eurofighter Typhoon: eurofighter typhoon; F-16: general dynamics f-16 fighting falcon; F/A-18: medonnell douglas f/a-18 hornet; Falcon 2000: dassault falcon 2000; Falcon 900: dassault
falcon 900; Fokker 100: fokker 100; Fokker 50: fokker 50; Fokker 70: fokker 70; Global Express:
bombardier global express: Gulfstream: gulfstream aerospace; Hawk T1: bae systems hawk; I1- 76: ilyushin i1-76; L-1011: lockheed 1-1011 tristar, MD-11: medonnell douglas md-1 1d MD-80:
medonnell douglas md-80; MD-90: medonnell douglas md-90; Metroliner: fairchild swearingen metroliner; King Air: beechcraft king air; PA-28: piper pa-28 cherokee; SR-20: cirrus sr20; Saab
2000: saab 2000; Saab 340: saab 340; Spitfire: supermarine spitfire: Tornado: panavia tornado;
Tu-134: tupolev tu- 134; Tu-154: tupolev tu-154; Yak-42: yakovlev yak-42).
Sports100 17 We use the following class name to Wikipedia entity name mapping:
[air hockey: air hockey; ampute football: amputee football; archery: archery; arm wrestling: arm wrestling; axe throwing: axe throwing: balance beam: balance beam: barell racing: barrel racing;
baseball: baseball; basketball: basketball; baton twirling: baton twirling; bike polo: cycle polo;
billiards: pool (cue sports); bmx: bmx; bobsled: bobsleigh; bowling: bowling; boxing: boxing; bull
riding: bull riding; bungee jumping: bungee jumping: canoe slamon: canoe slalom; cheerleading:
cheerleading; chuckwagon racing: chuckwagon racing; cricket: cricket; croquet: croquet; curling:
curling; disc golf: disc golf: fencing: fencing: field hockey: field hockey; figure skating men: figure skating; figure skating pairs: pair skating; figure skating women: single skating; fly fishing: fly
fishing; football: football; formula racing: formula one racing; frrisbe: frisbee; gaga: gaga; giant slalom: giant slalom; golf: golf; hammer throw: hammer throw; hang gliding: hang gliding: harness racing: harness racing; high jump: high jump; hockey: underwater ice hockey; horse
jumping: show jumping; horse racing: horse racing: horseshoe pitching: horseshoes; hurdles:
hurdling; hydroplane racing: hydroplane racing; ice climbing: ice climbing; ice yachting: iceboat;
jai alai: jai alai: javelin: javelin; jousting: jousting; judo: judo; lacrosse: lacrosse; log rolling:
logrolling (sport); luge: luge; motorcycle racing: motorcycle racing; mushing: mushing; nascar
racing: nascar racing; olympic wrestling: wrestling: parallel bar: parallel bars; pole elimbing:
pole climbing; pole dancing: pole dance; pole vault: pole vault; polo: polo; pommel horse:
pommel horse: rings: rings (gymnastics); rock climbing: rock climbing, rollerblade racing: inline
skating; roller derby: roller derby; rowing: rowing; rugby: rugby union; sailboat racing: sailing (sport); shot put: shot put; shuffleboard: shuffleboard; sidecar racing: sidecar; ski jumping: ski jumping; skydiving: parachuting; sky surfing: skysurfing; snow boarding: snowboarding; snowmobile racing: snowmobile; speed skating: speed skating; steer wrestling: steer wrestling; sumo wrestling: sumo; surfing: surfing; swimming: swimming: table tennis: table tennis; tennis: tennis; track bicycle: track cycling; trapeze: trapeze; tug of war: tug of war; ultimate: ultimate (sport); uneven bars: uneven bars; volleyball: volleyball; water cycling: aqua cycling; water polo: water polo; weightlifting: weightlifting wheelchair basketball: wheelchair basketball; wheelchair racing: wheelchair racing; wingsuit flying: wingsuit flying].
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We validate our claims by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section and in Appendix A.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
While the code and data are not provided, we fully disclose all the information needed to reproduce our methodology, including prompts, the publicly available multimodal LLM API that we used, and thorough implementation details in Section 4.1 of the main paper and Appendix A.3
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Releasing code and data not possible in our case.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc//pill wttp:ccppppcccceeeeeeeeee..c.ett httpa///guddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee ssi.iiliccyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The experimental setting is presented in the main paper in Section 4.1 The full and thorough experiment details provided in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the experiment statistical significance in Section A.3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Discussion is Appendix A.3.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research is conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussion about the potential negative and positive societal impact of this work is in Appendix A.4
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We not releasing data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Previous datasets, models and related methodologies that this work builds upon are correctly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "i0fqjkY3l2", "reviewer": "Reviewer_Poha", "summary": "The paper presents a method to curate datasets for visual entity recognition tasks. They rely on a multimodal LLM (Gemini Pro), which employs metadata information about the image (the caption) and the content of the Wikipedia page to improve the quality of the Entity Web-Li dataset. They further enrich the resource with the rationale regarding the relation between the image and the entity and several question-answer pairs about diverse range of entities appearing in the image.", "questions": "Could you clarify how the avg. relative improvement in table 3 is computed?", "limitations": "The authors do not clarify why they are not releasing the updated version of the dataset. if this is done for safety reasons, it should be clarified as part of the limitations.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents a methodology for improving the quality of the Entity Web-Li dataset and show its usefulness in a series of downstream experiments and ablation studies (on the OVEN benchmark and a series of finegrained datasets). The experiments are carried out thoroughly and well described.", "weaknesses": "The main contribution of the paper is the creation of an enriched version of the Entity Web-Li dataset, named REW, which however is not released to the public together with the paper (this is mentioned in the Checklist). I find this a major weakness of this project, given the complexity of the conducted work (as clearly described in the paper and the appendix) and would strongly encourage the authors to reconsider this decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The main contribution of the paper is the creation of an enriched version of the Entity Web-Li dataset, named REW, which however is not released to the public together with the paper (this is mentioned in the Checklist). I find this a major weakness of this project, given the complexity of the conducted work (as clearly described in the paper and the appendix) and would strongly encourage the authors to reconsider this decision.**\n\nWe thank the reviewer for their comment. We would like to point out that we also conduct experiments with publicly available LAION dataset (Table 5). Our intention is to release the annotations for the LAION dataset. Furthermore, our results are reproducible with the public Gemini-API, and now also with the open source models described in the “General response” of the rebuttal. \n\n\n**Could you clarify how the avg. relative improvement in table 3 is computed?**\n\nThe relative improvement for each dataset is (new_value - old_value) / old_value.\nFor each row, we compute the relative improvement separately for each of the six datasets in the table and then compute their average.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cugItBFBzP", "reviewer": "Reviewer_MtgR", "summary": "The paper deals with web-scale visual entity recognition, which consists of math a question(text)-image query to one of the 6M entities (wikipedia page title) of a base of reference. In the vein of previous works, the task is addressed with a generative text-to-image model, the challenge lying in building the training dataset of this model and the method to learn it (task/losses considered). The approach is based on the recently published dataset Entity-Web-LI which associates an entity (name) to the images of an image-caption dataset. The novelty lies in an additional module of curation that relies on a multimodal LLM (Gemini), which asserts the relevance of the image retrieved in the image-caption external dataset. The multimodal LLM is also asked to provide explanations on its curation. With the novel training dataset, a Git-Large model is trained and evaluated in the context of visual entity recognition (OVEN benchmark) and zero-shot fine-grained classification (5 benchmarks) with better performances than the recent paper the work is built. The work is completed by several experiments of analysis and ablation.", "questions": "* Can we have an idea of the additional computational complexity (and resource usage) due to Gemini?\n\n* In the same vein, is it possible to estimate the performance boost/drop with another external \"multimodal LLM\"?\n\n* The authors report that the code is not released (Checklist \"Experimental Result Reproducibility\") but do they at least plan to release the new \"mapping\" manually curated and used for fine-grained classification?", "limitations": "Lines 324-332 are specifically dedicated to the limitations of the approach. It highlights the dependence of the approach on the availability of relevant external knowledge (that is Wikipedia in the context of the paper) as well as the fact that the proposed approach is expensive in terms of computations. One can regret that the message is vague and generic, without specific insight into the actual proposed method, not to mention possible quantitative hints on the actual computational time (of complexity in terms of memory). It refers to Appendix A.2 but that last focuses again on the performance of the model. However, Appendix A.3.3 reports the usage of 256 TPUv3 during 15+44 hours for the models used in the paper. This only concerns the training of the GitLarge models, but an estimation of the additive resources used to build the dataset (that is the main novelty of the paper) with Gemini would also be relevant.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* the proposed approach is built upon a method [6] recently published at CVPR 2024, that is after the NeurIPS 2024 submission deadline. All the experimental evaluations are compared to this paper with a similar model (GIT-Large). They also report the results that [6] obtained with another model (GER-ALD) that has a similar complexity (400M parameters) but obtained better results that Git-Large on the task of large-scale entity recognition. In all cases, the proposed model has better performance.\n\n* the approach is evaluated on both the task of large-scale entity recognition (OVEN query and OVEN entity) and on five benchmarks of fine-grained classification in a zero-shot setting, with better performances than [6] in all cases, according to all metrics. It is worth noting that the experimental results on (zero-shot) fine-grained classification are new ([6] did not report such results) and that much more metrics/settings are reported in this paper than in [6] on OVEN. The authors also report experiments of visual matching with CLIP and Dinov2 on six benchmarks, showing that the proposed dataset allows a significant boost in performance.\n\n* the analyses and ablation in Section 4.3 are quite detailed and give a fair view of the contribution of each method component as well as its behavior in \"degraded mode\". In particular, using their smaller dataset, the authors report results using LAION", "weaknesses": "* It is regrettable to rely on a private model (Gemini-pro) to build the main contribution of the paper, which is the training dataset. There's no guarantee that this model will be stable over time, such that the proposed method (to build the dataset) may not be reproducible in a couple of months. This issue would nevertheless be partially mitigated if the dataset itself is released, although it still limits the interest of the method itself (e.g maybe another prompt would be required to get comparable results, or another multi-task learning to train the T2I model...). Surprisingly, this aspect is not addressed in the \"Limitations\" of Section 5.\n\n* The authors argue (lines 239-243) that \"the zero-shot performance of the multimodal LLM on OVEN\" is much lower than those obtained when it is used alone thus it \"suggest[s] that we are not merely distilling from the considered multimodal LLM\". However, it is hard to believe that the gain in performances over [6] is not essentially due to the use of an additive (large, complex and trained on many data) LLM. Beyond the fact the namely LLM is opaque (see above) the contribution seems to mainly consist of a RAG-like approach to filter and correct the entities associated with the images by [6], relying on a black box which makes the process difficult to actually understand. So yes, \"it works\", but the scientific contribution nevertheless seems limited in that sense.\n\n\n**minor**\n* line 543 \"if you answer\" --> \"if your answer\"\n* in Appendix A.5, the authors report that the mapping used for fine-grained classification was that proposed by OVEN \"then improved it through a careful manual review\". For the sake of reproducibility, it should be clear that this mapping will be released to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**It is regrettable to rely on a private model (Gemini-pro) to build the main contribution of the paper, which is the training dataset. There's no guarantee that this model will be stable over time, such that the proposed method (to build the dataset) may not be reproducible in a couple of months. This issue would nevertheless be partially mitigated if the dataset itself is released, although it still limits the interest of the method itself (e.g maybe another prompt would be required to get comparable results, or another multi-task learning to train the T2I model...).**\n\nWe agree and appreciate the reviewer's valuable feedback regarding open-source accessibility. To address this, we adapted our method to leverage the open-source Gemma 27B model, renowned for its strong reasoning capabilities and long context support. Please see the “General response” of this rebuttal for the results with PaliGemma and Gemma models. We will release the implementation of this variant.\n\n\n\n**The authors argue (lines 239-243) that \"the zero-shot performance of the multimodal LLM on OVEN\" is much lower than those obtained when it is used alone thus it \"suggest[s] that we are not merely distilling from the considered multimodal LLM\". However, it is hard to believe that the gain in performances over [6] is not essentially due to the use of an additive (large, complex and trained on many data) LLM. Beyond the fact the namely LLM is opaque (see above) the contribution seems to mainly consist of a RAG-like approach to filter and correct the entities associated with the images by [6], relying on a black box which makes the process difficult to actually understand. So yes, \"it works\", but the scientific contribution nevertheless seems limited in that sense.**\n\nWe would like to clarify the following points: \n1. We compare the results of Gemini to our model (13.3 versus 29.6 top-1 HM in the entity split). The gain is due to the fact that Gemini is a general purpose model, whereas our model is trained on a Gemini curated dataset for the specific task of entity recognition. So Gemini doesn’t work out of the box, but is able to curate the labels when used properly (more on this in point 3 below).\n2. We train the same GiT-Large model on two versions of the WebLI datasets, i.e curated and uncurated. We can see that the curation improves the results significantly (20.1 versus 29.6). \n3. Our scientific contributions in this work is about how to use LLMs for the curating data for web-scale entity-recognition. As evidenced by the results presented in Table 4, our findings highlight two key contributions for the community:\n    * LLMs excel in verification over direct prediction:  We show that a naive application of LLMs for direct prediction (Table 4, Row 1) can actually hinder performance compared to [6]. This is because they tend to generate generic entities, rather than very fine-grained ones required for this task. However, utilizing LLMs for verification purposes (Table 4, Row 3) leads to a significant performance boost.\n    * External knowledge enhances LLM reasoning for fine-grained entities:  We demonstrate that augmenting LLMs with external knowledge sources like Wikipedia (Table 4, Row 4) further improves their ability to reason about and identify fine-grained entities.\n\nTo also emphasize the generalizability of our approach beyond a specific LLM, we've included additional experiments with an open-sourced model in the “General response” of this rebuttal. This highlights the adaptability of our methodology across different LLM architectures, even when their internal workings are opaque.\n\n**Is it possible to estimate the performance boost/drop with another external \"multimodal LLM\"?**\n\nWe report the accuracy of our method with other open source models (PaliGemma + Gemma) in Table R3 of the “General response”. Even though there is a small drop in performance when using open source models, our method still outperforms the prior work. \n\n**The authors report that the code is not released (Checklist \"Experimental Result Reproducibility\") but do they at least plan to release the new \"mapping\" manually curated and used for fine-grained classification?**\n\nYes, the mapping in the appendix section A.5 of the paper will be released.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCmYSOMEvg", "reviewer": "Reviewer_LX9i", "summary": "The work proposes a data-centric approach, that leverages MLLMs to first verify the correspondence of the existing pre-training dataset (Entity-WebLI) and correct errors, ask them to produce rationales, and generate novel, query split-oriented QA pairs to train a language model. The refined dataset REW is curated to facilitate the training specifically for the task of visual entity recognition (VER).", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. With the LLM augmented dataset, this work achieves superior performance over previous PALI and other generative models on the task of Oven-Wiki. Additional experiments on fine-grained image classification also validate the effectiveness of the collected dataset. \n\n2. Filtering an entity-centric dataset within three steps: correction, entity rationale generation, and QA generation, could be of great help for future works that focus on synthetic datasets towards entity-specific domains. While rationale and QAs do help, the most significant performance gain for entity split arises from the correction and verification parts, according to Table 4. \n\n3. Extensive ablations studies confirm that additional rationales and QA pairs help with multi-tasking training on VER.", "weaknesses": "1. In L241, authors state that the trained model is better than MLLM itself which is used to produce filtered datasets. However, I did not see the results for Gemini Pro (in L226 they mention that) in Table 1.\n\n2. Lack of references: In L142 and L170, authors state that constrained decoding is used to guarantee the successful grounding of entity rationale generation. Works like GMEL [1] and AutoVER [2] should be the first to introduce such techniques into entity generation in multimodal contexts. \n\n3. Minor issue: This work mainly proposes a data-centric solution to GER [3], and I assume the base model architecture of GER and this work stays the same with a 0.4B decoder-only model and a CLIP visual encoder. In Table 1, why is GiT-Large with Entity-WebLI training getting a better performance compared to GER-ALD with the same dataset? Is it because of the newly added two supervisions L_Rationale and L_QA?\n\n\n[1] Shi S, Xu Z, Hu B, et al. Generative multimodal entity linking[J]. arXiv preprint arXiv:2306.12725, 2023.\n\n[2] Xiao Z, Gong M, Cascante-Bonilla P, et al. Grounding Language Models for Visual Entity Recognition[J]. arXiv preprint arXiv:2402.18695, 2024.\n\n[3] Caron, Mathilde, et al. \"A Generative Approach for Wikipedia-Scale Visual Entity Recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**In L241, authors state that the trained model is better than MLLM itself which is used to produce filtered datasets. However, I did not see the results for Gemini Pro (in L226 they mention that) in Table 1.**\n\nThis result is shown in the inline text in Section 4.2: “Finally, we report the zero-shot performance of the multimodal LLM on OVEN: it reaches 13.3 HM top-1 in the entity split and 29.5 HM top-1 in the query split”. Note that this is the zero-shot Gemini Pro model, which does not always generate entities that are sufficiently fine-grained. On the other hand, our fine-tuned GiT-Large model achieves 29.6 top-1 in entity split and 30.9 top-1 in the query split. We will make this more clear in the camera ready.\n\n**Lack of references: In L142 and L170, authors state that constrained decoding is used to guarantee the successful grounding of entity rationale generation. Works like GMEL [1] and AutoVER [2] should be the first to introduce such techniques into entity generation in multimodal contexts.**\n\nWe thank the reviewer for bringing these papers to our attention, we will definitely discuss them in the camera ready. Note, however, that they differ significantly, as they use retrieval to augment their models which is complementary to our approach.\n\n**Minor issue: This work mainly proposes a data-centric solution to GER [3], and I assume the base model architecture of GER and this work stays the same with a 0.4B decoder-only model and a CLIP visual encoder. In Table 1, why is GiT-Large with Entity-WebLI training getting a better performance compared to GER-ALD with the same dataset?**\n\nThe GER paper [6] does not utilize constrained decoding. We opted to apply constrained decoding to both the \"GiT-Large with Entity-WebLI\" baseline and our model (\"GiT-Large with REW-47M\") as it yielded a significant performance boost (+5 points) for these captioning models, while it doesn't change the performance of the GER models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FPwoOR1cAB", "reviewer": "Reviewer_9wiz", "summary": "This work focuses on an important and realistic visual task called visual entity recognition (image, query --> entity name).\nThis work proposes a LLM driven technique to generate and refine large scale training data by modifying incorrectly labeled entity names from Entity-WebLI. Several errors are identified in Entity-WebLI and this work proposes several techniques to address them.\nThe experiment on OVEN and OOD datasets confirmed the usefulness of the new dataset REW and ablation studies show important design decisions contributing to the performance.", "questions": "- I wonder how the REW benchmark generalize to OOD entities of OVEN (landmarks, animals, plants, car, aircraft, etc).\nCan you show some examples of the model predictions on visual entity categories not covered by the OVEN benchmark such as artwork, shopping items, etc. \n\n- It seems using rational data improves query split in OVEN. I wonder if the authors have any intuition about this.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Proposing LLM driven approach to refine labels in a visual entity recognition training dataset. Identifying several issues in the existing dataset creation pipeline and show LLM can help to correct them using additional context from meta data or wikipedia caption with ablation studies.\n- Thorough experiments and ablation studies confirm the design choices such as QA, rationale generation for performance improvement.", "weaknesses": "- I don't see major weakness of this work except using LLM for data generation might be technically limited. However, I believe the techniques used to correct entity labels and findings are useful and novel for large scale entity centric caption creation.\n- releasing data and code: Although the data will not be released, I think the reproducibility of experiments are supported by the experiments with 5 random seeds and also using open-sourced data such as LAION. The community will benefit from reproducing such large-scale entity recognition image data by using LAION and multimodal LLM such as LLaVA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**releasing data and code: Although the data will not be released, I think the reproducibility of experiments are supported by the experiments with 5 random seeds and also using open-sourced data such as LAION. The community will benefit from reproducing such large-scale entity recognition image data by using LAION and multimodal LLM such as LLaVA.**\n\nWe intend to release LAION annotations. We also intend to release the implementation of the open source Gemma variant presented in the “General response” of this rebuttal.\n\n\n**I wonder how the REW benchmark generalize to OOD entities of OVEN (landmarks, animals, plants, car, aircraft, etc). Can you show some examples of the model predictions on visual entity categories not covered by the OVEN benchmark such as artwork, shopping items, etc.**\n\nWe show results for shopping items, such as chairs, lamps and bicycles from the Stanford Online Products dataset with our model in the rebuttal PDF. We can observe that our model is able to predict the correct fine-grained classes, as the corresponding entities are contained in the 6M entity list of Wikipedia, which is much more extensive than the OVEN entities. \n\n**It seems using rational data improves query split in OVEN. I wonder if the authors have any intuition about this.**\n\nWe believe that the rationales help clarify the connections between entities and visual attributes. Furthermore, providing rationales as additional supervision improves the model's language processing capabilities, which is important for deciphering complex text queries. These improvements are particularly valuable for VQA tasks, where an understanding of both visual and linguistic information is essential for accurate question interpretation and answer generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive comments. The reviewers especially appreciate the “thorough experiments and ablation studies”, that our work “could be of great help for future works that focus on synthetic datasets towards entity-specific domains”, and better results compared to the previous methods.

We would like to address a few comments regarding the reproducibility of our approach. Firstly, we would like to point out that the results on the public LAION dataset are included in Table 5 (Right). We plan to release the curated annotations for this dataset. 

Secondly, we have run an additional set of experiments for the rebuttal with the open source PaliGemma and Gemma 27B models and the results are inline with the results when using the Gemini-Pro model. Since Gemma lacks visual input processing, we replaced direct image input with automatically generated captions. Specifically, we employed the open-source PaliGemma model to generate descriptive captions for each image using the prompt: "Describe the visual attributes of this image."  We then integrated these captions into the existing prompts outlined in Section A.3.1 by prepending the text: "Here are the visual attributes of the image: {paligemma_output}." We plan to release the code for this implementation.

We evaluated this approach on the 5M subset of WebLI and LAION as used in our ablation studies (Section 4.3). We compare the Gemma and Gemini-Pro variants of our method in Table R1 with the SOTA methods trained on the 5M subset of Entity-WebLI. We can see that in all cases our method gives substantial improvements over the prior work GER-ALD[6] and GIT-Large trained on Webli-Entity[6]. We will include the Gemma results with the full 47M dataset in the camera ready.

More detailed analyses are shown in Tables R2 and R3 of the rebuttal. Numbers in bold are the numbers from with Gemma, compared to unbolded numbers from the Table 5 (right) of the paper. We can observe that different losses improve the performance in a similar way. Biggest difference of performance between Gemini-Pro and Gemma variants comes from the QA loss. While Gemini-Pro has access to input images when generating QA pairs, Gemma generates QA pairs based on the PaliGemma caption (and rationale). This limits the variety of the generated QA pairs, resulting in a lower final accuracy. 

```
```
>**Table R1: Comparison to the prior work when using WebLI-5M and LAION-5M as the pretraining data.**
|                      | Pre-training Dataset | Entity Split HM | Query Split HM |
| :-------------------- | :-------------------- | :-------------: | :-------------: |
| **SOTA METHODS**         |                      |                 |                 |
| GER-ALD [6]           | WebLI-5M             |       10.2      |        -        |
| GiT-Large (Captioning) | WebLI-5M             |        9.1      |        5.6      |
| **OURS**             |                      |                 |                 |
| REW (Gemma)          | LAION-5M             |       **11.6**      |      **23.4**      |
| REW (Gemini-Pro)     | LAION-5M             |       13.4      |       28.2      |
| REW (Gemma)          | WebLI-5M             |       **14.2**      |       **24.3**      |
| REW (Gemini-Pro)     | WebLI-5M             |      16.0     |      28.2    |
```
```
> **Table R2: Accuracy of generated rationales and QAs in LAION-5M.** 
| Entity Loss | Rationale Loss | QA Loss | Gemini-Pro Entity Split | Gemini-Pro Query Split | Gemma Entity Split | Gemma Query Split |
|---|---|---|---|---|---|---|
| YES |  |  | 10.7 | 5.6 | **9.5** | **9.4** |
| YES | YES |  | 11.4| 6.9 | **10.6** | **9.7** |
| YES | YES | YES | 13.4 | 28.2 | **11.6** | **23.4** |
```
```
> **Table R3: Accuracy of generated rationales and QAs in WebLI-5M.**
| Entity Loss | Rationale Loss | QA Loss | Gemini-Pro Entity Split | Gemini-Pro Query Split | Gemma Entity Split | Gemma Query Split |
|---|---|---|---|---|---|---|
| YES |  |  | 14.1 | 5.4 | **11.9** | **5.8** |
| YES | YES |  | 14.6 | 6.7 | **13.3** | **6.3** |
| YES | YES | YES | 16.0 | 28.2 | **14.2** | **24.3** |
```
```
We address the reviewers’ comments in more detail in the corresponding sections.

Meta-review: The paper originally mixed reviews, although they were definitely leaning towards acceptance. The authors provided a detailed rebuttal and the paper has been discussed by the reviewers. The reviewers raised some concerns about missing experiments and some minor commment about presentation. The reviewers also asked for clarification about the collected dataset, and criticized the decision of relying on a private model (Gemini-pro) to build the dataset. The authors included in their responses new results and additional experiments with an open model; the reviewers where satisfied by that and, more in general, by the authors' rebuttal and they all maintained or increased their scores. Therefore there is a general consensus towards accepting the paper. The AC believes this is a solid submission and the authors are encouraged to carefully follow and implement all the reviewers' suggestion in the final revision of the paper.
Decision: Accept (poster)