Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach

Authors: mathilde  caron
alireza  fathi
cordelia  schmid
ahmet  iscen
google  deepmind
multimodal  llm
nematobrycon  palmeri
rationale & qas
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:52:18

Abstract: Web-scale visual entity recognition, the task of associating images with their corresponding entities within vast knowledge bases like Wikipedia, presents significant challenges due to the lack of clean, large-scale training data. In this paper, we propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Instead of relying on the multimodal LLM to directly annotate data, which we found to be suboptimal, we prompt it to reason about potential candidate entity labels by accessing additional contextually relevant information (such as Wikipedia), resulting in more accurate annotations. We further use the multimodal LLM to enrich the dataset by generating question-answer pairs and a grounded fine-grained textual description (referred to as "rationale") that explains the connection between images and their assigned entities. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks (e.g. +6.9% improvement in OVEN entity task), underscoring the importance of high-quality training data in this domain.

Main paper: 
 introduction: 
 Entities are at the core of how we represent and organize knowledge, as seen in prominent encyclopedias like Wikipedia, where each article is dedicated to a specific entity. In the field of computer vision, the task of visual entity recognition aims to identify entities within query images. This capability is not only a fundamental building block for various entity-aware visual understanding tasks, including "info-seeking" Visual Question Answering (VQA) [10, 22, 31] and News Content Understanding [4, 14, 24, 54] , but also has numerous commercial applications. Despite the progress made in recent years, current models often struggle with web-scale visual entity recognition. These models, typically trained on free-form image captions [9, 15, 56] , tend to hallucinate entities or output overly generic ones, leading to suboptimal performance. We hypothesize that the root of this issue lies in the lack of clean, large-scale training data specifically designed for visual entity recognition.Recent efforts have been made to address this problem by transforming existing captioning datasets into entity recognition datasets [7, 25] . For example, Caron et al. [7] propose to match each Wikipedia entity name to most similar captions in a large image-caption database [9, 46] and use the corresponding images as visual examples for the considered entity. Although this method has resulted in state-of-the-art performance, it still has significant limitations: the resulting datasets are often noisy, with poor matching between the image content and the candidate entity. First, a source of mistake is due to the ambiguity of language. For example the entity bishop of llandaff may refer both to a person and to a flower species [18, 35] . Second, there is some noise inherent to the used image-caption dataset that affects the results. For example in Fig. 1 (a), an image of a building is incorrectly linked to the Wikipedia entity Negative equity. This mismatch occurs because of the caption "Worst areas for negative equity", which is likely to have been extracted from an investment-related website. The irrelevant association of the caption with the building image results in the inaccurate connection to the Wikipedia entity. Third, the text embedding matching between Figure 1 : Two failure cases of the visual entity recognition dataset of [7] . Our proposed method overcomes these limitations by prompting a multimodal LLM to correct candidate entities. The LLM has access to relevant context such as the candidate entity Wikipedia page and the input image-caption pair. We also enrich the dataset with rationales and question/answer pairs covering diverse entities.entity name and caption is not always accurate. For instance, in Fig. 1 (b), the caption "nematobrycon espèce nematobrycon palmeri" is incorrectly matched with the entity name nematocampa resistaria, which are two distinct animal species (a fish and a moth). Moreover, another limitation of these datasets is that they typically focus on a single entity per image, which is a restrictive scenario, as most images contain multiple entities.In this paper, we overcome these different limitations by proposing a novel methodology to curate a high-quality, large-scale dataset for web-scale visual entity recognition leveraging the capabilities of modern multimodal Large Language Models (LLMs) available through public APIs [2, 15, 37] .Our approach is unique in that we do not rely on the multimodal LLM for direct annotation, which we found to be suboptimal. Instead, we prompt the LLM to reason about candidate entity labels by accessing additional contextual relevant information, such as the original image captions and external knowledge sources like Wikipedia. This approach significantly improves the annotation quality of the resulting dataset, as evidenced by our thorough ablation studies. Moreover, we employ the multimodal LLM to augment our dataset with rationales that explain the relationship between images and their corresponding entities. We observe in our experiments that training on this additional metadata improves the performance and visual entity understanding of the models. Finally, we address the previous limitation of focusing on a single entity per image by prompting the multimodal LLM to generate several question-answer pairs that cover a diverse range of entities in the image.We conduct extensive experiments to evaluate the effectiveness of our approach. The results demonstrate that models trained on our automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks, notably on the challenging Open-domain Visual Entity recognitioN (OVEN) benchmark [17] (e.g. +6.9% on the OVEN entity split and +3.8% on the OVEN query split). Remarkably, we obtain these results with moderate-size models, which are orders of magnitude smaller than competing approaches, highlighting the importance of high-quality training data in this domain. We further demonstrate the validity and effectiveness of our dataset when utilized as a memory base for approaches such as visual matching, showcasing its versability and potential for various applications. In summary, our contributions are threefold:• We introduce a novel methodology to curate a large-scale dataset for web-scale visual entity recognition, using a multimodal LLM as a verification and annotation tool. • We enrich the dataset with additional metadata, including question-answer pairs and rationales, generated by the multimodal LLM. • We demonstrate the effectiveness of our approach with thorough ablation study and by achieving state-of-the-art performance on web-scale visual entity recognition tasks. 
related work: 
 Visual entity recognition. Entities are key to knowledge representation and organization, as seen in prominent encyclopedias like Wikipedia. Visual entity recognition is the task of identifying entities based on visual queries [13, 43, 44] , and is a critical component of many complex applications. One such application is info-seeking VQA, which involves providing answers to questions about the detailed properties of finegrained entities [10, 22, 31] . Another application is entity-aware captioning [28, 34, 61] , a technique frequently employed in tasks such as news content comprehen-Figure 2 : LLM-Refined Entity-WebLI" (REW) dataset. We propose a method to refine the Entity-WebLI dataset of Caron et al. [7] by prompting a multimodal LLM to verify and correct Wikipedia entities. We also prompt the multimodal LLM to output visually grounded rationales and question/answer pairs about diverse attributes of the image. Complete prompts are in Appendix A.3.1.sion [1, 4, 14, 24, 54] . Recent research has expanded the scope of visual entity recognition to include web-scale and open-domain entities [17, 21, 41] . Of particular interest, Hu et al. [17] introduce the Open-domain Visual Entity recognitioN (OVEN) benchmark, which includes over 6 million entities from English Wikipedia, covering a broad range of concepts such as animals, buildings, organizations, landmarks, etc. Caron et al. [7] recently achieved state-of-the-art results on the OVEN benchmark by re-purposing auto-regressive generative models for this task. A key component of their approach is pretraining on a new entity-based dataset instead of captions. In this paper, we build on the work of [7] by improving their pretraining dataset using external multimodal LLMs.Using LLMs as annotation tools. Since their remarkable success and widespread accessibility, LLMs [36, 40, 53] have been used in many different ways to obtain better supervision for training various tasks [23, 32, 45, 59, 60, 62] . For example, LLMs have been used to generate question-answer pairs from Wikipedia pages [8, 31] or from transcribed video narrations [59] , while other works use LLMs to rephrase questions into sentences [60] . Of particular interest, Hsieh et al. [16] show that prompting LLMs to output rationales as additional supervision for training small models in a multi-task framework is an effective strategy. A recent related trend consists in prompting LLMs to generate high-quality instruction-following training samples [11, 50] , an approach which has succesfully been extended to multimodal tasks [26] . Unlike our work, these approaches prompt LLMs with text input only while we feed both image and text to multimodal LLMs. Another key difference is that we use the multimodal LLM as a verification and correction tool based on candidate annotations rather than using its raw output as supervision, which we find to be suboptimal for the task of web-scale finegrained entity recognition. 
method: 
 In this section, we describe how to leverage the capabilities of modern multimodal LLMs [15] in conjunction with external knowledge repositories such as Wikipedia to create a clean, large-scale training dataset specifically designed for web-scale visual entity recognition. We refer to the resulting dataset as "LLM-Refined Entity-WebLI" (REW) and an overview of our method is in Fig. 2 .Wikipedia-scale visual entity recognition. Following recent research in web-scale visual entity recognition [7, 17] , our goal is to train models capable of accurately matching any given image-text query (x v , x t ) to an entity e among a vast finegrained set E of possible entities. In this work, unless specified otherwise, the set of entities E consists of the 6 million entities from English Wikipedia.Each entity e ∈ E comes with an entity name t e corresponding to the entity Wikipedia page title.Entity-based pretraining dataset. Previous works have observed that auto-regressive image captioning models like GIT [56] or PaLI [9] have suboptimal results when transferred to visual entity recognition (see Tab. 1), due to the differences between captioning and entity recognition tasks [7, 17] .To address this, Caron et al. [7] propose training such models on entity-based data, not captions, and automatically create a new dataset called "Entity-WebLI" for this purpose. In short, for each Wikipedia entity, the authors find the most relevant image-caption pairs in WebLI through nearest neighbor search in the CLIP text embedding space [41] between encoded entity names and captions. They then replace the captions of the corresponding retrieved images with the considered entity name.In this work, we refer to the entities obtained in this manner as "candidate entities". Further details on the Entity-WebLI dataset are in [7] . We describe in the following how to improve over such a dataset by leveraging the capabilities of modern multimodal LLMs.Entity verification and correction. Our goal is to overcome the limitations of existing entity-based pretraining dataset discussed in the introduction. To improve the correspondence quality between images and entities, we propose to use existing multimodal LLMs to verify the assignment of an entity to an image. In particular, given an image x v and corresponding entity e from the Entity-WebLI dataset, we prompt a multimodal LLM with the task of verifying if x v is an image representing entity e. Interestingly, we make two important findings in our experiments (see Tab. 4). First, directly predicting an entity e is a more challenging problem than verifying the validity of the proposed entity.Intuitively, this is because multimodal LLMs either output too generic entities or hallucinate when they do not know the correct entity. This is expected since LLMs haven't specifically been trained to identify entities at very high levels of granularity. This results in sub-optimal performance for the finegrained entity recognition tasks, where the goal is to recognize non-generic finegrained entities.Second, we find that the verification by a multimodal LLM is more precise when it has access to external metadata such as the page content of the candidate Wikipedia entity e and the original caption.Intuitively, the Wikipedia content allows the multimodal LLM to know which visual attributes to look for in the image x v , while the original caption gives hints about the corrected entity when the multimodal LLM detects that the candidate e is not correct. Note that the corrected entity might not always coincide with an actual Wikipedia entity since we do not provide the list of the 6M Wikipedia entities to the LLM. We still include these corrected entities in our training dataset and simply constrain decoding to the 6M entities at inference time (see details in Sec. 4.1). Similar techniques have also been used into entity generation in multimodal contexts by works like GMEL [48] and AutoVER [58] . Our prompt for entity verification and correction is available in Appendix A.3.1.Generating rationales. At the same time as asking the multimodal LLMs to verify the entities assigned to training images, we also prompt it to provide a visually grounded rationale for its proposed entity. This allows to explain the connection between the image and the assigned entity as can be seen in the examples in Fig. 3 and Fig. 4 . We observe in our experiments that it improves the performance of the models for entity recognition (see Tab. 4). When training our model (see details in Sec. 3.3), we follow a multi-task learning strategy where we prepend task prefixes to the input examples so that the model can output differently based on whether it is asked to predict an entity name or a rationale.Generating question-answer pairs (QAs). The Wikipedia entity recognition OVEN benchmark [17] consists of two splits: an entity split and a query split. In the query split, images typically contains multiple entities, and the input question x t determines which entity should be recognized by the visual entity recognition models. We observe in our early experiments and in Tab. 1 that training on the Entity-WebLI dataset [7] consistently results in poor performance in the query split, while achieving state-of-the-art performance for the OVEN entity split. We hypothesize that this is due to how Entity-WebLI dataset is constructed (see details in Sec. 3.1). The k-NN search in the text embedding space favors short captions containing entity names. As a result, a single, unambiguous, entity is assigned to each image, a scenario quite different to the examples in the query split.To overcome this problem, we prompt the multimodal LLM to generate question-answer pairs for several, diverse entities in the input image (see prompt in Appendix A.3.1). The multimodal LLM has access to the input image, the verified/corrected entity as well as the rationale it previously generated. We empirically evaluate in the ablation study in Tab. 5 the impact of these on the question/answer pairs and resulting trained models.Auto-regressive generative models. Previous works have shown the effectiveness of generative approaches for entity recognition both in the pure NLP domain [12, 30, 39, 42, 49, 51] and, more recently, in visual entity recognition benchmarks [7, 17] . Motivated by their success, we perform entity recognition by generating Wikipedia entity names (i.e. page titles t e ) in an auto-regressive fashion. This is akin to a multimodal version of GENRE [12] or to the GER-CAPTION variant of [7] . Formally, we transform an input image-text query pair x = (x v , x t ) into a set of N d-dimensional embeddings X ∈ R N ×d formed by concatenating the visual encoder output of x v and the text tokenizer output of x t . We use an auto-regressive text decoder g(•) to generate the target text y.As detailed in the following paragraph, the target text can either be an entity name or a rationale. Specifically, the decoder predicts each text token y k from the target text (total length is K) given both the set of preceding token embeddings Y <k and the input image-text embeddings X. We train with a language modeling objective:EQUATIONwhere [; ] corresponds to the concatenation operation in the first dimension and ℓ is the softmax cross-entropy loss with label-smoothing [33] . We average this loss over minibatches of examples and update the weights of the visual encoder and text decoder with back-propagation.Multi-task learning. We train our models jointly with three distinct tasks: (i) predicting the verified/corrected visual entity, (ii) generating the rationale and (iii) answering the questions generated by the multimodal LLM. These three tasks are text generation tasks and follow the same framework introduced in the previous paragraph. They differ in the nature of the input text x t and target text y. For entity recognition and question answering, the target text y corresponds to a possible entity name and the input text x t corresponds to a question. For rationale generation, the target text y corresponds to the rationale generated by multimodal LLM and the input text x t consists simply of a prefix specifying to the model that this task is distinct from entity generation. Our final multi-task loss objective is:L Final = L Entity + L Rationale + L QAwhere L Entity , L Rationale and L QA correspond to entity, rationale and answer generation respectively. 
experimental setting: 
 We detail here the most salient experimental details. Full experimental setting is in Appendix A.3.Our training dataset builds upon the Entity-WebLI dataset [7] (see Sec. 3.1) which itself is based on WebLI [9] , a dataset already deduplicated against the train, val, and test splits of 68 common vision/vision-language datasets [9] . The Entity-WebLI dataset is further aggressively filtered against OVEN by removing any image which has a CLIP-score higher than 0.95 with an OVEN image [7] . This ensures that there is no downstream data leakage in our REW dataset. We build two versions of REW dataset: REW-5M (4.5M images) and REW-47M (47M images). Each training image is attached to a verified/corrected entity, a rationale and 3 question/answer pairs. In the main results section (Sec. 4.2) we train on REW-47M dataset for 600k steps while for analysis and ablation study (Sec. 4.3) we train on REW-5M for a shorter schedule (200k steps). We also validate our dataset refining methodology using the LAION dataset [46] as the image-caption base dataset.Downstream task: OVEN. We consider the entity and query splits of the OVEN benchmark [17] .For both splits, the goal is to output a Wikipedia entity given an input image and a question. OVEN validation and test splits are divided into seen and unseen entities. The seen examples correspond to entities present in the OVEN training set while unseen entities are a subset of entities not present in the training set. We report the harmonic mean (HM) of top-1 accuracy scores between "seen" and "unseen" entities [17] . We specify in our results if we report results before or after additional finetuning on the training set of OVEN ("+ seen finetune").Downstream task: finegrained datasets. We also report results on Oxford Flowers [35] , Sun397 [57] , Food101 [5] , FGVC-Aircraft [29] and Sports100 [17] finegrained datasets in the zero-shot mode. We choose these datasets since there is a direct mapping between their class vocabularies and the Wikipedia entities that our model is trained to output. The resulting class label to Wikipedia entity mappings are given in Appendix A.5.Inference. We perform decoding with beam search. When evaluating on OVEN, we discard all decoded texts that are not one of the 6M Wikipedia entities. This is akin to constraining the beam search only at the last decoding step. Constraining from the first decoding step is too costly in our implementation with such a large million-scale label space. For finegrained datasets, we perform constrained beam search decoding at all decoding steps since the label spaces are smaller.Model training implementation details. We use GiT-Large [56] : it consists of a visual encoder (CLIP-L/14 [41] ) and a 6-layer text decoder with internal dimension d = 768. Following [7] , the visual encoder is first pre-trained jointly on WebLI-100M [9] and Conceptual Captions-12M [47] while the decoder is randomly initialized. We use batch size of 4096, learning rate of 1e -5 for the visual encoder and 1e -4 for the decoder, label smoothing of 0.2 and no weight decay. We use standard inception crop data augmentation. For the multimodal LLM, we use Gemini Pro [15] . The public API is available at ai.google.dev.Comparison with the state of the art on OVEN. In Tab. 1, we observe that PaLI and GiT-Large models pretrained on captioning datasets have suboptimal performance, especially in the entity split. Intuitively this is because the entity split tackles finegrained entity recognition while query split is more reflective of a generic VQA task, which leverages the language understanding abilities learned from captioning. Hence, the query split task is more aligned with the captioning pretraining than the entity split task. In Tab. 1, we see that the model trained on our REW dataset instead of captioning data results in state-of-the-art performance in the OVEN benchmark, both before and after further OVEN finetuning on the "seen" classes. Notably, our model outperforms the captioning PaLI-17B model by large margins: +13.6 top1 HM test accuracy on the entity split and +3.8 on the query split, while using 42× less parameters.Finally, we report the zero-shot performance of the multimodal LLM on OVEN: it reaches 13.3 HM top-1 in the entity split and 29.5 HM top-1 in the query split. These numbers suggest that we are not merely distilling from the considered multimodal LLM as we outperform its performance on this benchmark by +10.3 on entity and +1.4 on query sets while using orders of magnitude less parameters. The analysis in Tab. 4 will further demonstrate the importance of using the multimodal Table 3 : Visual matching. We report top-1 accuracies of visual matching with CLIP or DINOv2 ViT-L/14 visual backbones. We compare two types of annotations for the visual matching memory database: either the candidate entities or our multimodal LLM corrected entities. We report the absolute improvements of using the latter compared to the former between parentheses as well as the relative improvement averaged across the six datasets in the last column. LLM as a verification and correction tool rather than a teacher since directly using its predictions as targets result in poor performance.Zero-shot transfer to finegrained datasets. In Tab. 2, we observe that the model trained on our proposed training dataset REW-47M transfers effectively to several finegrained datasets. The model trained on REW demonstrates superior transferability compared to the same model trained on captions or Entity-WebLI. This result shows the higher quality of the REW dataset.Using REW dataset as a memory base. We explore the potential of our REW dataset when utilized as a memory base for visual matching in Tab. 3, and for retrieval-enhanced contrastive (RECO) training [18] in Appendix A.1.1. Each image in the memory is either associated with the candidate entity from text k-NN matching (as in Entity-WebLI [7] ) or with the multimodal LLM corrected entity (our method for REW). In Tab. 3, we report the results of visual matching with two popular visual backbones [6, 41] and across six different finegrained visual entity recognition datasets for which we have the mapping from class label to Wikipedia entity. We see in Tab. 3 that our corrected entities lead to better visual matching performance across the board which suggest that they are better annotations, describing more accurately the content of the images. In fact, using our corrected entities boosts the performance in average by +10.5% relative improvement when considering CLIP [41] and by +13.9% relative improvement with DINOv2 [38] .Unless specified otherwise, models in this section are trained on the REW-5M dataset.Importance of entity verification and correction. In Tab. 4, we train models with different source of annotations for the target entities. First, we observe that directly using the multimodal LLM raw output as target (which is akin to a distillation scenario) results in suboptimal performance (row 1). By inspecting qualitative results, our hypothesis is that this can be attributed to the LLM's tendency to produce hallucinations, overly generic answers, or outputs in an incorrect format (long descriptive captions instead of entities). Second, we validate in Tab. 4 the importance of the multimodal LLM correction step compared to using the candidate entities from text k-NN matching as in Entity-WebLI [7] : this strategy improves the performance of the resulting model by more than 6 points (row 3 versus row 2). Finally, providing additional contextual information to the multimodal LLM results in a substantial boost of 1.7 points (row 4 versus row 3), a benefit that we further illustrate through our qualitative analysis in Fig. 3 . We note that the multimodal LLM refines the candidate entities 76% of the time, i.e. it validates the candidate entity 24% of the time (see Fig. 3c ).Qualitative analysis on the importance of the multimodal LLM correction. In Fig. 3 , we identify two typical failure cases of the multimodal LLM correction step when it lacks access to Wikipedia and original caption metadata. The first case (see examples in Fig. 3a ) involves the LLM making incorrect corrections by providing generic or hallucinated outputs. For example, it identifies an image of the Bronte baths as an outdoor swimming pool or it makes mistakes by recognizing incorrect plant or animal species. In contrast, the multimodal LLM with access to external metadata can get a hint about the correct entities by reading from the original caption. As a matter of fact, note that simply using the original captions as targets leads to suboptimal performance as shown by Caron et al. [7] . Intuitively, original captions are usually not in the form of an entity and can be noisy and not reflective of the visual entity of the image, as illustrated in the examples in Fig. 3b .Table 4 : Importance of entity verification and correction. We report HM top-1 accuracy on OVEN validation entity split. To isolate the effect of the entity target source, we train only with entity targets (i.e. only with loss L Entity ). We do not perform seen finetuning and evaluate the models directly after pretraining. All models are trained with the same pretraining images and use the same architecture. The image shows a plant with cordate leaves that are dark reddish-purple in color with prominent veins.Rationale: The fish has a purple body, a yellow tail, and a black stripe running down its side.Rationale: This is a freshwater hatchetfish, a type of fish with a long, thin body and a silvery sheen. The tail is forked.Rationale: The image shows an outdoor ocean pool with waves crashing over the edge.Rationale: An outdoor swimming pool filled with water and surrounded by concrete. The pool is on the edge of a cliff.Rationale: There are three motorcycles on a road in the desert.Rationale: The image shows a group of people riding motorcycles through a desert landscape.Rationale: The image is of a red t-shirt with white text that reads "What Boyfriend?".Rationale: A large black and white sailboat with a red stripe on the sail, heeling to starboard with a large crew on deck.Rationale: A large black and white sailboat moving quickly through the water with people on deck. Rationale: The image shows a small bird with a long, thin beak and a green body. The bird is hovering in front of a red flower.Rationale: It is a large, oval-shaped shell with a mottled green and brown color with a series of small holes along the edge.The image shows a brown and white oval-shaped shell with a rough texture and a hole near the top.Rationale: Multiple spools of ribbon with prominent transverse ribs, which is a defining characteristic of grosgrain. The second failure case of the multimodal LLM lacking access to metadata occurs when the model corrects information that it should not, likely due to a lack of knowledge about the entities involved (see examples in Fig. 3c ). This issue is reflected in the LLM rationale, indicating a need for further external information about the candidate entities. By contrast, the model with access to Wikipedia can read about the visual attributes which are characteristic of the candidate entity and look for these in the image. This process is reflected in the LLM rationale: for example, the model with Wikipedia access to the grosgrain entity page validates this entity with the rationale: "Multiple spools of ribbon with prominent transverse ribs, which is a defining characteristic of grosgrain" while the model without Wikipedia seems to lack specific knowledge about grosgrain.Generating rationales and QA pairs with multimodal LLM. In Tab. 5 (left), we probe the importance of providing external metadata to the LLM when generating rationales and question answer pairs. First, we observe in Tab. 5 (left) that the model without access to any metadata (row 1) has suboptimal performance. Incorporating external input, such as Wikipedia content and original captions, during the rationale generation process enhances the quality and pertinence of the rationale. Consequently, this leads to an improvement in performance (row 2). Importantly, this performance boost is even higher if the multimodal LLM can leverage the improved rationale when generating the QAs (row 3).Table 5 : Ablation study: (left): Generating rationales and QA pairs with multimodal LLM. (right): Multi-task training with generated rationales and QAs. We validate the robustness to the base image-caption dataset used by performing the latter ablation with both WebLI [9] and LAION [46] . We report HM top-1 accuracy on OVEN validation splits directly after REW training. In row 4, we notice that while it is important for the LLM to have access to the entity name during QA generation to improve entity split performance, it negatively impacts query split performance, as seen in the comparison between row 2 and row 4. This observation can be intuitively explained by the fact that when the LLM has access to the entity name, it is more likely to generate question/answer pairs focused on the main entity, rather than considering other attributes present in the image. However, these diverse attributes are often mentioned in the rationale, as demonstrated in the qualitative examples of Fig. 4 in Appendix. Therefore, having access to the rationale increases the variety of the generated QA pairs. Taking into account these observations, our default model (row 5) is configured to generate QAs by providing the model with access to both the entity and rationale. This approach aims to achieve a balance between strong performance in entity and query splits.Multi-task training with generated rationales and QAs. In Tab. 5 (right), we confirm the importance of multi-task training with the various output types obtained from the multimodal LLM. We conduct this experiment using two different base image-caption datasets: WebLI [9] (our default) and the publicly available LAION dataset [46] . Our results demonstrate that both rationale and QA objectives contribute to the improved performance of our model. Intuitively, rationales help clarify the connection between entities and visual attributes, while QA training encourages the model to focus on multiple entities within the image. Moreover, both objectives enhance the model's language understanding, which we found to be important especially for the query split.Robustness to the base image-caption dataset. Lastly, we observe that the results presented in Tab. 5 (right) are consistent across the different image-caption datasets used in our experiments. This suggests that our findings are robust and not specific to the WebLI dataset.Finally, we run an additional set of experiments with the open source PaliGemma [3] and Gemma 27B models [52] . We validate that the results are inline with the results when using the Gemini-Pro model [15] . Since Gemma lacks visual input processing, we replace direct image input with automatically generated captions. Specifically, we employ the open-source PaliGemma model to generate descriptive captions for each image using the prompt: "Describe the visual attributes of this image.". We then integrate these captions into the existing prompts outlined in Sec. A.3.1 by prepending the text: "Here are the visual attributes of the image:".We evaluate this approach on the 5M subset of WebLI and LAION. We compare the Gemma and Gemini-Pro variants of our method in Tab. 6 with the SOTA methods trained on the 5M subset of Entity-WebLI. We see in Tab. 6 that in all cases our method gives substantial improvements over the prior work GER-ALD [7] and GiT-Large trained on Entity-WebLI. In Tab. 8 of the Appendix, similarly to Tab. 5 (right) we evaluate the impact of the different losses when using the version of our method with open source models. We verify that conclusions are similar with private and open source models. As seen in Tab. 8, the main difference of performance between Gemini-Pro and Gemma variants comes from the QA loss. While Gemini-Pro has access to input images when generating QA pairs, Gemma generates QA pairs based on the PaliGemma caption (and rationale). This limits the variety of the generated QA pairs, resulting in a lower final accuracy.  
discussion: 
 Limitations. Our work relies on the use of multimodal LLMs with external knowledge bases, such as Wikipedia, for dataset creation and annotation. This dependence on using external data to prompt LLMs may limit the applicability of the approach in scenarios where such external knowledge is scarce or unavailable. Also, the proposed approach involves prompting the LLM to reason about candidate entity labels and generate additional metadata. This process is time-consuming and computationally expensive when considering multimodal LLMs with billions of parameters [15] , which may limit the scalability of the approach to even larger datasets. A direction for future work could be to overcome some of the limitations of OVEN we find in our study (Appendix A.2) and create additional benchmarks for web-scale visual entity recognition.We propose a novel methodology to curate a high-quality large-scale dataset, REW, for web-scale visual entity recognition using multimodal LLMs. Our approach significantly improves the quality of the dataset and bypasses the need for manual annotation. We also enrich the dataset with additional metadata, including question-answer pairs and rationales, generated by the LLM, which further improves the performance of the models. We achieve state-of-the-art performance on web-scale visual entity recognition tasks, highlighting the critical role of high-quality training data in this challenging domain. While our methodology for multimodal LLM-based data curation shows promise, we recognize significant opportunities for further enhancement. Integrating additional tools and external knowledge sources holds potential to improve its effectiveness. Furthermore, we anticipate that our approach can be broadly applicable to other visual tasks requiring extensive training data. Broader impact discussion is in Appendix A.4.The image shows a living room with a sofa, a chair, a coffee table, a rug, a lamp, and a window with vision blinds. We report the absolute improvements over the CLIP baseline between parentheses as well as the relative improvement averaged across the datasets in the last column. For reference, we also include the results from Iscen et al. [18] but note that they use a memory base 20× bigger (WebLI-1B).Cars [20] CUB [55] ImNet [44] Flowers [35] Places [63] A Appendix and supplemental material A.1 Additional resultsA.1.1 A different application of our dataset: memory base for RECO.We explore the potential of our dataset when utilized as a memory base for retrieval-enhanced contrastive training (RECO) [18] in Tab. 7. Each image in the memory is either associated with the original WebLI caption, with the candidate entity from text k-NN matching (as in Entity-WebLI [7] ) or with the multimodal LLM corrected entity (our method REW-47M). In Tab. 7, we show that using the multimodal LLM to correct entity entities results in a relative improvement compared to CLIP of +5.4% on average across the six datasets considered. Using the original dataset captions instead results in a smaller relative improvement of +4.4%. Lastly, we see in Tab. 7 that we are able to achieve performance comparable to or even better than RECO's results on some datasets, despite using a memory base that is 20 times smaller. This finding suggests that a smaller amount of high-quality annotated data can be just as effective, if not more so, than a larger amount of data with lower annotation quality.We see in Tab. 8 that the main difference of performance between Gemini-Pro and Gemma variants comes from the QA loss. While Gemini-Pro has access to input images when generating QA pairs, Gemma generates QA pairs based on the PaliGemma caption (and rationale). This limits the variety of the generated QA pairs, resulting in a lower final accuracy. The goal of this paper is to develop models capable of matching any given image-text query to an entity with a high precision. While the results in this work show that we improve upon the state of the art on the challenging web-scale visual entity recognition OVEN benchmark, we note that our best model still makes mistakes more than 70% of the time (see Tab. 1), which is far from optimal. While this shows that there is still a lot of room for improvement for further research in that direction, we provide a short analysis in this section of some limitations of the OVEN benchmark itself.In Fig. 5 , we show the top-5 predictions of our model for various examples from the OVEN validation set. We observe that, in some cases, our model's predictions are more accurate than the expected ground truth entities provided by the OVEN benchmark. This is because some of the positive entities in the OVEN validation or test set may have a better match within the negative entities. For instance, our model correctly predicts Echinocactus grusonii before the expected Barrel cactus entity, as the former is a more specific and accurate description of the cactus in the image. Similarly, our model predicts spotting scope instead of the expected telescopic sight, as the former is a more precise description of the visual entity in question.To address these limitations, we suggest considering the top-k predictions when comparing the performance of different models. For example, our best-performing model's HM score increases dramatically from 29.6 to 57.0 when we considered the top-10 predictions instead of just the top-1 prediction.Finally, the right side of Fig. 5 shows an example where the input question, Which type of animal is depicted in the image?, is not relevant to the input image, as there is no animal present. In this case, our model disregards the input question and outputs entities that are present in the image but are not animals such as hay or tractor for example. This raises the question of whether the model should prioritize the input question and provide a more relevant answer, even if it means potentially sacrificing accuracy. For instance, the model could output a response such as "There is no animal in the image," which would be more relevant to the user's query, but would not be strictly accurate in terms of entity recognition. We input the image to the multimodal LLM as well as the following prompts.Entity correction and rationale generation prompt. We include below our full prompt template for the entity correction and rationale generation process. We insert the original caption as {original caption}, the candidate entity as {candidate entity} and the Wikipedia page content summary of the candidate entity as {Wikipedia summary}.You are working on an entity recognition task. Is this an image of candidate entity? Your answer must be either 'YES' or 'NO'.Here is the definition of candidate entity: Wikipedia summary.If your answer is 'YES', you must use the definition of candidate entity to answer whether this is an image of a candidate entity.If your answer is 'NO', you must use the caption of the image original caption to describe the main object in the image with the most specific English Wikipedia article title, where the response follows the format '@response@'. " You must then explain your answer by describing the visual attributes of the image. If you answer is 'YES', your explanation MUST be based on the definition of candidate entity. If you answer is 'NO', your explanation MUST ONLY be based on the visual cues of the image, and it should NOT contain candidate entity. Your explanation must be concise. Your explanation MUST NOT exceed two sentences.Question/answer pair generation prompt. We include below our full prompt template for the question/answer pairs generation process. We insert the previously verified/corrected entity as {entity} and the previously generated rationale as {rationale}.You are working on a visual question answering task. This is an image of entity.Your rationale is the following: rationale. Your task is to generate 3 question/answer pairs describing the visual attributes of this image. The questions MUST be diverse and cover several entities of the image, including the main object in the image or image itself. The answers MUST be specific English Wikipedia article titles. The answers MUST be based on the visual content of the image and the provided rationale.The format for the question/answer pairs is Q:<question> A:<answer>. The questions MUST NOT contain What is the main object in the image?.We report the performance variance on our model trained on the small version of our dataset in order to assess the statistical significance of our results. We run this experiment five times with different random seeds. We perform the evaluation for each model separately and report in Tab. 9 the accuracy for each run in the different OVEN splits. We also report the mean over these five runs and the standard deviation.Our models are trained on 256 TPUv3. The short schedule 200k steps training lasts for 15 hours while the longer 600k steps training lasts for 44 hours. The full scope of our research project involved more runs than what is reported in the paper. This is because we conducted preliminary explorations and experimented with various design choices that ultimately did not yield successful results. While these failed experiments are not presented in the tables of the paper, we believe that they are an important part of any research project and contributed to the development of our final approach.Training and inference on REW dataset. We use GiT-Large [56] : it consists of a visual encoder (CLIP-L/14 [41] ) and a 6-layer text decoder with internal dimension d = 768. Following [7] , the visual encoder is first pre-trained jointly on WebLI-100M [9] and Conceptual Captions-12M [47] while the decoder is randomly initialized. We use batch size of 4096, learning rate of 1e -5 for the visual encoder and 1e -4 for the decoder, label smoothing of 0.2 and no weight decay. We use AdamW optimizer [27] and a cosine learning rate schedule with final learning rate of 0. We use standard inception crop data augmentation for the images. We set the maximum decoding length to 32 tokens and the maximum number of context tokens to 32 tokens as well. We find that 32 tokens is enough to comprehensively tokenize all the OVEN input questions as well as all the Wikipedia entity names. The decoding beam size is set to 30.Finetuning on OVEN train set. We finetune models on OVEN training set for 10,000 steps with a batch size of 256. Note that we equally balance the query and entity split contribution during finetuning. We choose the learning rate out of three values (1e -7 , 3e -7 , 1e -6 ) based on the HM performance on the OVEN validation set. Label smoothing is set at 0.1. Note that the finetuning schedule is relatively short (10,000 steps) because we observe that long finetuning (or equivalently, using a large learning rate) causes the model to forget about the unseen categories.Potential positive societal impacts. Our proposed methodology for curating a high-quality, largescale dataset for web-scale visual entity recognition using multimodal LLMs has the potential to significantly impact the field of computer vision. By improving the quality of training data, we can enhance the performance of models for various entity-aware visual understanding tasks, including info-seeking VQA or News content understanding, leading to more accurate and reliable results. Furthermore, our approach has the potential to democratize the process of dataset creation, reducing the time and resources required for manual annotation. This can enable researchers and practitioners with limited resources to create high-quality datasets.Potential negative societal impacts. However, it is essential to consider the potential risks and ethical implications of our work. The use of LLMs for dataset creation and annotation raises concerns about bias and fairness, as the models may reflect and perpetuate the biases present in the data they were trained on. Therefore, it is important to carefully evaluate and mitigate any potential biases in the datasets created using our approach. Additionally, the use of large-scale image-captions such as WebLI [9] or LAION [46] databases for entity recognition raises concerns about privacy and data protection, as the images and captions used for training may contain sensitive information. Therefore, it is essential to ensure that the datasets created using our approach are compliant with relevant privacy and data protection regulations and that appropriate measures are taken to protect the privacy and security of the data.In summary, our work has the potential to significantly impact the field of computer vision and enable the development of more accurate and reliable models for various entity-aware visual understanding tasks. However, it is essential to carefully consider and address the potential risks and ethical implications of our work to ensure that it is used for the benefit of all.We selected five finegrained datasets for evaluation because their class categories align with a subset of the Wikipedia entities that our model is trained to recognize. We obtained a preliminary mapping of class labels to Wikipedia entities from the authors of OVEN and then improved it through a careful manual review.Oxford Flowers [35] . Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We discuss the experiment statistical significance in Section A.3.2.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Discussion about the potential negative and positive societal impact of this work is in Appendix A.4.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We are not releasing data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Previous datasets, models and related methodologies that this work builds upon are correctly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Rationale:The image is of a red t-shirt with white text that says "What Boyfriend?". 
 
 Appendix:  


Decision: The paper originally mixed reviews, although they were definitely leaning towards acceptance. The authors provided a detailed rebuttal and the paper has been discussed by the reviewers. The reviewers raised some concerns about missing experiments and some minor commment about presentation. The reviewers also asked for clarification about the collected dataset, and criticized the decision of relying on a private model (Gemini-pro) to build the dataset. The authors included in their responses new results and additional experiments with an open model; the reviewers where satisfied by that and, more in general, by the authors' rebuttal and they all maintained or increased their scores. Therefore there is a general consensus towards accepting the paper. The AC believes this is a solid submission and the authors are encouraged to carefully follow and implement all the reviewers' suggestion in the final revision of the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a method to curate datasets for visual entity recognition tasks. They rely on a multimodal LLM (Gemini Pro), which employs metadata information about the image (the caption) and the content of the Wikipedia page to improve the quality of the Entity Web-Li dataset. They further enrich the resource with the rationale regarding the relation between the image and the entity and several question-answer pairs about diverse range of entities appearing in the image.
strength_weakness: strength_weakness: The main contribution of the paper is the creation of an enriched version of the Entity Web-Li dataset, named REW, which however is not released to the public together with the paper (this is mentioned in the Checklist). I find this a major weakness of this project, given the complexity of the conducted work (as clearly described in the paper and the appendix) and would strongly encourage the authors to reconsider this decision.
questions: questions: Could you clarify how the avg. relative improvement in table 3 is computed?
limitations: limitations: The authors do not clarify why they are not releasing the updated version of the dataset. if this is done for safety reasons, it should be clarified as part of the limitations.
paper_summary: paper_summary: The paper deals with web-scale visual entity recognition, which consists of math a question(text)-image query to one of the 6M entities (wikipedia page title) of a base of reference. In the vein of previous works, the task is addressed with a generative text-to-image model, the challenge lying in building the training dataset of this model and the method to learn it (task/losses considered). The approach is based on the recently published dataset Entity-Web-LI which associates an entity (name) to the images of an image-caption dataset. The novelty lies in an additional module of curation that relies on a multimodal LLM (Gemini), which asserts the relevance of the image retrieved in the image-caption external dataset. The multimodal LLM is also asked to provide explanations on its curation. With the novel training dataset, a Git-Large model is trained and evaluated in the context of visual entity recognition (OVEN benchmark) and zero-shot fine-grained classification (5 benchmarks) with better performances than the recent paper the work is built. The work is completed by several experiments of analysis and ablation.
strength_weakness: strength_weakness: * It is regrettable to rely on a private model (Gemini-pro) to build the main contribution of the paper, which is the training dataset. There's no guarantee that this model will be stable over time, such that the proposed method (to build the dataset) may not be reproducible in a couple of months. This issue would nevertheless be partially mitigated if the dataset itself is released, although it still limits the interest of the method itself (e.g maybe another prompt would be required to get comparable results, or another multi-task learning to train the T2I model...). Surprisingly, this aspect is not addressed in the "Limitations" of Section 5.

* The authors argue (lines 239-243) that "the zero-shot performance of the multimodal LLM on OVEN" is much lower than those obtained when it is used alone thus it "suggest[s] that we are not merely distilling from the considered multimodal LLM". However, it is hard to believe that the gain in performances over [6] is not essentially due to the use of an additive (large, complex and trained on many data) LLM. Beyond the fact the namely LLM is opaque (see above) the contribution seems to mainly consist of a RAG-like approach to filter and correct the entities associated with the images by [6], relying on a black box which makes the process difficult to actually understand. So yes, "it works", but the scientific contribution nevertheless seems limited in that sense.


**minor**
* line 543 "if you answer" --> "if your answer"
* in Appendix A.5, the authors report that the mapping used for fine-grained classification was that proposed by OVEN "then improved it through a careful manual review". For the sake of reproducibility, it should be clear that this mapping will be released to the community.
questions: questions: * Can we have an idea of the additional computational complexity (and resource usage) due to Gemini?

* In the same vein, is it possible to estimate the performance boost/drop with another external "multimodal LLM"?

* The authors report that the code is not released (Checklist "Experimental Result Reproducibility") but do they at least plan to release the new "mapping" manually curated and used for fine-grained classification?
limitations: limitations: Lines 324-332 are specifically dedicated to the limitations of the approach. It highlights the dependence of the approach on the availability of relevant external knowledge (that is Wikipedia in the context of the paper) as well as the fact that the proposed approach is expensive in terms of computations. One can regret that the message is vague and generic, without specific insight into the actual proposed method, not to mention possible quantitative hints on the actual computational time (of complexity in terms of memory). It refers to Appendix A.2 but that last focuses again on the performance of the model. However, Appendix A.3.3 reports the usage of 256 TPUv3 during 15+44 hours for the models used in the paper. This only concerns the training of the GitLarge models, but an estimation of the additive resources used to build the dataset (that is the main novelty of the paper) with Gemini would also be relevant.
paper_summary: paper_summary: The work proposes a data-centric approach, that leverages MLLMs to first verify the correspondence of the existing pre-training dataset (Entity-WebLI) and correct errors, ask them to produce rationales, and generate novel, query split-oriented QA pairs to train a language model. The refined dataset REW is curated to facilitate the training specifically for the task of visual entity recognition (VER).
strength_weakness: strength_weakness: 1. In L241, authors state that the trained model is better than MLLM itself which is used to produce filtered datasets. However, I did not see the results for Gemini Pro (in L226 they mention that) in Table 1.

2. Lack of references: In L142 and L170, authors state that constrained decoding is used to guarantee the successful grounding of entity rationale generation. Works like GMEL [1] and AutoVER [2] should be the first to introduce such techniques into entity generation in multimodal contexts. 

3. Minor issue: This work mainly proposes a data-centric solution to GER [3], and I assume the base model architecture of GER and this work stays the same with a 0.4B decoder-only model and a CLIP visual encoder. In Table 1, why is GiT-Large with Entity-WebLI training getting a better performance compared to GER-ALD with the same dataset? Is it because of the newly added two supervisions L_Rationale and L_QA?


[1] Shi S, Xu Z, Hu B, et al. Generative multimodal entity linking[J]. arXiv preprint arXiv:2306.12725, 2023.

[2] Xiao Z, Gong M, Cascante-Bonilla P, et al. Grounding Language Models for Visual Entity Recognition[J]. arXiv preprint arXiv:2402.18695, 2024.

[3] Caron, Mathilde, et al. "A Generative Approach for Wikipedia-Scale Visual Entity Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: This work focuses on an important and realistic visual task called visual entity recognition (image, query --> entity name).
This work proposes a LLM driven technique to generate and refine large scale training data by modifying incorrectly labeled entity names from Entity-WebLI. Several errors are identified in Entity-WebLI and this work proposes several techniques to address them.
The experiment on OVEN and OOD datasets confirmed the usefulness of the new dataset REW and ablation studies show important design decisions contributing to the performance.
strength_weakness: strength_weakness: - I don't see major weakness of this work except using LLM for data generation might be technically limited. However, I believe the techniques used to correct entity labels and findings are useful and novel for large scale entity centric caption creation.
- releasing data and code: Although the data will not be released, I think the reproducibility of experiments are supported by the experiments with 5 random seeds and also using open-sourced data such as LAION. The community will benefit from reproducing such large-scale entity recognition image data by using LAION and multimodal LLM such as LLaVA.
questions: questions: - I wonder how the REW benchmark generalize to OOD entities of OVEN (landmarks, animals, plants, car, aircraft, etc).
Can you show some examples of the model predictions on visual entity categories not covered by the OVEN benchmark such as artwork, shopping items, etc. 

- It seems using rational data improves query split in OVEN. I wonder if the authors have any intuition about this.
limitations: limitations: N/A

Comments: title: Response to the Official Comment by Reviewer Poha
comment: Thanks for the REW benchmark experiments and sharing LAION annotations with Gemma variant implementation. I think 8 is a fair evaluation of this work.
Thank you for the detailed answers - yes please do include the Gemma results and publish the curated annotations for LAION. I'm happy to increase my score to 6
We thank the reviewer for starting the discussion.

We would like to emphasize that we have already demonstrated the performance of our method using open-source models (PaliGemma+Gemma) in Tables R1, R2, and R3 of the "General Response".  

Open-source VLMs like PaliGemma and LlaVa do not support long contexts. To overcome this complexity, as the reviewer rightly points out, we employ a two-stage approach. First, we utilize PaliGemma (similar to LlaVa) to process the visual input and generate a detailed caption. Then, we feed the output from PaliGemma, along with with our prompt (Section A.3.1), to a more powerful LLM, Gemma 27B, chosen for its longer context support and stronger reasoning abilities. This two-stage approach allows us to effectively use our methodology with open-source models.

We can see in Table R1 that in all cases our method gives substantial improvements over the prior work GER-ALD[6] and GIT-Large trained on Webli-Entity[6]. We will include the Gemma results with the full 47M dataset in the camera ready.

We hope that this experiment and explanation addresses the reviewer's concerns. We are happy to elaborate further or discuss any other questions the reviewer might have.
Thank you for your reply. Please do release together with the paper the curated annotations for LAION. Regarding reproducibility more generally, I would not underestimate the complexities of reproducing results that are obtained from a proprietary model available only through an API access. For future research, I would highly encourage the authors to also include an available open source model with visual input (e.g. LlaVa).
The author's rebuttal was received. Thanks for clarifying the issues. 

Considering all the merits and weaknesses, I decided to update my score (6 -> 7) as it seems to be a fair evaluation of this work.