Open-domain Visual Entity Linking
Authors: Hexiang Hu
Yi Luan
Urvashi Khandelwal
Mandar Joshi
Kenton Lee
Kristina Toutanova
Ming-Wei Chang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Open-domain Visual Entity Linking
Vision and Language

Abstract: We introduce the task of Open-domain Visual Entity Linking (OVEN), targeting a wide range of entities including animals, plants, buildings, locations and much more. Given an image (e.g., an image of an aircraft), a text query (`What is the model?' or `What is the airline?'), and a multi-modal knowledge base (e.g., Wikipedia),  the goal is to link to an entity  (Boeing-777 or EVA Air) out of all entities in the knowledge base. We build a benchmark dataset (OVEN-wiki), by repurposing 14 existing image classification, image retrieval, and visual QA datasets. We link all existing labels to Wikipedia entities when possible, using a state-of-the-art entity linking system and human annotators, creating a diverse and unified label space. OVEN is a rich and challenging task, which requires models to recognize and link visual content to both a small set of seen entities as well as a much larger set of unseen entities (e.g., unseen aircraft models). OVEN also requires models to generalize to previously unseen intents that may require more fine-grained reasoning (`Who manufactured the aircraft in the back?'). We build strong baselines based on state-of-the-art pre-trained models and find that current pre-trained models struggle to address the challenges posed by OVEN. We hope OVEN will inspire next-generation pre-training techniques and pave the way to future knowledge-intensive vision tasks.

Under review as a conference paper at ICLR 2023
OPEN-DOMAIN VISUAL ENTITY LINKING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce the task of Open-domain Visual Entity Linking (OVEN), targeting a wide range of entities including animals, plants, buildings, locations and much more. Given an image (e.g., an image of an aireraft), text query ('What is the model?" or "What is the airline?'), and multi-modal knowledge base (e.g.. Wikipedia), the goal is to link to an entity (BoEING-777 or EVA AIa) out of all entities in the knowledge base. We build benchmark dataset (ikN-- by repurposing 14 existing image classification, image retrieval, and visual QA datasets. We link all existing labels to Wikipedia entities when possible, using a state-of-the-art entity linking system and human annotators, creating a diverse and unified label space. OVEN is rich and challenging task, which requires models to recognize and link visual content to both a small set of seen entities as well as much larger set of unseen entities (e.g., unseen aircraft models). OVEN also requires models to generalize to previously unseen intents that may re- quire more fine- grained reasoning ('What is the model of the a aircraft in the back?") ? ). We build strong baselines based on state-of-the pre-trained models and find that current pre-trained models struggle to address the challenges posed by OVEN. We hope OVEN will inspire next-generation pre-training tech-
niques and pave the way to future knowledge-intene inntggiveeneeivve vision tasks.
INTRODUCTION
Recent interest in knowledge-intensive visual applications such as KVQA (Shah et al. 2019) and OK-VQA (Marino et al. 2019) has demonstrated the value of grounding images to knowledge bases such as Wikipedia. However, while models for these applications focus on integrating information from the knowledge base, there has been little focus on systematic, broad- coverage approaches to the grounding problem itself. Existing work typically combines various closed-set set classifiers (e.g., ImageNet (Russakovsky al. 2015), COCO (Lin al. 2014), etc.) in ad-hoc manner, without clear formulation of image grounding as a general task.
In this paper, propose and formally define the task of Open-domain Visual Entity Linking (OVEN), with the goal of building vision systems that ground visual content to entities in large-scale multi-modal knowledge bases (such as Wikipedia). In contrast existing informal setups that rely on closed-set classifiers in an adhoc way, OVEN is open-domain where predictions cover a large space of entities governed by a knowledge base. The OVEN task takes as input with an image, text query[ that expresses intent with respect to the image, and a knowledge base which contains the entire set of entities, along with supporting text descriptions and relevant set of images for each. Given these inputs the goal is to predict an entity that is both physically present in the input image as well as satisfies the unambiguous intent expressed in the query. For instance, given the same image of an aircraft, different text queries such 'Which model is this?' or 'Waich airline is this?' can lead to different answers, i.e., BOEI NC-777 or EVA AIR. A strong OVEN model should learn to use all three inputs, í.e. the query, the image and the multi-modal knowledge base when making predictions. Figure I Iillustrates the input-output mapping of OVEN.
By connecting images entities in the knowledge base, OVEN creates universal entity space that organizes existing image labels. This setup poses challenges in the form of (1) generalization to UNSERN entities and (2) generalization to UNSEEN queries. For example, classic object recognition systems trained on FGVCAircraft (Maji et al. 2013) can recognize the ninety-six aircraft categories
A query be expressed different formats; in this paper, we choose use question reflect the intent.

---

Under review as a conference paper at ICLR 2023
Contest Image
Knswledge Bone (Kilipeda)
Text Quees
Vho marafaatuoo the
What place of equipment a placed image?
king
OVEN Models
Medernell deugles
Bridee
Bugutt Weyron
Shanderbuug Museum
Quipat Eatlyy (Klkipois linit)
Figure I: An illustration of the proposed OVEN task. Examples on the right are sampled from the constructed OVEN-Wiki dataset. OVEN aims at linking entities physically presented or revealed in the image.
defined in FGVCAircraft, but would necessarily fail to recognize BOBING-787 7,8,,6, as it does not belong to the pre-defined categories. In contrast, models for OVEN required recognize entities that were UNSEEN in the training data. This requirement reflects more realistic scenario, since there will never be enough training data to cover all knowledge base entities, especially when the number of knowledge base entities is constantly growing OVEN also evaluates a model's ability to understand real-world KB during training.
UNSEEN queries, since is impossible to observe all text queries
To benchmark OVEN, we construct OVEN- Wiki by repurposing 14 existing image classification, image retrieval, and visual QA datasets, and grounding all labels to the most prominent knowledge base, Wikipedia. The training, validation, and testing splits are formed in way such that models need to generalize to entities that are unseen during training. Grounding labels from different datasets into Wikipedia and combining datasets is challenging because of the ambiguity of language. For example, due to language polysemy, "Yoke' can represent a wooden beam or part of the construction of garment. 'Tornado' can be weather phenomena a type of airplane (PANAVIA ToRNadO). To reduce such ambiguity in the grounding, we take multiple steps to refine the labels, including the of state-of-the-aa textual entity linking system (De Cao et al.) 2020 For more accurate evaluation, a subset of examples is thoroughly annotated by human annotators: entity linking errors corrected and the ambiguous queries are rewritten that no other objects can be the answer. To ensure that the dataset is computationally manageable for the community, we use a 100k subset of Wikipedia entities.
Using multiple types of information (image, input query and multi-modal knowledge) is impor- tant for succeeding on OVEN, but existing pre-trained models like CLIP (Radford et al. (02T] or VLM (Wang et al.) 2021) cannot use all of the available information natively. We experiment with baselines applying different combinations of strong pre-trained models and conduct error analysis to show possible headroom. The contributions of the paper are as follows:
Towards the broader goal of linking visual entities to large open-domain KBs, we formalize the task of Open-domain Visual Entity Linking (OVEN)
We construct the first large-scale visual entity linking benchmark, OVEN-Wiki, by re-purposing 14 existing datasets with all of their labels grounded Wikipedia. A subset of the dataset is annotated by human annotators for high quality evaluation.
We build strong baselines based on state-of-the-art pre-trained models and find that the best performing systems are the ones that better leverage multi-modal knowledge.
2 TASK FORMULATION FOR OVEN
The proposed task of open-domain visual entity linking relies inputs that consist of text queries, images and a multimodal knowledge base (KB), all corresponding to a unified label space. The input image-text pairs
(x" _x") include text query x' expressing intent with respect to the corresponding image xP, Given a unified label space & which defines the set of all possible entities, entity
the input knowledge base K. {(e, (e))]e {} is set of triples, each containing its corresponding text description t(e) and (possibly empty) set of relevant images p(e). For instance, an entity SABATIA CAMPESTRIS would have a corresponding textual description t(e) = 'Sabatia campestris is a species of Sabatia and a set p(e) containing one
English Wikipedia has grown from 3M 6.5M articles in the last decade and continues grow.
2

---

Under review as a conference paper at ICLR 2023
more images from the corresponding Wikipedia page of SABATIA. CAMPESTRIS. We consider the combination of t(e) and p(e) the multimodal knowledge for the entity e.
In the current framework, we focus linking entities that physically present in the image This means that given an image, the target label is an entity that is present in the image and answers the query. There could be many different modeling choices for OVEN. As modeling choice, can learn a function fe to maximize the score of the target entity for the given image-query pair, using multimodal knowledge from the knowledge base.
Then, given a test example x = (x?, and the knowledge base of triples KC the function is used to make a prediction,
é' = arg max fe( "au",,t x., p(e), M(e))
(1)
eEE
where a large of entities in ع ESBEK EeNSEEN a combination of SEEN and UNSEEN entities. Note that all examples available during the training stage have entities in ESEEN-
2.1 COMPARING OVEN TO OTHER TASKS
OVEN a challenging recognition task By setting the text query to "What is the main object?', can recast OVEN into general object recognition task. However, OVEN is far more challenging compared to existing benchmarks because knowledge bases such as Wikipedia can provide unified label space. The number of entities in Wikipedia orders of magnitude larger than conventional benchmarks. We demonstrate the challenge of using large label space as follows-we start by selecting an image from standard benchmark and using CLIP (without fine-tuning) to encode it, then we scan 100K Wikipedia titles and make a prediction. Figure shows that extending the original label space 100K Wikipedia titles has introduced more closely related hard negative examples that turn many existing features into weaker signals. For example, models can no longer rely on just the color or the car body configuration to identify the ground-truth category D&EWOо NUBIRA, confusing it with other cars with similar configurations. Instead, fine-grained reasoning over visual attributes is required to make a correct classification.
In addition, the introduction of large label space also makes generalization to previously UNSEEN entities critical, which is not emphasized in most image recognition benchmarks. This is important because there would not be enough training data to learn classifiers for every possible entity in Web-scale KB such as Wikipedia, especially given that it grows over time.
Finally, knowledge bases such as Wikipedia cover entities of different granularities (e.g., the airplane manufacturer or the aireraft model), such that different entities would be preferred according to different text queries. Given the scenario in Figure 26 OVEN models should predict different entities given the same input image but different text queries, whereas recognition models can not handle this.
Hatshbusk
Wagen
Loganci
Dacwco Lacetti
Stobart Air
WIKIPOIIA
H] Station wagen
[5] Mazda Tribute
WIKIEEDIA
Demter 228
FoAkor
Plguosgworke
(a) Large entity sets
(b) Ambiguous intent
Figure 2: Why is OVEN challenging? We show that CLIP cannot solve OVEN out of the box. We CLIP encode images from standard recognition datasets and retrieve Wikipedia titles from 100K Wikipedia entities. Left: (a) As the of labels become large, difficult negatives start appearing. For example, DAEWOo LACeTT does not belong to the original label set. Right: (b) Intents can be ambiguous-DoRNEE and DORNIES FLUGZEUGKERS is the manufacturer.
228 is the model
OVEN is a specialized Visual QA task with KB grounding Due to the input text query, we can consider OVEN to be specialized VQA task with KB grounding. However, OVEN is not a VQA task as OVEN models are required to output the entity name inside a KB (same as text entity linking
https://en... wi ped i org/wi k /P e Sabat i caapeiaestaaaaaaaeeeeaaaee Sarrris_rraaaaa _rrrrraaasss.. jpg Extending this framework entities that not physically present the image (e.g. the inventor of the airplane) is also valid and useful-we leave this to future work.
'The omitted questions of Fig. 2a and Fig. 2b (hecause standard CI.IP model ignores it) are "what is the model of the car?" and "who manufactured this aircraft?"
3

---

Under review as a conference paper at ICLR 2023
tasks), while VQA models output free-form answers that contain yes yes-no/counting questions. Moreover, OVEN only focuses on the entities that are present in or can be directly inferred from the image while VQA tasks ask questions referring to entities not in the image. Another unique feature of OVEN is that covers over 100K fine-grained entities and this encourages the models to make use of the multimodal knowledge in general KB for unseen entities.
Knowledge Base as Input It is important to note that the knowledge base an input to the model. Therefore, the model should still work if the knowledge base gets updated, as the KB is a constantly evolving knowledge source. The two possible updates include: (1) editing an existing entity's 's text description or image set, and (2) growing the label space by adding new entities.
2.2 EVALUATION FOR OVEN
We evaluate OVEN with the goal of balancing performance between SEEN and UNSEEN entities using a harmonic mean, as shown below:
(ACCSEEEN, ACCUNSEEN) =
ACCSEEN
ACCUNSEEN
Here, AcCssen and AcCUNSEFN are the Macro accuracy over examples from all SEEN and UNSEEN classes, respectively. Harmonic mean equally weighs the importance of both SEEN and UNSEEN subsets, and penalizes models with a short barrel. More concrete details are provided in 3
3 THE OVEN-WIEE DATASET
Based our task formulation, we create benchmark dataset using comibation of automatic entity linking and human annotation. More specifically, we create OVEN-WiLE Wikii by combining 14 existing datasets originally created for image recognition, image retrieval, and visual question answering, and creating a unified label space grounded in Wikipedia entities. This dataset provides training and evaluation examples, and Wikipedia-basee KB for the proposed open-domain visual entity linking (OVEN) task. The complete list of datasets includes:
Image Recognition (or Retrieval) Datasets: ImageNet21k-2 k-P (Russakovsky al. 2015 Ridnik 2021), NNauuralis20117 (Van Horn et al.] 2018), Cars196 (Krause et al. 20133). SUN397 (Xiao al.
et al.
2010), Food101 (Bossard et al. 20014), Sports 100 (Gerry 2021), Aircraft (Maji et al. 2013).
Oxford Flower (Nilsback & Zisserman 2008), Google Landmarks v2 (Weyand et al. 2020)
Visual QA Datasets: VQA v2 (Gioyal et al., (017). Visual7W (Zhu et al.
2016) Visual
Genome (Krishna et al. 2017), OK- VQA (Marino et al. 2019), Text- VQA (Singh et al. 2019)
These datasets generally belong to two groups: image recognition (or retrieval) which provides diverse visual entities, defined as the Entity Split; and visual QA which provides visually-situtted natural lunguage queries, defined as the Query split.
We describe data collection, filtering and curation in Appendix A.1 In the following, we present the dataset statistics, and introduce evaluation protocols.
animaal 21.5%
Train Set
Val Set
Test Set
Gold Set
plant 15.4%
# unique queries
19,971
2,874
14,889
3,775
building 13.3%
# SEEN entities
10,213
1,925
10,213
705
location 11.0%
# SEEN examples
4.89M
65K
366K
2,419
organization 4.6%
# UNSEEN entities
0
2,089
10,149
710
food 4.2%
# UNSEEN examples
64K
364K
2.375
others 30.0%. # Total examples
4.89M
129K
730K
4.794
Distribution of super-categories of positive entities in dataset (See Fig Gin Appendix more details), Right: Statisties of different sets.
Figure 3: Dataset Statistics of the OVEN-Wik Wiki. Left:
Dataset Statistics Since goal is evaluate generalization to both existing and novel entities, we hold out half of the entities as UNSEEN and only include them for evaluation. We also hold out examples from SEEN entities to measure SEEN performance in the evaluation. Figure (left)
4

---

Under review as a conference paper at ICLR 2023
presents the general distribution of the super-categories for final collection of positive Wikipedia entities. Figure 3 (right) shows detailed statistics for queries and entities for each of the training (or fine-tuning), validation, test and gold splits | The number of SEEN/UNSEEN examples indicates the number of examples of which the positive entity labels are in the SEEN/UNSEEN split.
Human Annotated Gold Evaluation Besides evaluating on the automatically constructed dataset, we further perform a human annotation to create the gold evaluation data. Specifically, we hired over 30 dedicated annotators validate the entity links in <image, query, answer> triplets sampled from the test split. They were asked to re-annotate the triplets, with access to the visual context, ensuring that the query leads the correct Wikipedia entity answer. This process identified an approximate 8.04% of entity linking errors, and obtained 4,794 natural language queries, equally distributed over triplets originally sampled from the Entity and Query splits (i.e,, test splits). We asked the annotators to rewrite the queries so that no other object in the image can be the answer. As result, the percentage of unique queries in the total examples (3,775 out of 4,794) as shown in Table 3 (right) is significantly higher in the gold set than the other sets, which brings more challenges to query generalization for the gold eval set. We report results using the same evaluation metrics on the gold data, with respect to SEEN and UNSEEN entities. Figure provides glance at the human annotated gold data. More details about human annotation is in Appendix A.2
Evaluation Protocol To approach
mission of linking images Wikipedia entities without imposing overwhelming computational costs, we create an approximated Wikipedia knowledge base. Specifically, we include total of 85k top frequent and non-overlapping Wikipedia entities into dataset, which sums to a total of 105k entities. This step simulates practical entity linking scenarios, where a model required contrast against massive number of Wikipedia entities and infer the correct one for a query on the visual content. In this paper, all models predict among ~ 100k candidate entities. As stronger models emerge, we can consider a larger knowledge base in the future work.
We measure the model's 's performance using both the Entity Split (ES) and Query Split (QS). Specif- ically, we first compute the harmonic mean of Macro Fl over examples from SEEN and UNSEEN classes, as AcC.s and ACCas, according to Equation 2 Then we further calculate the harmonic mean between splits HM(ACCes, ACCos) to reward models that do well on both splits.
We use the validation data, which contains examples from subsets of both SEEN and UNSEEN entities, for model selection, and report the overall evaluation. For final performance, we measure performance on the test split, which covers ~98% positive entities (about half of them are unseen). We also report results on the Gold set, to measure model's 's generalization to a more diverse set of human queries.
BASELINE MODELS FOR OVEN
As performance on UNSEEN entities and text queries is important for OVEN, we focus on applying pretrained multimodal models which have previously reported promising zero-shot generalization.
4.1 EXISTING PRE-TRAINED MULTI-MODAL MODELS
We consider the following two representative pre-trained models.
CLIP: Image-to-Text Dual Encoder CLIP (Radford al. (221)) the most popular dual encoder model for images and text, where one encoder encodes images and the other encodes text. The biggest advantage of a dual encoder model that the embedding of all candidates in the knowledge base can be pre-computed, so that when new example arrives, only a dot product is required to compute the similarity between query input and entity representations. Based on the different combination of input information and entity information employed, there are fours variants of CLIP models, which make predictions based on cosine similarities between <z',t(e), t(e)> (CLIP T2T), <x', p(e)> (CLIP T2I), <z", p(e)> (CLIP 12I), and <x,, t(e)> (CLIP I2T). Without explicit notation, we refer CLIP 12T model as the default CLIP models. Note that all CLIP variants are depending on partial information pair queries with entities. However, CLIP's large-scale contrastive pre- anee-taining--ee with 400M image-caption pairs- makes a strong baseline in practice.
'Note that the models do not know which entities are present in the val/test/gold set, and must scan through the whole KB make predictions.
5

---

Under review as a conference paper at ICLR 2023
SimperVLM:
Image+Text Encoder with Text Decoder (SimpVLM) SimVLM (Wang et al. 2021 is pretrained model with an encoder that can jointly encode text and images, and outputs text with a decoder. Sim has demonstrated strong performance including Visual QA. Using this model for OVEN is straight-forward as we use the encoder to encode inputs (x²², x²), and use the decoder generate the target entity e. However, the generated text is unconstrained and might not belong to the set of possible entities &, In addition, this model also is not capable of using the entity images p(e). For this paper, we implement our version of SimVLM (Wang et al., 2021) (dubbed SimplerVL.M,
vision and language tasks,
where we remove the Convolutional Neural Networks (i.e.. Transformer only) and train the simplified model on the combined datasets of Fit400M (Yu al. 2022) and C4 (Raffel et al., 2020). The total # of image-text pairs used for training is similar to the CLIP model's pre-training dataset, and the # of text-only examples is the same as Wang et al. (2021).
Note that these two models do not make use of all information provided by OVEN. Next we extend these two pre-trained models to construct our baseline models that cover more types of information.
4.2 OUR BASELINE MODELS: MIXED-MODALA DUAL ENCODERS
In the following, we propose baseline models that can use all the information provided for given (x,, e) pair in OVEN: x", x², p(e). t(e). Note that it computationally infeasible to score possible entities in K. through a joint encoder of all components of the input and all components of each candidate entity, due to the large size of the candidate set. We therefore limit attention to scoring functions that decompose Given that both the input pair (x², x²) and the knowledge base information (p(e), t(e)) are mixed modality, call these models Mixed-Modality Dual Encoders. Note that the parameters of Dual Encoder @ have two components, i.e., left encoder parameters 0 and right encoder parameters ø, which are not shared for all the Mixed-Modality Dual Encoders in this paper.
dual encoders, i.e., fe (x", x', ((()) fo(2" x²) (p(c))). t(e)).
We study three baseline mixed-modality dual-encoders:
Dual SimplerVLM (D-SimpVLM) converts SimplerVLM into mixed modality dual encoder by using its encoder to embed the paired image and text information into a joint space.
CLIP Fusion adopts the pre-trained CLIP model as the featurizer to develop this system, via adding a 2-layer Multi-Modaa Transformer on top of the CLIP image and text features.
CLIP2CLIP adopts the pre-trained CLIP model via introducing a minimum set of non pre-trained parameters, only at for re-weighting the similarity scores. Particularly, it computes the cosine similarity between <x²,, t(e)>, <x', p(e)>, <x", ((e)>, and <x', ((E)), using the image and text encoders of CLIP, respectively. Then it aggregates these similarities by multiplying them with learnable vector that reflects importance weights.
For all three models, their left encoders and right encoders are using the same architecture, but differently parameterized. We fine-tune all their parameters (pre-trained or not) on the OVEN-Wiki.
Implementation Details We process all the images in dataset by resizing them to 224, and use standardization to normalize them. For natural language text, we perform tokenization based on the adopted pre-trained model's original vocabulary (e.g., CLIP). For SimplerVLM, we employ the vocabulary from T5 (Raffel et al. 2020). Additional details on input processing, model configurations, and fine-tuning ppper-parameters are provided in the Appendix B
EXPERIMENT
We first establish the main benchmark results in $5.1 We then conduct ablation studies to analyze models, and illustrate the key challenges of OVEN-Wiki in 5.2
5.1 BENCHMARK RESULTS
Main Results Results the validation set presented in Table including evaluation scores on both the Entity and Query split, as well as the overall combined score. We divide the methods into two categories, (1) zero-shot transfer models (w/ pretrained models only). and (2) fine-tuned models.

---

Under review as a conference paper at ICLR 2023
Entity Split
Query Split
Overall
Entity Split
Query Split Overall
SEEN UNSEEN SEEN UNSEEN
HM
SEEN UNSEEN SEEN UNSEEN
HM
Zero-shot Models
Fine-tuned Models
SimpVLM B
0.0
0.0
0.1
0.3
0.3
SimpVLM B 5.0
0.0
18.4
0.0
0.0
CLIP T2T
B
0.0
0.0
0.4
0.0
D-SimpVLM B 35.8
3.7
15.6
0.3
1.1
CLIPT2)
B
0.1
0.0
0.8
0.7
0.2
CLIP Fusion
B
4.5
19.7
1.1
3.4
CLIP 121
B
7.7
7.6
1.2
1.9
2,4
CLIP2CLIP
B 18.2
15.4
10.4
9.3
12.4
CLIP 12T
B
13.5
13.8
4.2
6.8
7.5
CLIP Fusion
L 36.7
7.9
27.3
1.3
4.3
CLIP 12T
1 17.7
17.3
4.2
8.6
8.5
CLIP2CLIP
L 19.8
15.9
16.0
12.1
15.5
Table 1: Comparison between the zero-shot and fine-tuned models the OVEN-Wiki validation set.
Entity Split
Query Split
Overall
Gold Eval
Model Size
SEEN
UNSEEN
SEEN
UNSEEN
HM
SEEN
UNSEEN
HM
CLIP 12T
Base Hase
13.8
12.2
5.2
6.5
8.0
12.1
14.2
13.1
CLIP Fusion
35.7
4.8
42.7
1.1
3.4
14.8
2.5
4.3
CLIP2CLIP
Base
19.1
13.1
13.0
9.5
12.9
19.1
16.1
17.5
CLIP 12T
Large
18.0
16.6
7.2
8.9
10.9
14.3
17.7
15.8
CLIP Fusion
Large
39.2
8.2
41.7
2.0
6.0
23.0
5.6
9.0
CLIP2CLIP
Large 20.5
15.8
13.3
12.9
15.1
21.0
18.7
19.8
Table 2: Results of top methods on the OVEN-Wiki test set and gold evaluation set.
There several interesting observations in Table First, not all fine-tuned models better than their pre-trained variants, particularly on UNSEEN entities. This showcases the unique generalization challenge in OVEN-Wiki, where models cannot be over-optimized for a specific set of entities that have more training data. Next, comparing encoder-decoder models such as SimplerVLN, LMM to other methods, we observe that the former struggles to link entities accurately. We find that the zero-shot SimplerVLM model often does not produce entity names, as the pretrained model never observed Wikipeida titles. The fine-tuned Simpler VLM models do not handle unseen entities well, since the models never see the UNSEEN Wikipedia titles from the KB while training OVEN-Wik Wiki data.
Comparing all Case- models, we observe that CLIP Fusion, where two new layers have been added on top of pretrained CLIP, shows very strong results on SEEN entities, for both the Entity and Query splits weak results on UNSEEN entities, thus leading to lower overall performance. The CLIP2CLIP model, on the other hand, is capable of retaining the cross-entity generalization performance while improving its prediction accuracies on SEEN entities. This suggests that adding more new (but non non-pre-tren pre-trained) parameters can harm the pre-trained models. Unsurprisingly, methods with large pre-trained models outperforms their smaller counterparts.
We select the best performing methods and evaluate them the test data, well as the human annotated gold evaluation set. Table shows that the results on the Test set are generally aligned with observations
the validation set. In the gold set, annotators provide more varieties of the input queries, which can challenge the model more. Interestingly, results on the gold set suggest that methods perform better the SEEN entities of the test split do necessarily transfer gold data split (e.g., CLIP Fusion vs. CLIP2CLIP), perhaps due to the aforementioned variation in queries.
Human Performance We conduct a study estimate the human performance OVEN-Wiki, via requesting 3 dedicated human annotators to answer 100 examples (sampled from gold evaluation set, answers are non- non-overlapee ooeoollllappigg. Because humans are ineffective searching"] we allow the annotators to use search engines (e.g., Google Image Search, Wikipedia Search, etc.), as long as the annotators can provide a valid Wikipedia entity name as answer (among 105K candidates). As result, human achieves 76% average accuracy, which is significantly higher than the best comparing systems
Fvvn with search engines, each annotator has used 254 seconds to complete one example. For comparison, the CLJPPCIIIP-I. -P-C.a.--.rr.... model is evaluated on this subset, and it achieves 25% average accuracy.
7

---

Under review as a conference paper at ICLR 2023
Query Repr.
Entity Split
Query Split
Overall
Entity Repr.
Entity Split
Query Split
Overall
x
SEEN
UNSEEN
SEEN
UNSEEN
HM
p(e) ((e)
SEEN
UNSEEN
SEEN
UNSEEN
HM
18.0
15.0
8.4
11.0
4.0
3.2
2.5
1.0
3.0
0.1
0.0
2.6
0.4
0.0
17.3
14,8
8.6
8.9
11.3
18.2
15.4
10.4
9.3
12.4
18.2
15.4
10.4
9.3
12.4
Table 3:
Ablation studies on CLIP2CLIP on the validation set. Left: Results of ablating input images (a:P) and input queries (a*)). Right: Results of ablating entity information (p(e): entity image; t(e): entity text).
5.2 ABLATION STUDIES AND ANALYSIS
We perform empirical studies analyze OVEN- Wikki using the best performing model, CLIP2CLIP. Particularly, these studies are related to (1) the importance of input representations for OVEN, (2) the effects of different knowledge information for entities, (3) how fine-tuning influences generalization, (4) how the size of the Wikipedia knowledge base influences model performance, and (5) error analysis on the best performing model.
Which input information is important for OVEN? Table (left) shows scores for CLIP2CLIP variants using different combinations of input query representations. We observe that the input image is most essential for the model when making predictions. Meanwhile, we see that query in combination with the image, can provide improvements to results on both the Entity split and the Query split, with larger margin on the latter. Additionally, models that only look at the question can also make a reasonable guess on the Query split, but not the Entity split.
Which entity knowledge is important for OVEN? Table 3 (right) provides a study on different com- binations of knowledge from Wikipedia, used as the representation for an entity (for the CLIP2CLIP model). Particularly, we observe that the Wikipedia entity image itself does not provide sufficient information to support strong OVEN performance, but it contains complementary information that helps the CLIP2CLIP model to better understand the entity text. As result the best performing model is achieved via combining Wikipedia entity image and entity title, which is also serves as the de facto setting for all Mixed- useful content on Wikipedia pages, such as the summary text. However, current experiments show that including summary text does not further improve Mixed-Modality Dual Encoders (i.e., overall results drops ~0.5%), potentially because they are often longer pieces of text with complex structure, which is oft-of-domain for the pre-trained models, Appendix C included the complete study.
Dual Encoders in this paper. We note that there is other
30
SEEN
400
CLIP2CLIP
Macro Acc
25
UNSEEN
Macro Ace
CLIP 12T
20
20
154
10
0
500
1,000
1,500
2,000
0
20
60
80
100
of training steps
# of candidates (x 1000)
(a) Impact of fine-tuning steps for unscen entities
(b) Impact of candidates in the KB
Figure 4: Left: Finc-tuning CLIP2CLIP for large of steps increases the SEEN entity accuracy but hurts the UNSEEN entity accuracy. Right: (b) Increasing knowledge base size makes the tasks difficult.
Does fine-tuning always help generalization? Figure (a) shows the validation scores of the CLIP2CLIP model, during fine-tuning on OVEN-Wiki's training split. Note that we only plot the results on Entity split as the representative, because the trends are similar on the Query split. It shows that longer training schedule does not converge to better generalization performance, particularly when evaluated on the UNSEEN entities. Because of this, we employ the early stopping strategy for model selection, and pick the model with the best UNSEEN validation performance.
How would the number of entities in KB influence the model's prediction? Figure (b) presents the accuracy of CLIP and CLIP2CLIP, as function of the # of total candidates to retrieve from. Here, we compute the accuracy by sub-sampling the negative candidates from KB to different sizes. We observe that when the retrieval candidate entities are only the positive entities (with 0 additional negatives). the performance of both CLIP and CLIP2CLIP is significantly higher than the 100K setting. Beyond this, as the KB size increases, model accuracy decreases.
8

---

Under review as a conference paper at ICLR 2023
6 RELATED WORKS
Learning to Recognize UNSEEN Categories There has been significant amount of prior work (Lampert et al.] 2014, Vinyals et al.) 2016 Liu et al. 2019) focusing on the generaliza- tion situation where information of novel categories are presented at test time. Zero-shot learning (ZSL) is one of such attempts that tackles learning new categories with zero images for training. To achieve such transfer. ZSL methods typically relies generating UNSEEN image classifiers based on corresponding semantic representations, in the format of mamually labeled attributes (Lamper) et al. 2014), unsupervised learned word vectors (Changpinyo et al. 2016), or pre-trained sentence embeddings (Kil & Chao 2021 Radford et al. 2021). Few-shot learning (FSL) (Vinyals et al. 2016) proposes more realistic setup, where learners have access limited number of visual exemplars during the model deployment. With this goal, FSL methods aim to extract the inductive bias of learning from the SEEN classes, such that the model can leverage in learning the UNSEEN classes, to avoid severe over-fitting Particularly, prior works either use adapted non panaaammetticc classifiers (Snell al., [017]] Rusu et al.) [2019] Ye et al. 22020), or meta-optimized linear classifiers (Finn et al.] 2017] Raghu et al. 2020) to incorporate the sew- UNSEEN support examples. Comparing to them, our proposed task exposes different challenges as we ask the model to make the best use of open-world Web knowledge (i.e. Wikipedia pages with images & figures), which contains textual semantic information and visual appearance of the entities in the open world.
Vision and Language Knowledge There has been efforts in combining knowledge into vision and language tasks, such as Visual QA (Shah et al. 2019 Marino et al. 2019; Chang et al. 2022 Chen et al. 2021) and entity-focused image captioning (Liu et al. [2200] Biten et al. 2019)) Among them, knowledge lewge-begeed VQA is most related to OVEN, but also differ in many aspects. Specifically, Chang et al. (2022) presents text QA dataset that requires understanding multi-modal knowledge in a KB. KB..Shhhh et al. (2019) propose to perform mowledge-based question answer tasks, centered around questions that resolves relational query over public Figures. Mcanwhile, Marino al. (2019) propose to answer questions where the answer is outside of the image context, to assess model's 's capability in understanding real-world knowledge More recently, Chen al. (202T) studies the zero-shot visual QA setting where some answers (out of a total of 500 frequent answers of general concepts) are unseen during the training. where a KB is supplied Comparing to them, OVEN steps back to the more fundamental problem of establishing the link between visual content and entity in the KB, but at larger scale and broader coverage. We believe that stronger models developed on OVEN would benefits such knowledge intensive visual QA tasks.
assist the model in answering unseen answers.
Entity Linking Entity linking (EL) is the task of grounding entity mentions in text by linking them to entries in a given knowledge base. Supervised EL (Milne & Witten 2008) has demonstrated its strong performance when all entities are in in-distribution during the evaluation. Because KB is updating all the time, recent works (Logeswaran et al. 2019 Botha et al.) 2020 Zhang al. T02T
De Cao et al. 2020; 2022) focus on a more realistic setting where entity linking needs to be achieved in the zero-shot, with a large portion of entities (to be evaluated) completely unseen during the training. OVEN is a visual analog of zero-shot EL., and targets at developing generalizable models that recognize entities unseen in the training. Among all EL literature, visually assisted EL Zheng et al. (2022) is most relevant to this work, whose goal is to use associated image of text to improve the precision of text EL. OVEN is different as its text queries do not mention the name of the entities, which put visual understanding and reasoning into the central position.
DISCUSSION
We introduced OVEN, a task that aims to link visual content to the corresponding entities in a knowledge base. While the focus of this paper is mostly on the task and the benchmark, our results point to several interesting future avenues for representation learning research. For example, the fact that Mixed-Modality Dual Encoders outperform CLIP models suggests that future work on large scale contrastive pretraining could benefit from jointly encoding or mixing modalities. This is especially critical for unseen entities whose representations are typically learned only during pretraining. In textual entity linking, De Cao et al. (2021) has shown that it is possible to obtain a high- perfoomnfoninn seq2seq model for entity linking. OVEN now provides a benchmark for evaluating such approaches for visual entity linking. In summary, we hope OVEN will drive future research on knowledge-inffee infused multimodal representation learning via visual entity linking.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
As our dataset, Le., OVEN- Wiki,, is composed of existing image recognition, image retrieval, and visual question answering datasets, we have introduced minimum risk of exposing additional social bias in our data. However, OVEN-Wiki is still at the risk of inheriting existing dataset biases. As result, we employed existing data curation strategies YYng et al. 2020) to reduce such potential risks. Besides such risk, OVEN-WiL also opens up new possibilities that alleviate ethical concerns in AI systems. Specifically, OVEN- Wiki is a dataset that targets advancing research for establishing stronger grounding between the visual content and knowledge base, which can potentially contribute to building more attributed visual systems, such as a visual question answering model that produces answers based on the linked Wikipedia page, with improved interpretability and controllability.
REFERENCES
Ali Furkan Biten, Lluis Gomez, Marçal Rusinol, and Dimosthenis Karatzas, Good news, every- one! context driven entity entity-aware aware captioning for news images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12466-1248 12475, 2019.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food- 101d-101-0 mining discriminative compo- nents with random forests. In ECCV, 2014.
Jan A Botha, Zifei Shan, and Dan Gillick. Entity linking in 100 languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7833-7845, 2020.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16495-16504, 2022.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In CVPR, pp. 5327-5336, 2016.
Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z Pan, Zonggang Yuan, and Huajun Chen. Zero-shot visual question answering using knowledge graph. In International Semantic Web Conference, pp. 146-162. Springer, 2021.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904, 2020.
Nicola De Cao, Gautier Izacard, Sehastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In ICLR, 2021.
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. Multilingual autoregressive entity linking. Transactions of the Association for Computational Linguistics, 10:274-290, 2022.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Gerry. Sports100: 100 sports image classification. http::::tppptttttt. www http:://wwwww.gggg...e kaggle com dat.sttst http:////w.ppppooaaas.ssssaairaa
2021. Accessed: 2022-09-26.
Yash Goyal, Tejas Khot, Douglas Summers -taay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Jihyung Kil and Wei-Lun Chao. Revisiting document representations for large-scale zero-shot learning. In EMNLP, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 33,R-13, 2013.
10

---

Under review as a conference paper at ICLR 2023
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(3), 2017.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classífication for zero-shot visual object categorization. TPAMI, 36(3):453 465, 2014.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint arXI1v2010.03773, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.
Lajanugen Logeswaran, Ming- Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. Zero-shot entity linking by reading entity descriptions. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3449-3460, 2019.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and paltern recognition, pp. 3195-3204, 2019.
David Milne and Ian H Witten. Learning to link with wikipedia. In Proceedings of the 17th ACM conference on Information and knowledge management, pp. 509-518, 2008.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE, 2008.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. 2020.
Tal Ridnik, Emanuel Ben- Asaf Noy, and Lihi Zelnik-Mand Imagenet-21k pretraining for the masses. In NeurlPS, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8876-8884, 2019.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604 PMLR, 2018.

---

Under review as a conference paper at ICLR 2023
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Sim Simple visual language model pretraining with weak supervision. ICLR, 2021.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In CVPR, 2020.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 547-558. 2020.
Han-Jia Ye, Hcxiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020.
Shuhei Yokoo, Kohei Ozaki, Edgar Simo-Serra, and Satoshi lizuka. Two-stage discriminative re-ranking for large-scale landmark retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1012-1013, 2020.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Wenzheng Zhang, Wenyue Hua, and Karl Stratos. Entqa: Entity linking as question answering. In International Conference Learning Representations, 2021.
Qiushuo Zheng, Hao Wen, Meng Wang, and Guilin Qi. Visual entity linking via multi-modal learning. Data Intelligence, 4(1):1-19, 2022.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
12

---

Under review as a conference paper at ICLR 2023
A DATASET CONSTRUCTION, ANNOTATION, AND ADDITIONAL STATISTICS
In this section, we describes the complete details on data collection, curation, entity linking, and show additional statistics of the processed dataset ($A.1). Then we also discuss how we train annotators to annotate our task, and provide the concrete annotation interface( A.2)
A.1 DATA COLLECTION & PRE-PROCESSING
Data Filtering Some of our member datasets have been reported to include non-imageable classes, classes with undesired social bias (Yang et al. 2020), or non-entity classes (e.g., numbers). Therefore, we apply filtering process to compose our dataset, based on the individual condition of each source dataset. Overall, to create the Entity split, we first apply a general safety filter (Yang et al.) 2020) to remove non-imageable labels, non-entity labels, and labels with social bias. To create the Query split, we employed three expert annotators to write heuristic policies to filter each VQA dataset, and ensure our task is focusing entity related questions. Concretely, questions related to counting, verification, or querying non-entity attributes (e.g., dates), are removed. Then we apply the same safety filter. We provide more details in Appendix.
Entity Linking Based on the filtered data, we developed a two-staged entity linking strategy to connect the label text to Wikipedia entities, both Entity and Query splits. First, we obtain exact match based entity candidates by querying the Wikipedia search API (with the auto-suggestion disabled) with the label text. We reject candidates whose landing pages are identified as disambiguation pages. The Wikipedia APl automatically redirects queries (in our case, labels) matching entity aliases their canonical form. For the labels which do not have an exact match in Wikipedia, we use a state-of-the-art text-based entity linker (i.e., GENRE (De Cao et al. (02T)) to obtain top candidate Wikipedia entity names. Finally, we link the label the top ranked entity whose landing page is not disambiguation page.
Preparing Multi-Modal Knowledge Using the entity linking process described earlier, we suc- cessfully connect total of 24,895 class labels in OVEN-WiK to corresponding Wikipedia entities. Overall, our dataset contains 20,801 unique entities. For the Entity split data, we generate synthetic text query based on the super-category information of the label (either provided by source dataset or mined from Wikidata To using templated language. For example, iNaturalist has provided detailed supercategory annotation on each class, such Plantae, Reptil etc. For dataset that do not provide this information, we use the super-category mined from Wikidata, which is publiclly crowd sourced and maintained. As result, our templated query generator produces the query "what is the species of the plant in this image?" for the entity "Eryngium a alpi numm,, whose super-category is P1 ana Due space limit, provide more explanation in Appendix. For all Wikipedia entities, we use the corresponding Wikipedia page and its associating melti-media content (e.g., information box images, etc.) as the source of multi-modal knowledge about entities.
13321
wIkN Original
uniqqee entiles
Original Answers
Entity Answers
VG
50,130
3,460
OK-VQA
4,214
1,600
Texi-VQA
19,500
3,562
VQA v2
26,748
4.337
Visual7W
7,588
1,945
Sports
Figure 5: Number of unique entities Entity split (left) and Query split (right). We compare it against the # of entities before applying pre-procsssing Note that VQA datasets contain massive non-entity answers, or collapsed answers, which leads to a large reduction in numbers after pre-processing.
Statistics on Entities Specifically, Figure 5 shows the number of unique entities in both the Entity and Query splits, where we compare the total number of entities in each source dataset against its original population (after applied safety filter). Note that for the Google Landmarks v2 (Gldv2)
'tttps: / http://wwwwwwww...o/////////////
Avaailable httph::t/////..... h/ww:///wwwww/... wi hwg.eiaaaaaaa....aaa.ooooo / /W///WiR Wikidats
13

---

animal Under review as a conference paper at ICLR 2023
dataset, employed the cleaned data split from Yokoo et al. (2020), where the total number of unique entities is significantly reduced. Because Gldv2 is automatically generated and has reported to contain noises particularly with tail entities Yokoo et al.
2020 removed entities with less
than 50 instances for a improved precision (further reduces the # entities in Gldv2 to ~6k).
Entity Super-Categories To give more details for the Figure in the main text, further present full super-category grouping information in Figure 6 As aforementioned, we have combined entities that belongs general groups (e.g.. "object", "item" groups) or unpopular groups (e.g., groups with less than entities) into the "others" group. We also merged some sub-categories into super-categories, e.g.. "location" "ecaation""patti +"par""+""""""""" "laak"" """iv"""""""""uuu "=mmomtain""lllooattin""" +"location", "building" ""bridde""""buuu
21.51
plant building
15.43
13.26
location
10.97
organization
4.57
food
4.23
vehicle
1.94
material
1.93
facility
1.64
person
1.57
sport
0.87
equipment
0.85
cloth 0.61
aircraft 0.53
game 0.46
activity 0.26
others
19.36
Figure 6: Distribution of the entities in our datasets (Grouped by their super category).
A.2 HUMAN ANNOTATION PROCEDURE & INTERFACE
In order to verify the quality of OVEN- Wiki and to provide human verified test set to evaluate on, we conduct human annotation on subset of test set. The annotators asked correct the errors the <image, query, answer> triplets. The details are as follows.
Original object image?
Annotation
Mozes docament Dkapanddal colors senelation.
Accorting imaga, come will padta reprosnnt comnec mear of
2. foot paste the correct widpecho tee
Arawer NVA such tirk
A the Wilipeia answeer of yalcall present in the Image??
Kn
image,
Don't anowers
Erite
Wikpedia Sheep_dog
Prtciles www.:/ewwwe..oeeaat..oo..o. dod
Figure 7: Annotation inferface
Annotation interface Figure illustrates the annotation interface. The left side of Figure aare are the input to the annotators which includes the original question, image and the answer (together with the wikipedia hyperlink). The annotators asked to complete the following questions:
14

---

Under review as a conference paper at ICLR 2023
1. Does the Wikipedia represent the correct meaning of the answer? Provide the Wikipedia link if not.
This question requires the annotators to correct the entity linking emrors The annotators use Google search to find the most suitable Wikipedia link if the provided one is not adequate. In our dataset, 8.4% of the entity links are reported wrong by more than 2 annotators.
2. Is the Wikipedia answer physically present in the image.
This question mainly aimed at filtering out the OCR examples which are out of our scope. One example is that the image about wall painted with the word "love" and the linked entity is the "love" Wikipedia. our dataset, 10.3% of the answers are reported not physically present in the image by more than 2 annotators.
3. Rewrite the question so that no other object can be the answer.
The annotators will rewrite the question is the answer is wrong ambiguous. Annotators will make sure that the question can not be answered without the image and that the answers can not be included the rewritten questions. In our annotation, 99.9% of the questions are being rewritten.
Instruction and Training We carefully design the training procedure to improve the annotation quality. We first conduct "Seef--tudy session" where the annotators will read the instructions and annotate a few toy examples. Then we conduct a "In-person tutorial" where we have an online video session in which we walk annotators through the full version of the instructions and discuss mistakes made in the self-study annotations. Finally we conduct a "Test exam" and the qualified annotators are accepted. In total, 30 annotators went through our training procedure and all of them were eventually accepted to work full-time on the main task.
Quality control
We have three way annotations where each examples are annotated by three annotators. We were giving regular feedback on the questions the annotators may have during the annotation and pointed out mistakes identified in annotators' past answers.
On average, it took annotators 4.6 minutes to answer each question with the time consumption slightly decreasing annotators get familiar with the task. The compensation rate for the task to be $17.8/hour which is higher than the minimum hourly wage in the US.
We filtered out all the examples where the wikipedia links are marked as wrong the Wikipedia answers are marked as "Not physically present in the image".
B IMPLEMENTATION DETAILS OF THE BASELINE SYSTEMS
In this section, we provide complete impelmentation details on the baselines for the OVEN task.
B. SIMPLERVLM DUAL SIMPLERVLM
As aforementioned, have simplified the architecture of original Sim VLM (Wang et al.) 2021 via using a Multi-Modal Transformer model (the same architecture as T5-Base by Raffel et al. (2020)) Specifically, we first extract non-overlapping 16x 16 patches from the input image, and perform linear transformation on them to get a sequendce of embeded patch embeddings, which then combined with the word embeddings of a tokenized text sequence to serve as the input. The Multi-Modas Transformer then take the combined sequence and contextualize the embeddings through self-attention (Vaswani et al.) 2017).
In the re- re-implemented SimplerVLM. we then feed the output contextualized embeddings into Text decoder (of same architecture), which can then produce text prediction. We first pre-train this model on the combined Fit400M (Yu et al.) 2022) and C4 (Raffel et al. 2020) dataset, under the PrefixLM objective defined in Wang et al. (2021) J. We optimize the model for 500K steps (with batch size = 4,096), using the Adafactor (Shazeer & Stern] 2018) optimizer, a initial learning rate of 0.01 and square root learning rate decay policy (with the final learning rate goes to le-4). Then we fine-tune the pre-trained SimplerVL., on the proposed OVEN-Win dataset, for 20k steps in total. The optimization is also done with the Adafactor optimizer, with the initial learning rate to be 0.001, the learning rate scheduler to be square root L.R decay, and the final learning rate to be le-6. During
15

---

Under review as a conference paper at ICLR 2023
the inference, the model is then directly predicting the name of the entity, given the input text query and the context image. W count the exact matched entity name to compute the Macro Fl accuracy.
In the Dual SimplerVLM, decoder, and use it for both the left and right encoders of the dual encoder model. Therefore, the left query encoder has the same initialization result dual encoders model on the training set of OVEN- Wiki, under a in-batch contrastive learning objective (Radford al. 202I), with a batch size of 4,096. Specifically, we optimize the model for 20K steps during the fine-tuning, again with Adafactor optimizer and a initial learning rate of 0.001.
take the pre-trained SimplerVLM (same above), remove its Text
the right entity encoder. We then fine-tune the
B.2 CLIP FUSION MODEL
As aforementioned, we implemented this Mixed-Modal Dual Encoder via taking pre-trained CLIP image and text encoders featurizers. Particularly, we build two 2-layer Transformer models, on top of two CLIP models as the left and right encoder, for encoding the query representation and the entity representation. respectively. The 2-layer Transformers follows the same architecture T5 Transformer Radford et al. (202)). but with layers, 12 attention heads, with each attention head of 64 dimensions, and the embedding size of 768. We then fine-tune this composed model on the WVEN-Wiki's training data, using the same contrastive objective as Dual SimplerVLM. We optimize the model for 20K steps in the fine-tuning stage, following the same hyper-parameter configurations as Dual SimplerVLM.
B.3 CLIP2CLIP MODEL
Different from CLIP Fusion, CLIP2CLIP is a model that adds minimum new parameters to the pre-trained CLIP encoders. Same as other models, we initialize both the query encoder and the target encoder separately with the pre-trained CLIP model. Specifically, we use the pre-trained CLIP encoders for both left and right encoders, to encode the image and text modality for both the query representation and the entity representation. We then compute the four dot product similarity scores on the <input image, target text>, <input text, target image>, <input image, target image>, and <input text, target text> pairs, which is then combined via a learnable similarity weights into one logit score. The make sure that the learnable similarity weights initialized properly, we perform grid search to find a roughly good similarity weights for the CLIP2CLIP model (using OvEN-Wiki training data). Then we took this similarity weights to initialize the CLIP2CLIP model and fine-tune all parameters on OVEN- Wiki's 's training set, under the same contrastive learning objective. Different from other models, given that this model has most of its parameters pre-trained, we realized that works the best to early stop the model. As result, we only fine-tune this model for 2k steps, with an initial learning rate of le-4, and a square root LR decay schedule with final learning rate of 1e-6. We select the model along the way with the best UNSEEN entity performances.
C ADDDITIONAL EXPERIMENTS
In this section, we provide experiments that is omitted in the main text due the space limit.
Alternative Evaluation Metries on OVEN-Wiki In addition to the primary evaluation metric, we also report the performance of dual encoder models using ranking based metrics, such as Recall @ 10 (R10) and Mcan reciprocal rank @ 10 (MRR10). The results are shown in Tad Table Overall speaking, the results are showing the same trend as our primary metric (i.e., Macro accuracy).
Complete Studies for Entity Representation Table (right) provides study on different com- bination of knowledge from Wikipedia to use as the representation for an entity (for CLIP2CLIP model). Particularly, we observe that image is always beneficial to include as it improve the results when combined with text description. Comparing to summary, the title information can be better used by the CLIP2CLIP model. The best combination is to have both Wikipedia image and Wikipedia title as the entity representation, which is the default configuration in this paper. We note that there many other useful content on Wikipedia pages that we have not yet exploited (e.g., Section text, tables, other floating images, efc.), which can potentially bring poential oportunity for further improvement.
16

---

Under review as a conference paper at ICLR 2023
Entity Split
Query Split
Overall
SEEN
UNSEEN
SEEN
UNSEEN
HM
Model Size
R10
MRRIO
RIO
MRRIO
RIO
MRRIO
RIO
MRRIO
R10
MRRIO
Zero-shot Models
CLIP
Base
0.0
1.4
1.1
0.0
CLIP T2I
Base
0.2
0.1
0.2
0.2
4.0
1.3
2.2
1.0
0.4
0.3
CLIP 121
Base
24.9
12.3
24.7
12.3
8.5
3.6
6.0
2.5
11.1
5.0
CLIP 12T
Base
40.2
22.1
37.7
21.2
16.0
7.5
12.0
6.5
20.5
10.8
CLIP 12T
Large
46.9
27.0
44.3
25.9
15.3
7.2
13.8
8.5
22.2
12.2
Fine-tuned Models
CLIP Fusion
Base
66.1
47.0
25.7
10.0
50.2
37.6
31,5
11.5
38.0
17.2
CLIP2CLIP
51.3
29.9
42.7
24.1
37.6
20.3
21.9
12.6
34.9
19.8
CLIP Fusion
Large
69.7
50.6
16.0
58.9
22.5
7.3
49.8
CLIP2CLIP
Large
55.4
32.0
45.8
25.4
45.4
26.2
31.3
19.6
42.8
25.2
Table 4: Comparison between the zero-shot and fine-tuned models the OVEN-Wiki validation Additional metrics are reported, i.e., Recall (R10) Mean Reciprocal Ranks (MRRI0).
Entity Split
Query Split
Overall HM
Gold Eval
SEEN
UNSEEN
SEEN MRRIO
UNSEEN
SEEN
UNSEEN
HM
MRRI0
MRRI0
MRRIO
MRRI0
MRRIO
MRRIO
CLIP 12T
B 39.9
22.1
34.5
18.7
15.7
8.1
16.1
9.1
22.4
12.2
34.9
21.2
40.0
26.0
37.3
23.4
CLIP Fusion
67.5
47.7
26.3
10.5
50.9
39.0
14.6
5.2
28.5
12.2
36.2
21.8
18.7
7.7
24.7
13.5
CLIP2CLIP
52.3
30.3
37.9
20.5
44.9
26.7
15.0
56.2
33.7
28.1
30.7
CLIP 12T 1. 46.7
27.1
42.8
24.5
14.9
7.9
15.9
9.2
23.1
12.9
38.5
22.8
45.5
30.4
41.8
26.1
CLIP Fusion
1. 70.1
51.3
37.8
16.8
57.8
43.6
18.1
7.2
35.5
16.8
48.6
31.2
29.0
13.4
36.4
18.8
CLIP2CLIP 54.4
31.8
43.8
24.2
40.6
23.3
31.6
17.6
33.6
18.8
56.6
33.4
51.4
53.9
Table 5: We report top methods the OVEN- Wiki test set and gold evaluation set. Additional metrics reported, f.e., Recall@ 10 (R10) and Mcan Reciprocal Ranks 10 (MRR10).
Entity Repr.
Entity Split
Query Split
Overall
Image
Title
Summary
SEEN
UNSEEN
SEEN
UNSEEN
HM
3.99
3.17
3.53
2.46
0.96
1.38
1.99
17.33
14.75
15.94
8.64
8.90
8.77
11.32
18.76
17.25
17.97
6.75
8.39
7.48
10.56
18.19
15.42
10.35
12.36
19.37
17.47
18.37
7.88
8.05
7.96
11.11
18.53
16.86
17.66
5.96
8.67
7.06
10.09
18.62
17.37
17.97
8.66
8.99
8.82
11.84
Table 6: Ablation study CLIP2CLIP, with different entity representation.
Error analysis We show top 5 predictions of CLIP2CLIP on OVEN-Wiki in Table with the left three examples being the erroneous predictions and the right example the correct prediction. From left two examples, CLIP2CLIP made mistakes among hard negative entities similar to the ground truth. On the third example, CLIP2CLIP seems to fail reason about the image and the query jointly.
17

---

Under review as a conference paper at ICLR 2023
Entity Split
Query Split
Query
Where this place?
Which species insect this?
What person holding?
What the boy's hand?
Image
Positive
Franciscoo City Hall
Polpita magniferalis
Remote
Gome controller
Old Courthouse (St. Louis)
Chalk-fronted corporal
ITnes
Game controller
Top
Utah State Capitol
Sheep moth
Rdio
PlayStation
San Francisco City Hall
Calopteron reticulatum
Playlist
Teddy bear
Top
Minneapolis City Hall
Diabrotica undecimpunctata
IPod
Suck monkey
Top
Legislature
Palpita magniferalis
Jukebox
Stuffed toy
Table Error Analysis on the CLIP2CLIP Model.
18


Reviews:
[{"review_id": "mQwq_WF3I3u", "reviewer": "Reviewer_iTQw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a dataset for vision-language entity linking. \n\nBoth queries (q) and entities (e) have image and text component, so a score needs to be defined for the quadruple (img(q), txt(q), img(e), txt(e)).\n\nThe training(~5m queries)/validation/test splits are automatically constructed from existing image recognition/retrieval and visual QA datasets. Separately, the paper provides ~5k manually corrected gold examples for evaluation. \n\nThe main baseline is mixed-modality dual encoders based on CLIP-large, achieving the harmonic mean of seen/unseen accuracies of ~15 on dev/test and ~20 on gold.  \n\n", "summary_of_the_review": "The paper identifies the interesting problem of vision-language entity linking and provides a dataset, but the utility of the dataset except for the manually annotated portion is somewhat unclear and the paper doesn't get to the bottom of the problem (i.e., not answering natural questions like what is the best possible performance). ", "strength_and_weaknesses": "STRENGTHS\n- The task is interesting without a doubt.\n- The human annotated set can be a high quality evaluation set for the task.\n- The baseline models are reasonable.\n\nWEAKNESSES\n- No performance upper bound. It is not clear if this problem is even solvable. In the first example in Figure 1, how can we expect the model to infer the plane manufacturer? There does not seem to be enough information in all modalities. (I think this example comes from the gold set? I wonder how the human annotators deemed it correct.)\n- On a related note, the baseline performance is really weak. This needs to be addressed in conjunction with the task feasibility. \n- Other than the gold set, the dataset itself is automatically constructed with basic means (Wiki API string matching + off-the-shelf text-only linker). The impact and implications of the flawed finetuning are not discussed. \n- I couldn't find some obvious baselines, such as doing only image-image or text-text matching. \n- The generative baseline SimplerVLM can/should do the same constrained decoding to ensure that the prediction is a valid entity.\n\n\n\n[Post-response] \n\nThanks for the response. I think the human performance is a valuable addition. I still think the paper could make the feasibility and baseline components more substantial (e.g., an actual analysis of impossible examples, instead of invoking existing datasets). I'd like to keep my score. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m31phEMNhW", "reviewer": "Reviewer_cfNC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new task, similar to the traditional text entity linking text where a textual entity is linked to its associated entry in Wikipedia. In this work, the authors propose to link \"visual\" entities to Wikipedia. Because the same entity could reasonably be linked to different Wikipedia entries (a photo of a car could be linked to \"car\" or the particular model), the authors propose to unambiguously link to Wikipedia by also incorporating a text query (e.g. \"what model is it?\").\nTo make progress on this task, the authors propose a new dataset, called OVEN-Wiki. The proposed dataset is constructed from existing datasets wherein all of the labels within the dataset are grounded to Wikipedia. In addition, the authors employed over 30 human annotators to annotate a subset of OVEN-Wiki.\nTo create a benchmark on this task, the authors formulate and evaluate a number of baselines using exsting VL models. They experiment and analyze the performance of using models like CLIP zero-shot at this task vs. finetuning models on their dataset.\nThe most successful baseline proposed by the authors leverages image-text pairs from the Wikipedia article and query image-text pair and essentially takes cosine distance to find the best match.", "summary_of_the_review": "The authors propose an interesting, impactful, and challenging new task called OVEN. They contribute a large-scale benchmark for this task and provide initial baseline results. The contributed dataset is of high quality and is the result of a large-scale human annotation effort. The baseline models are less novel, but are sufficient to establish a \"baseline\" for future work to compare against and beat. In sum, the paper makes a significant contribution likely to be of use to a wider audience and should be accepted.", "strength_and_weaknesses": "[Strengths]\nThe proposed task (OVEN) is highly interesting, realistic, and likely to be of high-impact. For example, methods developed on OVEN could reasonably be used to aid in many multimodal reasoning task, such as question answering or any tasks requiring reasoning over external knowledge. The proposed task is also highly challenging for most existing methods. For example, the Wikipedia articles often feature text and multiple images. A successful model should integrate the knowledge from the multiple images on the Wikipedia page to ensure a successful link.\n\nThe authors demonstrate that OVEN is an extremely challenging task requiring complex reasoning to adequately perform well. OVEN requires determining which of ~100k Wikipedia entities to link to. This is an enormous possible label space - scary even to those who work in zero-shot learning. Thus, the problem is a very interesting avenue for future research and is likely to get much interest.\n\nThe dataset is cleverly constructed and well-designed. The authors clearly invested significant resources in its construction, employing ~30 human linkers to gather thousands of ground truth links. The authors also cleverly re-use existing datasets from VQA and object recognition and note that some datasets feature diverse visual entities, while others feature diverse language queries - thus the dataset features an excellent balance of diversity.\n\nThe authors propose a number of baseline methods on their dataset that make use of CLIP or SimVLM and perform a detailed experimental analysis and ablation study. The authors make a number of key findings. For example, that not-all fine-tuned models were better than their untuned versions on the task. The authors' ablations are important and explore important aspects of the task or model such as how important input representations are for OVEN, the impact of fine-tuning on performance, and provide error analysis.\n\n[Weaknesses]\nOne weakness is the baseline models are quite simple / weak. I recognize that OVEN is primarily a dataset paper, however and that the primary contribution does not come from the baseline model, but rather the effort expended creating the dataset and formalizing the task.\n\nPerhaps I missed it, but how accurate are the human annotators at this task? Have the authors computed an inter-annotator agreement score? How often do humans agree on the Wikipedia entity link for the same image and query?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h6NJ8fRKKR1", "reviewer": "Reviewer_hwGY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents open-domain visual entity linking and contributes a large-scale dataset containing 4.89 Million and 730K examples in train and test sets respectively. The paper also presents several baselines for the task using state-of-the-art approaches. Visual entity linking for both seen and unseen entities have been reported. ", "summary_of_the_review": "The presented task is important and underexplored. The proposed dataset is large-scale but its coverage is limited and does not really try to solve visual entity linking as a standalone task. Evaluation measures are not well justified and some important baselines and literature have been dropped. \n\nPost-author response:\nI am not fully satisfied with the difference between VELD [A] and the proposed work provided in the author's response. Though the images in VELD come with captions and the proposed model (full) uses them, the obvious ablation for [A] is not to use any textual feature. They indeed perform such ablation, for example, in Table 3 in [A]. Therefore, VELD seems very relevant both model and dataset-wise. The setting proposed in this work is evaluating VQA and only implicitly visual entity linking. It would be great to perform VEL without any textual input (neither caption nor question, just an image and KG). With such a setting and appropriately defined task, the value of the proposed dataset will significantly increase.\n\nIt is good to see the response has now reported R@K and MRR-based results.\n\nI still have concerns about the baseline experiments. They are rather weak and do not really represent SOTA entity linking literature or multimodal entity linking literature. A simple hierarchical approach might have been a good competitor. For example, if a person is what is required to identify, I would rather prefer to use a specialized model for face recognition, and the same applies to other categories.\n\nWith these concerns, I am inclined toward sticking to my original rating. \n", "strength_and_weaknesses": "Strengths: \n\n+ A large-scale visual entity linking benchmark namely OVEN-Wiki has been presented. Visual entity linking (especially at large-scale) is an underexplored problem in the literature. It has utility in many downstream tasks such as knowledge-aware VQA, knowledge-aware image captioning, and in general better scene interpretation. Further, visual entity linking is also important in many applications such as news search and e-commerce. Therefore, this paper has definitely some value. \n\n+ Baselines using state-of-the-art pre-trained models have been presented. Evaluations are done for both seen and unseen entities. \n\nWeakness: \n\n- Literature and designed baselines are weaker. The following recent work also presents a dataset and model for visual entity linking task: \n\n[A] Qiushuo Zheng, Hao Wen, Meng Wang, Guilin Qi: Visual Entity Linking via Multi-modal Learning. Data Intell. 4(1): 1-19 (2022)\n\nTheir model looks closer to state-of-the-art entity-linking models in NLP literature. Why such models are not considered as one of the competitive baselines? Even the dataset presented in this paper can also be used to benchmark methods. Further, arguments in the paper such as “opens up possibilities to answer questions on entities that have not been learned before, and are thus UNSEEN.” Is not completely correct, as the following paper performs zero-shot VQA:\n\n[B] Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z. Pan, Zonggang Yuan, Huajun Chen: Zero-Shot Visual Question Answering Using Knowledge Graph. ISWC 2021: 146-162\n\t\nFurther, in a recent work namely WebQA, VQA on unseen object categories has also been shown. They also have proposed a dataset that has large coverage of visual entities. (One difference I see is they heavily rely on captions as well rather than completely doing visual reasoning). Nevertheless, it is a very relevant paper:\n\n[C] Yingshan Chang, Guihong Cao, Mridu Narang, Jianfeng Gao, Hisami Suzuki, Yonatan Bisk:\nWebQA: Multihop and Multimodal QA. CVPR 2022: 16474-16483\n \n- Regarding the proposed dataset: The dataset surely has some merits, especially the scale and diversity. However, it is not clear why one of the most prominent visual entities namely PER (or public figures) is not considered. I understand that including public figures also make the problem closer to face recognition at a very large scale. But, possibly it might have yielded a stronger dataset. There are several such public image datasets such as KVQA, oxford’s people in places that could have been used. Further, In NLP literature, the context has played important role in linking entities. The same may also be very important in visual entity linking. I am not sure if such things can be explored in the proposed dataset. Furthermore, how much role does natural language understanding play in the task? Are there complex queries ( as the paper says ambiguous queries are rewritten)? Isn’t it good to separate out visual entity linking from NLU tasks?\n\n- Regarding evaluation: Traditionally, rank@K and mean reciprocal ranks are used to measure entity linking. In problems like visual entity linking it is often important to get top-k predictions correct. It is not clarified why f-scores are preferred over such established and seemingly more suitable evaluation measures.\n\n- Does the paper consider some hierarchical approach where first-level high-level categories (such as animal, plant, building, etc.) are classified? I believe such an approach may improve performance. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "14XsX5LOIlp", "reviewer": "Reviewer_8tsg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly focuses on linking to the entity out of all entities in the knowledge base. A benchmark dataset is collected by linking all existing labels to Wikipedia entities when possible, using a state-of-the-art entity linking system and human annotators, creating a diverse and unified label space. It requires models to recognize and link visual content to both a small set of seen entities as well as a much larger set of unseen entities. Also, it requires models to generalize to previously unseen intents that may require more fine-grained reasoning.", "summary_of_the_review": "This paper formally defined a new task that reflects a more realistic scenario. And it collected a new dataset and proposed corresponding novel evaluation metrics.\nEven though there are some unclear clarifications, in general, it benefits the community.\nOverall, I'm leaning to accept it.", "strength_and_weaknesses": "*[Strength]*\n1. This work constructs a new knowledge-based dataset, reflecting a more realistic scenario that there will never be enough training data to cover all knowledge-base entities, especially when the number of knowledge-base entities is constantly growing.\n2. It defines a new evaluation metric for the proposed task.\n\n*[Weakness]*\n1. In equation 1, it will be better if add the concept that the prediction may be unseen in the question and image input.\n2. What are the two questions in Figure 2?\n3. Is the example mentioned in the paper defined as image-text pairs? If so, does that mean the unseen examples partially include unseen entities?\n4. Also, in the experiments section, there is seen/unseen entity split and query split, what is the relation between them and the seen/unseen examples in Figure 3?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject