Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose a new task, similar to the traditional text entity linking text where a textual entity is linked to its associated entry in Wikipedia. In this work, the authors propose to link "visual" entities to Wikipedia. Because the same entity could reasonably be linked to different Wikipedia entries (a photo of a car could be linked to "car" or the particular model), the authors propose to unambiguously link to Wikipedia by also incorporating a text query (e.g. "what model is it?").
To make progress on this task, the authors propose a new dataset, called OVEN-Wiki. The proposed dataset is constructed from existing datasets wherein all of the labels within the dataset are grounded to Wikipedia. In addition, the authors employed over 30 human annotators to annotate a subset of OVEN-Wiki.
To create a benchmark on this task, the authors formulate and evaluate a number of baselines using exsting VL models. They experiment and analyze the performance of using models like CLIP zero-shot at this task vs. finetuning models on their dataset.
The most successful baseline proposed by the authors leverages image-text pairs from the Wikipedia article and query image-text pair and essentially takes cosine distance to find the best match.
strength_and_weaknesses: [Strengths]
The proposed task (OVEN) is highly interesting, realistic, and likely to be of high-impact. For example, methods developed on OVEN could reasonably be used to aid in many multimodal reasoning task, such as question answering or any tasks requiring reasoning over external knowledge. The proposed task is also highly challenging for most existing methods. For example, the Wikipedia articles often feature text and multiple images. A successful model should integrate the knowledge from the multiple images on the Wikipedia page to ensure a successful link.

The authors demonstrate that OVEN is an extremely challenging task requiring complex reasoning to adequately perform well. OVEN requires determining which of ~100k Wikipedia entities to link to. This is an enormous possible label space - scary even to those who work in zero-shot learning. Thus, the problem is a very interesting avenue for future research and is likely to get much interest.

The dataset is cleverly constructed and well-designed. The authors clearly invested significant resources in its construction, employing ~30 human linkers to gather thousands of ground truth links. The authors also cleverly re-use existing datasets from VQA and object recognition and note that some datasets feature diverse visual entities, while others feature diverse language queries - thus the dataset features an excellent balance of diversity.

The authors propose a number of baseline methods on their dataset that make use of CLIP or SimVLM and perform a detailed experimental analysis and ablation study. The authors make a number of key findings. For example, that not-all fine-tuned models were better than their untuned versions on the task. The authors' ablations are important and explore important aspects of the task or model such as how important input representations are for OVEN, the impact of fine-tuning on performance, and provide error analysis.

[Weaknesses]
One weakness is the baseline models are quite simple / weak. I recognize that OVEN is primarily a dataset paper, however and that the primary contribution does not come from the baseline model, but rather the effort expended creating the dataset and formalizing the task.

Perhaps I missed it, but how accurate are the human annotators at this task? Have the authors computed an inter-annotator agreement score? How often do humans agree on the Wikipedia entity link for the same image and query?
clarity,_quality,_novelty_and_reproducibility: The paper is clear and of high quality. The authors provide numerous results in their supplementary material.
The dataset is highly novel and will be a useful multimodal reasoning task with important downstream use cases. 
The paper describes the baselines and method in sufficient detail to reproduce the approach the authors have taken. The dataset will  be publicly released and thus the paper is reproducible.
summary_of_the_review: The authors propose an interesting, impactful, and challenging new task called OVEN. They contribute a large-scale benchmark for this task and provide initial baseline results. The contributed dataset is of high quality and is the result of a large-scale human annotation effort. The baseline models are less novel, but are sufficient to establish a "baseline" for future work to compare against and beat. In sum, the paper makes a significant contribution likely to be of use to a wider audience and should be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper mainly focuses on linking to the entity out of all entities in the knowledge base. A benchmark dataset is collected by linking all existing labels to Wikipedia entities when possible, using a state-of-the-art entity linking system and human annotators, creating a diverse and unified label space. It requires models to recognize and link visual content to both a small set of seen entities as well as a much larger set of unseen entities. Also, it requires models to generalize to previously unseen intents that may require more fine-grained reasoning.
strength_and_weaknesses: *[Strength]*
1. This work constructs a new knowledge-based dataset, reflecting a more realistic scenario that there will never be enough training data to cover all knowledge-base entities, especially when the number of knowledge-base entities is constantly growing.
2. It defines a new evaluation metric for the proposed task.

*[Weakness]*
1. In equation 1, it will be better if add the concept that the prediction may be unseen in the question and image input.
2. What are the two questions in Figure 2?
3. Is the example mentioned in the paper defined as image-text pairs? If so, does that mean the unseen examples partially include unseen entities?
4. Also, in the experiments section, there is seen/unseen entity split and query split, what is the relation between them and the seen/unseen examples in Figure 3?
clarity,_quality,_novelty_and_reproducibility: The paper clarified its contributions and approach most clearly. 
In general, the paper is novel, can be reproduced and is of fair quality.
summary_of_the_review: This paper formally defined a new task that reflects a more realistic scenario. And it collected a new dataset and proposed corresponding novel evaluation metrics.
Even though there are some unclear clarifications, in general, it benefits the community.
Overall, I'm leaning to accept it.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents open-domain visual entity linking and contributes a large-scale dataset containing 4.89 Million and 730K examples in train and test sets respectively. The paper also presents several baselines for the task using state-of-the-art approaches. Visual entity linking for both seen and unseen entities have been reported. 
strength_and_weaknesses: Strengths: 

+ A large-scale visual entity linking benchmark namely OVEN-Wiki has been presented. Visual entity linking (especially at large-scale) is an underexplored problem in the literature. It has utility in many downstream tasks such as knowledge-aware VQA, knowledge-aware image captioning, and in general better scene interpretation. Further, visual entity linking is also important in many applications such as news search and e-commerce. Therefore, this paper has definitely some value. 

+ Baselines using state-of-the-art pre-trained models have been presented. Evaluations are done for both seen and unseen entities. 

Weakness: 

- Literature and designed baselines are weaker. The following recent work also presents a dataset and model for visual entity linking task: 

[A] Qiushuo Zheng, Hao Wen, Meng Wang, Guilin Qi: Visual Entity Linking via Multi-modal Learning. Data Intell. 4(1): 1-19 (2022)

Their model looks closer to state-of-the-art entity-linking models in NLP literature. Why such models are not considered as one of the competitive baselines? Even the dataset presented in this paper can also be used to benchmark methods. Further, arguments in the paper such as “opens up possibilities to answer questions on entities that have not been learned before, and are thus UNSEEN.” Is not completely correct, as the following paper performs zero-shot VQA:

[B] Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Z. Pan, Zonggang Yuan, Huajun Chen: Zero-Shot Visual Question Answering Using Knowledge Graph. ISWC 2021: 146-162
	
Further, in a recent work namely WebQA, VQA on unseen object categories has also been shown. They also have proposed a dataset that has large coverage of visual entities. (One difference I see is they heavily rely on captions as well rather than completely doing visual reasoning). Nevertheless, it is a very relevant paper:

[C] Yingshan Chang, Guihong Cao, Mridu Narang, Jianfeng Gao, Hisami Suzuki, Yonatan Bisk:
WebQA: Multihop and Multimodal QA. CVPR 2022: 16474-16483
 
- Regarding the proposed dataset: The dataset surely has some merits, especially the scale and diversity. However, it is not clear why one of the most prominent visual entities namely PER (or public figures) is not considered. I understand that including public figures also make the problem closer to face recognition at a very large scale. But, possibly it might have yielded a stronger dataset. There are several such public image datasets such as KVQA, oxford’s people in places that could have been used. Further, In NLP literature, the context has played important role in linking entities. The same may also be very important in visual entity linking. I am not sure if such things can be explored in the proposed dataset. Furthermore, how much role does natural language understanding play in the task? Are there complex queries ( as the paper says ambiguous queries are rewritten)? Isn’t it good to separate out visual entity linking from NLU tasks?

- Regarding evaluation: Traditionally, rank@K and mean reciprocal ranks are used to measure entity linking. In problems like visual entity linking it is often important to get top-k predictions correct. It is not clarified why f-scores are preferred over such established and seemingly more suitable evaluation measures.

- Does the paper consider some hierarchical approach where first-level high-level categories (such as animal, plant, building, etc.) are classified? I believe such an approach may improve performance. 

clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear and easy to read. Though there are a few language errors that need to be fixed such as:
Given these inputs the goal -> Given these inputs, the goal
For final performance, -> For the final performance,
De Cao et al. (2021) has shown -> De Cao et al. (2021) have shown
large scale contrastive pretraining -> large-scale contrastive pretraining

The proposed task, dataset, and model are not completely novel, as discussed in the weakness section. Regarding reproducibility, sufficient details are there in the paper to implement the models. Do the paper also plan to make dataset and baseline implementations public? 
summary_of_the_review: The presented task is important and underexplored. The proposed dataset is large-scale but its coverage is limited and does not really try to solve visual entity linking as a standalone task. Evaluation measures are not well justified and some important baselines and literature have been dropped. 

Post-author response:
I am not fully satisfied with the difference between VELD [A] and the proposed work provided in the author's response. Though the images in VELD come with captions and the proposed model (full) uses them, the obvious ablation for [A] is not to use any textual feature. They indeed perform such ablation, for example, in Table 3 in [A]. Therefore, VELD seems very relevant both model and dataset-wise. The setting proposed in this work is evaluating VQA and only implicitly visual entity linking. It would be great to perform VEL without any textual input (neither caption nor question, just an image and KG). With such a setting and appropriately defined task, the value of the proposed dataset will significantly increase.

It is good to see the response has now reported R@K and MRR-based results.

I still have concerns about the baseline experiments. They are rather weak and do not really represent SOTA entity linking literature or multimodal entity linking literature. A simple hierarchical approach might have been a good competitor. For example, if a person is what is required to identify, I would rather prefer to use a specialized model for face recognition, and the same applies to other categories.

With these concerns, I am inclined toward sticking to my original rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a dataset for vision-language entity linking. 

Both queries (q) and entities (e) have image and text component, so a score needs to be defined for the quadruple (img(q), txt(q), img(e), txt(e)).

The training(~5m queries)/validation/test splits are automatically constructed from existing image recognition/retrieval and visual QA datasets. Separately, the paper provides ~5k manually corrected gold examples for evaluation. 

The main baseline is mixed-modality dual encoders based on CLIP-large, achieving the harmonic mean of seen/unseen accuracies of ~15 on dev/test and ~20 on gold.  


strength_and_weaknesses: STRENGTHS
- The task is interesting without a doubt.
- The human annotated set can be a high quality evaluation set for the task.
- The baseline models are reasonable.

WEAKNESSES
- No performance upper bound. It is not clear if this problem is even solvable. In the first example in Figure 1, how can we expect the model to infer the plane manufacturer? There does not seem to be enough information in all modalities. (I think this example comes from the gold set? I wonder how the human annotators deemed it correct.)
- On a related note, the baseline performance is really weak. This needs to be addressed in conjunction with the task feasibility. 
- Other than the gold set, the dataset itself is automatically constructed with basic means (Wiki API string matching + off-the-shelf text-only linker). The impact and implications of the flawed finetuning are not discussed. 
- I couldn't find some obvious baselines, such as doing only image-image or text-text matching. 
- The generative baseline SimplerVLM can/should do the same constrained decoding to ensure that the prediction is a valid entity.



[Post-response] 

Thanks for the response. I think the human performance is a valuable addition. I still think the paper could make the feasibility and baseline components more substantial (e.g., an actual analysis of impossible examples, instead of invoking existing datasets). I'd like to keep my score. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear. The task is novel and interesting.
summary_of_the_review: The paper identifies the interesting problem of vision-language entity linking and provides a dataset, but the utility of the dataset except for the manually annotated portion is somewhat unclear and the paper doesn't get to the bottom of the problem (i.e., not answering natural questions like what is the best possible performance).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a benchmark for linking Wiki knowledge to a visual entity (e.g., an image with a label) given a query specifying what domain knowledge the entity is linking to. The authors provide human annotation ground-truth and several baselines for the proposed benchmark.

Strength:
A new benchmark called OVEN-Wiki has been proposed for visual entity linking that may support knowledge based vision-language tasks such as KVQA and Knowledge image captioning. 

Weakness:
As a dataset paper, it lacks a significant body of evaluations, metrics, potential challenges, and risks. The weakness has been raised by all the reviewers. 
justification_for_why_not_higher_score: All the reviewers and AC appreciated the contributions of the benchmark, however, it is not comprehensive enough, e.g., lacking strong baselines, task potentials, and challenges. Therefore, AC believes that the readers in the community cannot truly benefit from the work rather than just the acknowledgment of the benchmark.
Decision: Reject