Entity Retrieval for Answering Entity-Centric Questions
Authors: Hassan Shavarani, Anoop Sarkar


Reviewer paper summaries: Authors point out that standard dense and sparse retrievers fail on entity-focussed questions (relying on relevant past work). An entity-focussed retrieval step can replace these retrievers: given a question, link the entities to an entity set (like Wikipedia article titles) and retrieve the first paragraph (W tokens) from the article page.

The first set of analyses uses the "gold" entity linking in 2 entity focussed QA datasets: EntityQuestions and FactoidQA. They show that with the "gold" entity linking, the Entity Retriever returns documents with higher retrieval accuracy, NDCG, and MRR than existing dense (DPR, ANCE) and sparse (BM25) retrievers. These passages lead to higher downstream QA accuracy using fewer documents.

Next, when the "gold" entities are not labeled, they use the SPeL entity linker to identify relevant Wiki articles in the questions. They additionally evaluate the system on a dataset that does not contain explicit entity annotations: StrategyQA. They show that even with this noisy entity linking, the retriever leads to better retrieval performance than the baselines on the datasets.

The proposed approach is an intuitive system for entity-focused QA. With improvements in entity linking, the gap between "gold" and noisy labeling is bound to decrease. In some sense, the results with the "gold" entity linking demonstrate the ceiling of performance that can be achieved. It is an open question to identify what is required for the remaining headroom.
Summary of strengths: - Experiments with the gold entity linking set the ceiling for performance that can be achieved with the baseline Entity Retriever approach
    - Authors demonstrate that this linking allows for better retrieval with fewer documents than competitive sparse and dense methods
- The choice of entity linker was well-motivated
    - Results with the entity linker demonstrate the scope for improvement from the current SotA linkers to the gold entity linking
Summary of weaknesses: 1. Related work does not describe and draw connections to long lines of research in entity-centric (e.g. [1,2]) and hyperlink-focused (e.g. [3]) retrieval. Some of these systems use an entity linker as an explicit first step, like the proposed work.
2. The use of questions from the "500K most linked" Wikipedia pages for EntityQuestions raises two issues:
    1. It goes against the core contribution of EntityQuestions and ignores the tail entities. Your system could actually show even better improvement in the tail.
    2. It (possibly) doesn't demonstrate how entity linking may be affected by the entity's popularity. SPeL may work better on head entities than tail entities, meaning that ER results with SPeL may be worse on the tail.
3. Is entity linking necessary? What happens if you restrict DPR and BM25 to just the first 100 words of each Wikipedia page?
    - The concern here is that results may be better due to different underlying corpora rather than explicitly using entity information.
4. The paper is missing retrieval metrics with SPeL entity linking.
5. There is a concerning observation here: while retrieval accuracy, MRR, and NDCG seem to increase from ER50w (Entity Retriever with 50 words) to ER1000w, there is a much less noticeable increase in the downstream QA performance.

[1] Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, William W. Cohen. Differentiable Reasoning over a Virtual Knowledge Base. ICLR. 2020
[2] Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William Cohen. Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. EMNLP 2018
[3] Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, Caiming Xiong. Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering. ICLR. 2020
Comments, suggestions and typos: Clarifications
---
1. Lines 450-456: What do you do when the entity linker links 0 or more than 1 entity? What is ER100w in this context?
2. Lines 450-456: How do you estimate entity linking accuracy on StrategyQA?
3. Lines 459-461: What do you mean by this? So far, it was conveyed that all EntityQuestions and FactiodQA questions had entity annotation.
4. Table 5: Does the run-time of ERSp100w include the time for entity-linking?

Suggestions
---
1. Figure 1 does not convey much information since the proposed system is intuitive. It would be better to actually highlight how the passages from Dense/Sparse retrieval differ from passages retrieved by your system.
2. Lines 52-55: Doesn't this connect more to the previous paragraph about poor retrieval for entity-centric QA?
3. Lines 58-63: Maybe you should talk a bit more about why you think the proposed approach is expected to work better.
4. Lines 113-122: Description of ANCE can be compressed since it overlaps a lot with DPR. Ideally, you could have used a different retriever family like Colbert.
Overall assessment scores: 3