Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a brand new approach for entity retrieval, which leverages an encoder-decoder architecture to generate the target entity directly. To tackle the problem of invalid generation, a trie-constrained beam search is used for decoding. The author performs intensive experiments on entity disambiguation, entity linking, and document retrieval tasks and achieves new SOTA or competitive results on over 20 datasets. Although the paper does not come up with new architecture or elaborately designed neural components, I believe this paper is worth reading for the community, including how this paper redefines the problem.

Strengths:
1. A brand new perspective on entity retrieval. Clever use of text generation methods.
2. Compared to previous methods, GENRE is memory saving.
3. This paper provides comprehensive experiments on 3 important tasks, over 20 datasets, showing the robustness of this method. 

Weaknesses: 
1. Lack of clarity at many places: 
a) It's clearer to demonstrate how you reformulate the input and output for the 3 tasks, as they seem very different compared to previous methods
b) It's better to add some description about the baseline systems
c) I'm especially interested in the decoding details on end-to-end entity linking,  it's better to use a formular to demonstrate how you compute the log-probabilities.

Questions:
1. In the entity disambiguation task, it seems that for each inference step, GENRE only generates one entity. Is it possible to generate all entities simultaneously as what GENRE does in end-to-end entity linking? 
2. From table 5, an impressive result is that it seems that the pre-trained model is good at memorizing, even if the identifier is numeric. But from table 3, some of the baseline systems also benefit from pretrained models, while achieve much lower results. What do you think causes this difference? 

review_title: The paper proposes a brand new approach for entity retrieval, which leverages an encoder-decoder architecture to generate the target entity directly. Intensive experiments on entity disambiguation, entity linking, and document retrieval tasks prove the effectiveness of the approach.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper proposed to use autoregressive approach to solve entity-based problems. They proposed a uniform framework and showed that their model achieved the state of the art performance on 3 different types of tasks (~20 datasets). The GENRE model also significantly reduced the memory usage compared to previous models that stored a big memory table. It's also capable of linking novel entities at inference time. This paper is clearly written. The experiment results are convincing.

One limitation of this paper is that they required the vocabulary of the entities to be the ones that have a Wikipedia page. And that their model relied on copying the surface form of entities (as suggested in the paper). From the experiments, the "copying" approach worked very well on Wikipedia entities, that are often common entities. How can you improve the model to link rare entities not in the Wikipedia pages? 

Another concern is the efficiency at inference time. Compared to the models with a large entity memory whose retrieval is performed with Maximum inner product search, how is the efficiency of your decoding strategy?
review_title: Review #1: Great paper with convincing results.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Summary

This paper proposes to tackle the entity linking task using a sequence-to-sequence neural model, trained by producing unique entity names, in autoregressive fashion. The paper makes a case that this approach can scale better with larger entity vocabularies than previous methods with dedicated entity representations both in terms of memory as well as computation costs. The model is studied under a number of tasks including entity disambiguation, entity linking and document retrieval for question answering.

### Strong and weak points

Strong points:
- The work is well motivated: I believe there are, or will be, many systems where retrieving information about billions of entities is useful. Making simpler and more efficient models to deal with larger entity vocabularies is important.
- Doing seq2seq entity linking is really a novel idea and is surprising that it works so well for several of the datasets presented.
- The idea of constrained decoding using a Trie is neat and makes intuitive sense.
- The empirical evaluation in the paper is quite exhaustive, in terms of number of datasets.

Weak points:
- There is no discussion or analysis on the performance of this new model on “tail entities” (entities that have few examples in the training set). A believe such a discussion would be interesting for two reasons: (1) if one wishes to use this type of model for much larger entity vocabularies, it is likely that a larger fraction of entities will have low number of examples, and (2) one effect of contrastive learning (e.g., negative sampling) has on systems with explicit entity representations is some implicit training of _all_ entities, which is lacking in the described autoregressive proposal.
Finally on this topic of tail entities: the “IDs” experiment, shown in Table 5, indicates that when entity mention and decode target are not related, performance suffers. Similarly, for “tail entities” that are ambiguous with another “popular entity”, the model may be biased with the popular entity (results from “Cold Start” may support this hypothesis). Hard to say without some analysis.

- While the constrained decoding is really interesting, it was not clear from the paper how crucial this was for good performance. Is this something absolutely critical for performance overall, or does it provide a modest performance improvement? While it may be clear to the authors, it would be highly informative to explicitly describe the performance impact on unconstrained beam decoding.

### Recommendation

Overall, my recommendation is to accept this paper to ICLR. I believe the problem of representing entities in natural language systems is of high interest to the ICLR and NLP communities. The ideas in this paper are novel, the paper is well written and the empirical evaluation is well executed.

### Questions for authors

- See comment above regarding analysis of “tail entities”. Could any further insights on this topic be added to the paper? 
- See comment above about ablating the constrained beam decoding. This would not need to be a long or complex analysis. Simply 1 or 2 data points for the reader to understand the magnitude of the importance of this decoding.

review_title: Novel idea on entity retrieval/linking with well executed experiments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper introduces a new method to retrieve entity by auto regressively generating unique entity name as a sequence of word pieces, instead of pinpointing the ID representing an entity. This method stands out in novelty compared to existing various entity retrieval methods, which always assigns a single ID to each entity. Practically, the proposed method has two nice properties: (1) When the entity vocabulary is very large, this approach requires less parameter space and memory compared to other methods (as shown clearly in Table 4) (2) The model can address novel entities, which was unseen during the training. The paper is clearly written and extensively evaluated on three relevant tasks, entity disambiguation, entity linking, and entity retrieval.

I have one big concern with the current format of the presentation. In the current draft, It is not very clear whether the strong gain is coming from large scale pretr aining or the proposed method itself.  To this end, the ablations shown in Table 7 and Table 8 should be reported in the main paper, with clear explanations. As we all know, the model architecture (which is the focus of this paper) cannot be properly evaluated when the training set up is different (i.e., how much pretraining has been done, on what dataset?).  Could you elaborate on this? In appendix, there's only result tables without in-line explanations. 

The experiments on cold start, as well as table 5 which shows performances on entities divided by name match is pretty cool!

If space allows, adding some more analysis on what types of entities do this model do better compared to other methods would be interesting (would lengthier names easier or harder? would it do better on popular entities or more long tail ones? are these systems complementary to existing methods or mostly succeed and fail on the same set of examples?) 

 
review_title: Official Review #3

Meta-review: This is a novel, simple, and experimentally well-supported new idea for entity linking.  The key insight is to perform entity linking by producing meaningful entity names with seq2seq approaches, and the big surprise is how well this works experimentally (at least for wikipedia-style entities).  Very nice paper!

Decision: Accept (Spotlight)