Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to learn entity representations by matching entities to the context it occurs in. It also shows that using these representations is very effective for a wide variety of down-stream entity-centric tasks such as entity typing, linking, and answering entity centric trivia questions. They train the model using a corpus of entity linked Wikipedia contexts (sentences unto length 128 tokens). The context is encoded with a BERT model and the CLS representation is used as the representation of context. After obtaining the representation, they train the entity embedding (present in the sentence) to be similar to the context embedding. They test their embeddings on few down-stream entity-centric tasks — linking, typing and trivia question answering.

Strengths:
1. They try the entity representations on a wide variety of entity-centric tasks and get reasonable results.

Weaknesses:
1. The biggest weakness of the paper is wrt novelty. Masking out entities and training to context is not a new idea. As pointed by the paper, Yamada et al., 2017 have a very similar objective and it is not very clear from the paper what is the additional contribution that this paper makes. Is using pretrained LMs the major difference? If not, it would have been nice to see Yamada et al’s results with BERT. Over all, this paper needs to make its own contribution clear compared to Yamada et al., 2017.
2. The paper needs to be written more clearly at several places. Few examples are, even though in entity linking results (Table 1) the model achieves 83.0 with other papers achieving 90.9. I didnt see a discussion on how to close the gap. Even in the coNLL benchmark, the initial results of the paper is significantly behind. Claims like “CoNLL -Aida is known to be restricted and idiotic-synctatic domain” should be backed by detailed analysis or atleast a citation. Even after finetuning on the CoNLL benchmark, the result is 2.2 points behind state of the art and no discussions have been provided. As a result, I think the entity linking section needs major re-writing and explanation of the results.
3. The paper makes an interesting observation that masking of entities is better for typing tasks and it affects linking performance, because spelling features are really important for linking. It would be interesting to see a discussion on what could be done to remedy this. Because if we have to retrain entity embeddings for different tasks, then it goes against the hypothesis of the paper which is to use entity representations for a wide variety of down-stream tasks.
4. I found it confusing to read the setup in sec 5.4. especially where it says we represent each category with three random exemplars. Initially I thought 3 randomly sampled entities formed a category, which didnt make sense, but from figure 3, I think I understood that you first pick a category from Typenet and Wikipedia and then 3 entities are sampled from there. Is that correct? Regarding the results, can the poor results of Yamada et al., can be understood by the fact that it was trained using smaller number of categories? Also, why are the numbers wrt All entities left blank in Table 4. Given that your model is similar, I am assuming its easy to retrain Yamada et al and test it on the all entities benchmark?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
This paper aims to learn entity representations by aggregating all the contexts that an entity appears in based on English Wikipedia. 

The idea is very simple, basically it represents each entity as a vector, and also represents each context as a vector, and maximizes the cosine similarity between the two vectors using a negative-sampling training objective. The training process is similar as word2vec, and they leveraged all the hyperlinks in Wikipedia, and used BERT to encode the context (instead of learning a context vector for each word). 

As a result, the paper demonstrates that this set of entity embeddings are highly useful, and they were evaluated on 1) entity-level typing 2) entity linking 3) few-shot category reconstruction 4) answering trivia questions (TriviaQA).

I think this is a nice empirical paper and the experiments are thorough. If they are going to release the entity embeddings, that would also benefit the community a lot and also encourage more research in this direction.

I am a bit concerned about the novelty of the approach. It is a bit surprising that nobody has experimented with this before. It seems that Yamada et al took a very similar approach but used simple bag-of-words approaches to encode the context (instead of BERT).  To me, this paper may be better for the NLP community but it should be fine to the ICLR community too.

I am also not completely sure how strong the evaluation results are indeed, esp. related to the comparisons with Yamada et al, 2017, given the approaches are similar. 

- It seems to be on par with or slightly worse than Yamada et al 2017 on entity linking.

- For the category completion task, RELIC is doing much better than Yamada et al 2017 on more complex Wikipedia categories but I am not sure if it is a completely fair comparison. It’d be great if the authors can discuss all the key differences between the two approaches, from the model design to all the experimental details, that would help clear out all these confusions. I have read the related work section but can't figure out all the details.

- The entity typing results are very strong. I also like the TriviaQA experiments but the numbers are way behind the standard reading comprehension results. 


Minor comment:
- Why sec 5.3 (effect of masking) is listed together with other evaluation tasks? Isn’t better to move it to analyses/ablation studies?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper describes a new way of learning context dependent entity representations that are capable of encoding fine-grained entity types. This is realised by matching entities to all their contexts and thereby encoding all their properties.
The contribution is RELIC, a table of entity representations that are learned given the above objective. This table, RELIC, can be used in various tasks like entity typing, entity linking and question answering. Given a corpus of entities and their respective contexts, the probability of an entity occurring under its context is maximised (additionally, the probability is minimised for negative samples). The data is taken from a Wikipedia dump. Without adding further information, the RELIC table can directly be used for entity linking.
In the entity linking task, RELIC achieves a lower precision at two benchmarks (CoNLL-Aida, TAC-KBP 2010) than other approaches. This can be tackled by fine-tuning the RELIC table, which is done with the training set from the CoNLL-Aida task. With fine-tuning, comparable results are achieved.
For entity typing, RELIC embeddings of entities are used as input for a 2-layer FF network, which then outputs which types belong to the entity. The FIGMENT and TypeNet datasets are used as benchmark here and it is shown that the RELIC based approach outperforms the other approaches.
RELIC is also used for a category completion task, where a category is represented by a centroid of three randomly sampled entities belonging to it. The entities from RELIC are then ranked with the dot-product with the centroid. Two tasks are used to measure the performance here: the TypeNet completion task and a Wikipedia-based task. Compared to embeddings of a 2017 approach, the RELIC embeddings perform better. 
Finally, a QA task is performed using RELIC. Here, the questions are modelled as contexts and the entity which is closest in terms of cosine similarity is taken as answer. The performance is below the upper bound from 2018, but better than a classifier system from 2017.

Overall, the paper is well structured and manages to show that RELIC is a versatile tool on which various tasks can be performed. The technical details are well covered and the evaluation is done in a detailed way.
The contribution of the paper is mostly in the definition of context-entity pairs as input to a transformer model and in empirical evaluations. I specifically found the findings in 5.3 interesting. The technological contribution is rather limited.

Minor comments: At page 3, chapter 3.1 there is a word duplication […]to correctly match match[…] that should be corrected. Additionally, on page 5 it should be mentioned that the values in table 1 are precision values.
review_title: Official Blind Review #1

Meta-review: The paper describes an approach for learning context dependent entity representations that encodes fine-grained entity types. The paper includes some good empirical results and observations, but the proposed approach is very simple but lacks technical novelty needed to top ML conference; the clarify of the presentation can also be improved. 
Decision: Reject