Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a system for biomedical entity recommendation. The system performs entity recommendation in a pipeline manner with three steps: named entity recognition, relation extraction, and entity recognition. The paper shows that the recommendation accuracy can be enhanced if multiple ontologies are used for recognising and linking the entities. The paper is well-written and easy to follow

My major concern is that whether the experimental results provide sufficient evidence to support the hypothesis that the authors formulate: if multiple different ontologies are used for recognising entities, the accuracy of entity recognition increases. As shown in Figure 2, the accuracy of using all ontologies (ALS-all) outperforms any other dataset with only one ontology (e.g., ALS-chebi which only includes entities in CHEBI ontology). This might be a positive sign showing that other ontologies are contributing to model’s understanding about how to recommend CHEBI entities. However, it might be also simply because of the model’s learning curve: the data in ALS-all is a super set of ALS-CHEBI, and the model simply learns more with more available data, no matter which ontology the extra data comes from. And this can be evident by that the accuracy in Figure 2 (ALS-all>ALS-chebi>ALS-go>ALS-hp~=ALS-do) is actually roughly proportional to the data set size if we look at the size of each data set in Table 4. As such, the authors should consider to another baseline scenario where the entities are randomly sampled, without considering the ontologies.

Secondly, the rating matrix is not implicit feedback. Implicit feedback means the scenario where rating is not available and only users’ actions are available. In implicit feedback setting, the interaction between users and items is only recorded as 0 or 1 (interacted or not). Therefore, pairwise ranking is pervasively used for learning. The ALS algorithm is usually preferred in the setting of explicit feedback, rather than implicit feedback. 

Reasons to accept: well-written, data set can be useful
Reasons to reject: the design of the experiments cannot fully support the claim in the paper

review_title: Need to improve the experiment design

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Core review**:

This paper proposes a pipeline for the automatic extraction of entities and relations from the CORD-19 dataset and recommendation of relevant entities in those papers. The authors apply a system (MER, Couto and Lamurias, 2018) for the automatic extraction of entities which is seeded with 4 ontologies in the biomedical domain. A total of 9000 documents from CORD-19 are automatically annotated with entities, from which 100 documents are manually curated and a small subset is used to evaluate the inter-annotator agreement and manually annotate relevant relations (relations are defined in the ontologies used). Afterwards, a recommendation technique (LIBRETTI) is applied by considering author-entity-mention counts as the user-item-rating triplets. Thus, as I understand, the overall pipeline proposed has the purpose of suggesting an author of a paper in the annotated dataset with novel entities that might be of interest for that author and not mentioned in their papers.

The main hypothesis of the paper is that the use of ontologies from different domains for the NER phase improves the quality of the recommendation. While this hypothesis is effectively demonstrated by the experimental results, I believe this is a direct consequence of the fact that these ontologies are from different domains and thus mostly disjoint, which in my opinion renders the hypothesis self-evident. With respect to the results, the significance of the metrics provided is hard to evaluate without some comparison with alternative systems or at least some baselines.

Although I consider there is value in the proposal, since building such a recommendation system at a large scale (e.g., at the scale of Semantic Scholar or Google Scholar) would be extremely valuable for researchers, I still consider a significant effort is necessary both to improve the extraction and recommendation procedures, but also regarding manual annotation, evaluation, and the actual development of such a tool. Finally, I commend the authors for their effort. I recommend working on improving the motivation of the research; at least personally, it was hard for me to understand the underlying motivation for recommending entities, and I think this is a very important point.

**Reasons to accept:** The automatically and manually annotated corpora can be valuable resources for the community. Also, the source code is provided, which improves reproducibility and fosters collaboration.

**Reasons to reject:**  The significance of the results is hard to evaluate since there is no comparison with alternatives or baselines.

**Other considerations:**

- About the phrase "an IAA of 0.2978 which indicates fair agreement" -- It is my understanding that a fair Kappa agreement is considered above 0.70 unless the authors are using a different formulation, in which case I would recommend the authors to explicitly state and justify the threshold above which they consider the agreement "fair".

- The Precision, Recall and F1 scores are considered "high" by the authors, however, since there is no baseline or comparison it is hard to determine the significance of these measures.

- Recurrent typos in page 2: *"¡article,topic,cardinality¿"*, *"¡author,entity,rating¿"*, in Page 4: *"¡user, item, rating¿"*, ... (I believe this stems from the incorrect to use of $<$ and $>$ in LaTeX)
review_title: An interesting initial approach to recommendation systems in academic domains. The evaluation requires additional work.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors develop datasets for NER and RE extraction of COVID-19 literature. They used the generated data to generate concept recommendations.

Reasons to accept:
- The developed datasets might be valuable to further researchers

Reasons to reject:
- I do not understand why would it be useful to recommend concepts to someone. It might be because of my limited expertise on the field.


review_title: Deployment of recommender system based on multiple NER ontologies