Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Dear authors, 

This paper is about the development of a search engine for Proteins-compounds extracted from CORD-19 dataset. 
The pipeline of the work consists in extracting the entities from the abstracts using BERN and normalizing these entities. Then the system uses the BioBERT, previously tuned using the ChemProt dataset, for relation extraction between the entities (e1 for chemicals and e2 for proteins). Finally, they create an inverted index. 

I think this is a valuable work, well written, however a little confused in its structure. 

The introduction provides a good insight of the work, but I was expecting more information here. 
The related work section needs more work. This section talks about NER, BERN, but it does not present RW related to search engines, thus, when the authors state that this is the first search engine that “uses relation extraction to construct an inverted index of related biochemical entities”, I thought that I need more info about search engines provided in the RW. 

In Methods, the paper presents very well the datasets used, I have just some issues:
- Why do you consider CPR:0 as 1? Is there a certain relation there but ChemProt doesn't know what is it?
- Here the authors introduce new concepts, such as GENIA, and finally say what is BERN (a neural NER architecture with an integrated normalizer), but I think this should be said previously in the paper. 
- I didn’t quite understand what was used for the normalization

About Vapur, it seems a useful search engine, but there is no evaluation here. 
Are you planning some online evaluation? 

In Results and Discussion the authors only evaluate BERT vs BioBERT performance, which seems a bit limiting and expected that BioBERT performs better in a Biomedical related dataset. 

Further questions: 
- In vapur there is the display of similar entities, for example, similar genes as in Figure 3. I didn’t understand how the similarity is calculated here. 
- Are you considering the use of the full text? Some precious entities may be enclosured there. 

review_title: Useful search engine for biomedical entities COVID-19 related, but needs more work on the evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: Per author response, I've revised my review from 4 to 6 contingent on authors including the additional evaluation & clarifications they've described in the thread below.  

============

Vapur is a tool for finding related protein-chemical pairs. The idea is to (i) extract & normalize entities using BERN, (ii) identify relations using a ChemProt-supervised model, (iii) index papers based on related entity pairs.  Search results organized by relationships with other potential drug targets and entities of interest.  The authors evaluate the system's predictions in a small manual study.

Since the methods used to build the system are applications of existing methods, and the key innovation is in the interface, I recommend restructuring this paper like a demo paper.   
1) Make Figure 3 --> Figure 1.
2) Spend more time walking through how a user should use Vapur for search/exploration/discovery.  Taking us through a compelling example from the perspective of a real biomedical researcher user would be nice  (the current example in Figure 3 caption is a bit lacking in motivation)
3) Add to Related Work a review of other similar relation-oriented COVID19 systems.   Some examples include KDCovid (http://kdcovid.nl/about.html), EVIDENCEMINER (https://www.aclweb.org/anthology/2020.acl-demos.8/), and SemViz (https://www.semviz.org/), which all surface relational information between bio-entities for COVID-19.  I believe Vapur is sufficiently different from theirs because of the use of a relation-based inverted index, but I recommend the authors try to articulate this clearly earlier in their paper to help distinguish their work.
4) Move the details about the Methods (sec 3) to after the reader has familiarized with the system.  

Other writing feedback:
- Table 6 is not discussed in the text.
- Please include the Appendix from the GitHub containing domain-expert-checked instances in the final version of this paper.  Don't make us go look for it.

Questions:
- Why did you decide to use 1/0 binary classification of relation instead of predicting the class labels?  Were the relation class labels not useful for your particular user interface? This would be very interesting to discuss

Evaluation feedback:
- It's good you performed an evaluation for the relation extraction module.  When checking the Appendix, I realized the evaluation was on the binary level (i.e. does this express a relation or not?).  While this is fine to identify that errors are coming from entity normalization, I think your system could benefit from an error analysis on the sentences itself (i.e. what types of sentences does the system tend to fail on?  maybe split this out by the existing relation types in ChemProt?)

- I'm disappointed to not see a human evaluation of the system in front of (at least one) real user.  When we build such tools, it's important to get feedback from real biomedical researchers (e.g. given they issue K queries, do they find the results sensible/helpful?)  It doesn't have to be extensive (given this is a workshop submission), but some indication of having verified that this tool is actually something biomedical researchers would find helpful would give readers more confidence that system designs like Vapur are a good idea.




review_title: [revised -- lean accept assuming revisions] System demo of COVID-19 search tool; lacks proper system-level evaluation with a biomedical expert

final_score: rating: 7: Good paper, accept
review_content: Comments:
Introduction: The last paragraph staring with “in order to”, contains technical detail of the work. It would have been better to highlight the contribution and importance of the work and leave the technical description to the next sections. 

Figure 1 show the overall architecture of the work but the main component which is CORD-19 is not explicitly included. 

For this sentence “Vapur indexes only these abstracts, but it is able to return the linked full-paper,
as well.” What does it mean? As a reviewer I had to decipher it to mean this “ Vapur indexes into these abstracts, and is also able to return the linked full-paper.” Is that correct? (Only and as well do not go together).

 Section 3.3 there seem to be no need to reference BioBERT again. 
 
Where Chemprot dataset is introduced include the number of records (publications).

Table 1 would benefit from having a total row.

Correct: “In order too promote”.

As a reader I found myself going back and forth between sections to try to find how the components are introduced and what the contributions are. Both Results and Discussion and Conclusion sections have Vapur explained in them. I would suggest describing Vapur only in the Conclusion section. 

review_title: The paper presents an interesting work, but it is difficult to read. The paper would benefit from revision on the order the information is presented and removing some reptations.