Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
This work introduces a light weight retrieval and reranking model, as well as an easy-to-use web search service. Although the model has only 620 parameters and does not use the sophisticated Transformer variants like Bert,  it still ranks the second position in the third round of submission.

The writing quality of the paper is pretty good, and the logic is nice. I believe that the work is innovative.

There are still several weak points worth mentioning. 

1. It seems that the preliminary work of the model  come from (Almeida and Matos. In 2020). This work should be cited at the first moment when mentioned. Say, the third paragraph of the introduction, "BioASQ system in our work".

2. Please add citation to “BM25” in the section 2.2.

3. Are both the query and paragraph initialized by a pre-trained Word2vec model? There is no clear explanation in the experiment part, but the readers could get the above clue in the last paragraph. So, if "yes", declare them clearly earlier, pls. 

4. The effect of the re-ordering model on the third round of Baseline is weak, so does the UIowaS team. That's interesting. Any more analysis?

review_title: Light weight and efficient query-document linking algorithm with nice performance.

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a very light-weight reranking model for retrieving documents from the CORD-19 dataset. The impact is that the proposed model would encourage wider application in cases where computation resources are a constraint. The proposed model, while only having 620 parameters, performed surprisingly well (ranked 2nd in the 3rd round).

The paper is well-written and well-motivated. I specially like the 3-round of testing and corresponding "lessons learnt" parts. It offers some insights into the technical details that would cause substantial difference in the evaluation performances, and thus inspire future works.

One limitation is the lack of proper explanation on the baseline in the 3rd round. The paper mentioned a bug in feedback data but still not sure why the baseline performance is missing. Although it said "... it should correspond to our phase-I baseline results", but where are the numbers? How far is it off from the system "ours"?
review_title: A simple light-weight but strong performing reranking model

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Core review**:

This paper presents a currently in-development system for information retrieval in the CORD-19. Authors present a pipeline that is being evaluated in the TREC-COVID challenge and shows a clear advantage with respect to other approaches despite (or thanks to) being based on a simpler neural ranking component. While it may not necessarily be the top-performing approach, authors claim that the simplicity of the neural ranking component and the overall pipeline makes it more feasible to deploy online and use for real-time queries. However, the experimental data provided in the paper (which I briefly cross-checked w.r.t. the TREC-COVID leaderboard links provided in the paper) positions this system with a very good performance, among the top-3 in several metrics. A prototype of the system is deployed and usable online (I've been able to perform some queries while others have taken too long). 

In general, I believe this paper provides a valuable approach and a tool the community can benefit from, and I think it is very relevant for this workshop. My only suggestion is regarding the evaluation. As it is currently written in the paper, there is too much emphasis (in my opinion) on the idiosyncrasies of the TREC-COVID challenge, which makes the evaluation section unnecessarily long. On the other hand, I would appreciate some experimental results to back the hypothesis that this type of system is easier to deploy and faster to execute than the alternatives.

**Reasons to accept:** The paper presents a clear result with demonstrable effectiveness, as validated by experimental data that is publicly available and a prototype system available online.

**Reasons to reject:** Some claims regarding the scalability and practical advantages of this system require further experimentation to be completely justified.
review_title: A solid contribution with a usable prototype implementation