Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper describes an approach to the TREC COVID Search challenge. The proposed method combines an inverted index score using BM25 and a cosine similarity score based on neural representations derived from BioBERT-NLI. The paper's main contribution is the proposal of a hybrid system that can perform document retrieval without separate search and re-ranking steps using a pre-trained neural indexer.

1. A large concern about the paper is its incoherency in supporting its main contribution. The paper denotes NIR_{AVG} as its proposed method, but the model is neither the official submitted model (NIR_{[CLS]}) nor the best performing model in the additional run (ClinicalCovid-NLI). What's the rationale behind this choice?
2. It would have been nice to see quantitative analysis of models' performance regarding "Where does the model succeed or fail?"  The authors provide a qualitative description of a case where the model had lost points due to unjudged documents, but the explanation's qualitative nature leaves many questions open. Is it always the case that the model's low NDCG@10 scores in different topics due to this phenomenon? How does it improve as the base model changes to a stronger model?

Additional comments:
- The paper does not describe why certain documents were unjudged. One can guess that it's related to other teams' submissions, but it would be better to have the exact criteria described in the document.
- Why doesn't Table 2 have the numbers for Round 3?
review_title: System paper for TREC-COVID

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a simple and effective model for the COVID-19 challenge that combines similarity scores computed from neural representations and traditional inverted index scores (BM25). The details are extensively provided however the organization can be improved, as it's unclear what are the motivations for technical choices and baseline choices.

1. My biggest concern is that the contribution of this paper is not clear. It seems the biggest discovery is that BM25 score is very important for the model (and potentially for other models that participate in this challenge). While this can be a contribution, but the argument sounds weak to me. Isn't this a widely used tool even outside of this challenge? Furthermore, in the ablation study, when taking out BM25 components, the performance dropped to a point of unusable. This questions whether the neural part of this model is effective. It looks like the BM25 is doing heavy-lifting there.

2. What does the NIRR model add to this paper? It seems it performed worse than NIR but the paper does not offer any analysis.

3. It seems another contribution buried in the technical details is that "the NIR model...performs on par with the RF run with neural reranking, without the need for tasks specific training data." So the proposed method is actually unsupervised?  I am confused. Did it use round 1 judgement during round 2? and in round 3?
review_title: A detailed report of experiments with mediocre contribution.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Final update:  Per continued dialogue with the authors (see thread below), I've modified my rating of this work from 4 to 6, leaning toward acceptance under the belief that the authors will make the changes to their manuscript as they've described below.  I stand by my initial review as other reviewers also pointed out much of the same concerns; but I thank the authors for engaging & working toward what I believe will be a more clarified contribution.

=======

The authors latest response addressed many of my concerns, and I'd be happy to push for acceptance *contingent* on author commitment to make certain changes in the manuscript.

=======

I've increased my rating of this work from 3 to 4 based on the author response below.  Many more questions still remain (see my follow-up to author response).

=======

The authors provide a detailed documentation of their participation in the TREC-COVID challenge (Rounds 1-3).  Their system: (i) concatenates BioBERT sentence embeddings (Reimers and Gurevych 2019) with a standard inverted index over documents, and (ii) ranking documents using an averaging of BM25 scores and cosine similarities over the neural embeddings.  The system achieves moderate performance among automatic runs on the official TREC-COVID results.  

Regarding the method, there is not sufficient Related Work provided to understand how this method fits within the broader literature  (has this been tried before -- if so, citation? is this similar to something else w/ some modification?  -- if so, citation & explanation/motivation of differences?  is this entirely novel?)

Regarding the relevance feedback (RF) baseline, why is this a sensible choice over (a) BioBERT sentence embeddings & cosine similarities without BM25, and (b) BM25 without the BioBERT sentence embeddings?  Comparing against the latter baselines would be able to help explain how/where the described system is performing well/not.  

Regarding the evaluation, Table 1, while it is an accurate reporting of the official performance by this team, because the nature of the submissions differs between Rounds, it's hard to understand what learnings we can gain from this.  For example, RF appears for Round 1, but RF-RR for Round 2, and no baseline for Round 3.  Fusion appears for Round 3, but not Round 1 or Round 2.  NIRR appears for Round 2 and Round 3, but not Round 1.  It's really hard to see any meaningful patterns with results reported in this manner.  I believe the authors should've conducted separate controlled experiments to demonstrate efficacy of their method in a controlled setting, and supplement those results with the results from TREC-COVID, not base the entire paper on the TREC-COVID results.

Regarding the analysis in "Where does the model succeed or fail", the point about bias in TREC judgments is very interesting -- that is, their system returned a document at a high rank but was un-judged because no other system returned that document.  This could've been very interesting if the authors did a more thorough analysis -- how often does this happen? how big of an impact does this have on the performance scores (e.g. if these documents were judged, what would the scores look like?)   why might this model be behaving in this manner that's different from other systems?


Overall, this submission, while a perfectly fine detailed report about a team's submissions to a competition, does not propose a particularly powerful system or novel method (either of which would be great).  Nor does it perform enough careful experimentation/detailed analysis to glean any generalizable learnings from their experiences (which would also be great).
review_title: [revised - lean accept assuming revisions] Detailed report of participating system in TREC challenge; moderate performance; few learnings/insights

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This article describes IR methods and their evaluation on the context of the TREC-COVID challenge. The article is well written and structured, and the authors do a good job in providing the full context of the challenge, together with their own contributions. Given the space constraints, I believe the information is very well summarised.

With regards to the technical contributions, the methodology they applied to build and evaluate their models is sound, and the error analysis is insightful, providing useful ideas for future work. Maybe it would have been interesting to provide more details about the systems in Table 2. For instance, the different NLI and BioBERT versions should be explained, and also the difference between NIR_avg and NIR_cls (not just in the table caption). I think it would be more relevant to have a paragraph covering the Table2 variations than keeping Figure 2 (the example is interesting, but the figure takes too much space in my opinion).

Minor comments:
- NIR_avg has slightly different numbers for Round 1 NDCG@10 in Table 2 (0.614) and Table 3 (0.615)
- P1: "gains achieved with neural reranking are debated until recently": a bit of elaboration would help provide more context.
- P1: "large neural models pre-trained on language modeling—specifically BERT which uses bi-directional transformer architecture— achieves": should be "achieve"
- P4: "the document at rank 3 for shared no keywords": remove "for"
review_title: Well presented summary and extended experiments for robust TREC-COVID challenge system