Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a dataset that is annotated via a standard active learning framework, aiming at providing supervision for models to filter COVID-19 articles that are out of expert interests. This paper is well-motivated and well-written. Some elaborations appear to be unnecessarily extensive.

1. When the paper says "...however, the precision of the aforementioned query is highly arguable", there should be some statistics to justify this point. Afterall, this point establishes the motivation of this paper. Otherwise, the contribution is limited.

2. In Algo1, line 5, "i<= max" should be "i <= t_max". Further, it would be useful to present a curve of the acc(f_i) over time. This would illustrate the effectiveness of using active learning. It would be interesting to see if this is a log-like curve or some bumpy curve.

3. In footnote 10, it seems the data is ONLY available during the review time? I want to make sure if this is the case. For a closed dataset, even if it is available during the reviewing period, I would give it an immediate rejection, no matter how good the work is.

4. What is Fig 5? Not mentioned anywhere.

5. It would be perfect if this paper can even gently touch on this point: What is the impact of this dataset on other COVID challenges? This would highlight the usefulness of this data. There could be some domain issue that hinders its application elsewhere. But who knows.

-- PS --
While my rating is leaning positive to acceptance, it depends on the authors' response on point 3 above.
review_title: Well-motivated dataset work.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a task of filtering scientific literature for relevant artciles. (main contribution)
The authors frame the task as a binary classification problem, and use both BioBERT based classifier and conventional classifiers to solve the task. 
The second contribution is that the authors release an expert-curated benchamark set for the task.
Active learning is used to develop the training set, where BioBERT based classifier is used as the base classifier.

Strengths
- Although the proposed task focus on meeting the information needs from medicine, biology, chemistry and bioinformatics researchers, the task of filtering out scientific literature should have broader audiences. The proposed framework is generic and should be able to find use cases in other domains.
- The authors mention the dataset is available upon request

Limitations
- The binary classification setting seems oversimplified. The authors have defined several criteria which, in my opinion, can be used as targeted labels, because these criteria seem to target different groups of users. For example, practitioners who focus on treating patients may feel articles beloing to criteria 3 is relevant but those to criteria 5 irrevant. If the task is framed as a multi-class classification problem, it may further solve the overload issue.

Questions and suggestions:
- It would be interesting to see the performance of a simplest baseline that uses only query in Figure 3. In other words, you can use only the IF part of your Algorithm 2 (line 4) and assign all others `relevant'.

- Page 3: 'keyword query is used on the metadata of the articles' vs. Figure 1 caption 'Papers that match on these keywords in their title, abstract, or full text are included in the dataset.'. Are these two conflicting? Is full text used?
- Page 4: Line 4 in Algorithm 1, measuare accuracy on which set?
- Page 4: You didn't mention which classifier you use in Section 3, until in Section 5, you say the BioBERT based classifier is used as the base classifier. Maybe make it clear in Section 3.
- Page 4: is the seperate regression set randomly sampled?
- Page 5: The first part of Algorithm 1 is confusing. Should the training of BioBERT classifier not involve active learning? and the classifier is trained only once on the complete training set? The second part of Algorithm 2 (line 3-8) looks unnecessary.
- Page 6: Table 2, not sure why the bottom line of the first column is missing. Also, you may add \bottomline to Table 3.
- Page 6: not sure what Figure 5 is for, since you didn't mention it at all in the text.
- Table 2, 3: reporting only results on Relevant class should be enough, because it is a binary classification task and relevant class is what we really care.
- Page 7: suggest to simplify Section 6. For example, delete the first paragraph, which seems no need to repeat.
- Page 8: suggest to add citation after batch-aware methods, or explain what it is 
- Page 8: 'By conducting novel keywords extraction from the recent scientific literature, the CORA keyword-based query can be enhanced automatically with new terminology.' this part sounds not about future work
- The section 4 may benefit from rewriting. It is a bit of confusing. My understanding is at this stage you fine-tune BioBERT based classifier on the complete training set, and this process does not involve training the classifier several times (such as the while loop in Algorithm 1). Suggest to simplify the description.
review_title: A paper proposing new task and benchmark dataset

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors generated a benchmark dataset of relevant scientific information to COVID. They used active learning to build the training set and classified literatures into COVID relevant and non-relevant with high precision and recall. 

The paper is well written and includes sufficient technical details. The authors not only evaluated the performance the purposed method, but also evaluated the benchmark dataset using other classifiers.  The classifier fine-tuned on BioBERT outperforms other classifiers. 

Please clarify the following information in the paper:

The purpose of using active learning for classification is to reduce the number of labeled samples. Please provide the number of samples labeled by experts in Algorithm 1 and show the performance of classifier with the increase of labeled samples. 

The performance of classifier on two relevant and non-relevant sets were reported. In each of set, relevant and non-relevant literatures were included. What is the purpose of generating two sets?  

Two datasets were used: training and regression set. Both datasets were annotated by expert. What is the purpose of using two datasets and why only the regression dataset was used to compare with other classifiers?

review_title: A well-written paper requires minor clarifications