Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This article presents a comparison of methods for the classification of articles from the LitCovid collection. From all the evaluated methods, one based on BioBert performs better compared to other evaluated methods. The contribution of the paper is a comprehensive study of a broad range of methods to the multi-class classification of LitCovid. The classifiers are evaluated on a small set of papers from the CORD-19 collection. The article is clearly written and plenty of details. The authors provide the data and code, which would support reusability and reproducibility.

The performance on CORD-19 is significantly lower compared to the reported numbers for LitCovid. This might be due to several reasons, including the limited size of the CORD-19 articles used or the way manual annotation done by the authors might differ from the one performed in LitCovid.

The LitCovid collection already provides the classification, thus the classifier might have limited used unless a specific use case is specified in which this work would be used.

The title might be willing to include the specific collection on which the methods have been tested on, since the collections and the labels are provided by the LitCovid data set.

The article has been recently uploaded and currently has almost 28k PubMed articles. It might be relevant to mention when the documents were obtained. The authors mention 8k, thus probably it was obtained sometime in April.

In table 1, 2861 —> 2,861
review_title: Study of automatic classification of LitCovid categories.

final_score: rating: 7: Good paper, accept
review_content: The paper presents an exploration of the main document multi-classification algorithms, training and developing on the LitCovid dataset and testing on a small manually annotated portion of the much larger CORD-19 dataset. The LitCovid dataset includes labels for each document to 8 categories with about 1/3 of documents including multi-labels. The significance lies in the usage of these multi-classification algorithms for researchers, clinicians, epidemiologists, and policy makers to find the most relevant articles in a timely manner given the urgency of the pandemic. 

The paper is clear and well-written. The application of document classification algorithms to Covid-19 is novel, but the algorithms themselves are not.


Pros:
- An in-depth exploration of many different algorithms along with tuning for best performance.
- Contains both intrinsic and extrinsic evaluations.
- The discussion on data efficiency is quite interesting and can have a large impact on how much data is needed to annotate for future creation of gold standard datasets.
- An error analysis that suggest future work and updates.
- Important to be applying all research we have so far on these new datasets to help with Covid-19.



Cons:
- The manual annotation of the subset of CORD-19 seems to not be reliable as 34% of errors were annotation related.
- Guidelines and annotation process not provided for the CORD-19 set.
- The CORD-19 set is small and the performance does drop, suggesting that the models are not generalizable.
- No discussion of providing these models to LitCovid to help with the annotation labeling, even if it is a starting point.
- The accuracy is quite low for most algorithms and there is no discussion about this.
- Very little discussion on the very different distributions of the categories between LitCovid and CORD-19. 
review_title: Overall a good exploration of previous document classification algorithms applied to Covid-19.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides an empirical evaluation of several standard models' performances on a covid-19 related document classification task. Specifically, Covid-19 research papers are classified into one or more of eight subcategories based on their topic. Datasets for this task are constructed from LitCovid, and the Covid-19 Open Research Dataset. These datasets are shared online. Finetuned pre-trained language models are found to outperform more traditional models, such as SVM, CNN, or LSTM. In particular, BioBERT and Longformer are found to be most effective.

The paper introduces a multilabel document classification dataset which categorizes Covid19 related articles into one or more of eight subcategories. This dataset is distinct in that it represents a large body of work in the medical domain, united by a common subject. The data is drawn largely from LitCovid, with a supplemental test set drawn from the COVID-19 Open Research Dataset (CORD-19). The paper notes that models trained on the LitCovid data generalize poorly to the CORD-19 data, and theorizes that this is due to the models failing to ignore the overarching COVID-19 topic. It is unclear if this is the case, however, as, while CORD-19 includes papers related to other coronaviruses, COVID-19 is the overarching topic of both datasets.

This paper provides a thorough analysis of many different models' performance with thorough documentation. However, the efficacy of fine-tuned pretrained language models on document classification is well established in a large body of research. While this particular classification task is somewhat novel, demonstrating that finetuned pretrained language models are most effective upon it is a relatively limited contribution.

Pros
 - introduces a multilabel document classification task related to COVID-19
 - provides a dataset to support this classification task
 - provides a clear and thorough empirical evaluation of many different models' performances on this task

Cons
 - the empirical performance evaluation largely confirms trends shown by previous research
 - models trained on the provided dataset are shown to generalize poorly. The reasons for this are underexplored.
review_title: Review of Document Classification for COVID-19 Literature

Meta-review: Thank you for your submission and we are pleased to invite you to present this work at the workshop on Thursday (5:30-9:30pm PDT).

We look forward to hearing about your work.

Please plan on a 10 minute video presentation; pre-recorded is likely the best option.
Decision: Accept (Abstract only)