Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This manuscript describes an exemplary effort to address COVID-19 by bringing together much of the relevant literature into one corpus, CORD-19, and increasing its accessibility by providing a harmonized and standardized format convenient for use by automated tools. CORD-19 has been - and is likely to continue being - a critical resource for the scientific community to address COVID-19, and this manuscript not only reflects that importance, but also gives insight into the approach used, the design decisions taken, challenges encountered, use cases, shared tasks, and various discussion points. The manuscript is well-organized and readable, and (overall) an excellent case study in corpus creation. This manuscript is not only important for understanding the CORD-19 corpus and its enabling effect on current COVID-19 efforts, but is possibly also a historically important example of joint scientific efforts to address COVID-19.

Despite the critical importance of this dataset, there are several questions left unanswered by this manuscript, and it would be unfortunate to not address these before publication.

It would be useful to have a very clear statement of the purpose for CORD-19. The inclusion of SARS and MERS makes intuitive sense, but it is less clear why other coronaviruses that infect humans (e.g. HCoV-OC43) are not explicitly included - I am not a virologist, but neither will be most of the audience for this manuscript. While many of the articles that discuss these lesser known cornaviruses would be included anyway because they would also mention "coronavirus", this is not guaranteed. 

While it seems appropriate for document inclusion to be query-based, it is important to consider the coverage of the query. The number of name variants in the literature for COVID-19 or SARS-CoV-2 is rather large, and not all of these documents will include other terms that will match, such as "coronavirus". For example, how would a document that mentions "SARS CoV-2" but none of the query terms listed be handled? This is not a theoretical case: the title and abstract for PMID 32584542 have this issue, and I was unable to locate this document in CORD-19. In addition to minor variations such as this, there are many examples of significant variations such as "HCoV-19", "nCoV-19" or even "COIVD". Are these cases worth considering? If not, can we quantify how much is lost? And if we can't quantify it, this is a limitation.

How is the following situation handled: querying source A returns a document (e.g. the source has full text and that matches), but the version in source B does not return it (e.g. the source only has title & abstract, and they do not match). From the description, I would assume that the version from source A is used and the version from source B is ignored; is any reasonably useful data lost by not explicitly querying source B for its version?

There are other efforts to provide a repository of scientific articles related to COVID-19, and it would be appropriate to mention these, if only to indicate why CORD-19 has unique value. I am aware of LitCovid (Chen Q, Allot A, Lu Z. Keep up with the latest coronavirus research. Nature. 2020;579(7798):193), are there others?

There are also non-COVID-19 efforts to provide a large percentage of the literature in formats appropriate for text mining or other processing. One is (Comeau, Donald C., et al. "PMC text mining subset in BioC: about three million full-text articles and growing." Bioinformatics 35.18 (2019): 3533-3535.), which not only provides the full text of a large percentage of the articles in PubMed Central, but it is also kept up-to-date and converts all documents into a straightforward standardized XML format appropriate for text mining. While this effort is single-source, it specifically addresses some of the issues encountered in the creation of CORD-19 and the representation aspect of the "Call to Action".

review_title: Excellent description of a critical COVID-19 dataset, some questions remaining

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This is a paper that describes an important research dataset that has been produced during the Covid-19 epidemic. The CORD-19 collection is used for much research and some challenge evaluations. Even though this paper does not report any research results per se, and the paper is posted on the ArXiv preprint server, this version will give a citable description of the collection that will likely be widely referenced.

The authors describe well the process of dealing not only with the technical issues of processing heterogeneous scientific papers but also the non-technical issues, such as copyright and licensing.

The authors do not make any unreasonable claims, although I do question the value of this collection for non-computational researchers and clinicians. As the authors note, the collection is not complete, which is essential for clinical researchers and certainly for clinicians (who do not typically read primary research papers anyways, and tend to focus more on summations). But the dataset is of tremendous value to computational and informatics researchers, and that should be pronounced.

I appreciate the Discussion that points out the limitations of how scientific information is currently published, and how it could be improved. One other concern that could be addressed is how long the Allen Institute for AI, which is to be commended for this work, will continue to maintain this tremendously valuable resource.

review_title: Overview of a highly important Covid-19 dataset

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors present the CORD-19 data set and describe how it has been developed and continues to be developed. The CORD-19 data set is a valuable resource that provides access to the latest literature about COVID-19 and coronaviruses and it is updated daily with over 200k downloads. The generation of the CORD-19 requires a coordinated integration and processing effort that is significant. The contribution of this corpus is of high significance and will have a strong impact on the biomedical domain and support the development, for instance, of COVID-19 vaccines. The manuscript is clearly written and it is easy to understand.

The effort in providing a version of the latest literature in formats that can be processed by text analytics methods is excellent, using the latest of the available technology to do so. In the paper, it is mentioned in the manuscript that there are some problems in turning tables into structured format and the authors provide examples of issues that they have found. Table processing is done by IBM, who has as well a method for table processing that seems to be resilient to the problems mentioned and would be relevant to consider it for table processing (https://arxiv.org/abs/1911.10683).

The authors give an example of conflict from which it can be inferred that the same DOI might be linked to two different PubMed identifiers, the reviewer is curious why this might be the case and if an example could be provided.

When you mention “Classification of CORD-19 papers to Microsoft Academic Graph”, is this classification done by a method provided by the authors? is this classification provided as meta-data?

During my review the only typo I could find is:
* “other research activity.” —> “other research activities.”?
* “by not allowing republication of **an** paper”, an —> a

Please consider the following guideline for NLM trademarks: https://www.nlm.nih.gov/about/trademarks.html
review_title: CORD-19 is an excellent resource with an impressive integration work for the research community to fight COVID-19.

Meta-review: This is a resource of significant importance, and a high-quality paper describing it.

We look forward to a presentation about this at the workshop later this week!
Decision: Accept