\documentclass[11pt,a4paper]{article}
\PassOptionsToPackage{hyphens}{url}\usepackage{hyperref}    %
\usepackage[hyperref]{acl2020}
\usepackage{times}
\usepackage{latexsym}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{tabularx}
\renewcommand{\UrlFont}{\ttfamily\small}

\usepackage{xspace}   %

\usepackage{microtype}

\aclfinalcopy %

\setlength\titlebox{8cm}

\newcommand\BibTeX{B\textsc{ib}\TeX}

\newcommand{\covid}{\textsc{Covid-19}\xspace}
\newcommand{\cord}{\textsc{CORD-19}\xspace}
\newcommand{\sars}{\textsc{SARS}\xspace}
\newcommand{\mers}{\textsc{MERS}\xspace}
\newcommand{\swine}{\textsc{H1N1}\xspace}

\newcommand{\trec}{\textsc{TREC-COVID}\xspace}

\newcommand\kyle[1]{{\color{red}\{\textit{#1}\}$_{KL}$}}
\newcommand\lucy[1]{{\color{orange}\{\textit{#1}\}$_{LLW}$}}

\newcommand\todoit[1]{{\color{red}\{TODO: \textit{#1}\}}}
\newcommand\todo{{\color{red}{TODO}}\xspace}


\title{\cord: The \covid Open Research Dataset}

\author{
Lucy Lu Wang$^{1,}$\Thanks{ denotes equal contribution} \quad 
Kyle Lo$^{1,}$\footnotemark[1]  \quad 
Yoganand Chandrasekhar$^1$ \quad
Russell Reas$^1$ \quad 
\\
{\bf 
Jiangjiang Yang$^1$ \quad 
Douglas Burdick$^2$ \quad
Darrin Eide$^3$ \quad
Kathryn Funk$^4$ \quad 
} \\
{\bf
Yannis Katsis$^2$ \quad
Rodney Kinney$^1$ \quad
Yunyao Li$^2$ \quad
Ziyang Liu$^6$ \quad
} \\
{\bf 
William Merrill$^1$ \quad
Paul Mooney$^5$ \quad
Dewey Murdick$^7$ \quad
Devvret Rishi$^5$ \quad
} \\
{\bf
Jerry Sheehan$^4$ \quad
Zhihong Shen$^3$ \quad
Brandon Stilson$^1$ \quad
Alex D. Wade$^6$ \quad
} \\
{\bf
Kuansan Wang$^3$ \quad
Nancy Xin Ru Wang $^2$ \quad
Chris Wilhelm$^1$ \quad
Boya Xie$^3$ \quad
} \\
{\bf
Douglas Raymond$^1$ \quad 
Daniel S. Weld$^{1,8}$ \quad
Oren Etzioni$^1$ \quad
Sebastian Kohlmeier$^1$ \quad 
} \\ [2mm]
  $^1$Allen Institute for AI \quad $^2$ IBM Research \quad $^3$Microsoft Research \\
  $^4$National Library of Medicine \quad $^5$Kaggle \quad $^6$Chan Zuckerberg Initiative \\
  $^7$Georgetown University \quad $^8$University of Washington  \\
  {\tt\small \{lucyw, kylel\}@allenai.org}
}

\date{}

\begin{document}
\maketitle

\begin{abstract}
    The \covid Open Research Dataset (\cord) is a growing\footnote{The dataset continues to be updated daily with papers from new sources and the latest publications. Statistics reported in this article are up-to-date as of version \textsc{2020-06-14}.} resource of scientific papers on \covid
    and related historical coronavirus research.
    \cord is designed to facilitate the development of text mining and information retrieval systems over its rich collection of metadata and structured full text papers. 
    Since its release, \cord has been downloaded\footnote{\href{https://www.semanticscholar.org/cord19}{https://www.semanticscholar.org/cord19}} over 200K times and has served as the basis of many \covid text mining and discovery systems. In this article, we describe the mechanics of dataset construction, highlighting challenges and key design decisions, provide an overview of how \cord has been used, and describe several shared tasks built around the dataset.
    We hope this resource will continue to bring together the computing community, biomedical experts, and policy makers in the search for effective treatments and management policies for \covid.
\end{abstract}

\section{Introduction}

On March 16, 2020, the Allen Institute for AI (AI2), in collaboration with our partners at The White House Office of Science and Technology Policy (OSTP), the National Library of Medicine (NLM), the Chan Zuckerburg Initiative (CZI), Microsoft Research, and Kaggle, coordinated by Georgetown University's Center for Security and Emerging Technology (CSET), released the first version of \cord. 
This resource is a large and growing collection of publications and preprints on \covid and related historical coronaviruses such as \sars and \mers.
The initial release consisted of 28K papers, and the collection has grown to more than 140K papers over the subsequent weeks. Papers and preprints from several archives are collected and ingested through the Semantic Scholar literature search engine,\footnote{\href{https://semanticscholar.org/}{https://semanticscholar.org/}} metadata are harmonized and deduplicated, and paper documents are processed through the pipeline established in \citet{lo-wang-2020-s2orc} to extract full text (more than 50\% of papers in \cord have full text). We commit to providing regular updates to the dataset until an end to the \covid crisis is foreseeable.

\begin{figure}[tbp!]
  \centering
  \includegraphics[width=\columnwidth]{cord19_dset.png}
  \caption{Papers and preprints are collected from different sources through Semantic Scholar. Released as part of \cord are the harmonized and deduplicated metadata and full text JSON.}
  \label{fig:dataset}
\end{figure}

\cord aims to connect the machine learning community with biomedical domain experts and policy makers in the race to identify effective treatments and management policies for \covid. The goal is to harness these diverse and complementary pools of expertise to discover relevant information more quickly from the literature. Users of the dataset have leveraged AI-based techniques in information retrieval and natural language processing to extract useful information.

Responses to \cord have been overwhelmingly positive, with the dataset being downloaded over 200K times in the three months since its release. The dataset has been used by clinicians and clinical researchers to conduct systematic reviews, has been leveraged by data scientists and machine learning practitioners to construct search and extraction tools, and is being used as the foundation for several successful shared tasks.
We summarize research and shared tasks in Section~\ref{sec:research_directions}. 

In this article, we briefly describe: 
\begin{enumerate}[noitemsep]
    \item The content and creation of \cord,
    \item Design decisions and challenges around creating the dataset,
    \item Research conducted on the dataset, and how shared tasks have facilitated this research, and
    \item A roadmap for \cord going forward.
\end{enumerate}

\section{Dataset}
\label{sec:dataset}

\cord integrates papers and preprints from several sources (Figure~\ref{fig:dataset}), where a paper is defined as the base unit of published knowledge, and a preprint as an unpublished but publicly available counterpart of a paper. Throughout the rest of Section~\ref{sec:dataset}, we discuss papers, though the same processing steps are adopted for preprints. 
First, we ingest into Semantic Scholar paper metadata and documents from each source. Each paper is associated with bibliographic metadata, like title, authors, publication venue, etc, as well as unique identifiers such as a DOI, PubMed Central ID, PubMed ID, the WHO Covidence \#,\footnote{\label{footnote:who}\href{https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov}{https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov}} MAG identifier \citep{Shen2018AWS}, and others. Some papers are associated with documents, the physical artifacts containing paper content; these are the familiar PDFs, XMLs, or physical print-outs we read. 

For the \cord effort, we generate harmonized and deduplicated metadata as well as structured full text parses of paper documents as output. We provide full text parses in cases where we have access to the paper documents, and where the documents are available under an open access license (e.g. Creative Commons (CC),\footnote{\href{https://creativecommons.org/}{https://creativecommons.org/}} publisher-specific \covid licenses,\footnote{\label{footnote:pmc_covid}\href{https://www.ncbi.nlm.nih.gov/pmc/about/covid-19/}{https://www.ncbi.nlm.nih.gov/pmc/about/covid-19/}} or identified as open access through DOI lookup in the Unpaywall\footnote{\href{https://unpaywall.org/}{https://unpaywall.org/}} database).

\subsection{Sources of papers}

Papers in \cord are sourced from PubMed Central (PMC), PubMed, the World Health Organization's Covid-19 Database,\textsuperscript{\ref{footnote:who}} and preprint servers bioRxiv, medRxiv, and arXiv. The PMC Public Health Emergency Covid-19 Initiative\textsuperscript{\ref{footnote:pmc_covid}} expanded access to \covid literature by working with publishers to make coronavirus-related papers discoverable and accessible through PMC under open access license terms that allow for reuse and secondary analysis. BioRxiv and medRxiv preprints were initially provided by CZI, and are now ingested through Semantic Scholar along with all other included sources. We also work directly with publishers such as Elsevier\footnote{\label{footnote:elsevier}\href{https://www.elsevier.com/connect/coronavirus-information-center}{https://www.elsevier.com/connect/coronavirus-information-center}} and Springer Nature,\footnote{\href{https://www.springernature.com/gp/researchers/campaigns/coronavirus}{https://www.springernature.com/gp/researchers/\\campaigns/coronavirus}} to provide full text coverage of relevant papers available in their back catalog.

All papers are retrieved given the query\footnote{Adapted from the Elsevier COVID-19 site\textsuperscript{\ref{footnote:elsevier}}}:

\begin{quote}
\footnotesize\texttt{"COVID" OR "COVID-19" OR "Coronavirus" OR "Corona virus" OR "2019-nCoV" OR "SARS-CoV" OR "MERS-CoV" OR "Severe Acute Respiratory Syndrome" OR "Middle East Respiratory Syndrome"}
\end{quote}

\noindent Papers that match on these keywords in their title, abstract, or body text are included in the dataset. Query expansion is performed by PMC on these search terms, affecting the subset of papers in \cord retrieved from PMC.

\subsection{Processing metadata}
\label{sec:metadata_processing}

The initial collection of sourced papers suffers from duplication and incomplete or conflicting metadata. We perform the following operations to harmonize and deduplicate all metadata:

\begin{enumerate}[noitemsep]
    \item Cluster papers using paper identifiers
    \item Select canonical metadata for each cluster
    \item Filter clusters to remove unwanted entries
\end{enumerate}

\paragraph{Clustering papers}  We cluster papers if they overlap on any of the following identifiers: \emph{\{doi, pmc\_id, pubmed\_id, arxiv\_id, who\_covidence\_id, mag\_id\}}. If two papers from different sources have an identifier in common and no other identifier conflicts between them, we assign them to the same cluster. Each cluster is assigned a unique identifier \textbf{\textsc{cord\_uid}}, which persists between dataset releases.
No existing identifier, such as DOI or PMC ID, is sufficient as the primary \cord identifier. Some papers in PMC do not have DOIs; some papers from the WHO, publishers, or preprint servers like arXiv do not have PMC IDs or DOIs. 

Occasionally, conflicts occur. For example, a paper $c$ with $(doi, pmc\_id, pubmed\_id)$ identifiers $(x, null, z')$ might share identifier $x$ with a cluster of papers $\{a, b\}$ that has identifiers $(x, y, z)$, but has a conflict $z' \neq z$.  In this case, we choose to create a new cluster $\{c\}$, containing only paper $c$.\footnote{This is a conservative clustering policy in which any metadata conflict prohibits clustering.  An alternative policy would be to cluster if any identifier matches, under which $a$, $b$, and $c$ would form one cluster with identifiers $(x, y, [z, z'])$.}

\paragraph{Selecting canonical metadata} Among each cluster, the canonical entry is selected to prioritize the availability of document files and the most permissive license. For example, between two papers with PDFs, one available under a CC license and one under a more restrictive \covid-specific copyright license, we select the CC-licensed paper entry as canonical. If any metadata in the canonical entry are missing, values from other members of the cluster are promoted to fill in the blanks.

\paragraph{Cluster filtering} Some entries harvested from sources are not papers, and instead correspond to materials like tables of contents, indices, or informational documents. These entries are identified in an ad hoc manner and removed from the dataset.

\subsection{Processing full text}

Most papers are associated with one or more PDFs.\footnote{PMC papers can have multiple associated PDFs per paper, separating the main text from supplementary materials.}  To extract full text and bibliographies from each PDF, we use the PDF parsing pipeline created for the S2ORC dataset \cite{lo-wang-2020-s2orc}.\footnote{One major difference in full text parsing for \cord is that we do not use ScienceParse,\footnotemark~as we always derive this metadata from the sources directly.}\footnotetext{\href{https://github.com/allenai/science-parse}{https://github.com/allenai/science-parse}} In \cite{lo-wang-2020-s2orc}, we introduce the S2ORC JSON format for representing scientific paper full text, which is used as the target output for paper full text in \cord. The pipeline involves:

\begin{enumerate}[noitemsep]
    \item Parse all PDFs to TEI XML files using GROBID\footnote{\href{https://github.com/kermitt2/grobid}{https://github.com/kermitt2/grobid}} \cite{Lopez2009GROBIDCA}
    \item Parse all TEI XML files to S2ORC JSON
    \item Postprocess to clean up links between inline citations and bibliography entries.
\end{enumerate}

\noindent We additionally parse JATS XML\footnote{\href{https://jats.nlm.nih.gov/}{https://jats.nlm.nih.gov/}} files available for PMC papers using a custom parser, generating the same target S2ORC JSON format.  

This creates two sets of full text JSON parses associated with the papers in the collection, one set originating from PDFs (available from more sources), and one set originating from JATS XML (available only for PMC papers). Each PDF parse has an associated SHA, the 40-digit SHA-1 of the associated PDF file, while each XML parse is named using its associated PMC ID. Around 48\% of \cord papers have an associated PDF parse, and around 37\% have an XML parse, with the latter nearly a subset of the former. Most PDFs ($>$90\%) are successfully parsed. Around 2.6\% of \cord papers are associated with multiple PDF SHA, due to a combination of paper clustering and the existence of supplementary PDF files.

\subsection{Table parsing}

Since the May 12, 2020 release of \cord, we also release selected HTML table parses. Tables contain important numeric and descriptive information such as sample sizes and results, which are the targets of many information extraction systems. A separate PDF table processing pipeline is used, consisting of table extraction and table understanding.
\emph{Table extraction} is based on the Smart Document Understanding (SDU) capability included in IBM Watson Discovery.\footnote{\href{https://www.ibm.com/cloud/watson-discovery}{https://www.ibm.com/cloud/watson-discovery}} SDU converts a given PDF document from its native binary representation into a text-based representation like HTML which includes both identified document structures (e.g., tables, section headings, lists) and formatting information (e.g. positions for extracted text). \emph{Table understanding} (also part of Watson Discovery) then annotates the extracted tables with additional semantic information, such as column and row headers and table captions. We leverage the Global Table Extractor (GTE)~\cite{Zheng2020GlobalTE}, which uses a specialized object detection and clustering technique to extract table bounding boxes and structures. 

All PDFs are processed through this table extraction and understanding pipeline. If the Jaccard similarity of the table captions from the table parses and \cord parses is above 0.9, we insert the HTML of the matched table into the full text JSON. We extract 188K tables from 54K documents, of which 33K tables are successfully matched to tables in 19K (around 25\%) full text documents in \cord.
Based on preliminary error analysis, we find that match failures are primarily due to caption mismatches between the two parse schemes. Thus, we plan to explore alternate matching functions, potentially leveraging table content and document location as additional features. See Appendix \ref{app:tables} for example table parses.

\subsection{Dataset contents}

\begin{figure}[tbp!]
  \centering
  \includegraphics[width=\columnwidth]{papers_per_year.png}
  \caption{The distribution of papers per year in \cord. A spike in publications occurs in 2020 in response to \covid.}
  \label{fig:year}
\end{figure}

\cord has grown rapidly, now consisting of over 140K papers with over 72K full texts. Over 47K papers and 7K preprints on \covid and coronaviruses have been released since the start of 2020, comprising nearly 40\% of papers in the dataset.

\begin{table}[tbp!]
  \setlength{\tabcolsep}{.25em}
  \footnotesize
  \centering
  \begin{tabular}{p{34mm}p{15mm}p{17mm}}
    \toprule
    Subfield & Count & \% of corpus \\
    \midrule
    Virology & 29567 & 25.5\% \\
    Immunology & 15954 & 13.8\% \\
    Surgery & 15667 & 13.5\% \\
    Internal medicine & 12045 & 10.4\% \\
    Intensive care medicine & 10624 & 9.2\% \\
    Molecular biology & 7268 & 6.3\% \\
    Pathology & 6611 & 5.7\% \\
    Genetics & 5231 & 4.5\% \\
    Other & 12997 & 11.2\% \\
    \bottomrule
  \end{tabular}
  \caption{MAG subfield of study for \cord papers.}
  \label{tab:fos}
\end{table}

Classification of \cord papers to Microsoft Academic Graph (MAG) \citep{msr:mag1, msr:mag2} fields of study \citep{Shen2018AWS} indicate that the dataset consists predominantly of papers in Medicine (55\%), Biology (31\%), and Chemistry (3\%), which together constitute almost 90\% of the corpus.\footnote{MAG identifier mappings are provided as a supplement on the \cord landing page.} A breakdown of the most common MAG subfields (L1 fields of study) represented in \cord is given in Table~\ref{tab:fos}.

Figure~\ref{fig:year} shows the distribution of \cord papers by date of publication. Coronavirus publications increased during and following the SARS and MERS epidemics, but the number of papers published in the early months of 2020 exploded in response to the \covid epidemic. Using author affiliations in MAG, we identify the countries from which the research in CORD-19 is conducted. Large proportions of \cord papers are associated with institutions based in the Americas (around 48K papers), Europe (over 35K papers), and Asia (over 30K papers). 
 

\section{Design decision \& challenges}

A number of challenges come into play in the creation of \cord. We summarize the primary design requirements of the dataset, along with challenges implicit within each requirement:

\paragraph{Up-to-date} 
Hundreds of new publications on \covid are released every day, and a dataset like \cord can quickly become irrelevant without regular updates. \cord has been updated daily since May 26. A processing pipeline that produces consistent results day to day is vital to maintaining a changing dataset. That is, the metadata and full text parsing results must be reproducible, identifiers must be persistent between releases, and changes or new features should ideally be compatible with previous versions of the dataset. 

\paragraph{Handles data from multiple sources} Papers from different sources must be integrated and harmonized. Each source has its own metadata format, which must be converted to the \cord format, while addressing any missing or extraneous fields. The processing pipeline must also be flexible to adding new sources.

\paragraph{Clean canonical metadata} Because of the diversity of paper sources, duplication is unavoidable. Once paper metadata from each source is cleaned and organized into \cord format, we apply the deduplication logic described in Section \ref{sec:metadata_processing} to identify similar paper entries from different sources. We apply a conservative clustering algorithm, combining papers only when they have shared identifiers but no conflicts between any particular class of identifiers. We justify this because it is less harmful to retain a few duplicate papers than to remove a document that is potentially unique and useful.

\paragraph{Machine readable full text} To provide accessible and canonical structured full text, we parse content from PDFs and associated paper documents. The full text is represented in S2ORC JSON format \citep{lo-wang-2020-s2orc}, a schema designed to preserve most relevant paper structures such as paragraph breaks, section headers, inline references, and citations. S2ORC JSON is simple to use for many NLP tasks, where character-level indices are often employed for annotation of relevant entities or spans. The text and annotation representations in S2ORC share similarities with BioC \citep{Comeau2019PMCTM}, a JSON schema introduced by the BioCreative community for shareable annotations, with both formats leveraging the flexibility of character-based span annotations. However, S2ORC JSON also provides a schema for representing other components of a paper, such as its metadata fields, bibliography entries, and reference objects for figures, tables, and equations. We leverage this flexible and somewhat complete representation of S2ORC JSON for \cord. We recognize that converting between PDF or XML to JSON is lossy. However, the benefits of a standard structured format, and the ability to reuse and share annotations made on top of that format have been critical to the success of \cord.

\paragraph{Observes copyright restrictions} Papers in \cord and academic papers more broadly are made available under a variety of copyright licenses. These licenses can restrict or limit the abilities of organizations such as AI2 from redistributing their content freely. Although much of the \covid literature has been made open access by publishers, the provisions on these open access licenses differ greatly across papers. Additionally, many open access licenses grant the ability to read, or ``consume'' the paper, but may be restrictive in other ways, for example, by not allowing republication of a paper or its redistribution for commercial purposes. The curator of a dataset like \cord must pass on best-to-our-knowledge licensing information to the end user.

\section{Research directions}
\label{sec:research_directions}

\begin{figure}[tbp!]
  \centering
  \includegraphics[width=\columnwidth]{cord19_tasks.png}
  \caption{An example information retrieval and extraction system using \cord: Given an input query, the system identifies relevant papers (yellow highlighted rows) and extracts text snippets from the full text JSONs as supporting evidence.}
  \label{fig:tasks}
\end{figure}

We provide a survey of various ways researchers have made use of \cord.  We organize these into four categories:  \emph{(i)} direct usage by clinicians and clinical researchers (\S\ref{sec:by_clinical_experts}), \emph{(ii)} tools and systems to assist clinicians (\S\ref{sec:for_clinical_experts}), \emph{(iii)} research to support further text mining and NLP research (\S\ref{sec:for_nlp_researchers}), and \emph{(iv)} shared tasks and competitions (\S\ref{sec:shared_tasks}).

\subsection{Usage by clinical researchers}
\label{sec:by_clinical_experts}

\cord has been used by medical experts as a paper collection for conducting systematic reviews.  These reviews address questions about \covid include infection and mortality rates in different demographics \cite{Han2020.who-is-more-susceptible}, symptoms of the disease \citep{Parasa2020PrevalenceOG}, identifying suitable drugs for repurposing \cite{sadegh2020exploring}, management policies \cite{Yaacoube-bmj-safe-management-bodies}, and interactions with other diseases \cite{Crisan-Dabija-tuberculosis-covid19, Popa-inflammatory-bowel-diseases}.
\subsection{Tools for clinicians}
\label{sec:for_clinical_experts}
Challenges for clinicians and clinical researchers during the current epidemic include \textit{(i)} keeping up to to date with recent papers about \covid, \textit{(ii)} identifying useful papers from historical coronavirus literature, \textit{(iii)} extracting useful information from the literature, and \textit{(iv)} synthesizing knowledge from the literature. To facilitate solutions to these challenges, dozens of tools and systems over \cord have already been developed.  
Most combine elements of text-based information retrieval and extraction, as illustrated in Figure~\ref{fig:tasks}.
We have compiled a list of these efforts on the \cord public GitHub repository\footnote{\href{https://github.com/allenai/cord19}{https://github.com/allenai/cord19}} and highlight some systems in Table \ref{tab:other_tasks}.\footnote{There are many Search and QA systems to survey. We have chosen to highlight the systems that were made publicly-available within a few weeks of the \cord initial release.}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}

\begin{table*}[tbh!]
\small
\begin{tabularx}{\textwidth}{L{20mm}p{20mm}p{40mm}X}
    \toprule
    \textbf{Task} & \textbf{Project} & \textbf{Link} & \textbf{Description} \\
    \midrule
    \textbf{Search and \newline discovery} & \textsc{Neural Covidex} & \href{https://covidex.ai/}{https://covidex.ai/} & Uses a T5-base \cite{raffel2019exploring} unsupervised reranker on BM25 \cite{Jones2000APM} \\ \cline{2-4}
     & \textsc{CovidScholar} & \href{https://covidscholar.org}{https://covidscholar.org/} & Adapts \citet{Weston2019} system for entity-centric queries \\ \cline{2-4}
     & \textsc{KDCovid} & \href{http://kdcovid.nl/about.html}{http://kdcovid.nl/about.html} & Uses BioSentVec \cite{biosentvec} similarity to identify relevant sentences \\
     \cline{2-4} & \textsc{Spike-Cord} & \href{https://spike.covid-19.apps.allenai.org}{https://spike.covid-19.apps.allenai.org} & Enables users to define ``regular expression''-like queries to directly search over full text \\
    \midrule
    \textbf{Question answering} & \textsc{covidask} & \href{https://covidask.korea.ac.kr/}{https://covidask.korea.ac.kr/} & Adapts \citet{seo-etal-2019-real} using BioASQ challenge (Task B) dataset \citep{Tsatsaronis2015AnOO} \\ \cline{2-4}
     & \textsc{aueb} & \href{http://cslab241.cs.aueb.gr:5000/}{http://cslab241.cs.aueb.gr:5000/} & Adapts \citet{mcdonald2018deep} using \citet{Tsatsaronis2015AnOO} \\
    \midrule
    \textbf{Summariz-ation} & Vespa & \href{https://cord19.vespa.ai/}{https://cord19.vespa.ai/} & Generates summaries of paper abstracts using T5 \citep{raffel2019exploring} \\
    \midrule
    \textbf{Recommend-ation} & Vespa & \href{https://cord19.vespa.ai/}{https://cord19.vespa.ai/} & Recommends ``similar papers'' using Sentence-BERT \cite{reimers-gurevych-2019-sentence} and SPECTER embeddings \cite{specter2020cohan} \\
    \midrule
    \textbf{Entailment} & COVID papers browser & \href{https://github.com/gsarti/covid-papers-browser}{https://github.com/gsarti/covid-papers-browser} & Similar to \textsc{KDCovid}, but uses embeddings from BERT models trained on NLI datasets \\
    \midrule
    \textbf{Claim \newline verification} & SciFact & \href{https://scifact.apps.allenai.org}{https://scifact.apps.allenai.org} & Uses RoBERTa-large \cite{liu2019roberta} to find Support/Refute evidence for \covid claims \\ 
    \midrule
    \textbf{Assistive lit. review} & ASReview & \href{https://github.com/asreview/asreview-covid19}{https://github.com/asreview/ asreview-covid19} & Active learning system with a \cord plugin for identifying papers for literature reviews \\
    \midrule
    \textbf{Augmented reading} & Sinequa & \href{https://covidsearch.sinequa.com/app/covid-search/}{https://covidsearch.sinequa.com/ app/covid-search/} & In-browser paper reader with entity highlighting on PDFs \\
    \midrule
    \textbf{Visualization} & SciSight & \href{https://scisight.apps.allenai.org}{https://scisight.apps.allenai.org} & Network visualizations for browsing research groups working on \covid \\
    \bottomrule
\end{tabularx}
\caption{Publicly-available tools and systems for medical experts using \cord.}
\label{tab:other_tasks}
\end{table*}

\subsection{Text mining and NLP research}
\label{sec:for_nlp_researchers}

The following is a summary of resources released by the NLP community on top of \cord to support other research activities.

\paragraph{Information extraction}
To support extractive systems, NER and entity linking of biomedical entities can be useful. NER and linking can be performed using NLP toolkits like ScispaCy \cite{neumann-etal-2019-scispacy} or language models like BioBERT-base \cite{Lee2019BioBERTAP} and SciBERT-base \cite{beltagy-etal-2019-scibert} finetuned on biomedical NER datasets.
\citet{Wang2020ComprehensiveNE} augments \cord full text with entity mentions predicted from several techniques, including weak supervision using the NLM's Unified Medical Language System (UMLS) Metathesaurus \cite{Bodenreider2004TheUM}. 

\paragraph{Text classification} 
Some efforts focus on extracting sentences or passages of interest.
For example, \citet{Liang2020IdentifyingRF} uses BERT \cite{devlin-etal-2019-bert} to extract sentences from \cord that contain \covid-related radiological findings. 
\paragraph{Pretrained model weights} BioBERT and SciBERT have been popular pretrained LMs for \covid-related tasks. DeepSet has released a BERT-base model pretrained on \cord.\footnote{\href{https://huggingface.co/deepset/covid_bert_base}{https://huggingface.co/deepset/covid\_bert\_base}}   
SPECTER \cite{specter2020cohan} paper embeddings computed using paper titles and abstracts are being released with each \cord update.
SeVeN relation embeddings \cite{espinosa-anke-schockaert-2018-seven} between word pairs have also been made available for \cord.\footnote{\href{https://github.com/luisespinosaanke/cord-19-seven}{https://github.com/luisespinosaanke/cord-19-seven}}

\paragraph{Knowledge graphs} The Covid Graph project\footnote{\href{https://covidgraph.org/}{https://covidgraph.org/}} releases a \covid knowledge graph built from mining several public data sources, including \cord, and is perhaps the largest current initiative in this space. \citet{Ahamed2020InformationMF} rely on entity co-occurrences in \cord to construct a graph that enables centrality-based ranking of drugs, pathogens, and biomolecules. 

\subsection{Competitions and Shared Tasks}
\label{sec:shared_tasks}

The adoption of \cord and the proliferation of text mining and NLP systems built on top of the dataset are supported by several \covid-related competitions and shared tasks.

\subsubsection{Kaggle} 
\label{sec:kaggle}

Kaggle hosts the \cord Research Challenge,\footnote{\href{https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge}{https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge}} a text-mining challenge that tasks participants with extracting answers to key scientific questions about \covid from the papers in the \cord dataset. Round 1 was initiated with a set of open-ended questions, e.g., \textit{What is known about transmission, incubation, and environmental stability?} and \textit{What do we know about \covid risk factors?}

More than 500 teams participated in Round 1 of the Kaggle competition. Feedback from medical experts during Round 1 identified that the most useful contributions took the form of article summary tables. Round 2 subsequently focused on this task of table completion, and resulted in 100 additional submissions. A unique tabular schema is defined for each question, and answers are collected from across different automated extractions. For example, extractions for risk factors should include disease severity and fatality metrics, while extractions for incubation should include time ranges. Sufficient knowledge of COVID-19 is necessary to define these schema, to understand which fields are important to include (and exclude), and also to perform error-checking and manual curation. 

\subsubsection{TREC}

The \trec\footnote{\href{https://ir.nist.gov/covidSubmit/index.html}{https://ir.nist.gov/covidSubmit/index.html}} shared task \cite{trec-covid-jamia,voorhees2020treccovid} assesses systems on their ability to rank papers in \cord based on their relevance to \covid-related topics. Topics are sourced from MedlinePlus searches, Twitter conversations, library searches at OHSU, as well as from direct conversations with researchers, reflecting actual queries made by the community.  To emulate real-world surge in publications and rapidly-changing information needs, the shared task is organized in multiple rounds. Each round uses a specific version of \cord, has newly added topics, and gives participants one week to submit per-topic document rankings for judgment. Round 1 topics included more general questions such as \emph{What is the origin of COVID-19?}~and \emph{What are the initial symptoms of COVID-19?}~while Round 3 topics have become more focused, e.g., \emph{What are the observed mutations in the SARS-CoV-2 genome?}~and \emph{What are the longer-term complications of those who recover from COVID-19?} Around 60 medical domain experts, including indexers from NLM and medical students from OHSU and UTHealth, are involved in providing gold rankings for evaluation.  \trec opened using the April 1st \cord version and received submissions from over 55 participating teams.

\section{Discussion}
\label{sec:discussion}

Several hundred new papers on \covid are now being published every day. Automated methods are needed to analyze and synthesize information over this large quantity of content. The computing community has risen to the occasion, but it is clear that there is a critical need for better infrastructure to incorporate human judgments in the loop. Extractions need expert vetting, and search engines and systems must be designed to serve users. 

Successful engagement and usage of \cord speaks to our ability to bridge computing and biomedical communities over a common, global cause. From early results of the Kaggle challenge, we have learned which formats are conducive to collaboration, and which questions are the most urgent to answer. However, there is significant work that remains for determining \textit{(i)} which methods are best to assist textual discovery over the literature, \textit{(ii)} how best to involve expert curators in the pipeline, and \textit{(iii)} which extracted results convert to successful \covid treatments and management policies. Shared tasks and challenges, as well as continued analysis and synthesis of feedback will hopefully provide answers to these outstanding questions.

Since the initial release of \cord, we have implemented several new features based on community feedback, such as the inclusion of unique identifiers for papers, table parses, more sources, and daily updates. Most substantial outlying features requests have been implemented or addressed at this time. We will continue to update the dataset with more sources of papers and newly published literature as resources permit.

\subsection{Limitations}

Though we aim to be comprehensive, \cord does not cover many relevant scientific documents on \covid. We have restricted ourselves to research papers and preprints, and do not incorporate other types of documents, such as technical reports, white papers, informational publications by governmental bodies, and more. Including these documents is outside the current scope of \cord, but we encourage other groups to curate and publish such datasets.

Within the scope of scientific papers, \cord is also incomplete, though we continue to prioritize the addition of new sources. This has motivated the creation of other corpora supporting \covid NLP, such as LitCovid \citep{Chen2020KeepUW}, which provide complementary materials to \cord derived from PubMed. Though we have since added PubMed as a source of papers in \cord, there are other domains such as the social sciences that are not currently represented, and we hope to incorporate these works as part of future work. 

We also note the shortage of foreign language papers in \cord, especially Chinese language papers produced during the early stages of the epidemic. These papers may be useful to many researchers, and we are working with collaborators to provide them as supplementary data. However, challenges in both sourcing and licensing these papers for re-publication are additional hurdles.

\subsection{Call to action}

Though the full text of many scientific papers are available to researchers through \cord, a number of challenges prevent easy application of NLP and text mining techniques to these papers. First, the primary distribution format of scientific papers -- PDF -- is not amenable to text processing. The PDF file format is designed to share electronic documents rendered faithfully for reading and printing, and mixes visual with semantic information. Significant effort is needed to coerce PDF into a format more amenable to text mining, such as JATS XML,\footnote{\label{footnote:jats}\href{https://www.niso.org/publications/z3996-2019-jats}{https://www.niso.org/publications/z3996-2019-jats}} BioC \citep{Comeau2019PMCTM}, or S2ORC JSON \citep{lo-wang-2020-s2orc}, which is used in \cord. Though there is substantial work in this domain, we can still benefit from better PDF parsing tools for scientific documents. As a complement, scientific papers should also be made available in a structured format like JSON, XML, or HTML.

Second, there is a clear need for more scientific content to be made accessible to researchers. Some publishers have made \covid papers openly available during this time, but both the duration and scope of these epidemic-specific licenses are unclear. Papers describing research in related areas (e.g., on other infectious diseases, or relevant biological pathways) have also not been made open access, and are therefore unavailable in \cord or otherwise. 
Securing release rights for papers not yet in \cord but relevant for \covid research is a significant portion of future work, led by the PMC \covid Initiative.\textsuperscript{\ref{footnote:pmc_covid}}

Lastly, there is no standard format for representing paper metadata. Existing schemas like the JATS XML NISO standard\textsuperscript{\ref{footnote:jats}} or library science standards like \textsc{bibframe}\footnote{\href{https://www.loc.gov/bibframe/}{https://www.loc.gov/bibframe/}} or Dublin Core\footnote{\href{https://www.dublincore.org/specifications/dublin-core/dces/}{https://www.dublincore.org/specifications/dublin-core/dces/}} have been adopted to represent paper metadata. However, these standards can be too coarse-grained to capture all necessary paper metadata elements, or may lack a strict schema, causing representations to vary greatly across publishers who use them. 
To improve metadata coherence across sources, the community must define and agree upon an appropriate standard of representation.
\subsection*{Summary}

This project offers a paradigm of how the community can use machine learning to advance scientific research.
By allowing computational access to the papers in \cord, we increase our ability to perform discovery over these texts. 
We hope the dataset and projects built on the dataset will serve as a template for future work in this area. We also believe there are substantial improvements that can be made in the ways we publish, share, and work with scientific papers. We offer a few suggestions that could dramatically increase community productivity, reduce redundant effort, and result in better discovery and understanding of the scientific literature.

Through \cord, we have learned the importance of bringing together different communities around the same scientific cause. 
It is clearer than ever that automated text analysis is not the solution, but rather one tool among many that can be directed to combat the \covid epidemic. 
Crucially, the systems and tools we build must be designed to serve a use case, whether that's improving information retrieval for clinicians and medical professionals, summarizing the conclusions of the latest observational research or clinical trials, or converting these learnings to a format that is easily digestible by healthcare consumers.
\section*{Acknowledgments}

This work was supported in part by NSF Convergence Accelerator award 1936940, ONR grant N00014-18-1-2193, and the University of Washington WRF/Cable Professorship.

We thank The White House Office of Science and Technology Policy, the National Library of Medicine at the National Institutes of Health, Microsoft Research, Chan Zuckerberg Initiative, and Georgetown University's Center for Security and Emerging Technology for co-organizing the \cord initiative. We thank Michael Kratsios, the Chief Technology Officer of the United States, and The White House Office of Science and Technology Policy for providing the initial seed set of questions for the Kaggle \cord research challenge. 
We thank Kaggle for coordinating the \cord research challenge. In particular, we acknowledge Anthony Goldbloom for providing feedback on \cord and for involving us in discussions around the Kaggle literature review tables project. We thank the National Institute of Standards and Technology (NIST), National Library of Medicine (NLM), Oregon Health and Science University (OHSU), and University of Texas Health Science Center at Houston (UTHealth) for co-organizing the \trec shared task. In particular, we thank our co-organizers --  Steven Bedrick (OHSU), Aaron Cohen (OHSU), Dina Demner-Fushman (NLM), William Hersh (OHSU), Kirk Roberts (UTHealth), Ian Soboroff (NIST), and Ellen Voorhees (NIST) -- for feedback on the design of \cord.

We acknowledge our partners at Elsevier and Springer Nature for providing additional full text coverage of papers included in the corpus. 

We thank Bryan Newbold from the Internet Archive for providing feedback on data quality and helpful comments on early drafts of the manuscript. 

We thank Rok Jun Lee, Hrishikesh Sathe, Dhaval Sonawane and Sudarshan Thitte from IBM Watson AI for their help in table parsing.

We also acknowledge and thank our collaborators from AI2: Paul Sayre and Sam Skjonsberg for providing front-end support for \cord and \trec, Michael Schmitz for setting up the \cord Discourse community forums, Adriana Dunn for creating webpage content and marketing, Linda Wagner for collecting community feedback, Jonathan Borchardt, Doug Downey, Tom Hope, Daniel King, and Gabriel Stanovsky for contributing supplemental data to the \cord effort, Alex Schokking for his work on the Semantic Scholar \covid Research Feed, Darrell Plessas for technical support, and Carissa Schoenick for help with public relations.
\bibliography{cord19}
\bibliographystyle{acl_natbib}

\appendix

\section{Table parsing results}
\label{app:tables}

\begin{table*}[th!]
\centering
\small
\begin{tabular}{llL{40mm}}
    \toprule
    \textbf{PDF Representation} & \textbf{HTML Table Parse} & \textbf{Source \& Description} \\
    \midrule
    \raisebox{-0.6\totalheight}{\includegraphics[width=0.32\textwidth]{tables/pdf1.png}} & \raisebox{-0.6\totalheight}{\includegraphics[width=0.35\textwidth]{tables/parse1.png}} & From \citet{Hothorn2020RelativeCD}: Exact Structure; Minimal row rules \\ [2.0cm]
    \raisebox{-0.6\totalheight}{\includegraphics[width=0.32\textwidth]{tables/pdf2.png}} & \raisebox{-0.6\totalheight}{\includegraphics[width=0.35\textwidth]{tables/parse2.png}} & From \citet{LpezFando2020ManagementOF}: Exact Structure; Colored rows \\ [1.4cm]
    \raisebox{-0.6\totalheight}{\includegraphics[width=0.32\textwidth]{tables/pdf3.png}} & \raisebox{-0.6\totalheight}{\includegraphics[width=0.35\textwidth]{tables/parse3.png}} & From \citet{Stringhini2020SeroprevalenceOA}: Minor span errors; Partially colored background with minimal row rules \\ [2.0cm]
    \raisebox{-0.6\totalheight}{\includegraphics[width=0.32\textwidth]{tables/pdf4.png}} & \raisebox{-0.6\totalheight}{\includegraphics[width=0.35\textwidth]{tables/parse4.png}} & From \citet{Fathi2020PROGNOSTICVO}: Overmerge and span errors; Some section headers have row rules \\ [2.2cm]
    \raisebox{-0.6\totalheight}{\includegraphics[width=0.32\textwidth]{tables/pdf5.png}} & \raisebox{-0.6\totalheight}{\includegraphics[width=0.35\textwidth]{tables/parse5.png}} & From \citet{Kaushik2020MultisystemIS}: Over-splitting errors; Full row and column rules with large vertical spacing in cells \\
    \bottomrule
\end{tabular}
\caption{A sample of table parses. Though most table structure is preserved accurately, the diversity of table representations results in some errors.}
\label{tab:table_parses}
\end{table*}

There is high variance in the representation of tables across different paper PDFs. The goal of table parsing is to extract all tables from PDFs and represent them in HTML table format, along with associated titles and headings. In Table \ref{tab:table_parses}, we provide several example table parses, showing the high diversity of table representations across documents, the structure of resulting parses, and some common parse errors.
\end{document}


Peer review:
CORD-19 is an excellent resource with an impressive integration work for the research community to fight COVID-19.
The authors present the CORD-19 data set and describe how it has been developed and continues to be developed. The CORD-19 data set is a valuable resource that provides access to the latest literature about COVID-19 and coronaviruses and it is updated daily with over 200k downloads. The generation of the CORD-19 requires a coordinated integration and processing effort that is significant. The contribution of this corpus is of high significance and will have a strong impact on the biomedical domain and support the development, for instance, of COVID-19 vaccines. The manuscript is clearly written and it is easy to understand.

The effort in providing a version of the latest literature in formats that can be processed by text analytics methods is excellent, using the latest of the available technology to do so. In the paper, it is mentioned in the manuscript that there are some problems in turning tables into structured format and the authors provide examples of issues that they have found. Table processing is done by IBM, who has as well a method for table processing that seems to be resilient to the problems mentioned and would be relevant to consider it for table processing (https://arxiv.org/abs/1911.10683).

The authors give an example of conflict from which it can be inferred that the same DOI might be linked to two different PubMed identifiers, the reviewer is curious why this might be the case and if an example could be provided.

When you mention “Classification of CORD-19 papers to Microsoft Academic Graph”, is this classification done by a method provided by the authors? is this classification provided as meta-data?

During my review the only typo I could find is:
* “other research activity.” —> “other research activities.”?
* “by not allowing republication of **an** paper”, an —> a

Please consider the following guideline for NLM trademarks: https://www.nlm.nih.gov/about/trademarks.html
Rating: 9: Top 15% of accepted papers, strong accept
Confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature