Conclusion	In the future, we will further improve the CORD-19-NER dataset quality. We will also build text mining systems based on the CORD-19-NER dataset with richer functionalities. We hope this dataset can help the text mining community build downstream applications. We also hope this dataset can bring insights for the COVID-19 studies, both on the biomedical side and on the social side.

This dataset covers 75 fine-grained named entity types. CORD-19-NER is automatically generated by combining the annotation results from four sources. In the following sections, we introduce the details of CORD-19-NER dataset construction. We also show some NER annotation results in this dataset.

Comprehensive Named Entity Recognition on CORD-19 with Distant or Weak Supervision	We created this CORD-19-NER dataset with comprehensive named entity recognition (NER) on the COVID-19 Open Research Dataset Challenge (CORD-19) corpus (2020- 03-13). This CORD-19-NER dataset covers 74 fine-grained named entity types. It is automatically generated by combining the annotation results from four sources: (1) pre-trained NER model on 18 general entity types from Spacy, (2) pre-trained NER model on 18 biomedical entity types from SciSpacy, (3) knowledge base (KB)-guided NER model on 127 biomedical entity types with our distantly-supervised NER method, and (4) seed-guided NER model on 8 new entity types (specifically related to the COVID-19 studies) with our weakly-supervised NER method. We hope this dataset can help the text mining community build downstream applications. We also hope this dataset can bring insights for the COVID- 19 studies, both on the biomedical side and on the social side.

After we get the NER annotation results with the four different methods, we merge the results into one file. The conflicts are resolved by giving priority to different entity types annotated by different methods according to their annotation quality. The final entity annotation results (CORD-19-ner.json) with the file schema and detailed descriptions can be found in our CORD-19-NER dataset.

Introduction	Coronavirus disease 2019 (COVID-19) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The disease was first identified in 2019 in Wuhan, Central China, and has since spread globally, resulting in the 2019–2020 coronavirus pandemic. On March 16th, 2020, researchers and leaders from the Allen Institute for AI, Chan Zuckerberg Initiative (CZI), Georgetown University’s Center for Security and Emerging Technology (CSET), Microsoft, and the National Library of Medicine (NLM) at the National Institutes of Health released the COVID-19 Open Research Dataset (CORD-19) of scholarly literature about COVID-19, SARS-CoV-2, and the coronavirus group. Named entity recognition (NER) is a fundamental step in text mining system development to facilitate the COVID-19 studies. There is critical need for NER methods that can quickly adapt to all the COVID-19 related new types without much human effort for training data annotation. We created this CORD-19-NER dataset with comprehensive named entity annotation on the CORD-19 corpus (2020-03-13).

source: CZI (1236 records), PMC (27337), bioRxiv (566) and medRxiv (361). doi: populated for all BioRxiv/MedRxiv paper records and most of the other records (26357 non null). pmcid: populated for all PMC paper records (27337 non null). pubmed_id: populated for some of the records. Other keys: publish_time, authors and journal. The tokenized corpus (CORD-19-corpus.json) with the file schema and detailed descriptions can be found in our CORD-19-NER dataset.

CORD-19-NER Dataset ::: Corpus	The corpus is generated from the 29,500 documents in the CORD-19 corpus (2020-03-13). We first merge all the meta-data (all_sources_metadata_2020-03-13.csv) with their corresponding full-text papers. Then we create a tokenized corpus (CORD-19-corpus.json) for further NER annotations. Corpus Tokenization. The raw corpus is a combination of the “title", “abstract" and “full-text" from the CORD-19 corpus. We first conduct automatic phrase mining on the raw corpus using AutoPhrase . Then we do the second round of tokenization with Spacy on the phrase-replaced corpus. We have observed that keeping the AutoPhrase results will significantly improve the distantly- and weakly-supervised NER performance. Key Items. The tokenized corpus includes the following items: doc_id: the line number (0-29499) in “all_sources_metadata_2020-03-13.csv" in the CORD-19 corpus (2020-03-13). sents: [sent_id, sent_tokens], tokenized sentences and words as described above.

Results ::: Top-Frequent Entity Summarization	In Table TABREF34, we show some examples of the most frequent entities in the annotated corpus. Specifically, we show the entity types including both our new types and some UMLS types that have not been manually annotated before. We find our annotated entities very informative for the COVID-19 studies. For example, the most frequent entities for the type “SIGN_OR_SYMPTOM behavior" includes “cough" and “respiratory symptoms" that are the most common symptoms for COVID-19 . The most frequent entities for the type “INDIVIDUAL_BEHAVIOR" includes “hand hygiene", “disclosures" and “absenteeism", which indicates that people focus more on hand cleaning for the COVID-19 issue. Also, the most frequent entities for the type “MACHINE_ACTIVITY" includes “machine learning", “data processing" and “automation", which indicates that people focus more on the automated methods that can process massive data for the COVID-19 studies. This type also includes “telecommunication" as the top results, which is quite reasonable under the current COVID-19 situation. More examples can be found in our dataset.

CORD-19-NER Dataset ::: NER Methods	CORD-19-NER annotation is a combination from four sources with different NER methods: Pre-trained NER on 18 general entity types from Spacy using the model “en_core_web_sm". Pre-trained NER on 18 biomedical entity types from SciSpacy using the model “en_ner_bionlp13cg_md". Knowledge base (KB)-guided NER on 127 biomedical entity types with our distantly-supervised NER methods , . We do not require any human annotated training data for the NER model training. Instead, We rely on UMLS as the input KB for distant supervision. Seed-guided NER on 9 new entity types (specifically related to the COVID-19 studies) with our weakly-supervised NER method. We only require several (10-20) human-input seed entities for each new type. Then we expand the seed entity sets with CatE  and apply our distant NER method for the new entity type recognition. The 9 new entity types with examples of their input seed are as follows: Coronavirus: COVID-19, SARS, MERS, etc.

Viral Protein: Hemagglutinin, GP120, etc. Livestock: cattle, sheep, pig, etc. Wildlife: bats, wild animals, wild birds, etc Evolution: genetic drift, natural selection, mutation rate, etc Physical Science: atomic charge, Amber force fields, Van der Waals interactions, etc. Substrate: blood, sputum, urine, etc. Material: copper, stainless steel, plastic, etc. Immune Response: adaptive immune response, cell mediated immunity, innate immunity, etc. We merged all the entity types from the four sources and reorganized them into one entity type hierarchy. Specifically, we align all the types from SciSpacy to UMLS. We also merge some fine-grained UMLS entity types to their more coarse-grained types based on the corpus count. Then we get a final entity type hierarchy with 75 fine-grained entity types used in our annotations. The entity type hierarchy (CORD-19-types.xlsx) can be found in our CORD-19-NER dataset. Then we conduct named entity annotation with the four NER methods on the 75 fine-grained entity types.

Acknowledgment	Research was sponsored in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and SocialSim Program No. W911NF-17-C-0099, National Science Foundation IIS 16-18481, IIS 17-04532, and IIS-17-41317, and DTRA HDTRA11810026. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of DARPA or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation hereon. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

In Figure FIGREF30, we can see that our method can identify many more entities such as “pylogenetic" as a evolution term and “bat" as a wildlife. In Figure FIGREF30, we can also see that our method can identify many more entities such as “racism" as a social behavior. In summary, our distantly- and weakly-supervised NER methods are reliable for high-quality entity recognition without requiring human effort for training data annotation.

Results ::: NER Annotation Results	In Figure FIGREF28, we show some examples of the annotation results in CORD-19-NER. We can see that our distantly- or weakly supervised methods achieve high quality recognizing the new entity types, requiring only several seed examples as the input. For example, we recognized “SARS-CoV-2" as the “CORONAVIRUS" type, “bat" and “pangolins" as the “WILDLIFE" type and “Van der Waals forces" as the “PHYSICAL_SCIENCE" type. This NER annotation results help downstream text mining tasks in discovering the origin and the physical nature of the virus. Our NER methods are domain-independent that can be applied to corpus in different domains. In addition, we show another example of NER annotation on New York Times with our system in Figure FIGREF29. In Figure FIGREF30, we show the comparison of our annotation results with existing NER/BioNER systems. In Figure FIGREF30, we can see that only our method can identify “SARS-CoV-2" as a coronavirus.

Question: Did they experiment with the dataset?
Answer: Yes