DOI: 10.5061/dryad.3tx95x6dz
title: NLMChem a new resource for chemical entity recognition in PubMed full-text literature

description: Automatically identifying chemical and drug names in scientific
 publications advances information access for this important class of
 entities in a variety of biomedical disciplines by enabling improved
 retrieval and linkage to related concepts. While current methods for
 tagging chemical entities were developed for the article title and
 abstract, their performance in the full article text is substantially
 lower. However, the full text frequently contains more detailed chemical
 information, such as the properties of chemical compounds, their
 biological effects, and interactions with diseases, genes, and other
 chemicals.  We, therefore, present the NLM-Chem corpus, a
 full-text resource to support the development and evaluation of automated
 chemical entity taggers. The NLM-Chem corpus consists of 150 full-text
 articles, doubly annotated by ten expert NLM indexers, with ~5000 unique
 chemical name annotations, mapped to ~2000 MeSH identifiers. Using this
 corpus, we built a substantially improved chemical entity tagger,
 with automated annotations for all of PubMed and PMC freely accessible
 through the PubTator web-based interface and API. 
description: NLM-Chem corpus consists of 150 full-text articles from the
 PubMed Central Open Access dataset, comprising 67 different chemical
 journals, aiming to cover a general distribution of usage of chemical
 names in the biomedical literature. Articles were selected so that human
 annotation was most valuable (meaning that they were rich in bio-entities,
 and current state-of-the-art named entity recognition systems disagreed on
 bio-entity recognition.  Ten indexing experts at the
 National Library of Medicine manually annotated the corpus using the
 TeamTat annotation system that allows swift annotation project management.
 The corpus was annotated in three batches and each batch of articles was
 annotated in three annotation rounds. Annotators were randomly paired for
 each article, and pairings were randomly shuffled for each subsequent
 batch. In this manner, the workload was distributed fairly. To control for
 bias, annotator identities were hidden the first two annotation rounds. In
 the final annotation rounds, annotators worked collaboratively to resolve
 the final few annotation disagreements and reach a 100%
 consensus.  The full-text articles were fully annotated
 for all chemical name occurrences in text, and the chemicals were mapped
 to Medical Subject Heading (MeSH) entries to facilitate indexing and other
 downstream article processing tasks at the National Library of Medicine.
 MeSH is part of the UMLS and as such, chemical entities can be mapped to
 other standard vocabularies.  The data has been
 evaluated for high annotation quality, and its use as training data has
 already improved chemical named entity recognition in PubMed. The newly
 improved system has already been incorporated in the PubTator API tools
 (https://www.ncbi.nlm.nih.gov/research/pubtator/api.html).
description: We include the document of annotation guidelines, which makes it
 clear that the corpus can be combined with ChemDNER and BC5CDR corpora,
 which contain chemical name annotations, and name and MeSH annotations for
 chemicals respectively to further improve Chemical NER in biomedical
 literature.  The corpus has been divided
 into train/dev/test to facilitate benchmarking and comparisons. 
 The data annotations are inline in the BioC XML format, which is
 a minimalistic approach to facilitate text mining. We also maintain a copy
 here: https://www.ncbi.nlm.nih.gov/research/bionlp/