Introduction
In this paper, we describe our approach and present the results for our participation in the task 1, i.e. multilingual information extraction, of the CLEF eHealth 2018 challenge BIBREF0 . More precisely, this task consists in automatically coding death certificates using the International Classification of Diseases, 10th revision (ICD-10) BIBREF1 .
We addressed the challenge by matching ICD-10 terminology entries to text phrases in death certificates. Matching text phrases to medical concepts automatically is important to facilitate tasks such as search, classification or organization of biomedical textual contents BIBREF2 . Many concept recognition systems already exist BIBREF2 , BIBREF3 . They use different approaches and some of them are open source. We developed a general purpose biomedical semantic annotation tool for our own needs. The algorithm was initially implemented to detect drugs in a social media corpora as part of the Drugs-Safe project BIBREF4 . We adapted the algorithm for the ICD-10 coding task. The main motivation in participating in the challenge was to evaluate and compare our system with others on a shared task.
Methods
In the following subsections, we describe the corpora, the terminology used, the steps of pre-processing and the matching algorithm.
Corpora
The data set for the coding of death certificates is called the CépiDC corpus. Three CSV files (AlignedCauses) were provided by task organizers containing annotated death certificates for different periods : 2006 to 2012, 2013 and 2014. This training set contained 125383 death certificates. Each certificate contains one or more lines of text (medical causes that led to death) and some metadata. Each CSV file contains a "Raw Text" column entered by a physician, a "Standard Text" column entered by a human coder that supports the selection of an ICD-10 code in the last column. Table TABREF2 presents an excerpt of these files. Zero to multiples ICD-10 codes can be assigned to each line of a death certificate.
Dictionaries
We constructed two dictionaries based on ICD-10. In practice, we selected all the terms in the "Standard Text" column of the training set to build the first one which was used in the second run. In the first run, we added to this previous set of terms the 2015 ICD-10 dictionary provided by the task organizers. This dictionary contained terms that were not present in the training corpus. When a term was associated with multiple ICD-10 codes in our dictionary, we kept the most frequent one (Table TABREF4 ).
The first dictionary contained 42439 terms and 3,539 ICD-10 codes (run2) and the second one 148448 terms and 6,392 ICD-10 codes (run1).
Metadata on death causes were not used (age, gender, location of death).
Terms pre-processing
All the terms were normalized through accents (diacritical marks) and punctuation removal, lowercasing and stopwords removal (we created a list of 25 stopwords for this task). Then, each term was tokenized and stored in a tree data structure. Each token of a N-gram term is a node in the tree and N-grams correspond to different root-to-leaf paths BIBREF5 (Figure FIGREF6 ).
Matching algorithm
The goal of our algorithm was to recognize one or many dictionary entries in a raw text. An example is given in Figure FIGREF8 . For each raw text entry, the same normalization steps described above were performed first. The raw text was then tokenized. For each token, the algorithm looked for an available dictionary token depending on where it currently was in the tree. For example, the token "cardiaque" was possible after the token "insuffisance" but was not available at the root of the tree.
For each token, the algorithm used three matching techniques : perfect match, abbreviation match and Levenshtein match. The abbreviation match technique used a dictionary of abbreviations. We manually added nine frequent abbreviations after looking at some examples. The Levenshtein matching technique used the Levenshtein distance. It corresponds to the minimum number of single-character edits (insertions, deletions or substitutions) required to change one token into the other. The Lucene™implementation of the Levenshtein distance was used.
In Figure FIGREF8 , the algorithm used these three techniques to match the tokens "ins", "cardiaqu", "aigue" to the dictionary term "insuffisance cardiaque aigue" whose ICD-10 code is I509. As the following token "detresse" was not a dictionary entry at this depth, the algorithm saved the previous and longest recognized term and restarted from the root of the tree. At this new level, "detresse" was detected but as no term was associated with this token alone, no ICD-10 code was saved. Finally, only one term was recognized in this example.
Besides unigrams, bigrams were also indexed in Lucene™to resolve composed words. For example, "meningoencephalite" matched the dictionary entry "meningoencephalite" by a perfect match and "meningo encephalite" thanks to the Levensthein match (one deletion). Therefore, the algorithm entered two different paths in the tree (Figure FIGREF10 ). By combining these different matching methods for each token, the algorithm was able to detect multiple lexical variants. The program was implemented in Java and the source code is on Github.
Results
We submitted two runs on the CépiDC test set, one used all the terms entered by human coders in the training set only (run 2), the other (run 1) added the 2015 ICD-10 dictionary provided by the task organizers to the set the terms of run 1. We obtained our best precision (0.794) and recall (0.779) with run 2.
Table TABREF11 shows the performance of our system with median and average scores of all participants in this task.
Discussion
Surprisingly, adding more terms (run 1) did not improve the recall, which appears to be even slightly worse. The results were quite promising for our first participation in this task, using a general purpose annotation tool.
A limitation of the proposed algorithm that impacted recall was the absence of term detection when adjectives were isolated. For example, in the sentence "metastase hepatique et renale", "metastase renale" was not recognized even though the term existed. This situation seemed to be quite frequent.
Some frequent abbreviations were manually added to improve the recall in this corpora. Improvement at this stage may be possible by automating the abbreviation detection or by adding more entries manually.
In the past, other dictionary-based approaches performed better BIBREF6 . In 2016, the Erasmus system BIBREF7 achieved an F-score of 0.848 without spelling correction techniques. In 2017, the SIBM team BIBREF8 used a dictionary-based approach with fuzzy matching methods and phonetic matching algorithm to obtain an F-score of 0.804.
Further improvement may be possible by using a better curated terminology. We are currently investigating frequent irrelevant codes that may have impacted the precision. A post-processing filtering phase could improve the precision.
We also plan to combine machine learning techniques with a dictionary-based approach. Our system can already detect and replace typos and abbreviations to help machine learning techniques increase their performance.
Affiliation
DRUGS-SAFE National Platform of Pharmacoepidemiology, France
Funding
The present study is part of the Drugs Systematized Assessment in real-liFe Environment (DRUGS-SAFE) research platform that is funded by the French Medicines Agency (Agence Nationale de Sécurité du Médicament et des Produits de Santé, ANSM). This platform aims at providing an integrated system allowing the concomitant monitoring of drug use and safety in France. The funder had no role in the design and conduct of the studies ; collection, management, analysis, and interpretation of the data ; preparation, review, or approval of the manuscript ; and the decision to submit the manuscript for publication. This publication represents the views of the authors and does not necessarily represent the opinion of the French Medicines Agency.
english

Question: How long is the dataset?
Answer: 125383

125383 death certificates

Question: Do they use machine learning?
Answer: No

Unanswerable

Question: What are the ICD-10 codes?
Answer: International Classification of Diseases, 10th revision (ICD-10) BIBREF1

International Classification of Diseases