A Reproducible Approach with R Markdown to ,Automatic Classification of Medical Certificates in French
Authors: Giorgio Maria Di Nunzio, Federica Beghini, Federica Vezzani, Geneviéve Henrot (Università di Padova, 2017)
Topic: In-domain IR and IE

Abstract
In this paper, we report the ongoing developments of our first participation to the Cross-Language Evaluation Forum (CLEF) eHealth Task 1: ``Multilingual Information Extraction - ICD10 coding" AUTHOR. The task consists in labelling death certificates, in French with international standard codes. In particular, we wanted to accomplish the goal of the `Replication track' of this Task which promotes the sharing of tools and the dissemination of solid, reproducible results.

Introduction
When researchers use `traditional' methods of scientific publication to describe computational research, we, as readers and researchers, may incur into the so-called `reproducible research' problem AUTHOR. For example, a traditional conference paper usually specifies the relevant computations of the main approach, because the limitations of a paper medium prohibit a complete documentation, which would ideally include experimental data, parameter values, and the source code of the program. Those readers who wish to use the same approach of the paper, hence reproduce the results, must reimplement the whole process, which sometimes may be an unfeasible task. The extreme of reproducibility is `replicability', i.e. a perfect replica of a scientific experiment. The discussion of the difference between replicability and reproducibility is beyond the scope of this paper AUTHOR, and we will just point out that, in general, even in the most accurate replica of an experiment will be done by a different person, in a different lab, using different equipment. Researchers of different areas have identifyied the necessity for reproducibility, or reproducible research, as an attainable minimum standard for assessing the value of scientific claims AUTHOR. As Roger Peng suggests, ``one aim of the reproducibility standard is to fill the gap in the scientific evidence-generating process between full replication of a study and no replication. Between these two extreme end points, there is a spectrum of possibilities, and a study may be more or less reproducible than another depending on what data and code are made available''. Reproducibility matters because the lack of reproducibility in science causes significant issues for science itself, for other researchers in the community, and for public policy. For example, Nature published a special issue about ``Challenges in Irreproducible Research''} where the examined cases showed that there is [ \dots ] a growing alarm about results that cannot be reproduced. Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions and funders all have a part in tackling reproducibility. Among many other problems, the article showed that most of the drug validation studies (43 out of 67 studies) failed to reproduce. Another important case concerned Science, where the Editor-in-Chief retracted in 2015 a study of how canvassers can sway people's opinions about gay marriage because: `` (i) Survey incentives were misrepresented [ \dots ], (ii) The statement on sponsorship was false. [ \dots ]'' There are also cases of papers retracted by authors themselves because ``After carefully re-examining the data presented in the article, they identified that data of two different hospitals got terribly mixed. The published results cannot be reproduced in accordance with scientific and clinical correctness.'' as declared in the note of retraction of the paper ``Low Dose Lidocaine for Refractory Seizures in Preterm Neonates'' AUTHOR.

Conclusion
The aim of this work was to continue the work on the reproducible research approach that can be used as a baseline for further experiments. The performance of the system that uses a semi-manual spell-checking approach improved the baseline set by the original paper. The documentation produced for the reproducibility approach helped us to spot bugs during the implementation phase and we strongly believe that this type of actions should be supported more and more because, as reported by the analysis who tested the systems at CLEF eHealth “[ . . . ] still experienced varying degrees of difficulty to install and run the systems. [ . . . ] Analysts also report that additional information on system requirements, installation procedure and practical use would be useful for all the systems submitted, although documentation was overall more abundant and detailed compared to last year’s experiments. [. . . ] The results of the experiments suggest that replication is achievable. However, it continues to be more of a challenge than one would hope.”