Creation of an Annotated Corpus of Spanish Radiology Reports	This paper presents a new annotated corpus of 513 anonymized radiology reports written in Spanish. Reports were manually annotated with entities, negation and uncertainty terms and relations. The corpus was conceived as an evaluation resource for named entity recognition and relation extraction algorithms, and as input for the use of supervised methods. Biomedical annotated resources are scarce due to confidentiality issues and associated costs. This work provides some guidelines that could help other researchers to undertake similar tasks.

Oronoz:2015:corpus presented an annotated dataset in Spanish for adverse drug reactions analysis. There are different kind of medical reports. In our case, reports are very short, sentences are not always well formed and many of them have a telegraphic style. They contain spelling mistakes and the use of non-standard abbreviations and acronyms is frequent. This, added to the use of specialized language of the medical domain, makes the annotation task difficult. This work describes the annotation schema, the main guidelines and a brief analysis of the resulting corpus. We are evaluating the possibility of releasing the dataset publicly.

Discussion and Conclusions	We presented a manually annotated corpus of entities and relations in radiology reports written in Spanish. The goal was twofold: to have an annotated dataset available for evaluating NER and RE algorithms results and for training of supervised models and to present an annotation guideline that can be used by other researchers with similar needs. The creation of the corpus was not an easy task. Many annotation-revision iterations had to be performed in order to arrive to a stabilized annotation schema. According to what we expected, INLINEFORM0 improved in each annotation iteration step. Data had to be anonymized. Furthermore, the shortness of the texts, the abundance of abbreviations and acronyms (about 6% of the AE and FI are written as such and there are 105 different abbreviations or acronyms in 513 reports), the specificity of the medical language, the existence of multi-segment terms and the existence of relations between sentences, makes not only the NER and RE tasks, but also the annotation task a difficult one. The relation of findings with temporal terms, negation terms and uncertainty terms should be taken into account to determine their factuality.

This table exhibits the name of the relation and the entities involved in it. The relation occurs in, for instance, is always constructed between a finding and anatomical entity and explicits in which anatomical entity the finding occurred. A texture can be related by the texture relation to a finding or to an anatomical entity. Reports were annotated according to the following main guidelines: Finally, a concept, that we called multisegment term, was introduced: constructions like intra and extrahepatic had to lead to the annotation of the entities intrahepatic and extrahepatic. Examples of our annotated corpus are given in Figure FIGREF16 and FIGREF17 . In order to decrease the annotation time, entities, negation and uncertainty terms were pre-annotated automatically. Therefore, regular expressions, UMLS and a manually-created dictionary were used. Based on the annotation guideline, two native speakers of Spanish annotated the pre-annotated reports using brat  . Annotations wrongly made by the pre-annotation tool were corrected and missing concepts were included. Relations were introduced by the annotators. Overall, annotators worked for approximately a total of 160 hours on the generation of the corpus.

Introduction	The availability of annotated corpora from the biomedical domain, in particular for non-English texts, is scarce. There are two main reasons for that: the generation of new annotated data is expensive due to the need of expert knowledge and to privacy issues: the patient and the physician should not be identified from the texts. So, although the availability of annotated data is a highly valuable asset for the research community, it is very difficult to access it. We are interested in supporting physicians with automatic text processing methods, such as named entity recognition (NER), relation extraction (RE), and negation and uncertainty detection in Spanish radiology reports. The extraction of entities and relations from the reports could suggest possible medical problems, that might lead to surgical interventions, such as seen in Do:2013, Morioka:2016 and Lakhani:2009. To the best of our knowledge, there are no publicly available annotated datasets of Spanish medical reports for these tasks. For this reason, this work focuses on creating an annotated corpus of Spanish radiology reports. There are some datasets available for other languages in the clinical domain, eg. for English  ,  ,  , Swedish  , French  , Polish  and German  .

The abundance of negated findings (56%) might lead to the implementation of methods to detect negated findings in reports (see negex:2001, and CotikACLNegExSpanish for Spanish). The difference of criteria among the annotators helps us determine that the evaluation of NER systems is not an easy task. Finally, the pre-annotation helped us speed the annotation process, although it might have biased the annotation results.

Dataset Analysis	Among other entities, 4398 (405 different) AE, 2637 (745) FI, 1489 (51) NT and 109 (26) UT have been annotated. There are 2161 (750) occurs in and 1478 (164) negates, among other relations. There appear 470 abbreviations or acronyms corresponding to AE and 7 to FI. 7.89% (867 out of a total of 10987) of the relations are across-sentence relations. The inter-annotator agreement (IAA) for the final annotation schema is 0.89. It was calculated for the 61 reports annotated by both annotators on a token level using the Cohen's Kappa coefficient ( INLINEFORM0 )  .

In addition to that, the authors had various discussions to define the final annotation schema based on previous annotations revisions and annotators doubts. We assume that with a stable annotation schema and once the annotators have less doubts, the annotation process would be quicker.

Annotation process	We developed an annotation guideline, which we improved with three iterations of a process consisting of annotation and revision of doubts in the criteria. A set of 513 different kinds of ultrasound reports (e.g. kidney, abdominal, small parts) that were written in a hospital in Argentina were selected for annotation. They contain only one section that includes findings, conclusions and suggestions. The reports were anonymized by removing the date of the study, the report number and the patient identification number. Additionally, information about the physicians that performed the study was removed. Regular expressions have been used for this purpose considering the different ways of writing the title of the physicians (e.g. DR, Dr., doctor, Dra.), the doctor's names, the enrollment numbers and the order among these terms. Also names of the doctors appearing with titles or enrollments were searched to see if they appeared without titles and without enrollments and were removed. The main entities and characteristics that were annotated are presented in Table TABREF2 . An example is included for each case. Abbreviations and acronyms were only annotated for anatomical entities and findings. Table TABREF3 shows the main relations annotated.

Question: Did they experiment on this corpus?
Answer: No