D4.3 Update to Annotation guidelines, tools & training
Creators: name: Kreuzthaler, Markus
name: Aghaei, Sareh
name: Collins, Kris
name: Schulz, Stefan
name: Kalendralis, Petros
name: Kankainen, Kristian
name: De Zegher, Isabelle
Publisher: name: Zenodo
Year: 2023
Subjects: manual annotation guidelines
manual annotation tools

description: Manual annotations of clinical narratives are crucial for the adoption and evaluation of NaturalLanguage Processing (NLP) tools, which support an overall AI-assisted data curation approach withinAIDAVA. For a symbolic representation of clinical entities of interest and the way how they are related,normalisations that use international standards like SNOMED CT, FHIR or LOINC are crucial. For thisdeliverable, we updated the first version of the manual annotation guideline (see AIDAVA DeliverableD4.1), where requirements for annotation tooling were formulated with respect to the AIDAVA usecases, together with some initial annotation instructions. Grounded on this requirement analysis,INCEpTION was chosen as an annotation tool after a rigorous investigation of available annotationsoftware. A first manual annotation schema was developed and tested, with a focus on the use ofSNOMED CT and FHIR for the normalisation of the types of clinical entities (as annotating them withterminology codes) referred to by clinical narratives. Within this preparation phase, INCEpTION wasdeployed on all three clinical sites (MUG, NEMC, MUMC), with a first version of a consolidatedINCEpTION layer definition. A bi-weekly “train the trainers” session was started at the end of 2022,supporting a continuous transition into the piloting phase of the developed guideline, analysingexample narratives and how they should be annotated according to the first version of the guideline.Within the piloting phase lasting from January 2023 to May 2023, in communication with theresponsible clinicians, relevant attributes were identified, and a selection of them was used forupdating, testing and refinement of the annotation guideline. Annotators were recruited at all threedifferent sites and their feedback was taken into account for the customization and technical set up ofINCEpTION. Alignment with Deliverable D2.1 "Reference Ontology as a Global Data Sharing Standard"defining the AIDAVA Reference Ontology was identified as crucial, therefore this deliverable waspostponed for one month from April to May 2023.Building on the first version of the guideline delivered early January 2023, this updated descriptiveguideline provides a comprehensive framework and detailed instructions to ensure accurateannotation of clinical narratives. It covers crucial aspects like data standardisation and best practicesin annotation (including annotation tool, general principles, specific instructions, concrete examples,and quality control items), ensuring consistent, interoperable, and high-quality annotations. This isinvaluable for effective knowledge graph construction, data analysis, and knowledge extraction ascentral requirements in AIDAVA.The updated annotation instructions form the core of this deliverable, enabling to start the productivephase of the manual annotation. Manual annotation of texts is iterative and dynamic. It is, therefore,crucial to recognise potential updates and improvements that may arise during the productive phase.Factors that can contribute to the modifications and enhancements of the set of annotationinstructions include active feedback from the annotation team, new insights into text phenomena thatlead to annotator disagreements, updates in data requirements from use cases, and evolution ofproject objectives as a result of dissemination and communication activities during the project. Toensure consistency and minimise inconsistencies in the annotation work, a structured feedbackmechanism is established, involving documenting any challenges or updates in a shared document,and conducting meetings with the annotation team to address any emerging insights or challenges.