## 2 Materials and Methods  
Prototype conception and development was preceded by a systematic requirements analysis for annotation tooling. The results from the requirements analysis were formulated as user stories and grouped into \"must\", \"should\", and \"could\" requirements. Following an agile approach, the development took place in three development cycles. Each cycle focused on a defined set of user stories and produced new requirements for the next development cycle. The web app was implemented in Java SE 11 using the Spring Boot framework as a fast and easy way to benefit from the advantages of a Spring application. Vaadin 14 was used to develop a modern web graphical user interface with Java code using the framework's built-in UI components. We implemented three primary features going beyond the feature set of typical tooling choices for semantic annotation. 1. Collaborative annotation: Searching for pre-existing annotations from local sites or other participating sites for fitting terminology codes as a seamlessly integrated part of the annotation process itself allows users to see what local codes from the same or other participating sites have previously been annotated using these terminology codes. Annotators can profit from comparing their local codes to other local codes previously annotated with a terminology code to determine if the terminology code is fitting based on the equivalence or similarity of the local codes under consideration. 2. Medical Data Models (MDM) portal integration: As terminologies like SNOMED CT contain hundreds of thousands of codes, many existing annotations originating from many participating sites would be needed to regularly benefit from this feature while using CoAT. To give annotators access to more input in this early pilot phase of implementation, a web service with access to annotations from the Medical Data Models portal (MDM portal) was integrated [18; 19], effectively turning MDM into a large contributing annotation collaborator. The MDM portal is an online resource providing over 25,000 semantically annotated structured data acquisition forms [20], with coverage ranging from numerous electronic case report forms from clinical studies to documentation forms from clinical information systems including fully annotated versions of the 2015 state of intensive care documentation systems from both Bonn and Munster [21]. MDM integration was realized using its REST API. 3. Cross-terminology mapping: Medical ontologies and terminologies typically describe overlapping knowledge domains and are used to support different application scenarios, implying that annotating source data items with multiple semantic codes from different ontologies and/or terminologies may make annotation results more broadly applicable and practically useful. We therefore enabled CoAT to facilitate cross-/multiterminology annotation by seamlessly integrating the cross-terminology mapping information from the UMLS Metathesaurus [22] into CoAT for semi-automatic multi-terminology annotation. Since only metadata but no patient specific data is to be processed, the implementation poses no risks to patient privacy if used as intended. Risks to staff privacy were minimized by limiting the processing of staff-related data items to those strictly necessary to allow for authentication and authorization of users. The prototype was first evaluated regarding its usability in an intra-site application in Bonn. Three test users were recruited: one physician, one nurse, and one medical documenter. To evaluate the usability, these test users performed think-aloud tests [23] where they had to complete a given set of tasks in the tool that covered the whole annotation process including usage of the collaborative features. While doing so, the test users verbalized their thoughts and were recorded. The records were transcribed to document where users encountered challenges. The test lead judged users' success or failure in the completion of the tasks based on pre-defined expected results.