Abstract: Natural Language Processing (NLP) systems often make use of machine learning techniques that are unfamiliar to end-users who are interested in analyzing clinical records. Although NLP has been widely used in extracting information from clinical text, current systems generally do not support model revision based on feedback from domain experts. We present a prototype tool that allows end users to visualize and review the outputs of an NLP system that extracts binary variables from clinical text. Our tool combines multiple visualizations to help the users understand these results and make any necessary corrections, thus forming a feedback loop and helping improve the accuracy of the NLP models. We have tested our prototype in a formative think-aloud user study with clinicians and researchers involved in colonoscopy research. Results from semi-structured interviews and a System Usability Scale (SUS) analysis show that the users are able to quickly start refining NLP models, despite having very little or no experience with machine learning. Observations from these sessions suggest revisions to the interface to better support review workflow and interpretation of results.

Introduction: Electronic Health Records (EHRs) are organized collections of information about individual patients. They are designed such that they can be shared across different settings for providing health care services. The Institute of Medicine committee on improving the patient record has recognized the importance of using EHRs to inform decision support systems and support data-driven quality measures BIBREF0 . One of the biggest challenges in achieving this goal is the difficulty of extracting information from large quantities of EHR data stored as unstructured free text. Clinicians often make use of narratives and first-person stories to document interactions, findings and analyses in patient cases BIBREF1 . As a result, finding information from these volumes of health care records typically requires the use of NLP techniques to automate the extraction process. There has been a long history of research in the application of NLP methods in the clinical domain BIBREF2 . Researchers have developed models for automatically detecting outbreak of diseases such as influenza BIBREF3 , identifying adverse drug reactions BIBREF4 , BIBREF5 , BIBREF6 , and measuring the quality of colonoscopy procedures BIBREF7 , among others. Due to the complexity of clinical text, the accuracy of these techniques may vary BIBREF8 . Current tools also lack provision for end-users to inspect NLP outcomes and make corrections that might improve these results. Due to these factors, Chapman et. al. BIBREF2 have identified “lack of user-centered development" as one of the barriers in NLP adoption in the clinical domain. There is a need to focus on development of NLP systems that are not only generalizable for use in different tasks but are also usable without excessive dependence on NLP developers. In this paper, we have explored the design of user-interfaces for use by end users (clinicians and clinical researchers) to support the review and annotation of clinical text using natural language processing. We have developed an interactive web-based tool that facilitates both the review of binary variables extracted from clinical records, and the provision of feedback that can be used to improve the accuracy of NLP models. Our goal is to close the natural language processing gap by providing clinical researchers with highly-usable tools that will facilitate the process of reviewing NLP output, identifying errors in model prediction, and providing feedback that can be used to retrain or extend models to make them more effective. 2em

Question: How is annotation projection done when languages have different word order?
Answer: Word alignments are generated for parallel text, and aligned words are assumed to also share AMR node alignments.