Question: What are the three tasks they identify?
Evidence: Medical providers across the United States are required to document clinical visits in the Electronic Health Records. This need for documentation takes up a disproportionate amount of their time and attention, resulting in provider burnout BIBREF0, BIBREF1. One study found that full-time primary care physicians spent about 4.5 hours of an 11-hour workday interacting with the clinical documentation systems, yet were still unable to finish their documentation and had to spend an additional 1.4 hours after normal clinical hours BIBREF2.
Speech and natural language processing are now sufficiently mature that there is considerable interest, both in academia and industry, to investigate how these technologies can be exploited to simplify the task of documentation, and to allow providers to dedicate more time to patients. While domain-specific automatic speech recognition (ASR) systems that allow providers to dictate notes have been around for a while, recent work has begun to address the challenges associated with generating clinical notes directly from speech recordings. This includes inducing topic structure from conversation data, extracting relevant information, and clinical summary generation BIBREF3. In one recent work, authors outlined an end-to-end system; however, the details were scant without empirical evaluations of their building blocks BIBREF4. One of the simplistic approaches uses a hand crafted finite state machine based grammar to locate clinical entities in the ASR transcripts and map them to canonical clinical terms BIBREF5. This seems to perform well in a narrowly scoped task. A more ambitious approach mapped ASR transcripts to clinical notes by adopting a machine translation approach BIBREF6. However this performed poorly. To address the difficulty in accessing clinical data, researchers have experimented with synthetic data to develop a system for documenting nurse-initiated telephone conversations for congestive heart failure patients who are undergoing telemonitoring after they have been discharged from the hospital BIBREF7. In their task, a question-answer based model achieved an F-score of 0.80. This naturally raises the question of how well state-of-art techniques will perform in helping the broader population of clinicians such as primary care providers.
One might expect that the task of extracting clinical concepts from audio faces challenges similar to the domain of unstructured clinical texts. In that domain, one of the earliest public-domain tasks is the i2b2 relations challenge, defined on a small corpus of written discharge summaries consisting of 394 reports for training, 477 for test, and 877 for evaluation BIBREF8. Given the small amount of training data, not surprisingly, a disproportionately large number of teams fielded rule-based systems. Conditional random field-based (CRF) systems BIBREF9 however did better even with the limited amount of training data BIBREF10. Other i2b2/n2c2 challenges focused on coreference resolution BIBREF11, temporal relation extraction BIBREF12, drug event extraction BIBREF13 on medical records, and extracting family history BIBREF14. Even though the text was largely unstructured, they benefited from punctuation and capitalization, section headings and other cues in written domain which are unavailable in audio to the same extent.
With the goal of creating an automated medical scribe, we broke down the task into modular components, including ASR and speaker diarization which are described elsewhere BIBREF15. In this work, we investigate the task of extracting relevant clinical concepts from transcripts. Our key contributions include: (i) defining three tasks – the Medications Task, the Symptoms Task, and the Conditions Task along with principles employed in developing the annotation guidelines for them (Section SECREF2); (ii) measuring the label quality using inter-labeler agreements and refining the quality iteratively (Section SECREF3), (iii) evaluating the performance of the state-of-the-art models on these tasks (Section SECREF4), and (iv) a comprehensive analysis of the performance of the models including manual error categorization (Section SECREF5). The corpus we have created in this work is based on private, proprietary data that cannot be publicly shared. Instead, we are sharing the learnings from our experience that might be useful for the wider community as well as the detailed labeling guidelines as supplementary material in the extended version of this paper on arxiv.org.
Our key contributions include: (i) defining three tasks – the Medications Task, the Symptoms Task, and the Conditions Task along with principles employed in developing the annotation guidelines for them (Section 2.);
Answer: The Medications Task, the Symptoms Task, and the Conditions Task.