Venue: EMNLP 2020

MedFilter: Improving Extraction of Task-relevant Utterances through Integration of Discourse Structure and Ontological Knowledge

Sopan Khosla Shikhar Vashishth Jill Fain Lehman Carolyn Rose

Language Technologies Institute

Carnegie Mellon University, USA

{sopank,svashish,jf1,cprose}@cs.cmu.edu

###### Abstract

Information extraction from conversational data is particularly challenging because the task-centric nature of conversation allows for effective communication of implicit information by humans, but is challenging for machines. The challenges may differ between utterances depending on the role of the speaker within the conversation, especially when relevant expertise is distributed asymmetrically across roles. Further, the challenges may also increase over the conversation as more shared context is built up through information communicated implicitly earlier in the dialogue. In this paper, we propose the novel modeling approach MedFilter, which addresses these insights in order to increase performance at identifying and categorizing task-relevant utterances, and in so doing, positively impacts performance at a downstream information extraction task. We evaluate this approach on a corpus of nearly 7,000 doctor-patient conversations where MedFilter is used to identify medically relevant contributions to the discussion (achieving a \(10\%\) improvement over SOTA baselines in terms of area under the PR curve). Identifying task-relevant utterances benefits downstream medical processing, achieving improvements of \(15\%\), \(105\%\), and \(23\%\) respectively for the extraction of symptoms, medications, and complaints.

## 1 Introduction

In this paper, we propose a novel modeling approach that embodies insights regarding the organization of task-oriented conversations in order to improve performance at utterance classification over SOTA baseline approaches. Task-oriented conversations involve sharing task-relevant information that may be useful as the task ensues Liu et al. (2019); Kazi and Kahanda (2019). Unfortunately, human-to-human conversations are less well structured than expository text, which is more often the source material for information extraction and summarization. Expository text is typically structured top-down and organized around information flow. Task-oriented conversations, on the other hand, are typically organized around the task and knowledge of task structure provides an implicit scaffold for understanding. Thus speakers feel free to elide or imply important information rather than making it explicit. The challenges have been well documented Waitzkin (1989); Lacson et al. (2006). Prior work in utterance classification is a source of SOTA modeling approaches that perform relatively well despite these challenges while leaving much room for improvement.

Our evaluation in this paper specifically focuses on doctor-patient interactions. Doctor-patient interactions are task-oriented, expert-layerson interactions in which the concerns voiced by the layperson (e.g., symptoms), the underlying issue identified by the expert (e.g., complaint) and the prescribed solutions (e.g., medications) play a crucial part. Customer-service chats are another example of such dialogue. As in the general case, topic switching abounds: the doctor may jump from a question about a symptom to a statement providing initial assessment then back again, with or without waiting for a reply from the patient (which may, itself, be responsive or introduce a new concern). In addition, the participants make unequal contributions to different parts of the schema due to the inherent asymmetry between their roles in terms of knowledge and authority. Despite these challenges, humans are able to communicate very effectively in this way. Because of that, the issues increase as the conversation progresses and more shared context is built up, in part because of a certain amount of shared domain knowledge, despite differences in the extent and phrasing of it. In response to these insights, our proposed model, which we refer to as MedFilter, integrates elements of dis 

course structure and ontological knowledge to improve utterance classification, the impact of which is also observed in a downstream extraction task. We evaluate the approach on a corpus of nearly 7,000 doctor-patient interactions as a case study.

Our proposed method, MedFilter1, is illustrated in Figure 1 and described in detail in Section 3. Its architecture specifically reflects an awareness of the challenges above and begins to address them. In particular, the speaker's role (i.e., doctor, patient, and other) and position within the interaction are both introduced as structuring variables. Insights from ontological knowledge are also made available through a domain ontology: specifically, the Unified Medical Language System (UMLS) Bodenreider (2004). From a more technical perspective, the architecture introduces a novel Multi-Speaker BiLSTM to learn role-specific context representations. MedFilter also benefits from the incorporation of a hierarchical loss that jointly learns the coarse-grained task of predicting medical relevance to improve fine-grained topic-based utterance classification. The ability to extract medically relevant utterances from doctor-patient conversations and categorize them into the medical topics/categories has a substantial practical impact in medical practice Finley et al. (2018); Quiroz et al. (2019).

Footnote 1: [https://github.com/sopankhosla/MedFilter](https://github.com/sopankhosla/MedFilter)

## 2 Related Work

**Dialogue Summarization:** In addition to the challenges noted earlier in the paper, other linguistic phenomena such as backchannels, false starts, and topic diffusion are prominent in human-to-human conversations. They add noise, which challenges the capabilities of otherwise effective summarization approaches such as pointer-generator networks See et al. (2017); Liu et al. (2019b).

Some prior work has relied on an Information Extraction (IE) based approach to extract details about individual medical entities such as symptoms or medications Du et al. (2019); Selvaraj

Figure 1: Overview of MedFilter. MedFilter first encodes each utterance of the given conversation using a BERT-based encoder (A). The obtained utterance embedding is concatenated with contextual information like speaker role, position of utterance in the conversation, and ontological knowledge (B). This is then fed to a MS-BiLSTM (C1) for medical relevance identification. MS-BiLSTM leverages speaker role information to learn speaker-specific context for each utterance. This contextual representation is concatenated with the utterance embedding (C2) and passed through another MS-BiLSTM (C3) which focuses on fine-grained categorization. Both tasks are jointly learned. Refer to Section 3 for more details.

Figure 2: MedFilter as a part of extraction pipeline.

 

and Konam, 2019). However, recently, multiple studies (Lacson et al., 2006; Kocaballi et al., 2019; Liu et al., 2019, 2019; Park et al., 2019) have shown the benefits of using the topical structure in goal-oriented dialogues to improve summarization. Within that scope, Liu et al. (2019) introduce _key-point sequences_ that describe the logical topic flow of the summary of customer-service chats. They propose a hierarchical transformer to predict these topics (key-points) for each utterance and use them as auxiliary labels to guide the summarization.

This past work inspires our work in which we extend the approach and then apply it in the more challenging domain of doctor-patient interactions. We consider it more challenging both in terms of the number of utterances per conversation (avg. 225 vs 20) and topic switches (Kocaballi et al., 2019). To improve the key-point sequence utterance-level topic classification approach (Liu et al., 2019), we propose MedFilter that models speaker-specific context augmented with ontological knowledge and a hierarchical loss function.

**Intent Classification:** The problem of classifying utterances into medical topics/categories has many similarities with the task of utterance-level intent classification (Zhang et al., 2019; Budzianowski et al., 2018; Qu et al., 2019). In our case, medical categories act as coarse-grained intents that drive the content of the discussion. Much of the previous work in intent classification caters to creating better dialog agents that condition their responses on the intent of the previous utterance (Budzianowski et al., 2018; Bocklisch et al., 2017). For instance, Chen et al. (2019); Kim et al. (2017) propose intent classification as a text classification task where each utterance is considered a complete, independent command. However, this is not true in our case as the discussion about a medical category might range over multiple utterances, each dependent on context. Hence, we tackle the classification problem as a sequence-labeling task.

**Sequence Labeling in Dialogue:** Most prior work that employs sequence labeling for utterance classification in dialogues (Raheja and Tetreault, 2019; Liu et al., 2017; Jiao et al., 2019) evaluates their systems on dialogue-act classification (Shriberg et al., 2004, 1998) or emotion recognition datasets (Poria et al., 2019). In this paper, we adopt state-of-the-art modeling approaches from the emotion recognition task (Jiao et al., 2019, 2019) to serve as baselines in our evaluation since our task has not previously been benchmarked.

## 3 Proposed Method: MedFilter

The overall architecture of MedFilter is shown in Figure 1. The input to MedFilter is a transcribed clinical conversation \({\cal C}\) of form \(\{u_{1},u_{2},...,u_{n}\}\), where each \(u_{i}\) represents an utterance. Each utterance in the conversation is passed through a BERT-based encoder (Fig. 1A and Sec. 3.1) to get a fixed-dimensional representation. Contextual information such as speaker role, the utterance's position in the conversation, and ontological knowledge (Fig. 1B and Sec. 3.2) is then appended to the BERT representation. The encoding is input to the coarse Multi-Speaker BiLSTM (MS-BiLSTM) model (Fig. 1C1) followed by a fully-connected layer to classify the relevance of utterances for topical classification. The representation created by MS-BiSLTM (Coarse) is then concatenated with the utterance encoding (Fig. 1C2) and the resulting vector is fed to the fine-grained MS-BiLSTM (Fig. 1C3) to classify utterances into different medical categories (Sec. 3.3). MedFilter is jointly optimized on both classification tasks.

### BERT-based Encoder

Given the superior modeling capabilities of long-range dependencies in Transformer-based models (Vaswani et al., 2017), we use pre-trained BERT (Devlin et al., 2019) for encoding each utterance \(u_{i}\). We first encode each token in the utterance using BERT, i.e., \([h^{\texttt{BERT}}_{i1},h^{\texttt{BERT}}_{i2},...,h^{\texttt{BERT}}_{im}]\), where \(h_{ij}\) represents BERT-encoding of \(j^{th}\) token of \(u_{i}\). Now, following Reimers and Gurevych (2019), we use MEAN pooling for obtaining a representation for the entire utterance (\(h^{\texttt{Text}}_{i}\)). Since the original pretrained BERT model is trained on a general web corpus such as Wikipedia, it might not generalize well to our corpus. Therefore, we further fine tune the BERT model in a supervised manner for the task of predicting the utterance type.

### Contextual Information

In addition to encoding the text of an utterance, we also make use of the following types of contextual information.

**1. Speaker Role Info:** In conversations in general, speaker identity helps ground co-references like _I, You_. In doctor-patient conversations, each of the speakers play a specific role in the goals of the interaction. For example, the doctor is more 

likely to discuss medications than the patient. To allow the representation to be sensitive to speaker information, we map the speaker roles, namely, doctor, patient, and other, to a \(d\)-dimensional embedding (\({h}_{i}^{\texttt{speaker}}\)) which is learned during training and given to the model along with the text-based representation.

**2. Positional Info:** Clinical conversations often follow a pattern where topics like symptoms and complaints are discussed earlier in the dialog and prescribed medications are narrated in the middle or toward the end. To include this signal in MedFilter, we partition all the utterances in a conversation into \(k\) equal parts based on their position. For instance, if the conversation has \(40\) utterances and \(k=4\) then the initial \(10\) belong to \(1^{st}\) partition and the next \(10\) belong to \(2^{nd}\) and so on. Similar to speaker role information, a trainable embedding is associated with each partition (\({h}_{i}^{\texttt{position}}\)).

**3. Ontological Knowledge:** UMLS (Unified Medical Language System) Bodenreider [5] is a combination of a semantic network and a meta-thesaurus. The semantic network consists of a set of 127 broad subject categories, or semantic types, which provide a consistent categorization of all concepts represented in the meta-thesaurus. In MedFilter, we use Quick-UMLS Soldaini and Goharian [6], which identifies clinical mentions in an utterance and retrieves the associated UMLS Concept Unique Identifiers (CUIs) and semantic type, to inform our model about the type of medical phrases present in the input. We believe that types such as _Pharmacologic Substance_, _Syptoms_, and _Diseases_ can be helpful in correctly classifying the utterances. We assign a trainable embedding to each semantic type. However, since each utterance can contain multiple clinical mentions of varied semantic types, we average the semantic-type embeddings for each mention present in the utterance and pass it to the model (\({h}_{i}^{\texttt{semantic}}\)).

### Utterance Prediction

The classifier takes in the extended representation for each utterance \(u_{i}\) in the conversation given as

\[{h}_{i}=[{h}_{i}^{\texttt{Text}};{h}_{i}^{\texttt{speaker}};{h}_{i}^{ \texttt{position}};{h}_{i}^{\texttt{semantic}}].\]

To explicitly model the separate roles performed by each speaker (as discussed in Section 1), we propose a novel module **Multi-Speaker BiLSTM (MS-BiLSTM)** that includes speaker-level BiLSTMs to learn the context for each speaker type separately. We note, for example, that when the doctor is prescribing medications to the patient, she is more likely to expand on her previous utterance in order to discuss different details about the medicine, whereas the patient is most likely to give simple acknowledgments or ask questions in her turn. Having separate speaker-level BiLSTMs allows MS-BiLSTM to model this difference in the use of context.

MS-BiLSTM takes in \({h}_{i}\) and \(s_{i}\) (utterance's speaker) as input. \({h}_{i}\) is passed through a background BiLSTM (BiLSTM\({\tt BB}\)) and different speaker-level BiLSTMs (BiLSTM\({\tt B}\)). Thus, if there are 3 speaker roles in the conversation, then the extended representation for each utterance (\({h}_{i}\)) would be input to 4 BiLSTMs (1 background BiLSTM + 3 speaker BiLSTMs). The hidden representations from BiLSTM\({\tt BB}\) and BiLSTM\({\tt s}_{1}\) are combined using a sigmoid gate that is learned during training:

\[\hat{{h}}_{i}^{\texttt{bg}} =\text{BiLSTM}_{\texttt{bg}}({h}_{i})\] \[\hat{{h}}_{i}^{\texttt{a}j} =\text{BiLSTM}_{\texttt{g}_{j}}({h}_{i}),~{}\forall j\in\text{ speakers}\] \[g^{s} =\sigma(w_{g})\] \[{h}_{i}^{\prime} =g^{s_{i}}*\hat{{h}}_{i}^{\texttt{a}_{i}}+(1-g^{s_{i}})*\hat{{h}}_ {i}^{\texttt{bg}},\] \[{h}_{i}^{\prime} =\text{MS-BiLSTM}({h}_{i},s_{i}).\]

Each speaker-level BiLSTM (BiLSTM\({\tt B}\)) only receives gradients for that speaker's utterance (\(u_{i}\)) thus focusing on role-specific context. The gate between \(\hat{{h}}_{i}^{\texttt{a}_{i}}\) and \(\hat{{h}}_{i}^{\texttt{bg}}\) controls the relative importance of the role-specific and general-context representation learned by speaker-level and background BiLSTMs respectively.

In this paper, we focus on classifying an utterance into one or more out of three categories, namely _supromts_, _complaints_, and _medications_. However, these categories can be combined together to create a coarse-grained task of predicting if the utterances are _medically relevant_. We leverage this coarse-grained supervision to create a hierarchical model with a joint-learning loss.

Hierarchical Modeling:** In this architecture, the extended representation (\({h}_{i}\)) and the corresponding speaker role (\(s_{i}\)) are first passed through a coarse-grained MS-BiLSTM and a fully-connected layer followed by softmax to be classified into one of the two categories {_Medically Relevant_, _Irrelevant_}. The representation \({h}_{i}^{\texttt{coarse}}\) learned by this MS-BiLSTM would model the differences between medically relevant and irrelevant text which 

can also benefit fine-grained classification. Hence, \({\mathbf{h}_{i}^{\texttt{coarse}}}\) is concatenated with \({\mathbf{h}_{i}}\) and sent to the fine-grained MS-BiLSTM which focuses on the multi-label classification into the three categories discussed earlier:

\[{\mathbf{h}_{i}^{\texttt{coarse}}} =\texttt{MS-BiLSTM}_{\texttt{coarse}}({\mathbf{h}_{i}},s_{i}),\] \[{\mathbf{h}_{i}^{\prime\prime}} =[{\mathbf{h}_{i};{\mathbf{h}_{i}^{\texttt{coarse}}}}],\] \[{\mathbf{h}_{i}^{\texttt{fine}}} =\texttt{MS-BiLSTM}_{\texttt{fine}}({\mathbf{h}_{i}^{\prime\prime}}, s_{i}),\] \[{\mathbf{p}_{\texttt{coarse}}} =softmax({\mathbf{W}_{c}{\mathbf{h}_{i}^{\texttt{coarse}}}+{\mathbf{b}}}),\] \[{\mathbf{p}_{\texttt{fine}}} =\sigma({\mathbf{W}_{f}{\mathbf{h}_{i}^{\texttt{fine}}}+{\mathbf{b}}}).\]

Both tasks are jointly optimized and the hyperparameter \(\beta\) controls the relative strength of the medical-relevance classification loss (\({\mathbf{L}_{\texttt{coarse}}}\)):

\[{\mathbf{L}}={\mathbf{L}_{\texttt{fine}}}+\beta{\mathbf{L}_{\texttt{coarse}}}.\]

Such a loss function could also be used in other utterance classification tasks where classes follow a hierarchical structure. For instance, in emotion classification Poria et al. (2019), the fine-grained categories (e,g, happiness, anger, etc.) can be combined to create an _emotive_ class, and a coarse-grained classifier could be used to learn features that differentiate between _emptive_ and _neutral_ utterances.

## 4 Experimental Setup

### Corpus Description

Our data set comprises 6,862 annotated transcripts of real and de-identified doctor-patient conversations with an average of 225 utterances per conversation, primarily from the doctor and patient but occasionally including contributions from nurses, caregivers, and other attendees as well. The annotation guidelines were developed by a team of professional medical scribes and NLP experts. Annotators were trained to identify the medically-relevant utterances in a given conversation and assign one or more (out of 15 possible) tags to each utterance. Each of these tags represents a medical category like _symptom_, _previous medical history_, _diagnosis_ etc. Most conversations contain some informal, social interactions with utterances that are irrelevant to the downstream clinical tasks.2

Footnote 2: An example dialogue is included in Appendix (Sec. A.2).

In this work, we leverage the labels to train MedFilter on the task of utterance classification and focus on three categories, namely, _symptoms_, _complaints_, and _medications_, where _medications_ include past/current medications taken by the patient and prescriptions given by the doctor.3 We choose the above-mentioned categories as they are found in every office visit, and most closely generalize to other domains like customer-service chats. However, our approach can be easily generalized for capturing other aspects such as _previous medical history_, _diagnosis_, and _assessments_ as well. We set aside a random sample of 627 and 592 conversations for validation and testing respectively.

Footnote 3: Additional statistics are included in Appendix (Sec. A.1).

### Baselines

Since sequence-labelling models haven't been applied to utterance classification in doctor-patient conversations previously, we compare our proposed method, MedFilter, against baseline methods that give SOTA results on utterance-level emotion recognition data sets. **HiGRU-sf**Jiao et al. (2019) is a hierarchical gated recurrent unit (HiGRU) framework with an utterance-level GRU and a conversation-level GRU. **Bif-AGRU**Jiao et al. (2019) denotes a two-level BiGRU fusion model with uni-directional AGRU for attentive context representation. **UnFi-BiAGRU** is similar to BiF-AGRU, but uses a uni-directional GRU for contextual utterance representation and a bi-directional AGRU for attentive context. For implementation, we use the official code provided by the authors.45

Footnote 4: [https://github.com/wxjiao/HiGRUs](https://github.com/wxjiao/HiGRUs)

Footnote 5: [https://github.com/wkjiao/AGHUN](https://github.com/wkjiao/AGHUN)

#### Evaluation Metric:

We use the mean area under the PR curve (AUC), a widely used metric in multi-label classification setting Riedel et al. (2013); Mintz et al. (2009), as our evaluation metric. It is also used for early stopping and hyperparameter tuning.6

Footnote 6: Hyperparameters are included in Appendix (Table A4)

## 5 Utterance Classification Results

MedFilter performs better than any of the baseline approaches in assigning utterances in doctor-patient conversations to medically relevant categories. Table 1 presents the AUC scores for different utterance-labeling models on our test set. Each result is the mean of 5 independent runs with different seeds.

A BERT-based classifier that passes the mean of token-level embeddings through an FC layer 

gives a low score of \(33.5\) AUC. When the BERT encoder is fine-tuned along with the classification layer (BERT-FT), the performance jumps to \(36.2\) underlining the benefits of fine-tuning BERT Devlin et al. (2019). We also find that using Clinical BioBERT-FT (fine-tuned) does not beat BERT-FT. This is partly because the former is further pre-trained on MIMIC notes Alsentzer et al. (2019) which are much more formal than medical conversations and thus the additional knowledge does not transfer well to our corpus.

Adding context to BERT-based models , using, e.g. BiLSTM, gives substantial boosts. End-to-end fine-tuned BERT BiLSTM (BERT BiLSTM FT) performs worse than BERT-FT BiLSTM that passes fine-tuned BERT embeddings through a BiLSTM as non-learnable features. MedFilter, which further includes contextual information, uses MS-BiLSTM in place of BiLSTM, and optimizes a hierarchical loss, significantly outperforms all baselines and obtains \(1.4\) absolute AUC points over BERT-FT BiLSTM (\(2^{nd}\) best). It also surpasses emotion recognition SOTA methods like HiGRU-sf by \(4.1\) AUC points.

**Ablation Results:** To understand the importance of each module in MedFilter, we perform a cumulative ablation study (Figure 3). We find that removing individual modules results in notably reduced performance. The model that does not incorporate hierarchical modeling, shows a dip of \(0.4\) AUC points. This suggests that the information learned in the medical-relevance prediction layer aids the final classification task. Further, replacing MS-BiLSTM with a simple BiLSTM leads to a drop of an additional \(0.6\) AUC points, revealing the importance of modeling speaker-specific context. Without contextual information, we see a reduction of \(0.4\) AUC points. This shows that features like speaker role, position, and semantic types are essential for our task.

## 6 Impact of Utterance Classification on Downstream Medical Extraction

The results in the previous section portray the effectiveness of MedFilter at sorting important utterances in clinical conversations into medically relevant categories. Such filtering, when included in the pipeline (for example, as a pre-processing step), can assist downstream medical processing methods to focus on utterances that contain information pertinent for their tasks (Figure 2), by improving the signal-to-noise ratio in the input. In this section, we evaluate whether the use of MedFilter to prune irrelevant utterances is advantageous for symptom, medication, and complaint extraction.

### Task Setup

The extractor takes the conversation as input and outputs the discussed symptoms/medications/complaints within.

Conversation-level labels for all three extraction tasks are taken from a predefined set provided by the corpus annotators. For **symptoms**, they include \(14\) coarse-grained classes to represent different body systems (e.g., cardiovascular) and \(178\) fine-grained ones for the corresponding issues (e.g., palpitations). Given the small size of the training data, we use the coarse-grained body-systems for symptom extraction. We then manually curate a list of different symptoms corresponding to each body-system using UMLS and use their UMLS CUIs as labels.7

Footnote 7: Refer to Table A10 for the final list of symptom labels.

For **medications**, we manually link medication labels to their corresponding UMLS Bodenreider (2004) concepts and group them using hierarchies

\begin{table}
\begin{tabular}{l l l} \hline \hline  & **Methods** & **AUC** (x100) \\ \hline \multirow{3}{*}{**Baselines**} & UniF-BiAGRU & 40.9 (0.51) \\  & BiF-AGRU & 40.9 (0.37) \\  & HiGRU-sf & 43.1 (0.45) \\ \hline \multirow{3}{*}{**BERT variants**} & BERT & 33.5 (0.08) \\  & Clinical BioBERT-FT & 36.1 (0.11) \\  & BERT-FT & 36.2 (0.08) \\ \hline \multirow{3}{*}{**With Context**} & BERT BiLSTM FT & 44.5 (0.22) \\  & BERT-FT BiLSTM & 45.8 (0.16) \\ \hline
**Our Method** & MedFilter & **47.2** (0.26) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Utterance classification results on the test-set **(Avg. (std. dev.))**. Results on valid-set are shown in the Appendix. The improvements are statistically significant (p ¡ 0.01).

Figure 3: Cumulative Ablation Results



from NCI Thesaurus [14]. 8 We pass each medication name through QuickUMLS to get a list of possible CUIs for the term in UMLS. We take the candidate CUI with a similarity of 1 and find its NCI hierarchy in the UMLS metathesaurus. The four topmost nodes in the hierarchy are extracted, which act as the pseudo-label for that CUI. In order to reduce the class-imbalance, some of these hierarchies are combined to form a coarser label. This reduces the number of labels to 31. Finally, _Others_ label is added, which inhabits medicine names (in the test-set) that do not correspond to any of the previous 31 labels. This reduces the label count to \(32\) for medications.9

Footnote 8: [https://ncit.nih.gov](https://ncit.nih.gov)

**Complaints** in our corpus range from follow-up visits to disease names to vaccine requests. Similar to medication extraction, we leverage SNOMEDCT hierarchies10 to constraint the tag list to 11, where the first 10 represent diseases of different body systems and _Others_ encompasses complaints like follow-up, vaccine requests, medication refill requests, etc. (Table A11).

Footnote 9: Refer to Table A13 for the final list of medication labels.

We use the same train/val/test split as defined for the utterance classification experiments in Section 4.1. The performance of the extraction pipeline is evaluated on Micro and Macro-F1 scores.

### Extractor Details

All three extraction tasks are modeled as multi-label classification. We leverage a state-of-the-art medical entity-linking tool, QuickUMLS [14]61, that takes in a conversation and outputs UMLS CUIs corresponding to all identified candidate concepts. Concepts with a similarity measure of \(1\) are chosen as predictions. For symptom extraction, the predictions are compared against a manually created list of CUIs (presented in Appendix Table A12) for symptoms associated with each of the 14 Body Systems. The presence of a symptom of body-system \(b\) is determined by the presence of the predicted CUIs in the target list for that body system. We compare the NCI and the SNOMED-CT hierarchies of the predicted concepts against the label hierarchies for medications and complaints, respectively. Concepts that do not fit into one of the specific categories are grouped under the label _Others_. In the next section, we report the results for the best performing filtering thresholds.12

Footnote 10: [https://www.nlm.nih.gov/healthit/snomedct/index.html](https://www.nlm.nih.gov/healthit/snomedct/index.html)

Footnote 11: [https://github.com/Georgetown-IR-Lab/QuickUMLS](https://github.com/Georgetown-IR-Lab/QuickUMLS)

### Results

We find that the performance of the baseline medication and symptom extractor QuickUMLS (All Text) is substantially boosted by filtering out irrelevant utterances (Tables 2 and 3). Pruning medically irrelevant utterances using MedFilter (MR MedFilter) improves Micro F1 by \(29.3\) and \(4.7\) points for medication and symptom extraction, respectively. If only the medication/symptomatic utterances (MU/SU) are input to the extractors, the results improve further.

Results for complaint extraction are shown in Table 4. We find that the QuickUMLS extractor does not perform well on complaint extraction. However, consistent with the other two categories' trends, pruning irrelevant utterances before sending the conversation through the extractor improves performance. Micro-F1 score increases from 35.6 for All Text to 43.7 for CU MedFilter.

Pruning done using MedFilter seems to be more beneficial than BERT-FT BiLSTM (\(2^{nd}\) best utterance classifier in Table 1) for medication and complaint extraction, however they perform equally well for symptom extraction. This suggests that the benefits from the inclusion of discourse structure, domain knowledge, and a hierarchical loss function, do not transfer well to symptom ex 

traction. In Section 7, we investigate the kinds of utterance classification errors MedFilter makes, that need to be addressed to further improve the symptom extraction pipeline.

## 7 Discussion

**Why does contextual information help?** Ablation results (Figure 3) show that incorporating speaker role information and UMLS semantic-type information provides significant improvements in AUC scores for utterance classification. In Figure 4a, we plot the proportion of utterances from different medical categories against their speakers. While both parties contribute equally to symptom discussions, there is a clear asymmetry in the number of _medication_ and _complaint_ utterances spoken by the doctor and the patient, explaining the contribution of speaker role information in differentiating _medication/complaint_ utterances from others.

We also plot the distribution of the four most frequent UMLS semantic types present in the utterances of different medical categories (Figure 4b). For medications, we find that UMLS entities with semantic type _Pharmacologic Substance_ are present in more than \(55\%\) of the medication utterances indicating that its detection is a knowledge-dependent task. Similarly, and supporting our hypothesis, _Disease/Syndrome_ and _Sign/Symptom_ are the most frequent semantic types in complaint and symptom utterances, respectively.

Error Analysis:In this section, we present a deeper analysis of some of the systematic knowledge-extraction errors made by MedFilter that limit its performance in recognizing medicallyrelevant utterances.

1. **Informal Language:** The model sometimes overlooks informal references to symptoms. For instance, utterances such as _PT: I feel something unusual in my leg_ or _PT: My heart beats funny!_ discuss musculoskeletal and cardiovascular symptoms but do not use medical terms to refer to them. These patterns seem to be more frequent in patient utterances, likely because they are less familiar with medical terminology. Off-the-shelf entity-linkers, like QuickUMLS [28], do not transfer well to spoken medical conversations. They are unable to recognise the correct UMLS concepts (and semantic types) corresponding to the colloquial symptomatic phrases which reduces their effectiveness as features.13 For instance, for the utterance _PT: My heart is racing._, QuickUMLS outputs 1A rather than 1B:

Footnote 13: Sign/Symptom entities are identified in less than \(30\%\) of total symptomatic utterances (Figure 4b).

Input: PT: My heart is racing (A): PT: My [heart]bodypart is racing. (B): PT: My [heart]bodypart is racing\(|\))\(|\)system.

2. **Physical Manifestations of Symptoms: Internal symptoms often manifest themselves physically as a digression from the natural ability to

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Complaint Extraction** & **Macro F1** & **Micro F1** \\ \hline QuickIMLS (All Text) & 10.0 & 35.6 \\ \hline QuickTMLS (MR BERT-FT BiLSTM) & 10.9 & 40.3 \\ QuickTMLS (CU BERT-FT BiLSTM) & 11.1 & 43.0 \\ \hline QuickTMLS (MR MedFilter) & 11.1 & 40.7 \\ QuickTMLS (CU MedFilter) & **11.1** & **43.7** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results(%) for Complaint Extraction (CE). **MRR**=Medically Relevant, **CU**=Complaint Utterances.

Figure 4: Contextual Information: Different speaker roles contribute asymmetrically towards different medical topics/categories in the dialogue (Figure 4a). Furthermore, phrases with UMLS semantic types _Pharmacologic Substance_, _Sign/Symptom_, and _Disease/Syndrome_ occur quite frequently in medical, symptom, and complaint utterances respectively (Figure 4b).

 

perform typical activities. For instance, when the patient says _I can't do anything after I'm back from office_ or _I can only walk up one flight of stairs_, she might be implicitly mentioning a cardiovascular symptom. A sizeable subset of such examples includes usage of duration or frequency to convey the implicit deviation, like

```
Duration: PT: I only sleep for 4 hours, Frequency: PT: I go to the bathroom 10 times at night, Quantity: DR: I see you are up to 300 pounds now.
```

## 8 Future Work

For a system to correctly classify the samples of the above two categories, it needs both to generalize to patient-generated language, and to have a semantic understanding of whether the description strays from normal. Incorporating data from online self-disclosure sites like medical subreddits and discussion forums Basaldella and Collier (2019) during training might prove beneficial for learning better representations for such vocabulary. Concept normalization data sets Miftahutdinov and Tutubalina (2019); Lee et al. (2017) could also be leveraged in this regard. Our approach of training the BERT encoder separately from the context encoder would allow MedFilter to learn from such non-dialogue resources.

Extraction tasks (Section 6) mostly evaluate the ability of MedFilter to recognize utterances that contain the most information about the name or type of the medication, symptom, or complaint. However, to quantify the context-level benefits of MedFilter, especially the speaker-specific context modeling (MS-BiLSTM), on downstream processing, we need to evaluate the system on problems like regimen extraction Selvaraj and Konam (2019) or symptom summarization Liu et al. (2019b). Such tasks require utterance classification models to correctly identify utterances that discuss fine-grained details about the topic and would therefore evaluate a model's ability to solve multiple challenges like coreference resolution, speaker-specific context detection, thread identification, etc. Such an evaluation is a part of future work.

## 9 Conclusion

In this paper, we have proposed a novel text classification approach that specifically leverages insights into the organization of task-oriented conversations in order to improve performance at topic-based utterance classification over SOTA baseline approaches. In particular, we have demonstrated that our utterance classification model, MedFilter, benefits from discourse information, domain knowledge, speaker-specific context modeling, and a hierarchical loss to reach a new state-of-the-art performance on a doctor-patient interactions dataset. We find that using topic-based utterance classification in general, and MedFilter in particular, as a pre-processing step before medical extraction tasks, significantly improves the extraction scores. We believe that the contributions made in this work would also generalize to other kinds of expert-lay dialogue like customer-service chats.

## Acknowledgements

We thank the anonymous EMNLP reviewers for their insightful comments. We are also grateful to the members of the TELEDIA group at LTI, CMU for the invaluable feedback. This work was funded in part by Dow Chemical, University of Pittsburgh Medical Center, and Microsoft.

## References

* Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott (2019)Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, Minnesota, pp. 72-78. External Links: Link Cited by: SS5.
* D. Bocklisch, J. Faulkner, N. Pawlowski, and A. Nichol (2017)Rasa: open source language understanding and dialogue management. arXiv preprint arXiv:1712.05181. Cited by: SS1.
* O. Bodenreider (2004)The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research32 Database issue:D267-70. Cited by: SS1.
* a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1.
* P. Budzianowski, T. Wen, B. Tseng, I. Casanueva, G. Ultes, O. Ramadan, and Milica Gasic (2018)MultiwoZ-a large 

scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5016-5026.
* Chen et al. (2019) Cen Chen, Chilin Fu, Xu Hu, Xiaolu Zhang, Jun Zhou, Xiaolong Li, and Forrest Sheng Bao. 2019. Reinforcement learning for user intent prediction in customer service bots. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 1265-1268.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Du et al. (2019) Nan Du, Kai Chen, Anjuli Kannan, Linh Tran, Yuhui Chen, and Izhak Shafran. 2019. Extracting symptoms and their status from clinical conversations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 915-925, Florence, Italy. Association for Computational Linguistics.
* Finley et al. (2018) Gregory Finley, Erik Edwards, Amanda Robinson, Michael Brenndoerfer, Najmeh Sadoughi, James Fone, Nico Axtmann, Mark Miller, and David Suendermann-Oeff. 2018. An automated medical seribe for documenting clinical encounters. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_, pages 11-15, New Orleans, Louisiana. Association for Computational Linguistics.
* Jiao et al. (2019a) Wenxiang Jiao, Michael R Lyu, and Irwin King. 2019a. Real-time emotion recognition via attention gated hierarchical memory network. _arXiv preprint arXiv:1911.09075_.
* Jiao et al. (2019b) Wenxiang Jiao, Haiqin Yang, Irwin King, and Michael R. Lyu. 2019b. HiGRU: Hierarchical gated recurrent units for utterance-level emotion recognition. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 397-406, Minneapolis, Minnesota. Association for Computational Linguistics.
* Kazi and Kahanda (2019) Nazmul Kazi and Indika Kahanda. 2019. Automatically generating psychiatric case notes from digital transcripts of doctor-patient conversations. In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_, pages 140-148.
* Kim et al. (2017) Young-Bum Kim, Sungjin Lee, and Karl Stratos. 2017. Onenet: Joint domain, intent, slot prediction for spoken language understanding. In _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_, pages 547-553. IEEE.
* Kocaballi et al. (2019) Ahmet Baki Kocaballi, Enrico Coieras, Huong Ly Tong, Sarah J White, Juan C Quiroz, Fahimeh Rezazadegan, Simon Willcock, and Liliana Laranujo. 2019. A network model of activities in primary care consultations. _Journal of the American Medical Informatics Association_, 26(10):1074-1082.
* Lacson et al. (2006) Ronilda C Lacson, Regina Barzilay, and William J Long. 2006. Automatic analysis of medical dialogue in the home hemodialysis domain: structure induction and summarization. _Journal of biomedical informatics_, 39(5):541-555.
* Lee et al. (2017) Karhy Lee, Sadiid A Hasan, Oladimeji Farri, Alok Choudhary, andAnkit Agrawal. 2017. Medical concept normalization for online user-generated texts. In _2017 IEEE International Conference on Healthcare Informatics (ICHI)_, pages 462-469. IEEE.
* Liu et al. (2019a) Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019a. Automatic dialogue summary generation for customer service. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1957-1965.
* Liu et al. (2017) Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in DNN framework. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2170-2178, Copenhagen, Denmark. Association for Computational Linguistics.
* Liu et al. (2019b) Zhengyuan Liu, Angela Ng, Sheldon Lee Shao Guang, Ai Ti Aw, and Nancy F. Chen. 2019b. Topic-aware pointer-generator networks for summarizing spoken conversations. In _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019_, pages 814-821. IEEE.
* Miftahutdinov and Tutubalina (2019) Zulfat Mifttahutdinov and Elena Tutubalina. 2019. Deep neural models for medical concept normalization in user-generated texts. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_, pages 393-399.
* Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2_, pages 1003-1011. Association for Computational Linguistics.
* Park et al. (2017) Jihyun Park, Dimitrios Kotzias, Patty Kuo, Robert L Logan IV, Kritzia Merced, Sameer Singh, Michael Tanana, Efi Karra Taniskidou, Jennifer Elston Lafata, 

David C Atkins, et al. 2019. Detecting conversation topics in primary care office visits from transcripts of patient-provider interactions. _Journal of the American Medical Informatics Association_, 26(12):1493-1504.
* Poria et al. (2019) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 527-536, Florence, Italy. Association for Computational Linguistics.
* Qu et al. (2019) Chen Qu, Liu Yang, W Bruce Croft, Yongfeng Zhang, Johanne R Trippas, and Minghui Qiu. 2019. User intent prediction in information-seeking conversations. In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval_, pages 25-33.
* Quiroz et al. (2019) Juan C. Quiroz, Liliana Laranjo, Ahmet Baki Kocaballi, Shlomo Berkovsky, Dana Rezazadegan, and Enrico Coiera. 2019. Challenges of developing a digital scribe to reduce clinical documentation burden. _npj Digital Medicine_, 2(1):114.
* Raheja and Tetreault (2019) Vipul Raheja and Joel Tetreault. 2019. Dialogue Act Classification with Context-Aware Self-Attention. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers)_, pages 3727-3733, Minneapolis, Minnesota. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentencebert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. 2013. Relation extraction with matrix factorization and universal schemans. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 74-84.
* See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083.
* Selvaraj and Konam (2019) Sai P Selvaraj and Sandeep Konam. 2019. Medication regimen extraction from medical conversations. _arXiv_, pages arXiv-1912.
* Shriberg et al. (1998) Elizabeth Shriberg, Rebecca Bates, Paul Taylor, Andreas Stolcke, Daniel Jurafsky, Klaus Ries, Noah Coccaro, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? _Language and Speech_, 41(3-4):439-487.
* Shriberg et al. (2004) Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The ICSI meeting recorder dialog act (MRDA) corpus. In _Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004_, pages 97-100, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
* Sioutos et al. (2007) Nicholas Sioutos, Sherri de Coronado, Margaret W. Haber, Frank W. Hartel, Wen-Ling Shaiu, and Lawrence W. Wright. 2007. Nci thesaurus: A semantic model integrating cancer-related clinical and molecular information. _J. of Biomedical Informatics_, 40(1):30-43.
* Soldaini and Goharian (2016) Luca Soldaini and Nazli Goharian. 2016. Quickumls: a fast, unsupervised approach for medical concept extraction. In _MedIR workshop, sigir_, pages 1-4.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Waitzkin (1989) Howard Waitzkin. 1989. A critical theory of medical discourse: Ideology, social control, and the processing of social context in medical encounters. _Journal of Health and Social Behavior_, pages 220-239.
* Zhang et al. (2019) Chenwei Zhang, Yuliang Li, Nan Du, Wei Fan, and S Yu Philip. 2019. Joint slot filling and intent detection via capsule neural networks. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5259-5267.

 



## Appendix A Dataset Details

### Dataset Statistics

The de-identified doctor-patient dialogue corpus used in this work was made available by University of Pittsburgh Medical Center (UPMC) and Abridge AI Inc.. Most of the conversations in this corpus are follow-up encounters between cardiovascular/general medicine doctors and patients. Figure A1 shows the distribution of the number of utterances in each conversation. The number ranges from as low as \(3\) to as high as \(1521\) with a mean of \(225\). The proportion of medically relevant utterances in a conversation is quite low (Table A1). As shown, utterances that belong to the three categories combined make up less than \(10\%\) of the conversation portraying the amount of noise present in doctor-patient conversations with regards to further medical processing.

In Table A2, we show the average position in the doctor-patient conversation where the speakers start discussing different medical topics. Several of the the encounters are follow-up discussions about a pre-existing complaint. Therefore, patient's current condition with respect to the complaint is often discussed earlier in the conversation. This is generally followed by a discussion about different body systems and associated symptoms that may be bothering the patient, which allows the doctor to prescribe suitable medications.

The above-mentioned flow is merely an ideal depiction of the logical path that could be followed in the dialogue. However, real conversations in the corpus contain multiple topic-switches. For example, discussion of a symptom could be followed by medication which could then lead into a discourse about another symptom and so on.

### Example Conversation

An example conversation (details modified) from our corpus is shown in Table A3. Utterances 4,5,8,9

in the conversation discuss a symptom, with the Patient's reply (A3:5) acting as an important information about confirmation of the presence of that symptom. Furthermore, A3:8 and A3:9 provide additional details about the physical activities that cause the symptom. Although, the symptom name is discussed only in A3:4, information presented in the other utterances plays an important role in the clinical note. Similarly, utterances A3:11,12,13 discuss the medication _Baclofen_. Doctor prescribes the medication in A3:11. She provides further information like frequency of usage in A3:12, and dosage in A3:13, all of which is extremely important for regimen extraction. A3:19 contains names of two medications however it is not a medication utterance. This is case because the utterance does not discuss any medication that the patient is currently taking or being prescribed. The doctor is merely comparing the benefits of her prescribed medication against two popular pain pills.

### Symptom and Medication Extraction Labels

In addition to identifying the type of each utterance, corpus annotators also provide a class label to the symptoms and medications from a predefined set. For symptoms, guidelines include 178 classes of the form <_Body System>: <symptom>_ (e.g. Cardiovascular: Palpitations). Given the small size of the training data, instead of predicting given symptom classes, we predict the body system with which a symptom is associated (Table A10). Table A12 contains the list of target UMLS CUIs for each body system that are used as labels for Symptom Extraction. Please note that the list is manually curated and therefore is not exhaustive. For medications, we manually link each medication label in our training-set to its corresponding UMLS Bodenreider [2] concept and group them using hierarchies from NCI Thesaurus Sioutos et al. [6] (Table A13).

## Appendix B Hyperparameters

All our experiments are performed on a single Nvidia GeForce GTX 1080 Ti GPU. For MedFilter and other BERT-based baselines, we divide the conversations into windows of 128 utterances to ensure fair comparison against BERT-BiLSTM FT, which cannot process more than 128 utterances at a time due to GPU constraints. Other hyperparameters are presented in Table A4. We perform manual tuning on the entire range of hyper-parameters. AUC under the PR curve metric was chosen to select the best configuration. Results were not very sensitive to different non-zero values of \(\beta\).

## Appendix C Utterance Classification

### PR Curves

Figures A2 and A3 show the precision-recall curves for each category separately. MedFilter improves utterance classification for all three categories. For symptom classification, the AUC scores improve from \(52.2\) to \(53.5\). However, symptom extraction results (Section 6 in the main paper) suggest that most of this improvement is on identifying utterances that discuss fine-grained details about symptom discussion and not on recognizing the utterance that contains the actual symptom name.

### Performance on Validation Set

Table A5 shows the performance of different utterance classification models on the validation set.

 

Similar to the trend shown on test-set, MedFilter beats all of the baselines reaching a score of \(50.5\) AUC points.

## Appendix D Downstream Medical Extraction

### Micro-F1 vs Threshold

Figure A4 and A5 show how the performance of medication (ME) and symptom extraction (SE) varies against different utterance topic prediction probability thresholds. We plot the results for BERT-FT BiLSTM and MedFilter for brevity. Micro F1 scores for ME increase monotonically when the threshold is increased from \(0\) to \(\sim 0.75\) (Figure A4). This suggests that QuickUMLS medication extractor has low precision that is substantially improved when we prune irrelevant utterances. However, the graph for SE (Figure A5) shows that the extractor's performance is dominated by its recall. Pruning helps with improving the precision however does not help with the low recall. This explains the lower gains as compared to ME when using topic-based utterance classification in the SE pipeline (Table 3 in the main paper).

### Oracle Results

Table A6 contains results for medication extraction when medically relevant (MR) or medication (MU) utterances are chosen using an oracle (MR/MU Oracle). Similarly, oracle results for symptom and complaint extraction are shown in Tables A7 and A8, respectively.

We find that there is still a substantial room for improvement in the symptom extraction pipeline. By just improving the topic-based utterance classifier, one can observe a potential jump of 5 Micro-F1 points in symptom extraction. However, we do not observe this trend for medication extraction where the topic-classification done by MedFilter performs much better than the Oracle.

#### d.2.1 Why does MedFilter perform better than Oracle on Medication Extraction?

Extraction experiments (like medication extraction or symptom extraction) evaluate the performance at the conversation-level. So, where the medication 

name gets extracted from within the conversation is irrelevant to the task.

Oracle picks utterances that would be sufficient for a human to identify the medications discussed in the dialogue. However, they might not be adequate for an automatic string-matching based extractor like QuickUMLS. Since QuickUMLS uses non-contextual surface-level features to identify medication names, it would look for phrases (in the input given to it) that match the surface requirements. So, it is possible for the Oracle utterances not to contain the proper surface-level forms that QuickUMLS could leverage for extracting medications. Furthermore, the utterances categorized as medication utterances by MedFilter on the other hand, even though incorrect, might contain the medication names in the form QuickUMLS expects, thus improving the score over the Oracle. One should note, however, that a perfect downstream extractor would not suffer from these side-effects.

### Supervised Extractor

For symptom extraction (SE), we also show the benefits of using topic-based utterance classification on a supervised-classification based SE approach that leverages a BiLSTM with attention (BiLSTM-Attn) for the problem of predicting the symptoms present in a conversation.

#### d.3.1 BiLSTM-Attn

Each utterance in the conversation is passed through the embedding layer and a BiLSTM layer to obtain a contextualized representation.

\[h_{i}=\texttt{BiLSTM}(e(s_{i}),h_{i-1}) \tag{2}\] \[\mathbf{H_{i}}=\left\{h_{1},h_{2},...,h_{n}\right\}\]

where \(e(.)\) is the embedding function. The final state of the BiLSTM is re-weighted using attention calculated as shown in Equation A1.

\[h_{final} =[\vec{h_{0}};h^{\rightarrow}_{n-1}] \tag{3}\] \[\mathbf{S} =\mathbf{H_{i}}h_{final}\] \[\mathbf{A} =softmax(\mathbf{S})\] \[h^{\prime}_{final} =\mathbf{H_{i}^{T}}\mathbf{A}\]

This allows our model to pay attention to important utterances in the conversation to extract symptom information. We pass \(h^{\prime}_{final}\) through a linear classifier and a sigmoid layer to get logits for each possible symptom label (Table A10).

#### d.3.2 Experimental Setup

Similar to the QuickUMLS based extractor, we use Micro and Macro F1 scores to evaluate the performance of the supervised extraction pipeline. BiLSTM-Attn (All Text) model takes in the entire conversation as input, whereas the other variants are given only a subset of utterances. MR Oracle/MedFilter models are trained on the medically relevant utterances as output by the oracle. Similarly, SU Oracle/MedFilter models are trained on the Oracle symptom utterances in each conversation in the training-set. Therefore, topic-based classification is used as a pre-processing step in the pipeline.

#### d.3.3 Results

We present the results for symptom extraction (SE) using a BiLSTM-Attn model in Table A9. We find that using topic-based utterance classification to remove irrelevant utterances before passing the conversation through the BiLSTM-Attn improves 

the SE performance of the pipeline (\(4\) point jump in Micro F1). The results are further improved when the Oracle symptom utterances (SU Oracle) are input to the BiLSTM-Attn.

[TABLE:A2.T3][ENDT 



**Medication Labels**

DFCBM/Chemical Modifier/Toxin

DFCBM/Dietary Supplement

DFCBM/Drug or Chemical by Structure

DFCBM/Food or Food Product

DFCBM/Industrial Aid

DFCBM/Natural Product

DFCBM/Pharmacologic Substance/Adjuvant

DFCBM/Pharmacologic Substance/AA Blood or Body Fluid

DFCBM/Pharmacologic Substances/AA Cardiovascular System

DFCBM/Pharmacologic Substance/AA Digestive System or Metabolism

DFCBM/Pharmacologic Substance/AA Integumentary System

DFCBM/Pharmacologic Substance/AA Musculoskeletal System

DFCBM/Pharmacologic Substance/AA Nervous System

DFCBM/Pharmacologic Substance/AA Organs of Special Senses

DFCBM/Pharmacologic Substance/AA Respiratory System

DFCBM/Pharmacologic Substance/Anti-Infective Agent

DFCBM/Pharmacologic Substance/Anteoplastic Agent

DFCBM/Pharmacologic Substance/Biological Agent

DFCBM/Pharmacologic Substance/Action Channel Blocker

DFCBM/Pharmacologic Substance/Chemopreretive Agent

DFCBM/Pharmacologic Substance/Combination Medication

DFCBM/Pharmacologic Substance/Endothelin Receptor Antagonist

DFCBM/Pharmacologic Substance/Enzyme Inhibitor

DFCBM/Pharmacologic Substance/Hormone Therapy Agent

DFCBM/Pharmacologic Substance/Immunotherapeutic Agent

DFCBM/Pharmacologic Substance/Prostaglandint Analogue

DFCBM/Pharmacologic Substance/Protentive Agent

DFCBM/Pharmacologic Substance/Protein Synthesis Inhibitor

DFCBM/Physiology-Regulatory Factor

Activity/Clinical or Research Activity/Intervention or Procedure

Manufactured Object/Diagnostic, Therapeutic, or Research Equipment

Others