SlowGuess/ACL_2020_OCR huggingface dataset:
The superscript  $k$  denote the index of the training sample, and  $l$  is the index of the candidate.  $K$  and  $L$  are the number of samples and candidates respectively.  $\widehat{y}_l^k$  is the true label of the training sample.
There could be more than one answer in a dialogue window. In the inference phase, we reserve all the candidates whose matching score is higher than the threshold of 0.5. Since the training process is performed in the window size, the inference phase should be the same situation. We also obtain the dialogue-level results by updating the results of windows as aforementioned.
In this section, we will conduct experiments on the proposed dataset. It is worth to note that we are not going to compare MIE with (Du et al., 2019) and (Lin et al., 2019), because a) they all employed sequential labeling methods, leading to different evaluation dimensions from ours (theirs are more strict as they must give the exact symptom positions in the original utterance), and b) their approaches were customized for sequential labeling paradigm, thus cannot be re-implemented in our dataset.
We use pretrained 300-dimensional Skip-Gram (Mikolov et al., 2013) embeddings to represent chinese characters. We use Adam (Kingma and Ba, 2015) optimizer. The size of the hidden states of both feed-forward network and Bi-LSTM is 400. We apply dropout (Srivastava et al., 2014) with 0.2 drop rate to the output of each module and the hidden states of feed-forward network for regularization. We adopt early stopping using the F1 score on the development set.
We compare MIE with several baselines.
1) Plain-Classifier. We develop a basic classifier model that uses the simplest strategy to accomplish the task. The input of the model are the utterances in the window. We concatenate all the utterances to obtain a long sequence, and encode it using a Bi-LSTM encoder, then we use self-attention to represent it as a single vector. Next, the vector is fed into a feed-forward classifier network. The output labels of the classifier consist of all the possible
candidates. The encoder adopts category-specific parameters.
2) MIE-Classifier. To develop a more competitive model, we reuse MIE model architecture to implement an advanced classifier model. The difference between the classifier model and MIE is the way of obtaining  $q_{c}$  and  $q_{s}$ . Instead of matching, the classifier model treats  $c_{c}^{utt}$  and  $c_{s}^{utt}$  directly as  $q_{c}$  and  $q_{s}$  respectively. Thanks to the attention mechanism in the encoder, the classifier model can also capture the category-item pair information and the status information to some extent. To further examine the effect of turn-interaction, we develop two classifiers as we do in MIE. MIE-Classifiersingle treats each utterance independently, and the probability score of each utterance is calculated. The model uses a max-pooling operation to get the final score. MIE-Classifier-multi considers the turn-interaction as MIE-multi does.
The experimental results are shown in Table 4. From the results, we can obtain the following observations.
1) MIE-multi achieves the best F-score on both window-level and dialogue-level full evaluation metric, as we expected. The F-score reaches 66.40 and 69.28, which are considerable results in such sophisticated medical dialogues.
2) Both of the models using multi-turn interactions perform better than models solely using single utterance information, which further indicates the relations between turns play an important role in dialogues. The proposed approach can capture the interaction. As a proof, MIE-multi achieves a  $2.01\%$  F-score improvement in dialogue-level full evaluation.
3) Matching-based methods surpass classifier models in full evaluation. We think the results are rational because matching-based methods can introduce candidate representation. This also motivates us to leverage more background knowledge in the future. Note that in category and item metrics, MIE-classifiers are better at times, but they fail to correctly predict the status information.
4) Both MIE models and MIE-classifier models overwhelm Plain-Classifier model, which indicates the MIE architecture is far more effective than the basic LSTM representation concatenating method.
5) Dialogue-level performance is not always better than window-level performance in full evalua
6466
Window-level	Dialogue-level
Model	Category	Item	Full	Category	Item	Full
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Plain-Classifier	67.21	63.78	64.92	60.89	49.20	53.81	53.13	49.46	50.69	93.57	89.49	90.96	83.42	73.76	77.29	61.34	52.65	56.08
MIE-Classifier-single	80.51	76.39	77.53	76.58	64.63	68.30	68.20	61.60	62.87	97.14	91.82	93.23	91.77	75.36	80.96	71.87	56.67	61.78
MIE-Classifier-multi	80.72	77.76	78.33	76.84	68.07	70.35	67.87	64.71	64.57	96.61	92.86	93.45	90.68	82.41	84.65	68.86	62.50	63.99
MIE-single	78.62	73.55	74.92	76.67	65.51	68.88	69.40	64.47	65.18	96.93	90.16	92.01	94.27	79.81	84.72	75.37	63.17	67.27
MIE-multi	80.42	76.23	77.77	77.21	66.04	69.75	70.24	64.96	66.40	98.86	91.52	92.69	95.31	82.53	86.83	76.83	64.07	69.28
tion. In our experiment, the classifier-based models perform better in window-level than dialogue-level in full evaluation. The possible reason is error accumulation. When the model predicts results the current window does not support, the errors will be accumulated with the processing of the next window, which will decrease the performance.
To further analyze the behavior of MIE-multi, we print the confusion matrix of category-item predictions, as shown in Figure 3. We denote the matrix as  $A$ ,  $A[i][j]$  means the frequency of the circumstance that the true label is  $i$  while MIE-multi gives the answer  $j$ .
Figure 3: Illustration of the confusion matrix of MIEmulti. Darker color means higher value. The figure in the axis is the category-item pair index of a total number of 71. Values of orange blocks are 0.
Figure 4: Case illustration of attentions: a) attention heat map of category-item pair for each utterance; b) attention heat map of status for each utterance; c) attention heat map for the fourth utterance in the window.
We study the matrix and find that MIEmulti failed to predict Symptom:Limited mobility, Symptom:Nausea, Symptom: Cardiomyopathy, and Test: Renal function test, which are emphasized by orange blocks  $(A[i][i] = 0)$  in Figure 3. The
Table 4: The experimental results of MIE and other baseline models. Both window-level and dialogue-level metrics are evaluated.
Patient: I have atrial fibrillation, heart failure, anemia and loss my appetite.
Doctor: Hello! How long did them last? Did you examine blood routine?
Patient: Yes.
Doctor: Is there coronary heart disease?
Patient: No.
(a)
Patient: I have atrial fibrillation, heart failure, anemia and loss my appetite.
Doctor: Hello! How long did them last? Did you examine blood routine?
Patient: Yes.
Doctor: Is there coronary heart disease?
Patient: No.
(b)
Patient: I have atrial fibrillation, heart failure, anemia and loss my appetite.
Doctor: Hello! How long did them last? Did you examine blood routine?
Patient: Yes.
Doctor: Is there coronary heart disease?
Patient: No.
(c)
possible reason is that they rarely appear in the training set, with frequency of  $0.63\%$ ,  $2.63\%$ ,  $2.38\%$  and  $1.25\%$ , respectively. The results reveal that the data sparse and uneven problems are the bottlenecks of our approach.
5.5 Case Discussion
In this part, we will analyze some cases to verify the effectiveness of the model with best performance, e.g. MIE-multi. Particularly, we investigate an example shown in Figure 4. To determine whether the candidate Symptom:Coronary heart disease (patient-negative) is mentioned in the window, we should focus on the interaction between the adjacent pair located in the last of the window. This adjacent pair is a question-answer pair, the category-item pair information is in the question of the doctor while the status information is in the answer of the patient. In this case, MIE
6467
Patient: What is the effect of sinus arrhythmia?  
Doctor: Sinus arrhythmia is normal in general. Don't care about it unless you feel unwell significantly.  
Patient: I'm feeling unwell so much (because of the sinus arrhythmia).
MIE-single symptom:sinus arrhythmia (unknown) MIE-multi symptom:sinus arrhythmia (patient-positive)
Figure 5: Predictions of MIE-single and MIE-multi. The gray string is the implicit reason.
single does not predict right result due to its independence between utterances, while MIE-multi manages to produce the correct result.
For better understanding, we utilize visualization for matching module and aggregate module. Figure 4(a) is the attention heat map when the category-item pair information vector  $c_c^{can}$  matches the utterances category representations  $H_c^{utt}$ . We can observe that the attention values of the mention of coronary heart disease are relatively high, which illustrates that the model can capture the correct category-item pair information in the window.
Figure 4(b) is the attention heat map when the status information  $c_{s}^{can}$  matches the utterances status representation  $H_{s}^{utt}$ . The attention values of the expressions related to status such as "Yes" and "No" are high, and the expression "No" is even higher. So MIE-multi can also capture the status information in the window.
We also visualize the interaction between the fourth utterance and the other utterances. In Figure 4(c), the score of the fifth utterance is the highest, which is in line with the fact that the fifth utterance is the most relevant utterance in the window. In this way the model successfully obtains the related status information for the category-item pair information in the window.
In a nutshell, MIE-multi can properly capture the category-item pair and status information.
The Effectiveness of Turn Interaction
We demonstrate a case in Figure 5 that can explicitly show the need for turn interaction, where MIE-multi shows its advancement. In this case, the label Symptom:Sinus arrhythmia (patient-positive) requires turn interaction information. Specifically, in the third utterance, the patient omits the reason that makes him sick. However, under the complete context, we can infer the reason is the sinus arrhythmia, since the patient consulted the doctor at the beginning
of the window. The model need to consider the interaction between different utterances to get the conclusion. Interaction-agnostic model like MIE-single makes prediction on single utterance, and then sums them up to get the final conclusion. Consequently, it fails to handle the case when the expressions of category-item and status are separated in different utterances. As a result, MIE-single only obtains the category-item information Symptom:Sinus arrhythmia, but the status prediction is incorrect. In contrast, MIE-multi is able to capture the interaction between different utterances and predicts the label successfully.
In this paper, we first describe a new constructed corpus for the medical information extraction task, including the annotation methods and the evaluation metrics. Then we propose MIE, a deep neural matching model tailored for the task. MIE is able to capture the interaction information between the dialogue turns. To show the advantage of MIE, we develop several competitive baselines for comparison. The experimental results indicate that MIE is a promising solution for medical information extraction towards medical dialogues.
In the future, we should further leverage the internal relations in the candidate end, and try to introduce rich medical background knowledge into our work.
This work is supported by the National Natural Science Foundation of China (No.61533018, No.61922085, No.61906196) and the Key Research Program of the Chinese Academy of Sciences (Grant NO. ZDBS-SSW-JSC006). This work is also supported by Beijing Academy of Artificial Intelligence (BAAI2019QN0301), the Open Project of Beijing Key Laboratory of Mental Disroders (2019JSJB06) and the independent research project of National Laboratory of Pattern Recognition.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (almost) from Scratch. Journal of machine learning research, 12(Aug):2493-2537.
6468
Nan Du, Kai Chen, Anjuli Kannan, Linh Tran, Yuhui Chen, and Izhak Shafran. 2019. Extracting symptoms and their status from clinical conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 915-925.
Gregory Finley, Erik Edwards, Amanda Robinson, Michael Brenndoerfer, Najmeh Sadoughi, James Fone, Nico Axtmann, Mark Miller, and David Suendermann-Oeft. 2018. An automated medical scribe for documenting clinical encounters. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11-15.
Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015.
Xinzhu Lin, Xiahui He, Qin Chen, Huaixiao Tou, Zhongyu Wei, and Ting Chen. 2019. Enhancing dialogue symptom diagnosis with global attention and symptom graph. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5032-5041.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Nikola Mrkšić, Diarmuid Žáeghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788.
Katta G Murty and Santosh N Kabadi. 1987. Some np-complete problems in quadratic and nonlinear programming. Mathematical programming, 39(2):117-129.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, ICLR 2017.
Christine Sinsky, Lacey Colligan, Ling Li, Mirela Prgomet, Sam Reynolds, Lindsey Goeders, Johanna Westbrook, Michael Tutty, and George Blike. 2016. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Annals of internal medicine, 165(11):753-760.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Özlem Uzuner, Brett R South, Shuying Shen, and Scott L DuVall. 2011. 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552-556.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Robert Wachter and Jeff Goldsmith. 2018. To combat physician burnout and improve care, fix the electronic health record. *Harvard Business Review*.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496-505.
Liu Yang, Qingyao Ai, Jiafeng Guo, and W Bruce Croft. 2016. anmm: Ranking short answer texts with attention-based neural matching model. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 287-296.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018.
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1118-1127.
6469