Dialogue Medical Information Extraction with Medical-Item Graph and Dialogue-Status Enriched Representation
Authors: Lei Gao
Xinnan Zhang
Xian Wu
Shen Ge
Yefeng Zheng
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue medical information extraction
Multi-label text classification
Graph neural network
Natural language processing
TL;DR: Extracting medical item status pairs from patient doctor conversations with GNN

Abstract: The multi-turn doctor-patient dialogue includes rich medical knowledge, like the symptoms of the patient, the diagnosis and medication suggested by the doctor. If mined and represented properly, such medical knowledge can benefit a large range of clinical applications, including diagnosis assistance and medication recommendation. To derive structured knowledge from free text dialogues, we target a critical task: the Dialogue Medical Information Extraction (DMIE). DMIE aims to detect pre-defined clinical meaningful medical items (symptoms, surgery, etc.) as well as their statuses (positive, negative, etc.) from the dialogue. Existing approaches mainly formulate DMIE as a multi-label classification problem and ignore the relationships among medical items and statuses. Different from previous approaches, we propose a heterogeneous graph to model the relationship between items. We further propose two consecutive attention based modules to enrich the item representation with the dialogue and status. In this manner, we are able to model the relationships among medical items and statuses in the DMIE task. Experimental results on the public benchmark data set show that the proposed model outperforms previous works and achieves the state-of-the-art performance.

Dialogue Medical Information Extraction with Medical-Item Graph and Dialogue-Status Enriched Representation
Lei Gao; Xinnan Zhang; Xian Wu, Shen Ge and Yefeng Zheng
Tencent Youtu Lab, Jarvis Research Center (lexlgao, xinnanzhang, kevinxwu, shenge, yeffenghhengg@eeenent.coom
Abstract
Docter-Patient Conversation
Patient: (E B052 H-Eff 1188, /"g24
(Hello, doctor, What the resson for the frequent ventricular promature beats? Is serions?)
The multi-turn pector-pptient dialogue includes rich medical knowledge, like the symptoms of the patient, the diagnosis and medication sug- gested by the doctor. If mined and represented properly, such medical knowledge can benefit a large range of clinical applications, includ- ing diagnosis assistance and medication rec- ommendation. To derive structured knowledge from free text dialogues, we target a critical task: the Dialogue Medical Information Extrac- tion (DMIE). DMIE aims to detect pre-defined clinical meaningful medical items (symptoms, surgery, etc.) as well as their statuses (posi- tive, negative, etc.) from the dialogue. Exist- ing approaches mainly formulate DMIE as multi-label classification problem and ignore the relationships among medical items and sta- tuses. Different from previous approaches, we propose heterogeneous graph model the relationship between items. We further propose two consecutive attention based modules to en- rich the item representation with the dialogue and status. In this manner, we are able to model the relationships among medical items and sta- tuses in the DMIE task. Experimental results on the public benchmark data set show that the proposed model outperforms previous works and achieves the state-of-the-art performance.
Doctor: R20 R2.% 10 SESR 13 8032 1: R-C. (UN pe T TR 11 EE TT
will provide you with online guidance for medical treatment. Frequent ventricular premature beats are problems with the conduction system the hear. Whether it is seriuss or not can be determined according to the specific situation. Have you an electrocardingram))
Patient:
1t it At f5 2 1E C- (Hl, Mir & #h (Is there any danger from low blood pressure? haven't done an clectrocardiogram. How treated?)
Doctor:
-- 22441195 & & GJ T -0- $4 WE 1816 EUSE R0 (Low blood pressure is not necessurily dangerous, It is betler to find premaauree ventricular benis throughout the day, frquenyyy ablation can be used for treatment if necessary.)
24-hour dynamic Bbetrocardiogran
many
Patient: Kf iti W&
(OK, thank you doctor.)
Extracted Labels
Category
Item
Status
Label
Symptom
Frequent Ventricular Premature Bears
Patient-Poo Pos
Label
Symptom
Hypotemsion
Unknown
Label 3 Surgery
Radiefroquency Ablation
Doctor-Poo
Label
Electrocardiogram
Patient-Nee Negent-Negg
Label
Test
Electrocardiogram
Doctor- Pos
Figure I: An example of the DMIE task, translated from Chinese. Given the whole dialogue, five labels are extracted. Each label consists of category, item and its status. The label and the corresponding evidence in dialogue are highlighted with the same color.
1 Introduction
diagnosis assistance and medication recommenda- tion. To this end, we target a critical task: the Dialogue Medical Information Extraction (DMIE). Given multi-turn doctor doctor-patient dialogue, the DMIE task aims to label this dialogue with pre- defined medical items and their corresponding sta- tuses. We use an example from the MIEACL data set' (Zhang et al., 2020) to illustrate the DMIE task. As shown in Figure 1, this dialogue receives five labels in total. Each label consists of three sub-components: Category, Item and Status, and
The digitization of medical systems over the past decade has accumulated massive medical data. Among various medical data, the multi-turn doctor- patient dialogue contains rich medical knowledge, like the suggested medical test according to patient symptom descriptions, the recommended medica- tion according to doctor's diagnosis results, etc. If such medical knowledge can be properly extracted and represented, it can enable the development of a large range of clinical applications, like discase
"Equal contribution. Conresponding author.

---

their combination provides detailed medical de- scriptions. As shown in Table 1, all items are grouped into four types of categories: Symptom, Surgery, Test and Other info. And the Other info include mental state, sleep situation, whether drink- ing and etc. In addition to items and their cate- gories, the MIEACL dataset also defined five sta- tuses to provide fine-grained state for each item. For example, for item whose category is Symp- tom, the status Posittiv indicates that this patient mentions he/she has this symptom, and the status Dector-Negative indicates that the doctor de- nies this symptom in the dialogue. For an item with category of Test, the status Doctor-Positive indicates that the doctor recommends the patient to take this test. For example, in Figure 1, Label 4 cessst-leccooardiggnaaee Patient-Neg indicates that the patient mentioned he/she did not have an worth mention-
Table 1: Details of the labels in a DMIE dataset: MIEACL.
Caleggrry
Backache, Perspiration. Hiceups, Nausea, Cyanosis, Fevor, Fatigue, Abdominal discomfort.
Symptom
Patient-Postivve (appearidone! normal)
Interventional treaiment, Radiofrequency ablation.
Surgery
[eert bypass surgery.
Parient Negative (ahcem/nntt done! abnormal)
Stent implantation.
B-mode alirasonogmphhy
CT examination,
Doctor: Positive (diggosed//ugges)
CT angiography. CDFL Ultrisonogrpphy, MRI,
Thyroid function test,
Doctor Negutive (exceule/@eprecatee)
Tresdmill test,
Sleep. Dier, Defocation, Smoking. Drinking.
Mental condition.
heterogeneous graph enrich the representa- tions of items. In particular, we use the prior co- occurrences statisties and relationship information between items to guide the information transmis- sion in the heterogeneous graph; Thirdly, we enrich the item representations with dialogue context and semantic information of status by introducing two attention based components, Item Updating with Dialogue-Aware Attention Module and an Item Re- fining with Status-Aware Module. Finally, we use the joint representation which combines the item, status and dialogue representations as well as their mutual relations to predict the set of proper labels for this dialogue. We evaluate our approach on the MIEACL benchmark, and the results show that the proposed model outperforms all baselines. Fur- ther ablation study shows the effectiveness of our framework model the diverse relationship among items, dialogues and statuses.
clectrocardiogram test before. ing that the statuses of an item in DMIE tasks are not mutually exclusive, like Label 4 and Label 5, an item can have two statuses at the same time.
Most existing approaches either focus on rec- ognizing a subset of items or directly formulate DMIE as a classification task. Among existing DMIE works, Lin et al. (Lin et al., 2019) and Du et al. (Du al., 2019) only exploited the symptom items. They first recognized symptom terms from dialogues and then classified their corresponding statuses; Zhang et (Zhang al., 2020) proposed a deep neural matching model which actually a multi-lahel text classification (MLTC) model. Each combination of item and status refers to an indepen- dent class label. Recently, Li et al. (Li et al., 202 Ib) modeled the extraction developed multi-gne caaliiiranullartty transformer which can effectively capture the interaction between role- enhanced crossturns and integrate representations of mixed granularity. Xia et (Xia al., 2022) proposed a speaker- aware co-attention framework to address the intricate interactions among different utterances and the correlations between utterances and candidate items. All aforementioned methods disregard the relations among items as well as the relations between items and statuses.
Overall, the main contributions are as follows:
generation process, and
We propose a novel model for the DMIE, which introduces the interactions among med- ical items, incorporates the doctor-patient dia- logue information to item representations and explores the item-status relationships.
Rather than directly formulating DMIE as a multi-label text classification task, we design a status-aware item refining module for the DMIE task, which effectively captures the re- lationship between items and statuses.
In this paper, propose novel approach to model 1) the relations among items and 2) the re- lations between items and statuses in the DMIE task. Firstly, we adopt pre-trained language model to learn representations for dialogues, items and statuses respectively; Secondly, we introduce
The experimental results on the MIEACL benchmark dataset show that our model signif- icantly outperforms the state-of-the-a ahe mod- cls, achieving a new SOTA performance."
Phtps//giituuhcoom/MMMMEEEEM

---

2 Related Work
different labels, researchers began to exploit the knowledge of label correlations. Seq2emo (Huang et al., 2021) used a bi-directional LSTM (Hochre- iter and Schmidhuber, 1997) decoder to implicitly model the correlations among different emotions; CorNet (Xun et al., 2020) designed a computational unit which was actually an improved version of a fully connected layer to learn label correlations, enhancing raw label predictions with correlation knowledge and output augmented label predictions; LDGN (Ma al., 2021) employed dual Graph Con- volution Network (GCN) (Kipf and Welling, 2017) to model the complete and adaptive interactions among labels based on the statistics of label co- occurrence, and dynamically reconstruct the graph in joint way; (Zhang et al., 2021) designed a multi-task model, in which two additional label co-occurrence prediction tasks were proposed enhance label relevance feedback.
2.1 Dialogue Medical Information Extraction
Extracting medical information from doctor-pation dialogues has been receiving growing research in- terests. Shi al. (2022) formulated dialogue medi- cal information extraction as slot filling task and proposed to extract symptoms from dialogues. Lin et al. (2019) detected symptom items as well their status in a pipeline fashion: first recognize symp- tom words from the dialogue and then classify them to predefined items and statuses. However, Lin al. (2019) can only detect the items that are ex- plicitly mentioned in the dialogue and ignore the implicitly mentioned ones. To identify implicitly mentioned symptoms, Du al. (2019) proposed a Seq2Seq model to generate a sequence of symp- toms and their statuses in an end-to-end manner. Recently, Zhang et al. (2020) utilized an attention based matching module to obtain category-specific representation and status-specific representation from the doctor-patiens dialogues. The two ac- quired representations are further used to extract the targeted medical information. Li et al. (2021b) incorporated word-level information by using a nattice-basede encoder and a proposed relative po- sition encoding method. Additionally, they pro- posed a role access controlled attention mechanism to incorporate utterance-level interaction informa- tion. Xia et al. (2022) proposed speaker-aware dialogue encoder with multi-task learning, which incorporates the speaker's identity into the model. They also presented a co-attention fusion network to combine the information from different utter- ances and handle the intricate interactions among them, as well as the correlations between the utter- ances and candidate items.
In addition, there are also some other meth- ods to improve model performance. For exam- ple, LightXML (Jiang et al., 2021) introduced a probabilistic label tree based clustering layer and a fully connected dynamic negative sampling layer text representations to improve accuracy while controlling computational complexity and model sizes. For other NLP tasks like slot filling, slot correlations is learned with a self-attention mecha- nism (Ye et al., 2021). However, slot filling aims to extract the slots to fill in parameters of the user's s query which is different from the targeted MIE task in this paper (classify dialog into predefined item status combinations).
Although DMIE could be treated as special multi-label text classification task, the aforemen- tioned methods are not able to process status in- formation, which is not included in the ordinary MLTC datasets. Thus, we design an item refin- ing with status-aware attention module to learn the cross attention between statuses and items to gen- erate the status enhanced item representation.
Existing DMIE approaches treat each medical item independently. As result, the correlations among medical items are ignored in modeling. Dif- ferent from the above works, we employ a hetero- geneous graph to directly model the correlations between medical items. Furthermore, we intro- duce an item updating with dialoguc-aware atten- tion module to infer implicit mention of medical items from dialogues.
Approach
3.1 Problem Delinition
Given a dialogue with multiple consecutive turns of conversations, the objective of the DMIE task is to detect all predefined items, with their correspond- ing categories and mentioning statuses. Following (Zhang et al., 2020), we formulate the DMIE task as follows.
2.2 Multi-Label Text Classification
The research of Multi-Label Text Classification (MLTC) focuses on two topics: document represen- tation learning and label correlation learning. For a better understanding relations between text and
Given a dialogue D, we need to predict the labels

---

Dialogue (Status) -Awre Structure
F1.1 Fes
[|]
Add & Norm
Predication
Feed Forward
Pooling
Item-Wise Feature Cross
Add & Norm
Item Refining with Status Aware Attention (IS)
Mult--Head Attention
Items atand to Dialogue. (Status)
Item Updating with Dialogue-Award Aware Attention (ID
Hp
Add & Norm
Interactive Heterogeneaus Graph Convolution (IIGGC)
Multi-Head self-Attention
Items: attend to Items
Item Co-occurrence Heterogeneous Graph
Figure 2: The workflow of the proposed model, which consists of three main components: Item Interactive Heterogeneous Graph Convolution (IIHGC) module, Item Updating with Dialogue-Awan Attention (ID) module and Item Refining with Status-Aware (IS) module.
[v..}] E {0, (ix where and S represent the item set and status set, respectively, and Vi.j denotes whether the item E with status j E $ is mentioned this dialogue. Here /| is the total number of predefined items, and |S| = 5 is the total number of statuses, i.e., Ratient-Positive Patient- Negative, Doctor-Positive, Doctor-Negative and Unknown. These items can be grouped with set of C categories, with mapping r(-) where rii) E C assigns the corresponding category, i.e., Symptom, Surgery, Test or Other info to the item
3.3 Item Interactive Heterogeneous Graph Convolution
Items in DMIE tasks belong to four categories: Symptom, Surgery, Test and Other info, and the prior information between item pairs can be defined by the relationships between their categories. In Figure 1, when the patient said that he/she had the symptom of frequent ventricular premature beats, the doctor advised him to perform electrocardio- grams testing and radiofrequency ablation surgery. As result, we could observe a "Symptom-Test" re- lationship between frequent ventricular premature beats and electrocardiograms, and "Symptom- Surgery" relationship between frequent premature beats and radiofrequency ablation. Such co-occurrence relationship between items is one prior knowledge that can be used the DMIE task. As shown in Figure 2, we designed an Item Inter- active Heterogeneous Graph Convolution (IHGGC) module to explicitly model the correlation among items.
3.2 Dialogue, Item and Status Encoder
frequet ventricular
We denote a dialogue D = [ui, 1/2 un} with n utterances, where
[wf, w ...,, w and of is the j-th token in i-th utterance in the dia- logue. We treat 1) long sequence (D) tokens, [D] = m² and feed to the pre-trained lan- guage model to generate the contextualized rep- resentations Hp = [ hipi) HD E D|xd where h; E d is the contextualized representation of the i-th token with dimension of d.
3.3.1 Graph Construction
We use an undirected heterogeneous graph to model item correlations with different types of nodes and edges. We establish the edges between items based on their categories and the frequency of their co-occurrence in the corpus. Formally, our graph is denoted as G = (V,E). Each node v E V represents an item with a corresponding
For pre-defined items and statuses, we feed their text descriptions into a pre-trained language model then use the mean-pooling of the last layer outputs to initialize their latent features H1 Ihi....,hy h,,,,, H, E R xd and Ils Th....,hisil Hs E R|

---

category 7(D) E C. Each edge (Vi, Uj) & represents co-oceurrence between items ey and "j from different categories. For example, the co-occurrence of frequent ventricular premature beats and electrocardiogram in Figure 1 gives an edge ("Frequent Ventricular Premature Beats", "Symptom-Tes"" "Eleecrocanddogran"") in our het- crogeneous graph. In addition, the co-occurrence patterns between item pairs obtained from training data could contain noisy information. To reduce noise, we remove all edges with the number of co-occurrences less than four.
where represents concatenation and H; has a dimension of K d' d features.
Finally, we aggregate messages from all sub- graph with different relationships for each node:
UR
Hacc,
Σ
Hrz
where Hr, is the representation of item i calcu- lated from the r-th subgraph and [R] is the total number of relationship categories included in the heterogeneous graph.
In setting, the final output of node i, HHGG, has dimension of d features.
3.3.2 Heterogeneous Graph Convolution
3.4 Item Updating with Dialogue-Aware Attention
After graph construction, utilize heteroge- neous graph convolution module model the item interactions, which the IIHGC module in Fig- ure 2. The IIHGC module takes the item representa- tion H₁ = [hi...., h z] and the constructed graph as inputs and outputs high-level features HHGC-
In the DMIE task, each item corresponds to nor- malized name. The content of the doctor-patien dialogue can help
determine which item is mentioned, and its corresponding status. In this pa- per, we utilize item-dialogue cross attention mech- anism to enrich the information of item represen- tation for better classification, which is denoted as the Item Updating with Dialogue- Attention (ID) module in Figure 2.
We first decompose the entire heterogeneous graph into several homogeneous subgraphs accord- ing the types of edge relationships, and then ap- ply independent Graph Attention Network (GAT) (Velickovic et al., 2018) on each subgraph to ex- tract higher-level aggregated features. The graph attention network of each subgraph refines the rep- resentation of nodes by aggregating and updating information from its neighbors via helti-head at- tention mechanism. For given edge from node i to node j. the attention coefficient of the k-th head at is calculated as follows:
We obtain the dialogue representation HD. item representation H, from the dialogue encoder and the item encoder, respectively. Then we update the item representation through the aforementioned Item Interactive Heterogeneous Graph module and obtain the representation HIGC- Next, in the ID module, we use the attention mechanism similar to the Transformer block (Vaswani et al., 2017) to model the relation between the dialogue and items:
exp (LonkyRoLU(e (a (WINDWS5)
a/3 DEEN; eXp (LeakyRoLU(aT (WFA, ([(he)
Huac AMM Hoo.. MultiHeadi Hucc, Huoc, Hheel),
where d' is the dimension of the head attention,
#i. AdMRNorm (f MoltiHeadd Maace Ho. HD));
W E Rd' xd Wh d xd R2d are model parameters, N₁ is the neighbors of node i and denotes the concatenation operation.
Hoo = AdI8Norm (H) FFN(H)))
where Add&Norm, MultiHead and FFN are the standard Transformer operations (Vaswani et al., 2017). Since there is no autoregressive or causal relationship between items, we do not use attention masks in the self attention layer.
Next, the representation of the k-th head for node obtained by linear combination of attention coefficients and corresponding features:
wh1))) JEN
3.5
Item Refining with Status-Aware Attention
where is the activation function. After that, get the final information- aggregated output of node i in this subgraph by concatenating the generated K independent atten- tion heads outputs:
One major difference between the DMIE task and the conventional multi-label text classification task that it needs to assign a status to each item. There- fore, in this paper, we design an Item Refining with Status- Aware Attention (IS) module in Figure 2,
H. 1

---

enrich the representation of items. The IS mod- ule uses mechanism similar the ID module to refine the representation of items:
4.2 Evaluation Metries and Settings
Following the settings of (Zhang al., 2020), we use the MIE-Macro Precision, MIE-Macro Recall and MIE-Macro Fi score as our main evaluation metrics, which produce the metrics over the av- erage performance of each sliding window. It is worth mentioning that, different from ordinary Macro-Average metrics, the MIE-Macro Precision and MIE-Macro Recall is set to by (Zhang et al., 2020) if the prediction is empty no labels. To reduce the impact of samples with no labels, we additionally use Micro-Average metrics for further comparison. For a fair comparison, we apply the same train/dev/test dataset split as (Zhang et al., 2020), and select the best performing model in the validation set for testing.
1i Adai&Neem(H1 MultiHead HIp, Hyd)), His Add&Norm| ip MuhiHead(H Ha. Hs)), His Add&Nomm| His (s)(
where Hi is the refined status-aware item rep- resentation. Following the ID module, we also dis- card the attention mask in the self attention layer in the IS module.
window with
3.6 Label Prediction and Optimization Objectives
After the above procedures, we obtain the pooled dialogue representation ha MeanPooling ())). item embedding H1, item-graph interacted repre- sentation HHGC, dialogue-award item represen- tation HID and status-aware item representation HIs. We predict the [S]-dimension status of item by:
We utilize the Chinese Roberta (Cui et al., 2019) as the encoder in our model, which base ver- sion of the pre-trained Chinese RoBERTa-wwm- ext. The ID, IIHGC, and IS have the same dropout rate, attention heads, and hidden layer dimensions as the original RoBERTa model, which are 0.1, 12, and 768 respectively. We employ the AdamW opti- mizer for training model, with learning rate of 2e-5 for the parameters in the RoBERTa encoder and 2c-4 for the of the model. We incorpo- rate an early stop mechanism to avoid overfitting, whereby the training is stopped there no im- provement the MIE-Macro Fl or MIE-Micro Fl on the validation set over 10 epochs. Running an experiment with these settings will roughly take 9 hours single NVidia V V100 GPU.
(W.. ffha, (ha, Hxoo,)
1fbaa (lka, H1s:)l).
f(h1. h2) = [Nallabs((ss b2)]
where the sigmoid activation function and W. E x9d are the model parameters used for classification.
The proposed model is trained with the binary cross entropy loss:
4.3 Baseline Models
To demonstrate the effectiveness of our proposed model, we compare it with seven strong baselines:
Σ
Σ Biy log(br))) se.5) log(1 0e.))
basic clas- sification model which employs the top-level representation hCLS to perform medical infor- mation extraction.
RoBERTACIS: RoBERTacI
4 Experiments
4.1 Dataset and Preprocess
We validate our model on the MIEACL benchmark from an online medical consultation website and proposed by (Zhang et al., 2020). (Zhang et al., 2020) divided each dialogue into multiple parts using sliding window, the sliding step is and the window size is 5. In this manner, dialogue with k conversation turns will be split into windows size 5. total, the 1,120 dialogues in MIEACL are divided into 18,212 windows.
LightXML. (Jiang et al., 2021): LightXMI. adopts end-to-end training, label clustering and dynamic negative label sampling im- prove model performance.
CorNet (Xun et al., 2020): CorNet adds an extra module learn label correlations to en- hance raw label predictions and output aug- mented label predictions.

---

Table 2: Results(%) on the MIEACL dataset, where P stands for Precision, R stands for Recall, and FI is the standard Fl metric. IIHGC+ID+IS is our proposed model, which can observed to outperform over all baselines. indicates the results from original paper.
Table 3: Quantitative analysis(%) of our model with different variants.
Metric
Metric
Variants IHHG
79.
78,20
77.57
74.38
74.45
MIF-Macro
2
79.84
79.05
78.31
76.25
73.86
75.04
78.97
79.36
78.01
75.12
74.86
74.99
82.19
80.49
81 46
RoBERTaaLs
77,80
75.40
75.35
75.77
69.90
72.72
Full
V
82.77
83.60
82.54
81.35
81.89
81.62
LightXMI.
74.72
70.81
71.38
12.45
62.17
66.92
CorNet
67.72
65.66
MIE
69.59
65.89
66.05
69:43
61.77
65.38
SAFE'
73.86
MGT
75.30
71.70
72.70
We can also see that our proposed method out- performs the conventional Multi-Label Text Clas- sification (MLTC) approaches (Jiang al., 2021; Xun al., 2020). This is mainly due that the MLTC models do not consider the complex rela- tionships between the conversation contents and items, as well as the corresponding relationships between items and status. Our model outperforms these models by more than 1 % and 14% on MILE- Macro F1 score and Micro FI score, respectively.
IHGC+ID=1S
82,77
83,60
82.54
81.35
81.89
81.62
MIE (Zhang al., 2020): MIE focuses on learning an improving representation using deep matching architecture that takes into account dialogue-tune turn interactions to capture the category-item pair information and the status information, and feed them both into classifier for text classification.
To confirm that the improvement does not merely come from the adoption of pre-trained language model, we compare our model with RoBERTacLS. The performance of RoBERTacIS shows that the adoption of a high-quality pre-trained model is helpful for the DMIE task. Nevertheless, our method surpasses this baseline by additional 7.19% and 8.9% on MIE-Macro FI score and Micro FI score, respectively, showing the effectiveness of the introduction of our IIHGC, ID and IS modules.
SAFE (Xia et al., 2022): SAFE a speaker- aware model that employs a co-attention fu- sion technique with multitask learning and graph networks.
MGT (Li al., 2021b): The MGT model proposes a Multi- Guati-Gramularity Transformer to fully capture the interaction between role- enhanced cross-turns and integrate mixed granularity representations.
4.5
Ablation Study
4.4 Results and Discussion
4.5.1
Results of Different Model Variants
We report the experimental results all comparing algorithms on the MIEACL datasct in Table 2. Our approach outperforms previous work (Zhang et al., 2020) on the DMIE task. The combi- nation of IIHGC, ID, and IS achieves 82.54% MIE- Macro Fl and 81.62% Micro Fl scores, outper- forming the previous work by 16.49% and 16.24% respectively. The performance gain mainly due to the introduction internal correlations between items and statuses. Our model not only introduces prior information between items, but also uses the self-attention mechanism to effectively update and refine item representations. Please note that since we need to calculate the Micro metrics in addition the Marco metrics, for consistency consideration, we re-run the code of (Zhang et al., 2020) on the benchmark data set instead of directly citing the numbers reported (Zhang et al., 2020).
We conduct an ablation study by gradually strip- ping components to examine the effectiveness of each component in our full model (IHGG++D+IS), The experiment results are illustrated in Table 3. Variant, is the most basic variant which only uses the dialogue representation and item embeddings for label prediction. Benefiting from the introduc- tion of item information, the model Variant, has effectively improved the text classification based approach (RoBERTacLs). However, due to the lack of medical item correlation information, the refer- ence relationship between dialogue and item, and the corresponding relationship between item and status, Variant; gets the worst performance among all variants. The performances of Variantz2-41 are better than Variant,, which proves the effectiveness of the introduced IIHGC, ID and IS components. single ID module brings the largest improvement

---

Table 4: Results (6) on the MIEACL Dataset with Different GNN models
MIE Macro Metric
Micro Metric
Metheds
P
R
F1
P
R
FI
MIE
69.59
65,89
66.05
69.45
61.77
65.35
ITTGC with /Velickovic
$2.77
IHHC with (Cooso et al., 2020)
82,56
83.54
82.35
81.43
81.40
81.41
IHGG with (Chen 2020)
82.42
82.65
81.34
81.72
81.24
81.48
ITGG with (Li al., 2021a)
81.15
83.76
81.67
81.15
81.69
81.41
Gree)
Sivep
13k
6:4k
Hodey
Deint Velee Veriss Votle, rubllen
wele
Vwing torke natlent
Figure 3: Model performance distribution under differ- ent random seeds.
Figure 4: The distribution of medical item+status label frequency on the MIEACL. benchmark dataset.
compared to the baseline variant. With the com- bination of IIHGC, ID and IS modules, our full model is able to obtain improvements of 4.97% on the MIE-Macro F1 score and 7.17% on the Micro Fl score over Variant Overall, removal of any in- troduced components will cause the classification performance drop, which confirms the contri- bution of each component in the final model and proves their effectiveness.
tion of item and its specified status. As shown in Figure 4, the label distribution the actual medi- cal environment naturally long tailed. Therefore, such long-tai tail bias may have impact on the perfor- mance of the model and its practicability. Here we divide all the labels into three groups according to their frequencies in the training set to explore the performance of the model: the major group (Group
I, F 90), the mid-frequency group (Group 2, 20 F 90), and the long-tail group (Group 3, 1 F S 20). We calculate the accuracy of each la- bel and add into corresponding group average accuracy. As shown Table 5, we find that the per- formance of all methods gradually declines when the the label frequency becomes smaller, but the performance of the full model is constantly higher than all variants and decreases in slower rate. In Group can see that the full model achieves 28.65% improvement compared with the baseline variant model. We speculate the reason due the incapability of the baseline variants to under- stand the semantics of labels, which restricts their performance. The full model not only generates fine-grained item representations for different dia- logues, but also discovers the implicit semantics of long-tail labels.
In Section 3.3, we use Graph Attention Network (GAT) introduced in (Velickovic et al., 2018) to model the relations among items. There are also other graph neural network approaches like (Corso al., 2020; Chen et al., 2020; Li et al., 2021a). Here we compare the performance of the IIHGC equipping with different GNN models. As shown in Table 4, we can find that equipping with any of these four graph models, the proposed method consistently outperforms SOTA works.
We further explore the sensitivity the model to random seed setting, and the results are presented in Figure 3. Here five different random seeds were used. We find that the full model performance re- mains stable under different random seeds and is consistently better than other model variants. To sum up, our proposed method can effectively im- prove the performance and robust in training.
4.5.3 Results of Different Dialogue Lengths
4.5.2 Results of Different Label Frequencies
Medical conversations varied in length which is usually determined by the complexity of patients' diseases. The MIEACL dataset adopts a conven-
Now we study the performance on labels with var- ied frequencies. Here a label means the combina-

---

Table Labels with different frequencies. Groups 1, 2. and 3 correspond to the divisions in Figure 4.
Table 7: Results(%) on the MIEACL datasct, Compari- with LLM approaches
Component
Group
MIE-Macro Metric
Micro Metric
Variants
IHAC
Methods
R
P
FI
27.05
GPT-3.5 Turbo
46.14
40.46
43.11
39.37
29.96
34.03
2
50.87
44.70
27.57
GPT4
54.92
18
58.33
44.36
56.72
49.78
83.60
81.35
81.62
v
82.87
48.79
31.38
52.43
Full Model
83.91
65.12
55.68
Table 6: Fl (%) of dialogues with different lengths.
Component
Length Group
Variants
IIHC
ID
1S
I S 300
300
"Laree'" "grind ner pange their "ratta"" andriic sas
77.54
69.14
3
81.62
$7.63
Decto Hegite
gary
Bodel satonc surth saten can rithaur or oemannn canee winng to grenci
76.79
Che uocarara cartn outor lad
tw meat reveraaton arib
81.76
85.83
Full Model
82.34
87.69
Livourt iratone the oous bull tncee aticrsot statu morpees
Dockic Ang care sating uo and mediartor ouamecat Hou sara outect hor paso
Dockin ave you trududt
decar prosetee macciee..
tional data annotation scheme by dividing the di- alogue into multiple windows to predict the item mentions and their statuses at the window level. As a result, the longer conversations will be divided into more instances which could bring in data bias in training. In this part, we divide the test set into two groups according to the length of the dialog compare the model variants: length less than equal to 300 (Group 1) and length greater than 300 (Group 2). The experimental results are shown in the Table 6. Our full model still significantly outperforms other model variants in both groups. It is worth mentioning that only the performance of the Variant; decreases as the input length in- creases. The performance of the remaining variants Group 2 surpassed their performance Group 1. This is mainly because longer dialogues con- tain more information. Though the baseline model Variant; cannot capture this information well, the other variants with one or more of our proposed components effectively extract these key informa- tion from different perspectives. The empirical results clearly demonstrate the superiority of our model on longer dialogues, which contain richer information.
Duice aoter on lave nentt encanee antmun restrat
actific soare
couutise stalt caneggoai lgg etak ex
cade and
Satar Fatont
yerate Fricur horn Desacw
Dedee
Figure 5: The One-Shot Prompt Template of the DMIE Task for GPT-3.5 and GPT-4
both underperformed SOTA MIE works and the proposed IIHGC by a large margin. This is may be due to, for this particular MIE task, the supervised models are more suitable than the general LLM approaches.
Conclusions
We introduce heterogeneous item graph to model item correlations, as well as two attention based modules learn a dialogue-status enriched joint representation for Dialogue Medical Information Extraction (DMIE). Instead of formulating DMIE as an ordinary multi-label text classification prob- lem, we consider the item-status relationship and model this relationship explicitly. Extensive ex- periments demonstrated that each component of our proposed model could bring performance gains to the baseline model, and that their combination further improved the result and achieved the state- of-the-art performance on the MIEACL benchmark. We also evaluate the performance of GPT-3.5- --55.-tturbo and GPT-4. In future, we will attempt to include textual names of items and status modeling.
4.5.4 Comparison with LLM
We also evaluate the performance of GGPT-3.5-turbl and GPT-4 the DMIE task. Here we design specific prompt and adopt one-shot learning setting. The prompt template can be found in Figure 5. The experimental results are shown in Table We can find that GPT-4 outperformed GPT-3.5-turba but

---

References
Xiaoming Shi, Sendong Zhao, Yuxuan Wang. Xi Chen, Ziheng Zhang, Yefeng Zheng, and Wanxiang Che. 2022. Understanding patient query with weak su- pervision from doctor response. IEEE Journal of Biomedical and Health Informatics, 26(6):2770- 2777.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020. Simple and deep graph convo- lutional networks. In Proceedings of the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1735. PMLR.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurlPS, pages 5998-6008.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Velickovic. 2020. Principal neighbourhood aggregation for graph nets. In Pro- ceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20) 20, Red Hook, NY, USA. Curran Associates Inc.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liø, and Yoshua Bengio. 2018. Graph attention networks. In ICLR (Poster).
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang. and Guoping Hu. 2019. Pre- training with whole word masking for chinese bert. arXiv preprint 1XXx::906600801
Yuan Xia, Zhenhui Shi, Jingho Zhou, Jiayu Xu, Chao Lu, Yehui Yang. Lei Wang, Haifeng Huang, Xia Zhang, and Junwei Liu. 2022. A speaker-aware co-attention framework for medical dialogue infor- mation extraction. In EMNLP, pages 4777-4786. Association for Computational Linguisties.
Nan Du, Kai Chen, Anjuli Kannan, Linh Tran, Yuhui Chen, and Izhak Shafran. 2019. Extracting symp- toms and their status from clinical conversations. In ACL (1), pages 915-925.
Guangxu Xun, Kishlay Jha, Jianhui Sun, and Aidong Zhang. 2020. Correlation networks for extreme multi-label text classification. In KDD, pages 1074- 1082.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short- tort-term memory. Neural Comput., 98)):1335 1780.
Chenyang Huang. Amine Trabelsi, Xuebin Qin, Naw- shad Farraque, Lili Mou, and Osmar R. Zaiane. 2021. Seq2emo: A sequence multi-label emotion classi- fication model. NAACL-HLT 2021, pages 4717- 4724.
Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li, and Emine Yilmaz. 2021. Slot self- attentive dialogue state tracking. In Proceedings of the Web Conference 2021, WWW '21, page 1598 1608, New York, NY. USA. Association for Computing Machinery.
Ting Jiang, Deging Wang, Leilei Sun, Huayi Yang. Zhengyang Zhao, and Fuzhen Zhuang. 2021. Lightxml: Transformer with dynamic negative sam- pling for high-performance extreme multi-label text classification. In AAAI, pages 7987-7994.
Ximing Zhang, Qian-Weit Zhang, Zhao Yan, Ruifang Liu, and Yunbo Cao. 2021. Enhancing label corre- lation feedback in multi-label text classification via multi-task learning. In Findings ACI/IJCNIP 2021, pages 1190-1200.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In ICLR 2017.
Yuanzhe Zhang, Zhongtao Jiang, Tao Zhang, Shiwan Liu, Jiarun Cao, Kang Liu, Shengping Liu, and Jun Zhao. 2020. MIE: A medical information extractor towards medical dialogues. In ACL, pages 6460- 6469.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. 2021a. Training graph neural net- works with 1000 layers. CoRR, abs/2106.07476.
Mei Li. Lu Xiang, Xiaomian Kang, Yang Zhao, Yu Zhou, and Chengqing Zong. 2021b. Medical
A Limitations
term and status generation from chinese clinical dia- logue with multi-granuue granularity transformer. IEEE ACM Trans. Audio Speech Lang. Process., 29:3362-3374
This paper aims to extract predefined medical items and status from doctor-patient dialogues. One ma- jor limitation is that our approach needs pre-defined medical items and status, this is also the common limitation for all DMIE approaches. Currently the pre-defined items and starus cover common dis- eases and symptoms, when applying to relatively rare diseases, new medical items and status labels need to be introduced, and new training data needs to be labeled.
Xinzhu Lin, Xiahui He, Qin Chen, Huaixiao Tou, Zhongyu Wei, and Ting Chen. 2019. Enhancing dialogue symptom diagnosis with global attention and symptom graph. In EMNIP//JCNI.P (1), pages 0032-5041.
Qianwen Ma, Chunyuan Yuan, Wei Zhou, and Songlin Hu. 2021. Label-specific dual graph neural network for lalti-label text classification. ACL/JJNLP 2021, pages 3855-3864.

---

B Ethics Statement
this paper, we target the Dialogue Medical In- formation Extraction task. Although the proposed model achieves statc-of-the-art results, it aims to assist doctors in online diagnosis and the construc- tion of Electronic Medical Records (EMRs) rather replace the doctors. Our model can relieve the doctors from the tedious, consucinn and er- ror prone tasks of labelling relevant medical items and status, but inexperienced doctor may be over- reliant on it. Therefore, is necessary to introduce additional quality controls to avoid the abuse of our model.


Reviews:
[{"review_id": "U3wCdBH8QC", "reviewer": "Reviewer_4JHj", "summary": "", "questions": "Is there any previous work that uses a status-aware item refining module for the DMIE task? Is this the FIRST effort to deviate from formulating DMIE as multi-label classification problems? \n\nIs there a specific reason for not including research on graph convolution in the related work section?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 4JHj for the valuable comments. Please kindly find the point-to-point responses below.\n\n**Q1**: Is there any previous work that uses a status-aware item refining module for the DMIE task? Is this the FIRST effort to deviate from formulating DMIE as multi-label classification problems?\n\n\n**A1**: We have reviewed relevant papers on the DMIE task again and found few method that uses a status-aware item refining module to enhance doctor-patient dialogue representation. For those DMIE related papers, we found that the \"A Speaker Aware Co Attention Framework for Medical Dialogue Information Extraction\" also introduces an item representation module, however it does not fully model the internal connections among items and the interaction between items and status.\n\nThe existing works mainly formulate DMIE as a classification task, using only dialogue information (speakers and utterances). While our proposed method explicitly models the relationship between dialogue, item, and status, and fully utilizes prior knowledge of the co-occurrence relationship between items, resulting in improved performance over existing works.\n\n\n**Q2**: Is there a specific reason for not including research on graph convolution in the related work section?\n\n**A2**: Thanks for your great advices. We will add more discussion on graph convolution models in the related work section. In this paper, we use the graph convolution model to mine the relations among items which in turn improves the performance of the DMIE task. The graph convolution model itself is not major focus of this paper.\n\nFollowing your great advices, in addition to the graph attention network [1] used in this paper, we conduct experiments on three recent graph works[2][3][4]. From the table below, we can find that equipping with any of these four graph models, the proposed method consistently outperforms SOTA works. \n\n\n|     **Model**     | **Macro Metric**| | | **Micro Metric**| | |\n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n|        | Precision| Recall | F1 | Precision| Recall | F1 |\n| CorNet       | 73.07 |71.43 |71.00| 67.72| 63.72| 65.66|\n| MIE        | 69.59| 65.89| 66.05| 69.43| 61.77| 65.38|\n| IIHGC with [1]        | 82.77 | 83.60 | 82.54 | 81.35 | 81.89 | 81.62 |\n| IIHGC with [2]        | 82.56 | 83.54 | 82.35 | 81.43| 81.40 | 81.41 |\n| IIHGC with [3]        | 82.42| 82.65 | 81.74 | 81.72| 81.24 | 81.48 |\n| IIHGC with [4]        | 81.15| 83.76 | 81.67 | 81.15| 81.69 | 81.41 |\n\n\n1. Graph attention networks, ICLR 2018.\n2. Principal Neighbourhood Aggregation for Graph Nets, NeurIPS 2020.\n3. Simple and Deep Graph Convolutional Networks, ICML 2020\n4. Training Graph Neural Networks with 1000 Layers, ICML 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. One of the notable strengths of the paper lies in the robustness of its experimental findings. The results obtained from the MIEACL benchmark dataset demonstrate a significant superiority of our model over existing state-of-the-art alternatives, thereby achieving an impressive new level of state-of-the-art performance. This outcome underlines the effectiveness and potential real-world applicability of the proposed model.\n\n2. The paper is well-written and provides detailed explanations of the model, making it easy to follow and understand.", "reasons_to_reject": "1. The authors assert the introduction of a novel model for the DMIE task, emphasizing interactions among medical items and the incorporation of doctor-patient dialogue information into item representations, along with an exploration of item-status matching relationships. However, I find the contributions to be relatively weak. While the use of a heterogeneous graph and attention-based modules holds potential, it lacks a strong selling point that sets it apart from similar approaches. Additionally, it's worth noting that this isn't the first instance of attempting the DMIE task. The paper could benefit from further clarifying how their approach distinguishes itself from existing methods and addressing the lack of novelty in certain aspects of the proposal.\n\n2. Limited evaluation of recent graph methods: The paper does not include a comprehensive evaluation of recent graph methods, which may limit the understanding of the proposed model's performance relative to the state-of-the-art. Moreover, comparisons with only seven baselines can raise concerns. \n\n3. Insufficient investigation of related research and inadequate referencing.", "questions_to_authors": "Is there any previous work that uses a status-aware item refining module for the DMIE task? Is this the FIRST effort to deviate from formulating DMIE as multi-label classification problems? \n\nIs there a specific reason for not including research on graph convolution in the related work section?", "paper_topic_and_main_contributions": "Paper Topic:\nThe paper focuses on extracting and structuring rich medical knowledge from multi-turn doctor-patient dialogues. This knowledge includes patient symptoms, doctor diagnoses, and prescribed medications. If effectively mined and represented, this medical knowledge can greatly benefit various clinical applications, such as diagnosis assistance and medication recommendations.\n\n\nMain Contributions:\n1. Extraction of Medical Information from Multi-turn Dialogues: One of the main contributions of the paper is addressing the task of \"Dialogue Medical Information Extraction (DMIE)\" from multi-turn doctor-patient dialogues. This task aims to detect predefined medical items (e.g., symptoms, surgeries) and their corresponding statuses (positive, negative) in the dialogues.\n\n2. Heterogeneous Graph Modeling and Attention-based Modules: Unlike conventional approaches, the paper proposes the use of a heterogeneous graph to model relationships between medical items and statuses. Additionally, the paper introduces two consecutive attention-based modules that enrich the representation of medical items by incorporating both the dialogue context and status information. This approach allows for modeling the relationships among medical items and statuses in the DMIE task.\n\n3. Performance Enhancement and State-of-the-Art Results: Experimental results demonstrate that the proposed model outperforms previous methods and achieves state-of-the-art performance. Particularly, the model shows superior performance on a public benchmark dataset compared to prior works.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LKcZ7QeKG0", "reviewer": "Reviewer_DNoq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer DNoq for the valuable comments. Please kindly find the point-to-point responses below.\n\n\n**Q1**: Comparison with newer large pretrained language models would be nice\n\n**A1**: Following your great advice, we conduct experiments on gpt3.5-turbo and gpt4. We design a prompt for the MIE task and use one-shot learning setting. The experimental results and the prompt can be found below. \n\n\n|     **Model**     | **Macro Metric**| | | **Micro Metric**| | |\n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n|        | Precision| Recall | F1 | Precision| Recall | F1 |\n|gpt3.5-turbo\t|46.14|\t40.46|\t43.11|\t39.37|\t29.96\t|34.03|\n|gpt4|\t54.92|\t62.18|\t58.33|\t44.36\t|56.72|\t49.78|\n| IIHGC    | 82.77 | 83.60 | 82.54 | 81.35 | 81.89 | 81.62 |\n\nWe can find that gpt4 outperformed gpt3.5-turbo, but both underperform SOTA works and the proposed IIHGC by a large margin. This is may be due to for this particular MIE task, the supervised models are more suitable than the general LLM approaches.\n\nBelow is the template of prompt (one-shot setting):\n\n```\ntemplate = \"\"\" \n# Predefined Information\n## Symptoms\n- Difficulty in moving\n- Tremors\n- Palpitations\n- Back pain\n- Dizziness\n- Burping\n- Abdominal discomfort\n- High blood pressure\n- High blood sugar\n- Difficulty breathing\n- Chest tightness\n- High blood lipids\n- Nausea\n- Vomiting\n- Chest pain\n- Fatigue\n- Sweating\n- Fever\n- Shock\n- Fainting\n- Cold\n- Cough\n- Runny nose\n- Headache\n- Stomach discomfort\n- Stiffness\n- Cyanosis\n- Diabetes\n- Anemia\n- Edema\n- Angina\n- Hyperthyroidism\n- Premature beat\n- Irregular heartbeat\n- Atrial septal defect\n- Atrial fibrillation\n- Heart failure\n- Myocardial infarction\n- Congenital heart disease\n- Myocardial ischemia\n- Ventricular septal defect\n- Myocarditis\n- Coronary heart disease\n- Cardiomyopathy\n- Cardiac hypertrophy\n## Tests\n- Electrocardiogram\n- Color ultrasound\n- Cardiac enzymes\n- Physical examination\n- Angiography\n- Ultrasound\n- CT\n- Complete blood count\n- Thyroid function\n- Chest X-ray\n- B-ultrasound\n- Kidney function\n- Treadmill\n- CTA\n- Blood pressure measurement\n- MRI\n## Surgery\n- Intervention\n- Radiofrequency ablation\n- Bypass\n- Stent\n## General Information\n- Sleep\n- Diet\n- Mental state\n- Bowel movements\n- Smoking\n- Drinking\n\nInstruction：Given a dialogue between a patient and a doctor, based on the predefined information above, extract the predefined \"symptoms\", \"tests\", \"surgery\", or other \"general information\" mentioned in the dialogue, and judge their \"status\". The definition and description of the status are as follows:\n- Doctor Positive: The doctor determines that the patient has a symptom, the situation corresponding to the general information exists, and a test or surgery is needed\n- Doctor Negative: The doctor determines that the patient does not have a symptom, the situation corresponding to the general information does not exist, and no test or surgery is needed\n- Patient Positive: The patient indicates that they have a symptom, the situation corresponding to the general information exists, and they have undergone a certain test or had a certain surgery\n- Patient Negative: The patient indicates that they do not have a symptom, the situation corresponding to the general information does not exist, and they have not undergone a certain test or had a certain surgery\n- Unknown: It is mentioned in the dialogue, but it is not known which specific status it corresponds to\n\nExample:\n# Patient-Doctor Dialogue\nDoctor: Are you currently taking any oral medications for treatment? How long have you noticed this? How is your sleep?\nPatient: My sleep is very good. Discovered 2 years ago. It was found during a physical examination at the age of 19, more than 3 years ago\nDoctor: Have you ever been treated?\nPatient: This is last year's electrocardiogram, the doctor prescribed some medicine. I have been taking Danshen tablets. Sometimes I eat,\nDoctor: Your arrhythmia does not have much impact on fertility, it is recommended that you avoid fatigue and emotional excitement. You can take some Wenxin granules to improve symptoms. In addition, it is recommended that you check the electrocardiogram and cardiac enzymes when you are not feeling well.\n# Extraction results and corresponding status (only based on predefined information)\n- Test: Electrocardiogram-Status: Doctor Positive\n- Test: Cardiac enzymes-Status: Doctor Positive\n- Test: Electrocardiogram-Status: Patient Positive\n- Test: Physical examination-Status: Patient Positive\n- Symptom: Arrhythmia-Status: Unknown\n- General Information: Sleep-Status: Patient Positive\n- Symptom: Fatigue-Status: Unknown\n\n# Patient-Doctor Dialogue\n{}\n# Extraction results and corresponding status (only based on predefined information)\n\"\"\"\n```\n\n\n**Q2**:  Is the cited Zhang 2020 paper table 4 score of 66.40 F1 supposed to be correlate somewhere with this papers’ Table 3 MIE Micro F1 65.38?\n\n**A2**:  In Zhang 2020 paper table 4, the score of 66.40 F1 is the MIE-Marco metric which corresponds to 66.05 in Table 2 of this paper. Since we need to calculate the Micro metrics in addition to the Marco metrics, for consistency consideration, we re-run the code of Zhang 2020 on the benchmark data set instead of directly citing the numbers reported in Zhang 2020's paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is an important problem\n- Meaningful improvement over baselines", "reasons_to_reject": "- Comparison with newer large pretrained language models would be nice", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors describe their system for classifying dialogue medical information including one of several overall categories (symptom, surgery, test, other info), more detailed items (e.g. stent implantation), and status (e.g. patient-positive). Their system outperforms several baselines and contains performance analysis related to most frequent categories.\n\nIn general the paper is well written, including relevant descriptions of the problem, related works, baselines, and performance analysis.\n\nThe system description seems like it provides necessary information; this combined with code will allow users to understand the system.\n\nMinor Comments\n-------------------\n4.1 line 413-414 has a repeat reference\n\nLooking at the results of this works’ cited paper (https://aclanthology.org/2020.acl-main.576.pdf) which is also the dataset paper, there was breakdown by both window/dialogue level as well as by category/item (and full). Is the cited Zhang 2020 paper table 4 score of 66.40 F1 supposed to be correlate somewhere with this papers’ Table 3 MIE Micro F1 65.38? (if yes, perhaps put a small footnote on why it’s different would be good)\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QMiMQRZgkR", "reviewer": "Reviewer_hgH8", "summary": "", "questions": "- Table 1: How are disorders or pharmacological substances modeled? Are they included in the \"Other info\" category?\n- L. 530ff: How many random seeds were used to explore the variability of results?\n- L. 437ff (sect. 4.3): more details about the hyperparameters are missing, and also the GPU type or number of training hours. Moreover, given that this is a sophisticated model that needs long pretraining, the authors could have brought out the discussion about its advantages compared to other frameworks for the dialogue task; especially, when large volumes of data are hard to obtain.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer hgH8 for the valuable comments. Please kindly find the point-to-point responses below.\n\n**Q1**: Table 1: How are disorders or pharmacological substances modeled? Are they included in the \"Other info\" category?\n\n**A1**: For the targeted MIE task, as shown in Table 1, four types of items and five kinds of status are introduced. The items as well as the status are predefined. Our task is to determine whether a combination of item and status can be infered from current input dialogue. \"Other info\" is a separate category, which includes sleep, diet, mental state, bowel movements, smoking, and drinking. The complete list of items can be found below.\n\n```\n## Symptoms\n- Difficulty in moving\n- Tremors\n- Palpitations\n- Back pain\n- Dizziness\n- Burping\n- Abdominal discomfort\n- High blood pressure\n- High blood sugar\n- Difficulty breathing\n- Chest tightness\n- High blood lipids\n- Nausea\n- Vomiting\n- Chest pain\n- Fatigue\n- Sweating\n- Fever\n- Shock\n- Fainting\n- Cold\n- Cough\n- Runny nose\n- Headache\n- Stomach discomfort\n- Stiffness\n- Cyanosis\n- Diabetes\n- Anemia\n- Edema\n- Angina\n- Hyperthyroidism\n- Premature beat\n- Irregular heartbeat\n- Atrial septal defect\n- Atrial fibrillation\n- Heart failure\n- Myocardial infarction\n- Congenital heart disease\n- Myocardial ischemia\n- Ventricular septal defect\n- Myocarditis\n- Coronary heart disease\n- Cardiomyopathy\n- Cardiac hypertrophy\n## Tests\n- Electrocardiogram\n- Color ultrasound\n- Cardiac enzymes\n- Physical examination\n- Angiography\n- Ultrasound\n- CT\n- Complete blood count\n- Thyroid function\n- Chest X-ray\n- B-ultrasound\n- Kidney function\n- Treadmill\n- CTA\n- Blood pressure measurement\n- MRI\n## Surgery\n- Intervention\n- Radiofrequency ablation\n- Bypass\n- Stent\n## Others\n- Sleep\n- Diet\n- Mental state\n- Bowel movements\n- Smoking\n- Drinking\n```\n\n\n**Q2**: L. 530ff: How many random seeds were used to explore the variability of results?\n\n**A2**: Five different random seeds were used.\n\n\n\n**Q3**: L. 437ff (sect. 4.3): more details about the hyperparameters are missing, and also the GPU type or number of training hours. Moreover, given that this is a sophisticated model that needs long pretraining, the authors could have brought out the discussion about its advantages compared to other frameworks for the dialogue task; especially, when large volumes of data are hard to obtain.\n\n\n**A3**: We utilize the hfl/chinese-roberta-wwm-ext as the encoder in our model, which is a base version of the pre-trained Chinese RoBERTa-wwm-ext. The ID, IIHGC, and IS have the same dropout rate, attention heads, and hidden layer dimensions as the original RoBERTa model, which are 0.1, 12, and 768 respectively. We employ the AdamW optimizer for training our model, with a learning rate of 2e-5 for the parameters in the RoBERTa encoder and 2e-4 for the rest of the model. We incorporate an early stop mechanism to avoid overfitting, whereby the training is stopped if there is no improvement in the MIE-Macro F1 or MIE-Micro F1 on the validation set over 10 epochs. Running an experiment with these settings will roughly take 9 hours on a single Nvidia V100 GPU.\n\n\n**Q4**: Some technical aspects about how the dialogue representation is obtained are missing. I could not find a explicit statement to explain whether the graph from the IIHGC model is obtained for each 5-turn window, and trained separately, or whether the graph is obtained once from the full DMIE dataset.?\n\n**A4**: The graph is obtained once from the training set of the DMIE dataset. Section 3.1.1 of the paper discusses the steps to construct the graph. Each node v represents an item with its corresponding category, and the graph's edges are inferred from the co-occurrence relationships among items. In detail, if the co-occurrence of two items (within the same dialogue) exceeds four times in the training set, they are considered to be connected by an edge. The type of the edge is determined by the types of two connected nodes.\n\n\n**Q5**: Not much information is provided about how the term variation is processed. Medical term variation/normalization modules could increase the results. Since data from doctor-patient encounters are very scarce, these components could be explored as a complement to neural network frameworks.?\n\n**A5**:  Thanks for your great advice. We totally agree that medical term variation/normalization modules could increase the performance of MIE task. In this paper, we use the item refining module to handle term variations between doctor-patient conversations. We first encode the representation of items using a language model, and introduce the relationship graph between items to further enrich the representation of items. Then, the representations of these items and conversations are fed into a cross attention network to perceive the mention of items. The acquired representations of items may help to provide term variation/normalization functions and allievate the data scarce problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new architecture to extract medical information in dialogue data. Dialogue context and the correlation between medical entities is incorporated. \n- An extensive evaluation including ablation analysis and experiments with variants of dialogue length and label frequencies. I recommend the authors to include conduct a qualitative analysis or an error analysis in future work; this could shed more light to understand the model's results.\n- Overall, I think this contribution is well writen, reports extensive work and could be of interest of the conference audience.\n", "reasons_to_reject": "- Some technical aspects about how the dialogue representation is obtained are missing. I could not find a explicit statement to explain whether the graph from the IIHGC model is obtained for each 5-turn window, and trained separately, or whether the graph is obtained once from the full DMIE dataset.\n- Not much information is provided about how the term variation is processed. Medical term variation/normalization modules could increase the results. Since data from doctor-patient encounters are very scarce, these components could be explored as a complement to neural network frameworks.\n", "questions_to_authors": "- Table 1: How are disorders or pharmacological substances modeled? Are they included in the \"Other info\" category?\n- L. 530ff: How many random seeds were used to explore the variability of results?\n- L. 437ff (sect. 4.3): more details about the hyperparameters are missing, and also the GPU type or number of training hours. Moreover, given that this is a sophisticated model that needs long pretraining, the authors could have brought out the discussion about its advantages compared to other frameworks for the dialogue task; especially, when large volumes of data are hard to obtain.\n", "paper_topic_and_main_contributions": "This work reports some experiments conducted on the DMIE dataset (Zhang et al. 2020) for a medical information extraction task using Chinese doctor-patient dialogues. The authors present a neural-network-based model using the correlation between each medical entity pair (modeled as a relationship between nodes in a graph, IIHGC module); the relation between each dialogue representation and each item representation; and the relation between each item and its status. A representation of each dialogue is obtained by applying a mean pooling technique on the aggregation of these information types. Then a sigmoid function is applied to train and optimize the model parameters. The authors compare their architecture with previous models: MIE (Zhang et al. 2020), LightXML (Jiang et al 2021), CorNet (Xun et al. 2020), Roberta CLS, MGT (Li et al., 2021) and SAFE (Xia et al., 2022). The evaluation metrics were obtained by averaging results in each sliding window of 5 turns. The authors's results outperform those obtained with the other models, and include an ablation analysis. Experiments are also repeated using different frequencies (f) of labels in the dataset (f>90, 20<f<90, 1<f<20) and with longer (>300 turns) or shorter dialogues (<300 turns).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- I did not find.", "Typos_Grammar_Style_and_Presentation_Improvements": "- L. 414: please, at the beginning of the sentence, change \"(Zhang et al., 2020)\" to \"Zhang et al. (2020)\"; use Latex \"citet{}\"\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the idea and the results shared in this paper but had mixed excitement scores. I believe that the suggested edits would be helpful to improve the current version of the paper. The authors are encouraged to: 

- include the results added during the rebuttal and add a comprehensive analysis,

- better situate their contribution and correct the references as suggested by R1,

- explain how term variations are handled (see comment by R3 after the rebuttal),

- The limitations are the ethics sections should also be moved to the main text.|This work proposed improved methods for the dialogue medical information extraction (DMIE) task, resulting in non-negligible improvements over the current best-performing systems on the corresponding dataset. Specifically, they develop a heterogenous graph modeling procedure to account for interactions between the different extraction types and attention models to augment the input representation to better include dialogue context and relationships between medical terms in the dialogue. Additional ablation studies are performed to elucidate additional relevant issues and add to understanding of the dynamics of the proposed method.

== Quality == The strongest element of the paper is the strength of the primary empirical results. The proposed method makes intuitive sense and is shown to perform well on the DMIE datasets, pushing the SotA forward.  Additionally, the paper is well-structured and easy to understand. Both reviewer 4JHj and DNoq suggest additional experiments that were provided during the rebuttal period. I would recommend including some of these results in the camera ready version (even if in an appendix if necessary) as they further support the efficacy of the proposed method.

== Clarity == Overall, the paper is well-motivated, well-structured, and easy to understand. While it would benefit from general polishing of the writing, it is well-written overall, but i recommend adding the additional results, references, and discussion as recommended by the reviewers.

== Originality == The authors claims methodological innovation, but it is really a case of applying more recent methodologies to this particular application. While this is still important (and they got it to work), they are not describing new methods that are easily adaptable to other problems, etc. This, it is a targeted innovation.

== Significance == Medical information extraction is an important and potentially impactful problem. From an application perspective, the proposed method achieves strong empirical results, pushing the SotA forward and thus will minimally be used as a baseline for future work. However, it is a domain-specific improvement using existing methods and doesn't make algorithmic advances that would be useful to the broader community.
Decision: Accept-Findings