Towards a Reliable and Robust Dialogue System for Medical Automatic Diagnosis
Authors: Junfan Lin
Lin Xu
Ziliang Chen
Liang Lin
Venue:  (2021)

Abstract: Dialogue system for medical automatic diagnosis (DSMAD) aims to learn an agent that mimics the behavior of a human doctor, i.e. inquiring symptoms and informing diseases. Since DSMAD has been formulated as a Markov decision-making process, many studies apply reinforcement learning methods to solve it. Unfortunately, existing works solely rely on simple diagnostic accuracy to justify the effectiveness of their DSMAD agents while ignoring the medical rationality of the inquiring process. From the perspective of medical application, it's critical to develop an agent that is able to produce reliable and convincing diagnosing processes and also is robust in making diagnosis facing noisy interaction with patients. To this end, we propose a novel DSMAD agent, INS-DS (Introspective Diagnosis System) comprising of two separate yet cooperative modules, i.e., an inquiry module for proposing symptom-inquiries and an introspective module for deciding when to inform a disease. INS-DS is inspired by the introspective decision-making process of human, where the inquiry module first proposes the most valuable symptom inquiry, and then the introspective module intervenes the potential responses of this inquiry and decides to inquire only if the diagnoses of these interventions vary.

Under review as a conference paper at ICLR 2021
TOWARDS A RELIABLE AND ROBUST DIALOGUE SYs- TEM FOR MEDICAL AUTOMATIC DIAGNOSIS
Anonymous authors Paper under double-blind review
ABSTRACT
Dialogue system for medical automatic diagnosis (DSMAD) aims learn an agent that mimics the behavior of a human doctor, i.e. inquiring symptoms and informing diseases. Since DSMAD has been formulated as a Markov decision- making process, many studies apply reinforcement learning methods to solve it. Unfortunately, existing works solely rely on simple diagnostic accuracy to justify the effectiveness of their DSMAD agents while ignoring the medical rationality the inquiring process. From the perspective of medical application, it's criti- cal to develop agent that is able produce reliable and convincing diagnosing processes and also is robust in making diagnosis facing noisy interaction with patients. To this end, we propose novel DSMAD agent, INS-DS (Introspec- tive Diagnosis System) comprising of two separate yet cooperative modules, i.e., inquiry module for proposing symptom-inquiries and an introspective module for deciding when to inform a disease. INS-DS is inspired by the introspective decision-makee process of human, where the inquiry module first proposes the most valuable symptom inquiry, then the introspective module intervenes the po- tential responses of this inquiry and decides to inquire only if the diagnoses of these interventions vary. We also propose two evaluation metrics validate the reliability and robustness of DSMAD methods. Extensive experimental results demonstrate that INS-DS achieves the new state-of-the-art under various exper- imental settings and possesses the advantages of reliability and robustness com- pared to other methods.
INTRODUCTION
Dialogue system for medical automatic diagnosis (DSMAD) aims to learn an agent to collect pa- tient's information and make preliminary diagnosis in interactive manner like a human doctor. This task increasingly grasps the attention of researchers because of its huge industrial poten- tial (Tang al. 2016). Similar to other task-oriented dialogue tasks (Lipton et al. 2018 Wen al.) Yan et al. 2017 Lowe et al. 2015). DSMAD is composed of a sequence of dialogue-based inter- actions between the patient and the agent, which can be formulated as a Markov decision process and resolved by reinforcement learning (RL) (Mnih al. 2015) Van Hasselt et al., 2016). Although several frameworks have been proposed (Xu et al. 2019: Wei et al. 2018 Peng et al. 2018 Tang et al. 2016). DSMAD is still far from being applicable. because these works only evaluate the agent based on the accuracy of diagnosis, but ignoring the importance of robustness and reliability for practical medical applications. The two major shortcomings of the current DSMAD methods summarized below.
Unreliable symptom-inquiry and disease-diagnosis.
reasonable to measure DSMAD by di- agnosis accuracy since the accuracy is the ultimate goal of the task. However, in unilateral pursuit of high accuracy, DSMAD agent pays less attention to the rationale of the diagnosis process, reducing the trust of users. For example, DSMAD agent might jump into conclusion without inquiring about any symptom. As long as the diagnosis is correct, such an agent will still get positive reward. In this sense, the correctness of diagnoses not sufficient to reflect the performance of DSMAD, and might lead the agent to make a hasty diagnosis without interactions. Moreover, DSMAD should learn to make consistent disease-diagnóse according the symptom-disease relation in the training data, insensitive to the noise happened during training.
Sensitive to small disturbance. Almost all of the current DSMAD methods combine the operation of symptom-inquiry and disease-diagnosis together and allow models to make the sequential deci-

---

Under review as a conference paper at ICLR 2021
sion in black-box manner (hang & Zhu 2018 Koh & Liang 2017) without regulations, resulting in a system vulnerable to the noise during the interaction process. If we place one of the inquired
symptoms to the self-report (to ensure the information is consistent between two cases), the agent sensitive to noise would make different diagnoses.
To this end, we propose a novel DSMAD agent, Introspective Diagnosis System (INS-DS) (Fig. D and two new evaluation metries in terms of reliability and robustness. The diagnosis logic of INS-DS draws on the introspective pecision-making process of human doctors. In real life, human doctors come into conclusion if they believe that more inquiries make difference. In INS-DS, the inquiry module is responsible for selecting the most valuable symptom to be inquired about, while the introspective module intervenes the potential answers of this inquiry to decide whether to inquire the symptom or inform the disease. Specifically, the introspection module assigns different possible answers to the inquiry resulting in multiple one-step-look-abee whether the diagnosis results of these states are going to be varied. If the predicted results are the same, which means that inquiring the most valuable symptom inquiry would result in the same diagnosis, INS-DS will inform the disease instead. Otherwise, the agent would inquire about the symptom. Such mandatory introspection makes the inquiry more disease-related because the agent is not allowed to make diagnosis until the agent has collected sufficient symptom information. It also makes the disease-diagnosis more consistent because the disease can only be informed when the comprehensive hypothesis test is passed.
dialogue states. Then it inspects
To quantify the reliability and robustness of DSMAD, we also propose two novel evaluation metrics, namely, reliability and robustness.
Reliability. The purpose of the reliability metric is quantify how confident the diagnosis is from the perspective of the model (internal trust or Int.) and user (external trust or Ext.). The internal trust to testify whether the diagnoses made by the model is insensitive to the task-irrelevant factors, e.g., the sampling noise and parameter initialization. Specifically, for Int., we adopt the expected diagnostic probability of a set of bootstrapping models. These bootstrapping models are initialized with different parameters and trained on re-sampled data with replacement, to reduce the effect of the parameter initialization and the data sampling. Therefore, the higher Int. is, the less sensitive the diagnosis result is to the noise in the training process. The external trust is proposed to indicate the trust degree of a diagnosing process to users. Intuitively, patients are more likely to believe in the diagnosis made from the agents who request symptoms like human doctors. According this intuition, for Ext., we compute the symptoms overlap ratio based on the co-occurrence between symptoms and diseases in the diagnostic dialogue dataset. The higher the level of Ext., the more likely the agent is to inquire symptoms like a human doctor.
Robustness. As for robustness, we draw on the inspiration from the noted adversarial attack (Ku- rakin et al. 2018) in machine learning models, which generally uses samples with subtle modifica- tion that is indistinguishable for human but may be different for a machine to prove the vulnerability of a model. Our evaluation metric for the robustness is the final unaltered proportion of correct diagnoses after feeding the model with the attack samples constructed according to the formulas in Sec. 5
The extensive experimental results evidence that INS-DS achieves the superior performance com- pared to other DSMAD baselines under various settings and possesses the advantages of reliability and robustness. We also conduct human evaluations our INS-DS and achieve significant improve- ment in the aspects of diagnosis validity, symptom rationality as well as topic transition smoothness.
RELATED WORK
The task-oriented dialogue system is designed to accomplish specific tasks, like the ticket, restaurant booking, online shopping etc. (Lipton et al.) 2018 Wen et al.) Yan et al. (017)) Most of the current lask-oriented dialogue systems adopt the framework of reinforcement learning (Mnih et al.) 2015
Lipton et al. 2018 Li et al. 2017). and some adopt the sequence-to-sequene style for dialogue
generation Madooto et al. 2018 Wu et al. 2019
Lei et al.) 2018).
For medical dialogue system, due to a large number of symptoms, reinforcement learning is a better
Kao al. 2018 Peng et al., 2018). [Tang et al. (2016) apply Deep Q-Network (DQN) (Mnih et al. 2015) to diagnose using synthetic data. While Weil al. (2018) first did experiments on real-world data using DON. To include explicit medical induc- tive bias for improving the diagnostic performance, Xu et al. (2019) proposed an end-to-end model
choice for topic selection (Tang et al. 2016
2

---

Under review as a conference paper at ICLR 2021
guided by a symptom-dissee dissese knowledge graph. KR-DQN applies the predefined conditional prob- ability of symptom and disease to transform the Q-values estimated by DQN. However, it's often difficult to get a knowledge graph in real life. Moreover, most of these methods integrate symptom- inquiry and disease-diagnosis actions into one single reinforced policy network without considering the essential difference between sympton-inquiry and dissase-diagnosie allowing the agent to jump into conclusions rashly to avoid the possible penalty in the diagnosing process. Different from these methods, our approach divides the actions for symplom-inquiry and disease-diagnosis into two sep- arate but cooperative neural modules. The idea of drawing the lesson from human is not new. Ling al, (2017) improve the performance of diagnosis for clinical documents by replicating the memory recall and attention mechanism in the human decision-making process. They selected evidence from the external clinical source according to the hint sentences the given clinical document and then applying reinforcement learning to filter out the useless evidence. Different from DSMAD, instead of interacting with the patient, their RL agent is used to decide whether the matching evidence is useful not. And similar (Xu et al. 2019) and (Wei et al. 20188), Ling et al. (2017) also rely on the agent to choose to stop but not through an introspective stop mechanism as ours.
As for evaluation, aware of that only final diagnosis accuracy is not sufficient, current methods measure the performance of the diagnosing process by computing the hitting rate of the inquired symptoms. Lei et al. (2018); MMdotto et al. (2018); Wu et al. (2019) compute the entity Fl scores. Xu al. (2019) use the matching rate compared all asked symptoms while Tang et al. (2016): Kao et al. (2018); Peng et al. 2018 measure by the average number of queried positive symptoms of each user goal. However, the recorded symptoms in user goal are generally a small subset of the actual disease-relevant symptoms in the whole dataset. Therefore, measuring DSMAD according to the recorded symptoms in each user goal could not reflect the actual performance of DSMAI) in general. Different from them, one of our proposed metrics in this paper, namely, external trust, measures the matching rate according to the symptom-disease co-occurrence of the whole dataset. Besides, we also propose the metrics for evaluating the model's confidence in the decision and the robustness to the noisy input.
3 PRELIMINARY
3.1 DIALOGUE SYSTEM FOR MEDICAL AUTOMATIC DIAGNOSIS
Medical automatic diagnosis is to interact with the users, actively collect useful information, and conduct preliminary diagnosis according to the collected information. In this task, user goal in the dataset includes a self-report (e.g.. "Doctor, my child has a headache. What happened to him?"), a set of explicit symptoms, set of implicit symptoms and ground-truth disease. Usually, a patient simulator is built upon user goal to create an interactive environment. Given a user goal, the patient simulator starts the dialogue with the self-report, and then answers the inquiries from the DSMAD agent according to the recorded symptom information in the user goal. DSMAD agent chooses to inquire the next symptom or inform the disease according the dialogue state in each turn. When the agent inquires a symptom, it would get meaningful feedback if the symptom is presented in the explicit symptoms or implicit symptoms of the user goal. And if the agent chooses inform a disease, the discourse will be ended and the agent will receive feedback on the ground-trun disease. In this paper, we adopt the same patient simulator used in (Xu et al., 2019).
Generally, DSMAD consists of three elementary components: 1) natural language understand- ing (NLU), which is used to parse the language input of the patient to fill the semantic symptom slots; 2) dialogue management (DM), which used to track the dialogue history and predict next dialogue state according to the gathered information; 3) natural language generation (NLG), which used to generate natural language for either symptom inquiry or disease diagnosis. Different from works (Xu et al. 2019. Wei et al. 2018), this paper focuses on the decision-making part in DM, that's, using the reinforced agent to either inquire symptom or inform disease. The NL. and NLG components used in this paper are the same as (Xu et al., 2019).
3.2 NOTATION
We design our agent in deep reinforcement learning framework (Mnih et al. 2015). The medical au- tomatic diagnosis can be formulated as finite-horizon Markov decision process (MDP), defined by the tuple (S, .A p, r), where the state space S and the action space A are discrete. Patient commu- nicates with the dialogue system by request (inform the self-report) or confirm/deny/not- symptom. There are three types of symptom states, positive, negative and not-sure, represented by value 1, 1 and 0 respectively. We also provide binary indicator for each symptom to indicate
3

---

Under review as a conference paper at ICLR 2021
State tracker
Max value
Symptom-inquiry Module
Policy network
Want symptom
Intervention
Introspection Module
Biut with sym 11
Diseuse with sym i-
Request symptom
Are those same disesse? Inform diseese
Sias with sym. D
Classifier
Disease with sym. 1=
Reward
Fix1 sym.
Discase sym.
INS-DS with sympiom- cimplom-inquiry and introspection module. In the vymptom-inqury module, the current dialogue state 8, is fed to the policy network choose candidate symptom Then, in the introspective module, 8, further intervened by setting the value symptom with all possible values generate multiple one-step-look-alee 5/41 s. These 8441 8 are forwarded to the discase classifier to predict the discases. The introspection module will choose to inform the discase if all predicted discases are the same, otherwise, the agent will continue inquire about the symptom
Figure 1:
whether symptom has been mentioned. At each turn, the dialogue state E S is composed of the value and the binary indicator of each symptom. The state tracker DM records mentioned symp- toms in the dialogue and produces the up-to-date dialogue state at time t. And the action space consists of the actions of symptom-inquiries and disease-diagnoses. Without causing conflict, M denotes the number of the diseases and N denotes the number of the symptoms. Therefore the size of the action space is N + +.. The unknown state transition probability p S A [0, oc) rep- resents the probability of the next state $1+1 S given the current state 8, S and action at E A. The patient simulator emits a bounded reward r S x A [Tmin, mmax] on each transition.
The target of reinforcement learning to maximize the expected accumulated rewards BI where is the discount factor to adjust the horizon of the foresight. In this pa- per, we adopt the classic q-learning to optimize the policy with parameter 0, via minimizing the temporal-difee :enpeeeediifferenc::
min E (s ( wl(Q(ss 0): 7t max Q(x4(($41.
4 INTROSPECTIVE DIAGNOSIS SYSTEM
Generally, disease-dinn diagnosss is the most critical step because only the correct diagnosis result could make the patient get the correct treatment. With sense of security and responsibility, doctors in- trospect their decisions by imagining all the potential outcomes (Rubin) 1974) before making the final diagnoses. Inspired by this diagnosing process, propose introspective diagnosis sys- tem (INS-DS), which includes a symptom-inquiry module to select a symptom to be inquired about, and an introspection module determine whether inquire about that symptom or inform the predicted disease. Different from the most popular diagnosis systems which treat symptom-inquij and disease- dinagnoss equally (Xu et al. 2019 Wei al. 2018), our method only allows the disease- diagnosis to be made after considering all possible future outcomes in order to make the decision more reliable and interpretable. Also, INS-DS requires more correlation between the symptom- inquiry and the disease-diagnosis In INS-DS, only when the candidate symptom can trigger differ- diagnosis results will the symptom be inquired about. And only when the most valuable inquiry proposed by the symptom-inquit module makes difference to the future diagnosis will the pre- dicted disease be informed. INS-DS pipeline is illustrated in Fig. I Next, we elaborate the details of the two modules.
Symptom-inquiry module. At time f, the state tracker gathers historical symptom information to generate the current dialogue state 8t. Then 8; is input to the symptom-inquiry module, which is responsible for selecting the most valuable symptom inquiry. As mentioned previously, 8, consists of two vectors, one for the presences of symptoms and another for the visitation indicators. The two vectors are fed into two different Multi-Layered Perceptrons (MLPs) with parameters 6, whose outputs are fused by Hadamard multiplication to produce the final Q values of all possible inquiries, i.e. qe E R.N. Both ML.ss consist of two fully-connected layers. And only the symptom inquiry
4

---

Under review as a conference paper at ICLR 2021
with the maximum Q value will be selected as candidate action and then be forwarded to the introspection module.
Introspection module. The introspection module would examine the necessity of the symptom inquiry by predicting potential diseases caused by all possible answers of the symptom inquiry. If the potential diseases are different, meaning that the symptom will cause differences in determining the discase, the agent would choose to request the symptom. Otherwise, the agent will inform the disease. Specifically, after receiving the most valuable symptom i from the symptom-inquin module, the introspection module intervenes all possible outcomes if the inquiry were made. This process is named as Intervention demonstrated in Fig. [ which is an important ability for our Homo sapiens ancestors achieve global dominion (Pearl 2018). To do so, the intervention function replaces the symptom value of symptom in 8; with all possible values (i.e., -1, O and 1) and sets the visitation indicator of the symptom 1, produce several imaginary states $/-1 S (with the same size as the number of the possible answers to the inquiry). These one-step-look-abeee states are then forwarded disease classifier produce their corresponding diagnoses. The classifier is an MLP with three fully-connected layers.
Training objective. In the introspection module, we adopt a classifier to infer the diseases of poten- tial outcomes of intervention. The classifier is trained with input-label pairs (st, dr )ss for E [0, T] collected after each training episode with length ' using crossyentrpp.. In order to optimize the inquiry policy, we employ Deep Q-learning following (Mnih et al., 2015). and two important DQN tricks, i.e. target network Q' with parameter 6' and experience relay are adopted (Van Hasselt et al., 2016). In order to combine the introspection with the policy optimizing, the objective of INS-DS is derived from Equ.
We use (s2, og; 0) to denote the expected discounted sum of rewards, after taking an action az under state St- Different from traditional temporal difference whose proposed action the actual executed action, our policy with introspection is updated according to
min (s (Q((s,, az; 6) - r(st. Introspection(s., a+)) - max Q' (8)+ (2)
where 0' is the parameters of the target network updated by 0' a8 + a)0'' where is the polyak factor. Introspection(s, a) denotes the process in the introspection module. We use e-greedy exploration at training phase for exploration, selecting a random action according to probability e. We store the agents experienced transition at each time-step to a experience replay buffer.
EVALUATION CRITERIA
Most diagnosis systems use accuracy to evaluate performance (Wei et al. 2018 Xu et al. 2019) However, we argue that only accuracy is insufficient to evaluate a diagnosis system as medical application. In particular, two major shortcomings observed in prior works (Wei et al. 2018 Xu al. 2019) are the lack of reliability and robustness. To this end, we propose two novel metrics, namely, reliability and robustness, complements to accuracy.
5.1 RELIABILITY
In real life, prudent doctors tend to be cautious when faced with unfamiliar cases while unpro- fessional doctors may ask for some irrelevant symptoms that cause patients' distrust. To quantify whether a DSMAD agent is reliable, we introduce two new evaluation criteria for reliability, namely, Internal Trust (Int.) and External Trust (Ext.).
Internal trust. In order to model the internal trust of a diagnosis system, we adopt the bootstrapping method to produce multiple diagnosis results from 100 models, (k), k E [1, 100). We employ random sampling strategy with replacement to generate training set for each model. The models are initialized with different random parameters. By considering the noises of data sampling and parameter initialization, the expected diagnosis prediction of the bootstrapping models ought to be less sensitive to these noises. We calculate the expectation of the diagnoses from these models by:
D Σ
1
Print. (dlp) = Pr (mmmto)(dd))) = E Emmii) [m/ = d), and Int.(m) =
Print (m(p:)lpe).
D
(3)
where p denotes patient and denotes a disease, m is the target model and D is the size of test patients.
5

---

Under review as a conference paper at ICLR 2021
External trust. As for modeling the external trust, calculate the coverage ratio of the men- tioned symptoms during the diagnosing process. Specifically, we first calculate the symptom-disean co-occurrence Prext. (Symptoml d) from the whole dataset. Then calculate the external trust of each dialogue using the average probability of the mentioned symptoms corresponding to the ground-truth disease. Denote the set of symptoms inquired by the target model when interacting with the patient p; as {Symp.j.j j E [1, Hp.l] where Hp, is the number of the inquiries. Formally, the final external trust of the target model m is the averaged external trust of all test dialogues as
D
Hp.
1
FxL(m) = D Hp.
Prest. (Symp.,yy
(4)
where D is the size of the test patients and dz is the ground-truth disease of patient i. In order to reach higher external trust, the system is required inquire disease-relevan symptoms which are frequently inquired about in real life.
5.2 ROBUSTNESS
Robustness refers to the ability of a system to stay functioning correctly in the presence of noisy inputs or stressful environmental conditions. Enlightened by the adversarial attack theory of current machine learning models (Kurakin et al., 2018), in the context of the medical dialogue system, we consider the robustness of model measured by the invariance under the noisy inputs which are designed to have little impact on the final diagnosis. Owing to the particularity of the medical diag- nosis, that is, subtle changes in symptoms might lead to different diagnoses (for example, reversing the presence of a symptom might associate the patient with a complete distinct disease), we design two relatively innocuous patterns of noises symptoms in a user goal: 1) Given the symptom- disease relationship, we move O ~ k symptoms with the weakest correlation to the ground-truth disease from explicit symptoms to the implicit symptoms: 2) Add 1 ~ l symptoms to the user goals that are mostly correlated to the ground-truth disease. To quantify the robustness of the model m, we calculate the unaltered ratio of the correct diagnoses given the noisy test set P', formally:
Rub.(m; P') = Ep (m)(') = m = =d),
(5)
where p is the original data of the noisy data p', and d² is the ground-truth disease. The robustness is calculated on the patients whose original diagnoses are correct (i.e., m(p) = d'). This is because the misdiagnoses made by robust system could be attributed the inadequate symptom information from the patients. Therefore, a robust system is reasonable to change its diagnoses with more or less symptom information from these patients.
6 EXPERIMENTS
We begin by introducing the two open-released DSMAD benchmarks used in this paper and the baseline approaches. Then, we compare different methods w.r.t. diagnosis accuracy as well as our proposed metrics. Moreover, we have conducted a human evaluation of different approaches to better demonstrate the superiority of our method from the human expert perspective.
Benchmarks. In this paper, all experiments are conducted using the two open-released medical diagnosis datasets, i.e., MuZhi (MZ) and DingXiang (DX). MZ is a synthesized dataset proposed by Wei et al.) (2018), which includes 586 training and 142 test records with 66 symptoms and 4 diseases; DX in Xu et al.] (2019) consists of 423 training and 104 test records with 41 symptoms and 5 diseased] Records in MZ. are synthesized so that its records are clean and structural. Different from MZ. records in DX Xu et al. 2019) is collected in real life, so that the raw self-report corpus are maintained in these records. In both datasets, the diagnosis record contains the ground- truth disease, the explicit symptoms (i.c. the symptom information in self-report), the implicit symptoms (i.c. the symptom information mentioned during the discourse).
Baselines. In this section, we focus on evaluating and analyzing the performance of current DSMAD baselines, i.e., Basic DQN (Wei et al., 2018), KR-DS (Xu al. 2019), including our proposed INS- DS. Basic DQN makes the first attempt to apply the classical reinforcement learning framework, deep n-newwork (Miih et al. 2015), to dialogue-based medical automatic diagnosis system. Based
'The diseases in MZ are infantile bronchitis (Bronch.), upper respiratory tract infection (U.R.L.. infantile diarrhea (1.D.) and infantile dyspepsia (Dysp.): and the diseases for DX are allergic rhinitis (A.R.). upper respiratory tract infection (U.R.I.), infantile diarrhea (I.D.), infantile hand-foot-and-moone pneumonia (Pneu.). For brevity, In the experiment section, disease names in the tables are ahbreviations.
discase (H.FM) and

---

Under review as a conference paper at ICLR 2021
Table
Diagnosis accuracy of different baselines on MZ dataset and DX datasets
MZ
DX
Method
LD.
Dysp.
U.R.I.
Bronch.
Overall
A.R.
U.R.I.
Pneu.
H.F.M.
LD.
Overall
SVM-ex
0.89
0.28
0.44
0.71
0.59
0.5
0.92
0
0.8
0.95
0.64
SVM-ex&im
0.91
0.34
0.52
0.93
0.71
0.7
0.96
0.35
0.75
0.9
0.74
Basic DQN (2018)
0.65
0.7
0.79
0.55
0.7
0.9
0.73
KR-DS (2019)
0.96
0.39
0.5
0.97
0.73
0.9
0.67
0.3
0.95
0.9
0.74
Our IN
0.87
0.55
0.60
0.82
0.73
0.75
0.96
0.3
0.95
0.8
0.76
Table 2: Dialogue performance on DX dataset
Table 3: Robustness on MZ dataset.
Method
Acc.
Match rate
#turns
Method
NS.1 NS.2
NS.3
Basic DQN 2018
0.731
0.110
3,92
Basic DON (0188)
0.669
0.785 0.699
Sequicity (2018)
0.285
0.246
3.40
KR-DS (2019)
0.883
0.864 0.806
KR-DS (
0.740
0.267
3.36
KR-DS--rlation 2011
0.760
0,836 0.785
Our INS-DS
0.760
0,290
5.84
Our INS-DS
0.840 0.883 0.835
on this method, KR-DQN incorporates the pre-defined symptom-disease conditional probability to transform the output of DQN to improve diagnostic performance. For fair comparison, we adopt the same NLU from (Xu al., 2019) for all baselines. Following the same settings as (Wei al. 2018) and (Xu et al.) 2019). we also includes the supervised learning baselines SVM-ex (using explicit symptoms as input), SVM-ex&im (using explicit and implicit symptoms as input) as well as Sequicity (Lei et a 2018) were not trained or evaluated with interactive processes, we only evaluate the accuracy of these methods to highlight the effectiveness of the RL baselines.
sesquence-t-ssqueee
model). Since these supervised learning methods
Training Details. In general, the maximum discourse length of these baselines is 22. Rewards are critical to reinforcement learning. We follow the reward settings in (Xu et al., 2019). where +44 reward is used to encourage successful diagnosis and -22 to punish misdiagnosis. For each turn, -1 used to penalize the failure to hit one of the recorded symptoms in the user goal. The philosophy of the scale of the reward is to balance the maximum accumulated rewards (+44) and minimum accumulated rewards (-22+-1 x2)=444). Empirically, the size of the replay buffer is 1c6, the discount factor y is set as 0.95, the polyak factor is 5c-3, the learning rate is lc-4 and the optimization method is Adam (Kingma & Ba 2014). The maximum interaction runs for each experiment is 1e6. At the end of the training, the lastest model with the best diagnosis accuracy on training set is adopted for evaluation. As for training 100 bootstrapping DSMADs and synthesizing noisy test datasets, we adopt different random seeds keep them various.
As for the training data for the disease classifier, we use data in replay buffer which is collected along training. Specifically, at each training step, the RL agent interacts with the patient simulator and generates the dialogue state comprised of the symptom value vector and the binary visited indicator vector, as mentioned in Sec.3.1. At the end of each discourse, each of these dialogue states are paired with the disease label and stored into the experience replay buffer. When updating the disease classifier. batch of training data is sampled from the replay buffer, and then the classifier takes the symptom value vectors and disease labels as the training inputs and labels, respectively. The classifier is training along the training of the RL policy.
6.1 QUANTITATIVE RESULTS
In this part, we compare different baselines with our method using the metries diagnosis accuracy as well as our proposed DSMAD metrics for reliability and robustness. Note that, the results of the basclines are the best number reported in their papers if provided. Otherwise, the results are calculated by running the open/reproduced codes.
Accuracy of diagnosis. We compared the performance of our method and the state-of-art methods by diagnostic accuracy as well as matching rate, as shown in Tab. _ and Tab. which strictly follows the same settings as (Xu et al. 2019). Method Sequicity (Lei et al. 2018) is only evaluated on DX because requires raw language as input. And the results of each disease of Basic DQN in Tab. 2 is missing because the results are from the paper (Xu et al.) 2019). According to these tables, our approach exceeds or reaches the comparable diagnosis accuracy of KR-DS which is guided by the external knowledge. In the Tab. we observed that the accuracy of diseases "infantile diarrhea"
7

---

Under review as a conference paper at ICLR 2021
Table 4: Interal/external trusts and diagnosis entropy on MZ and DX datasets.
MZ
DX
Method
Int.
Ext.
Ent.
Int.
Ext.
Ent.
Basic DQN (Wei et al. 2018
0.6298
0,4054
0.6351
0.7428
0.0135
0.4563
KR-DS (Xu et al. 2019)
0.7999
0.4740
0.2204
0.7438
0.1632
0.3663
Our INS-DS
0,8508
0.4832
0.0414
0.7916
0.4327
0.0614
(I.D.) and "infantile bronchitis" (Bronch.) of baseline methods are much higher than the accuracy of the other two diseases. Aware of the similarity of these diseases ("infantile diarrhea" is similar to "infantile dyspepsia" and "infantile bronchitis" is similar to "upper respiratory tract infection"), concludes that the baselines method have learnt biased diagnoser to obtain higher overall diagnosis accuracy. And our approach has achieved a more balanced performance. This implies our method is better distinguishing two similar diseases instead of biasing one of them. While on DX, Basic DQN has more balanced results but its overall accuracy is lower. As shown in Tab. 2 our INS-DS has the most average interaction turns. This is because INS-DS not allowed to jump into informing a disease until the introspective results are converged. The higher matching rate of INS-DS means that mentioned symptoms of INS-DS are more likely presented in the records.
Reliability analysis. We evaluated DSMAD baselines using reliability measures and the results are shown in Tab. 4 We found the internal trust of our model has obvious ad advantages over other methods (at least 0.048). This indicates that the diagnosis of our INS-DS less sensitive to the noise during the training phrase, such as sampling error and parameter initialization. Basic DQN has the lowest internal trust on both benchmarks. To better understand the variance of diagnosis results, we also calculated the entropy (Ent.) of diagnoses from the bootstrapping models. Specifically, the entropy of diagnoses from the bootstrapping models for each test patient, and then the averaged entropy is reported in Tab. 4 Method with higher entropy indicates that the method is more likely to produce various diagnosis with training noise. Our INS-DS has the close to zero entropy, which also evidences that the diagnoses from INS-DS more consistent and are more in line with the symptom-diseass relation in the benchmarks. In terms of external trust, our model still achieves the highest score. Since the external trust is calculated based the symptom-disease co-occurrence, it indicates that our INS-DS can inquire about more discase-relevant symptoms, so its diagnosis is more convineing the patients. For Basic DQN and KR-DS, their external trust across MZ and DX are dramatically different. This is because the symptom- -diseoe co-occurrence of DX is much denser than MZ's so the values in the co-occurrence matrix of DX are generally smaller. And if the method does not inquires about those frequent symptoms DX, the overall external trust calculated by averaging small probabilities is likely to be smaller.
Robustness analysis. A robust system is expected to maintain its correct diagnosis after incon- sequential modification of its input. We set up three noisy test dataset according to the formulas proposed in Sec. 5.2 Specifically, (a) NSS. consists of noisy samples constructed by randomly moving 0~2 symptoms (which are mostly irrelevant with the ground-truth disease) from self-report to the implicit symptoms slots; (b) As for NSS.2 and N.S.3, we append and 2 symptoms most related with the ground-truth discase to the explicit symptoms. respectively. We repeated the above process 10 times and calculated the averaged unaltered proportion of the correct diagnoses using Equ. (5) The results are presented in Tab. We only evaluated the robustness on MZ dataset since As discussed in Sec.5.2, higher unaltered propor-
the self-reports in DX are natural language tion means that the correct diagnosis is more robust to noisy information. We found that INS-DS achieved superior performance on these noisy datasets.
As for N.2 and NSS,, the results of all approaches decreased when more disease-related symp- toms are added. It indicates that current DSMAD methods are prone to memorize the pattern of the diagnosis dialogue record instead of learning the ground-trutt -ruth symptom-disease relationship. This is also contradictory our real-life experience, indicating the difference between current DS- MAD methods and human cognition process. While by incorporating the introspective process, INS-DS could maintain a high invariance, which means that INS-DS are better at exploiting the ground-truth symptom-disease relation than other methods. It's worth noting that KR-DS also has good performance on these three settings since KR-DS provided with a pre-defined symptom- disease knowledge graph. As we observed in Tab. 3 the performance of KR-DS without the graph,
Itss possible but time-consuming revise the self-report of DX in order evaluate the robustness on DX.
8

---

Under review as a conference paper at ICLR 2021
70, User At noon on 10th, my --wekkoold baby began komiling and diarthea He's getting loose and yellow The fever began night Mertil Lyuch esles, the fever recurred and Then subsided on the zoming the 12th. On tse smelly yellow-grewn stool He had in upport stomari this moming.
T0, Uset My baby caluhc a cold the beginoling of this year and is still Coughing last weok, He took and hetamines for tour days, and it hosn't Improved The day before yestarbay, I had cold, which scemed be infortious by my taby. present, can koouhhvith the pileggn, sympioos aggear.
stogped vomiting, but getting
kinds of
11, agents Your boby have
T1, agant Your baby may have
T1, gant: Does your boby get nousee?
Agent Your bubyy may have
T1A Does your baby
T1, Agent: Has your buby
122 Usen have no idoa.
Love basel congesisn?
gol bung moist rale)
infantile
T3 T3..gmmt 0X, has your buby got lung moist
upper respératory infection
T2. User: not
Usen
Infantile diarrhea.
diarrhea.
rale?
gest: Does your baby
Agent: Does your baby have egil pelleen stool?
T4, Usen 1 don't know:
Lave d sxpppee
T5 Agemt Does your luby have green stol?
74, Usec. Yes, he has.
T4, User laave no idea. A Accorting symptoms tlink your baby mag hove pnoumonia.
T6, User
sure.
75. A Does your buby
17, Ammb Does your luby had lnss
get nutniing nose?
appetite
766 User don't know.
L3, Lisee Yes, he does.
17, A gent: OK, may your
T9. gegt Your baby may have infantile
beby has cought allergie
diarrhen.
rhinitis
Figure 2:
Two visualized conversation results on DX dataset of basic DQN, KR-DS and our INS-DS methods. In each table, the first line is self-report of the patient. The resulted dialogues from the left right are Basic DQN, KR-DS and INS-DS respectively. The correct mentioned symptoms are highlighted with orange boxes, and the correct diagnosis is blue underlined.
named KR-DS-relation*, is severely degraded. Since the policy of INS-DS without the introspection module has the similar structure as Basic DQN, we conclude that the introspection module enables INS-DS to better perceive the symptom -isooasee dependencies and obtain the similar performance with KR-DS which has the pre-defined symptom-disease relationship.
6.2 QUALITATIVE RESULTS
For intuitive recognition of baselines, provide some visual results, as shown in Fig. 2 The first line of each running example is the self-report of the patient and the second line is dialogues of DQN, KR-DS, and INS-DS consecutively. Basic DQN often gives reckless diagnosis without inquiring about more symptoms. KR-DS did make the attempts to inquire some symptoms but have limited ability to grasp useful symptoms. Our INS-DS performs better in acquiring relevant symptoms and is more cautious in conclusion with more reasonable interactions.
It is not sufficient to evaluate a dialogue system merely by automatic metric, invited three students, who are in pursuit of a medical doctorate, to perform human eval- uations based on three aspects: 1) diagnosis validity, the correctness of final diagnosis: 2) symplom rational- ity, the rationality of requested symptoms based on the medical knowledge; 3) topic transition smoothness, the smoothness of the inquiry logic irrelevant to the disease. Different from the diagnosis accuracy in Sec.6.1, par- ticipant were asked to score the diagnoses validity not only referring to the ground-truth disease but also the symptom-inqurry process. We generated dialogues of the test dataset by using each DSMAD baseline incorporated with NLU and NLG modules from Xu et al. (2019). Then asked the participants to score each of the randomly shuffled dialogues (to ensure a double-blind rating process) from 1 to 5 (integer). At the end, we re- ceived 104*3=412 assessments for each index and for each approach. The larger the score, the better the dialogue. The averaged scores and standard deviations are shown in Fig, From the perspec- tive of human experts, our INS-DS exceeds the other two methods in the three standards, especially in the aspect of symptom rationality and topic transition smoothness. The standard deviations are about 1. 1 The standard deviation is large since the scores are discrete integers.
4.5
35
2,501
2,50
3,35
2:036
3:121
2:45
630
3.25
25
2.18
Disgroois validiy
Syssplan
inssitoo smoothoes
Bast DQN
#KR w
Figure 3: Results of human evaluation on DX dataset, our INS-DS has exceeded in diagnosis validity, symptom rationality and topic smoothness.
7 CONCLUSION
In this work, we heuristically designed the INS-DS model mimic the rational decision-making process in real life. Our approach achieves superior performance in different evaluation metrics, including our proposed reliability and robustness metrics. Experimental results evidence that our method can better understand the symptom-disease relation and therefore is insensitive to other irrelevant noisy factors.
From the left right of the scores in Fig. standard deviations are 1.10, 1.11, 1.09, 1.06, 1.24, 1.16, 0.95, 0.87 and 1. 1.04, respectively
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hao-Cheng Kao, Kai-Fu Tang, and Edward Y Chang. Context-aware symptom checking for dis- ease diagnosis using hierarchical reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. corr abs/ 1412.6980 (2014), 2014.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume 70, pp. 1885- 1894. JMLR. org, 2017.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang. Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS 17 Competition: Building Intelligent Systems, pp. 195-231. Springer, 2018.
Wengiang Lei, Xisen Jin, Min- Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to to-suueeee architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties, vol- ume 1, pp. 1437-1447, 2018.
Xiujun Li. Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. End-to-end task- completion neural dialogue systems. In Proceedings of the Eighth International Joint Conference Natural Language Processing, volume pp. 733 733-743,11 2017.
Yuan Ling, Sadid A Hasan, Vivek Datla, Ashequl Qadir, Kathy Lee, Joey Liu, and Oladimeji Farri. Learning to diagnose: assimilating clinical narratives using deep reinforcement learning. Pro- ceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 895-905, 2017.
7achary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng. Bhg-networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems. Thirty- Second AAAI Conference on Artificial Intelligence, 2018.
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 285, 2015.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Mem2seq: Effectively incorporating knowl- edge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 1468-1478, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Judea Pearl. Theoretical impediments to machine learning with seven sparks from the causal revo- lution. In Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pp. 3-3, 2018
Yu-Shao Peng, Kai-Fu Tang, Hsuan- Lin, and Edward Chang. Refuel: Exploring sparse features in deep reinforcement learning for fast disease diagnosis. In Advances Neural Information Processing Systems, pp. 7333-7342, 2018.
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
Kai-Fu Tang, Hao-Cheng Kao, Chun-Nan Chou, and Edward Y Chang. Inquire and diagnose: Neural symptom checking ensemble using deep reinforcement learning. In Proceedings of NIPS Workshop on Deep Reinforcement Learning, 2016.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, volume 2, pp. 5. Phoenix, AZ, 2016.
10

---

Under review as a conference paper at ICLR 2021
Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-Fai Wong, and Xiangying Dai. Task-oriented dialogue system for automatic diagnosis. Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), volume 2, pp. 201-207, 2018.
TH Wen, D Vandyke, N Mrkšíc, M Gašíc, LM A network-based end-to-end trainable task-oriented dialogue system. In 15th Conference of the European Chapter of the Association for Computational Linguistics.
PH Su, S Ultes, and S Young.
Chien-Sheng Wu, Richard Socher, and Caiming Xiong. Global-to-local memory pointer networks for task-oriented dialogue. ICLR, 2019.
Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang, Jianheng Tang, and Liang Lin. End-to-end knowledge-routed relational dialogue system for automatic diagnosis. AAAI, 2019.
Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. Building task-oriented dialogue systems for online shopping. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Quan-shi Zhang and Song-Chun Zhu. Visual interpretability for deep learning: survey. Frontiers of Information Technology &amp; Electronic Engineering. 19(1):77-39, 2018.


Reviews:
[{"review_id": "WTtDpzZzO9O", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting medical diagnosis dialogue system. It considers cooperation between two separate modules: an inquiry module that manages symptom inquiries and an introspective module to decide when to inform the diagnosis based on available symptoms. The paper also proposes two evaluation metrics based on reliability and robustness of the models. Results show the usefulness of the proposed model in terms of these evaluation metrics along with a human evaluation study.\n\nStrengths:\n\n- The idea of considering the symptom inquiries and the diagnosis decision making as a separate cooperative process in a deep reinforcement learning setting is interesting.\n\n- The proposal of how to meaningfully evaluate the proposed models using two new metrics is very useful. \n\n- Experimental results appear to be solid and the comparisons with related work are also meaningful.\n\nWeaknesses:\n\nThis paper appears to be an incremental piece of work based on the prior work on the same task. In my opinion, the main limitation of the paper is its novelty in terms of the core idea as it claims to mimic the decision making process of the human in defining the introspective system - however, this idea and the framework presented in Figure 2 is not new (see Ling et al. \"Learning to Diagnose: Assimilating Clinical Narratives using Deep Reinforcement Learning\"). Furthermore, the core modules of the paper are essentially same as Xu et al. 2019 as also acknowledged by the authors in Section 3.1.\n\nFew other comments: \n\n- More clarity needed for the notion of contradictory diagnosis and correct diagnosis as they appear to be ambiguous.\n\n- Not sure about the rationale behind discriminating the symptoms between explicit and implicit - why was it necessary to consider them differently for modeling?\n\nOverall, the proposed methodology is solid and the work would be beneficial to the community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SCxQNdf67Rw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe authors propose a new method for an RL agent-based , interactive medical dialogue. The method has two core components: a symptom inquiry module which selects a new symptom to inquire about a time step t, and an introspection module which looks ahead one time step to evaluate *if* knowing the outcome of this inquiry will change the predicted disease of the patient at time step t + 1. If the predicted disease at t + 1 would not change on the basis of issuing the symptom query at time t, the system returns the diagnosis, else it issues the query and repeats this process until convergence. The authors argue that this mimics the human decision making process and leads to a more principled diagnostic procedure. \n\nThe authors also propose two new metrics by which to evaluate medical dialogue systems. Reliability, which measures how sensitive the model is to random fluctuations during training (internal) and how well the selected symptoms align with human doctors (external). Robustness, which measures how well the model performs under an application-specific adversarial attack.\n\n##########################################################################\n\nRationale for score:\n\nI proposed a recommendation of reject due mainly to the deep concerns I have about the experimental setup. The two example vignettes the authors have provided are both seriously flawed both with respect to the accuracy of the ground truth (symptoms and diagnoses) and with respect to the example dialogues (see below). These issues will substantially impact at least one of novel metrics proposed by the authors (external reliability), which arguably is the most important metric. Due the experimental issues, I am unable to trust this metric and, unfortunately, the authors' method is no better (and in some cases substantially worse) on the second most important metric (in my assessment), diagnostic accuracy. \n\n##########################################################################\n\nPros:\n\n1. The proposed approach is novel and interesting and addresses an important problem. I particularly like the introspection aspect. I do however wonder if authors could provide some theoretical insight into situations where their one-step look ahead procedure will and will not converge? e.g. is the proposed method always guaranteed to terminate after a finite number of steps or are there situations when it might never return a diagnosis without a hard, pre-defined step cutoff?\n\n2. The figure is clear and extremely helpful.\n\n3. The introduction of new metrics other than diagnostic accuracy is new and somewhat well motivated.\n\n##########################################################################\n\nCons:\n\n1. There seems to be very little theoretical connection between internal and external reliability, as described. It might be better to break these out into two different metrics.\n\n2. Internal reliability is not fully defined. From equation (3), it appears to be the average probability from k=100 bootstrapped models. That, to me, seems to be a way to *reduce* aleatoric uncertainty but does not measure model sensitivity to perturbations. Something like the standard deviation of this bootstrapped probability would measure that however. \n\n3. Even though the authors are using a pre-existing dataset, I have deep concerns about how realistic the simulated datasets are, and how it may specifically impact their proposed metrics. For example, I showed the conversation about infantile diarrhea to my clinical colleague who is a practicing neonatologist and board certified pediatrician. Several things that were indicated to be the correct symptoms (orange boxes) were completely nonsensical in her assessment. In her opinion, no physician would ever describe a baby who has trouble feeding as \"anorexic\" and a \"stomach murmur\" is not a term she has ever heard. Likewise in the second example her opinion is that none of the systems collected sufficient information to render a diagnosis of pneumonia, and that actually upper respiratory infection (agent 1) provides the most sensible diagnosis, and the symptom requested by INS-DS (egg pattern stool) makes little sense.  Unfortunately, if these examples are representative of the rest of the data, it casts doubts on the strength of the authors claims. Their external reliability metric rests on how well the ground truth symptoms are captured by a system, and it seems that there could be systemic issues with the ground truth symptoms and diagnoses. The authors report some human assessment with three medical students, but it is hard to understand how the evaluation was done due to insufficient detail, especially in light of the two examples provided in the paper. \n\n4. Unfortunately, the authors proposed approach is not as accurate as other methods in terms diagnostic accuracy, and sometimes by a substantial margin. As the authors argue, other factors are certainly important (reliability and robustness), but all are secondary to diagnostic accuracy in my opinion. If the system you are interacting with is robust to noise that's great, but if it performs worse on average, then you will do worse in the long run as a potential user. \n\nMinor typos/confusing sentences:\n- \"Dialogue system for medical automatic diagnosis (DSMAD) aims at learning an agent to collect patient’s information and make preliminary diagnosis\"\n\n- \"but ignore the importance of robustness and reliability for practical medical application\"\n\n- \"Despite the contradictory diagnoses, the agent will still get a positive reward from the upper one since the diagnosis is correct.\" (what is the upper one?)\n\n- \"resulting in the system vulnerable to the noise happened during the interactions.\"\n\n- \"As for evaluation, aware of the insufficient validity of final classification accuracy and the number of query steps, current methods measure the performance of the diagnosing process by computing the hitting rate of the inquired symptoms.\"\n\n- \"Therefore, measures DSMAD according to the recorded symptoms in each user goal could not reflect\nthe actual performance of DSMAD in general.\"\n\n- \" Different from the most popular diagnosis systems which laving equal emphasis on symptom-inquiry and disease-diagnosis\"\n\n- \". And only when the most valuable inquiry proposed by the symptom-inquiry module makes no difference to the future diagnosis will the predicted disease be imformed\"\n\n- \"Emperically, the size of the replay buffer is 1e6,\"\n\n- \"Trusts for reliability. \" (Section header, should be tests?)", "extra_scores": {}, "extra_text": {}}, {"review_id": "Np0-NxD_I4d", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, authors proposed a novel dialogue system for medical automatic diagnosis (DSMAD) called INS-DS. There are three components in the general DSMADs, which are NLU, DM and NLG. In this paper, NLU and NLG components are adopted from Xu et al., 2019. Authors focused on designing decision-making parts in the dialogue management (DM) component. INS-DS includes two modules: an inquiry module and an introspective module which is inspired by the introspective process when humans make decisions. Authors also introduced two metrics evaluating the reliability and robustness of general DSMAD agents.\n\nAuthors demonstrated INS-DS achieved state-of-the-art performance compared with other DSMAD agents and performed better with respect to robustness and reliability. My concerns are mainly related to the experimental settings and the detailed comments are listed as follows.\n\n#### Major Comments:\n1. (Table 4) Evaluating and improving the robustness and reliability of DSMAD agents is one of the major contributions in this manuscript. Authors have evaluated the robustness and reliability of DSMAD agents based on proposed metrics. One of the further experiments authors can conduct is to check the performance of DSMAD agents trained on MZ or DX and test on the other. This is a general and widely-accepted approach to demonstrate the agents' robustness and reliability.\n\n Both DX (527 conversations) and MZ (710 conversations) are relatively small datasets and there are only two diseases (I.D. and U.R.I) that appeared in both of the datasets. The performances regarding I.D. are pretty consistent but this is not true for U.R.I. The symptoms related to U.R.I. may be different across these two sets but there should be a reasonable overlap. Could you train the model on DX and test it on MZ focusing on I.D. and U.R.I? This would be more convincing to demonstrate the agent's robustness and reliability and this helps understand the inconsistency of diagnosis accuracies regarding U.R.I.\n\n2. (Table 1) For Basic DQN and KR-DS, their performances in Ext. (External trust) across MZ and DX are dramatically different. Is there any reason that can explain this difference especially considering that DX dataset was proposed along with KR-DS?\n\n3. (Table 2) It's better to include the baseline performances (test dataset without noise) for each of the agents. Otherwise it is hard to see whether agents are robust to noise or not under different settings (NS.1, NS.2 and NS.3 ).\n4. (Table 3). If more related symptoms are appended, the tasks should be relatively easier for humans. But NS.3 is consistently worse than NS.2 across all agents. Could you elaborate on this observation?\n\n5. (Figure 3) How does the diagnosis validity correlate with the ground-truth diagnostics for the test set? The diagnosis validity score ranges from 2.8 to 3.2 which is far from the best score 5. Is this due to the inconsistency between the ground-truth diagnosis and the diagnosis made by students?\n\n6. (Page 7, robustness analysis) In the robustness analysis, authors made use of noise test sets to demonstrate the agents' robustness. For humans, the diagnosis should be invariant with respect to the orders of the explicit symptoms or implicit symptoms. Based on this assumption, it is natural to augment the train set by permuting/sampling symptoms and this should further improve the performances of all models. Have you applied this augmentation strategy in the training phase?\n\n\n#### Minor Comments:\n1. (Figure 3) It is helpful to include the error bars (or all three data points) to show the variance of scores assigned by students.\n\n2. (Table 2) Robusteness -> Robustness\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "734BN_BtTwW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for automatic medical diagnosis in a dialog system which is comprised of two modules: one which proposes symptoms to inquire about, and another which decides whether to go ahead with the inquiry or inform a disease. The second module makes the decision by looking ahead and checking whether the symptom wold case differences in determining the disease. The authors argue that existing systems have only been evaluated with respect to diagnosis accuracy, and introduce two additional metrics to evaluate the reliability and robustness of the method. They evaluate their method and compare with existing state-of-the-art methods on two datasets. Furthermore, they do a small human evaluation and report results on how evaluators perceive the diagnosis validity, symptom rationality, and topic transition smoothness of different methods. Overall, the paper is well motivated and written, and the authors show equal or better results compared to other state-of-the-art methods.\n\nI have the following questions/comments. Given clarifications in an author response, I would be willing to increase the score. \n- Can you add more information about the datasets/augmented test sets used (e.g., train/dev/test size). What data is the disease classifier trained on? What do the diseases in Table 4 represent?\n- Some details about the training are missing. For example, how many seeds were used? How long does it take for the training to converge using each method?\n- For the results of the DX dataset, is the same NLU model used for all three models? If not, how does that impact the results?\n- What do the numbers in Table 2 indicate? Are they accuracy results on the noisy test-sets?\n- The section \"Accuracy of diagnosis\" is not well written. Sequicity is introduced without any discussion on what it is. Additionally, new methods are introduced at the end of the paragraph. Why are these methods not used in Table 1? What is the intuition behind different results for different diseases? I suggest rewriting this section to first introduce the methods that are used, and then discussing the results. It might make sense to put this section before the \"Trusts for reliability\" section on page 6.\n- In the same section, the sentence \"And the results of each disease of Basic DQN is missing because the results are from the paper\", it is not clear that refers to table 4.\n- Since you discuss Figure 2 in detail in section \"Qualitative Analysis\", I suggest removing it from section \"Accuracy of diagnosis\" as mentioning it just briefly in this section is confusing.\n- There is no discussion of the errors that the proposed approach is making. Are the errors the same as the ones made with the other methods?\n\nThe text has spelling errors:\n - Page 3, P2: measures DSMAD -> measuring\n - Page 3, P3: dataset is includes -> dataset includes\n - Page 3, P4: paper is the same -> paper are the same\n - Page 4, P1: record -> records, produce -> produces, Remove And from the beginning of the sentence\n - Page 4, P2: imformed -> informed\n - Page 8, P1: an reckless -> a reckless (what does a reckless diagnosis refer to?)\n - Page 8, P2: ask -> asked. Remove And from the beginning of the sentence\n - Page 8, P3: mimics -> mimic, could -> can, therefore insensitive -> therefore is insensitive\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel approach to a dialog-based  automated medical diagnosis, and present promising empirical results. The focus of this work is on robustness and reliability besides just the accuracy of diagnosis, which appears to be an important aspect in medical applications. The paper is clearly written and well-motivated. However, in there are still several concerns raised by the reviewers, and the paper may require a bit of extra work to be ready for publication.
Decision: Reject