Task-oriented Dialogue System for Automatic Disease Diagnosis via Hierarchical Reinforcement Learning
Authors: Kangenbei Liao
CHENG ZHONG
Wei Chen
Qianlong Liu
zhongyu wei
Baolin Peng
Xuanjing Huang
Venue: ICLR 2022 Submitted (2022)
Keywords: Dialogue System
Automatic Disease Diagnosis
Hierarchical Reinforcement Learning

Abstract: In this paper, we focus on automatic disease diagnosis with reinforcement learning (RL) methods in task-oriented dialogues setting. Different from conventional RL tasks, the action space for disease diagnosis (i.e., symptoms) is inevitably large, especially when the number of diseases increases. However, existing approaches to this problem typically works well in simple tasks but has significant challenges in complex scenarios. Inspired by the offline consultation process, we propose to integrate a hierarchical policy of two levels into the dialogue policy learning. The high level policy consists of a master model that is responsible for triggering a low level model, the low level policy consists of several symptom checkers and a disease classifier. Experimental results on both self-constructed real-world and synthetic datasets demonstrate that our hierarchical framework achieves higher accuracy and symptom recall in disease diagnosis compared with existing systems.


Under review as a conference paper at ICLR 2022
TASK-ORIENTED DIALOGUE SYSTEM FOR AUTO- MATIC DISEASE DIAGNOSIS VIA HIERARCHICAL RE- INFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, we focus on automatic disease diagnosis with reinforcement learning (RL) methods in task-oriented dialogues setting. Different from conventional RL tasks, the action space for disease diagnosis (i.e., symptoms) is inevitably large, especially when the number of diseases increases. However, existing approaches to this problem typically works well in simple tasks but has significant challenges in complex scenarios. Inspired by the offline consultation process, we propose to integrate hierarchical policy of two levels into the dialogue policy learning. The high level policy consists of a master model that is responsible for triggering a low level model, the low level policy consists of several symptom checkers and a disease classifier. Experimental results on both self-constructed real-world and synthetic datasets demonstrate that our hierarchical framework achieves higher curacy and symptom recall in disease diagnosis compared with existing systems. Besides, both our datasets and codes will be released.
INTRODUCTION
With the development of electronic medical records (EMRs), researchers explore different machine learning approaches for automatic diagnosis (Shivade al. (013). Although impressive results have been reported for the identification of various diseases (Jonnalagadda et al. 2017 Doshi Velez jet al. 2014), they rely on well established EMRs which are labor-intensive to build. Moreover, each diagnosis requires EMR of the patient, while those systems lack the ability to obtain information through interaction with patients.
To relieve the pressure for constructing EMRs, researchers (Wei al. 2018 Xu et al. 2019a) in- troduce task-oriented dialogue system to request symptoms automatically from patients for disease diagnosis. They formulate the task Markov Decision Processes (MDPs) and employ reinforce- ment learning (RL) based methods in the policy learning for the agent. Existing frameworks utilize the setting of flat policy that treats diseases and all related symptoms equally. However, although RL-based approaches show positive results for symptom acquisition, when it comes to hundreds of diseases in real environment, flat policy is quite impractical.
In general, particular disease is often related to certain group of symptoms. As shown in figure [ present the correlation between diseases and symptoms, where x-axis represents symptoms and y-axis is the proportion of diseases related. From figure [ there are obvious differences in the distribution of symptoms for different diseases. In offline consultation, doctors also do the pre- examination and triage according to the different symptoms that patient suffered, then doctors in different departments will make more detailed diagnosis. This method significantly reduces the workload of individual doctors and enables them to be more specialized in certain field.
In the reinforcement learning algorithm, we found that Hierarchical Reinforcement Learning (HRL) Parr& & Russell 1998; Sutton et al. 1999) conforms to this moblem-solvinn logic. Recently, HRL, in which multiple layers of policies are trained to perform decision making, has been successfully applied to different scenarios, including course recommendation (Zhang et al., 2019), visual dia- logue (Zhang et al.) 2018), relation extraction (Feng et al.) 2018), etc. The natural hicrarchy of target tasks are modeled either manually automatically, which motivates us classify diseases

---

Under review as a conference paper at ICLR 2022
symptoms
chiidion's bronchits
upper respiottory infection
Whhlddren functional dyspepsla
Intirntte
The disease distribution over symptoms in the real-world dataset (see section 3.1 stands for symptoms and y-axis is the proportion. Each bar describes the disease distribution given symptom.
Figure I:
x-axis
into different groups following the setting of departments in the hospital, and design hierarchical structure for symptom acquisition and disease diagnosis.
In this paper, we classify diseases into several groups and build a dialogue system with hierarchy of two levels for automatic disease diagnosis using HRL methods. The high level policy consists of model named master and the low level policy consists of several workers and disease classifier. The master is responsible for triggering a model in the low level. Each worker is responsible for inquiring symptoms related to certain group of disease while discase classifier is responsible for making the final diagnosis based on information collected by workers. The proposed framework imitates group of doctors from different departments diagnose patient together. Among them, each worker acts like a doctor from a specific department, while the master acts like a committee that appoints doctors interact with this patient. When information collected from workers are sufficient, the master would activate a separate disease classifier to make the diagnosis. Models the levels are trained jointly for better disease diagnosis. We our model in three large real-world datasets and a synthetic dataset for the evaluation of our model. Experimental results demonstrate that the performance our hierarchical framework outperforms other state-of-the-art approaches.
We summary our contribution as follows: 1) We propose a new common-senun common-sensible method for au- tomatic diagnosis. It simulates the real scene of clinical practice, and assigns patients to different workers through high-level policy, thereby reducing the action space and improving training effi ciency. Also, the method can be compatible with different network models and training strategies; 2) We evaluate our method on three public datasets and a synthetic datasets to demonstrate the supe- riority of our framework with higher accuracy of disease and recall of symptoms; 3) We construct detailed analysis of the effectiveness of each part of this method to confirm that is reasonable and effective.
FRAMEWORK
In this section, we introduce our hierarchical reinforcement learning framework for disease diagno- sis. We start with the hierarchical policy setting and then introduce every part of method. We further introduce reward shaping techniques which help us improve the model performance.
2.1 HIERARCHICAL POLICY
To reduce the large action space, we extend the RL formulation to hierarchical structure with two- layer policies for automatic diagnosis. Following the options framework (Sutton et al. 1999), our framework illustrated as in Figure a). There are five components in our framework: master, workers, discase classifier, internal critic and user simulator.
Specifically, we divide all the diseases in into subsets D1. D2,..., Dr. where D U D2 ....U Dr D and D; Dj 0 for any i # j. Each D, is associated with set of related symptoms S C S. While worker w is responsible for collecting information about symptoms of S.
At turn L, The state 8 will be encoded as one-hot vectors that reflects the status of each symptom, i.e., 8, |bT 6T ... b/s __. and the master decides whether to collect symptom information (picking one worker interact with user for several turns) inform the diagnosis result (picking disease classifier to output the predicted disease). When agent inquires the symptom i, the patient will respond with treesfalse/unknown and agent will update the corresponding symptom statuses b;
2

---

Under review as a conference paper at ICLR 2022
Figure 2: The training and inference process of model. (a) The framework of our hierarchical reinforcement learning model with two-layer policies. (b) Illustration of the diagnosis process of our model with interactions between models in two levels. w* is the action invoking worker w² and d is the action invoking disease classifier.
High level policy
Master
ExtrMIc
Warkker
Warther
Warker is
Actediing
disaler
Low level
Asguest sirsson
User Simulattrr
Classiler
Disesse
Intriiic rowerd
Internal
Cfftic
Roquesiing the anstrrdaa iumpltmn aicantitee wurks
(a) Masker-worker framework (training)
(b) Master-worker framework (inference)
R3 When agent informs disease, the dialogue will be terminated success/fail in terms of the correctness diagnosis. Note that the not-requested symptoms are all encoded as b [0, 0, 0]. After each action executed, the agent will receive reward based the result of the action.
An illustration of the diagnosis process with interactions between models in two levels are presented
in (eguree[b) As for internal critic, it is responsible for both returning intrinsic reward to the worker and telling whether the subtask of the invoked worker is finished. In addition, the user simulator is applied to interact with our model and return extrinsic reward.
2.1.1 MASTER
The action space of master is nM = fw'li 1,2, 11.,....,,,JU) U The action w' indicates activating worker w² while is primitive action which means activating the disease classifier. At each turn f, the master takes the dialogue state 8t S input and takes action at Am according to it's policy. An extrinsic reward rf will be returned to master from the environment.
Once the master activates worker, this worker will interact with user for N turns until the subtask is terminated. Only after that master can take a new action and observe a new dialogue state. So, the learning problem of master can be formulated as Semi-Markov Decision Process (SMDP), where the extrinsic rewards returned can be accumulated the immediate rewards for the mas- ter (Ghavamzadehi 2005). That to say, after taking an action ot the reward r," for master can be defined as:
Ei= 7 riter is at w {
r =
ri. if
a. = d
Where s' is the observed dialogue state master after it takes an action am, n! the next action when the state is s'.
The objective of master is to maximize the extrinsic reward, thus we can write the master's loss function follows:
C(0m) = Es pm Bm ((y Qm am am; 01
(2)
where y = nm + N max.m: (s' an
a am), 6m the network parameter at current iteration, a, is the network parameter of previous iteration and 1 is the fixed-length buffer of samples for master.
2.1.2 WORKER
The worker wi corresponds to the set of diseases D, and the set relevant symptoms S,, and the action space of worker w' is S, At turn if worker w' is invoked, the current state of master s, will be passed to worker wi, then worker w will extract the corresponding state representations; from 8, and take of input to generate action of E A.
3

---

Under review as a conference paper at ICLR 2022
After taking action dj A,,, the dialogue is updated into 8++1 and worker w² will receive an intrinsic reward r from the module of internal critic. So the objective of worker is to maximize the expected cumulative discounted intrinsic rewards. The loss function of worker w can be written
[(0 Eg ot! By [( (in a' (2))))))
(3)
where = y² + The m nd Qu (s" a²; (0w, Tw is the discounted factor of all the workers, 0e is the network parameter at current iteration, 0% the network parameter of previous iteration and B. is the fixed-length buffer of samples for worker w².
2.1.3 DISEASE CLASSIFIER
Once the disease classifier is activated by master, will take the master state & as input and output vector p RRIP, which represents the probability distribution over all diseases. The disease with highest probability will be returned to the user as the diagnosis result. Two layers of Multi-Layers Perceptron (MLP) is utilized here for the disease diagnosis.
2.1.4 INTERNAL CRITIC
The internal critic is responsible for generating intrinsic reward r to worker w² and judging the termination condition for the worker. our task, worker is terminated as failed when the number of subtask turns reaches 7sub, and terminated as successful when correct symptom is requested by the agent. After that, positive reward will be returned if the worker requests a correct symptom or disease. If there repeated actions generated the number of subtask turns reaches T'sub r would be negative. Otherwise, r would O.
2.1.5 USER SIMULATOR
Following (Wei al. 2018) and (Xu et al. 2019a), set up user simulator to interact with the agent. At the beginning of each dialogue session, the user simulator samples a user goal from the training randomly. The simulator will initialize the dialogue session base the explicit symptoms, and interacts with the agent based on the implicit symptoms. One dialogue session will be terminated as successful if the agent make the correct diagnosis, or failed if the informed disease is incorrect or the dialogue turn reaches the maximal turn T'.
2.2 REWARD SHAPING
In practice, the number of symptoms patient suffers from is much less than the size of symptom set S, which results in sparse feature space. In other words, it is hard for the agent to locate the symptoms that the truly suffers from. In order encourage master choose a worker that can discover more positive symptoms. we follow (Peng et al.) 2018) and use the reward shaping method add auxiliary reward to the original extrinsic reward while keeping the optimal reinforcement learning policy unchanged.
The auxiliary reward function from state 8t to 8t+1 is defined as f(r,, st-1) 7)Φ($141) d(st). Ø(s) is the potential function, Φ(s) counts the number of correct symptoms informed for given state 8, > 0 hyper-parameter which controls the magnitude of reward shaping and S the terminal state set. Thus, the reward function for master will be changed into R = rt + f(r,, 8(41).
2.3 TRAINING
Both the master policy m and each worker's policy x are parameterized via Deep Q- Network (ullkarni et al. 2018 Mnih et al., 2015). In DQN, the action is often selected following an e-greedy policy. In our hierarchical framework, both the master and the workers behave following their own c-greedy policy for training and greedy policy for testing. During the training process, we store (st, at" rm S14N) in T and (s), at rř, s1+1) in B At each training step, we perform expe- rience replay to update the current networks for both master and workers in n and 13"" respectively, while the target networks fixed during experience replay.
4

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTAL DATASETS
We evaluate our framework one simulated public datasets: SymCat, and three medical dialogue datasets: MZ-4 (Wei et al., 2018). MZ-10 and Dxy (Xu et al. 2019b). MZ-10 comes from our expansion the basis of MZ-4, that can be close to more complex dialogue environment in real medical dialogue.
3.1 REAL-WORLD DATASET
MZ-4 An existing dataset collected from real world for the evaluation of task-oriented DS (Wei et al., 2018). After proposed, it became one of the popular datasets for evaluating automatic medical systems. This dataset includes 4 diseases, 230 symptoms and 1733 user goals. Most of the existing RL-base methods have been tested on this dataset.
A newly constructed real-world dataset contains 41 16 user goals that belong to 10 dis- eases, including typical diseases of digestive system, respiratory system and endocrine system. The raw data is collected from the pediatric department on a Chinese Online Healthcare Community! Each user record consists of the self-report provided by the user and conversation text between the patient and a doctor. We hire experts with medical background to identify symptom expressions and label them with three tags ("True", "False" " or "UNK"") to indicate whether the user suffers this symp- tom. After that, experts manually link each symptom expression to concept on SNOMED CT T Symptoms extracted from self-report are treated as explicit symptoms and the ones extracted from conversation are implicit symptoms. At the same time, to ensure that the dataset is not too sparse, we only keep the symptoms with the top-100 frequency. The real-world dataset we constructed contains 41 16 user goals, of which 80% for training 10% for validating and 10% for testing.
MZ-10
Dxy A Dialogue Medical dataset (Xu et al. 2019b) contains data from a prevalent Chinese online healthcare website] This dataset contains 527 user goals, including 5 diseases and 41 specific symptoms. Both datasets are already in the form of symeposs pairs.
For real-world dataset, we compare the similarities of the counts of symptoms in each discase on the training set, and divide the diseases with higher similarity (above 0.5) into a group.
3.2 SYNTHETIC DATASET
However, the number of disease and user goals in real-world dataset is still limited. In order to show the effectiveness of the HRL method, we build a synthetic dataset (SD) following (Kao et al., 2018) for further analysis of the method. It constructed based pymptom-disease database called SymCar) There are 801 diseases in the database and we classify them into 21 departments (groups) according to International Classification of Diseases (ICD-10-CM) We choose 9 representative de- partments from the database, each department contains top 10 diseases according to the occurrence rate in the Centers for Disease Control and Pre-vention (CDC) database.
In SymCat database, each disease is linked with set of symptoms, where each symptom has probability indicating how likely the symptom is identified for the disease. Based on the probability distribution, we generate record one by one for each target disease. Given disease and its related symptoms, the generation of a user goal follows two steps. First, for each related symptom, we sample the symptom based on the probability. Second, symptom is chosen randomly to be the explicit one (same as symptoms extracted from self-report RD) and rest of true symptoms are treated implicit ones. The synthetic dataset we constructed contains 30,000 user goals, including 90 diseases and 267 symptoms, of which 80% for training and 20% for testing.
Table I shows the details of all dataset used in this paper.
hhtt://muuziii hwwp://uuhiibaiuu..oom com
'htts://dxy.com/
www.symcattcom
5

---

Under review as a conference paper at ICLR 2022
Table I: Overview of the Dataset.
Name
I of user goal
f of diseases
ave. $ of im. sym.
: of sym.
MZ-4
1733
5.46
230
MZ-10
3745
10
6.90
100
Dxy
527
5
1.67
41
SymCat
30,000
90
2.60
266
PERFORMANCE ON REAL- WOALD DATASETS
We train our model following the setting in Appendix A.1 A.I and compare the effectiveness different models in terms of accuracy, average turns and match rate.
4.1 MODELS FOR COMPARISON
We compare model with some state-of-the-arr reinforcement learning models for disease diag- nosis.
Flat-DQN: This is the agent of (Wei et al. 2018), which has one layer policy and an action space including both symptoms and diseases.
HRL-pretrained:
This is a hierarchical model from (Kao et al. 2018). The setting is similar to ours, however, the low level policy pre-trained first and then the high level policy is trained. Besides, there is no disease classifier and the diagnosis made by workers.
REFUEL: This is reinforcement learning method with reward shaping and feature rebuild- ing (Peng et al. 2018). uses a branch reconstruct the symptom vector to guide the policy gradient.
GAMP: This is GAN-hased policy gradient network (Xia al. 2020). uses the GAN network to avoid generating randomized trials of symptom, and add mutual information to encourage the model select the most discriminative symptoms.
KR-DS: This is an improved method based on Flat-DQN (Xu al. 20196). It integrates relational refinement branch and a knowledge-routed graph to strengthen the relationship between disease and symptoms.
FT7: FIT is a novel non-RL. framework for inquiring and diagnosing tasks (He et al. 2020). propose self-attention diagnosis model and use multi-modal variational autoencoder (MVAE) as probabilistic model dealing with partially observed data to improve the diagnostic accuracy.
4.2 OVERALL PERFORMANCE
Table show the overall performance of different models respectively. In MZ-4 dataset, most of the methods do not report the average turns and match rate of the dialogue, so only compare the accuracy of the disease. Observed from MZ-4 dataset, our method outperforms all the meth- ods by least 9%. In MZ-10 dataset, the advantages of our model become more obvious. In comparison with flat-DQN, our method outperforms in both accuracy and match rates. When the number of symptoms increases, the symptom-disean and symptom-disent more complicated, which make Flat-DQN difficult to deal with. Our method makes the distribution of symptoms in each type of discase more logical by dividing diseases in different groups. So it can hit more symptoms during dialogue and provide more information for disease diagnosis.
relationship becomes
Also, the results Dxy dataset shows similar conclusions. Compared with other methods, our to make
method can inquire more appropriate symptoms through interactions with patients, better decisions. In Dxy dataset, our methods outperforms other method both accuracy and match rate by larger magnitude, which means method is more reasonable, efficient and logical.

---

Under review as a conference paper at ICLR 2022
Table 2: Overall performance on dataset. We conduct each experiment (Flat-DQN and HRLs)
times, and the reported number is the average, other numbers are copied from the corresponding literature, denotes missing numbers, and Acc., M.R, Avg. T are the abbreviations of Accuracy, Match Rate and Average Turns respectively.
Dxy
M7.4
MZ.10
Model
Acc.
M.R.
Avg. T
Acc.
M.R.
Avg.
Acc.
M.R.
Avg. T
Flat-DON
0.731
0.110
3.92
0.681
0.062
2.53
0.408
0.047
19.51
KR-DS
0.740
0.267
3.36
0.730
REFUEL
0.757
0.718
GAMP
0,769
0.179
2.68
0.730
FIT
0,811
0.726
HRL (w/o grouped)
0.894
0.132
9.14
0.807
0.004
4.51
0.734
0.114
9.18
HRL (w/o discriminator)
0,512
16.84
0.120
12.93
0.330
17.49
HRL (ours)
0.848
0,426
14.97
0.820
0.127
10.16
0.747
0.363
17.37
4.3 ABLATION STUDIES
To verify the effects of the main components of method, further conducted series of abla- tion studies. Because our HRL method consists of two parts, master-worker structure and disease classifier, we try
divide symptoms into different group (Denoted as HRL (w/o grouped)) and remove the separate disease discriminator (Denoted as HRL (w/o discriminator)). Note that when we remove the master-worker structure and leave the work of judging the disease the master, the model will degenerate into simple Flat-DQN model.
The result shown in Table From the results, it can be seen that dividing diseases into different groups can effectively improve the recall of symptoms. Comparing HRL (w/o grouped) and Flat- dqn, a single disease classifier can greatly improve the efficiency of disease classification, while avoiding the dilemma caused by the uneven distribution of discases and symptoms. Note that the implicit symptoms and the number of user goals in the Dxy dataset are limited, the advantages of the overall HRL method over its various parts are not very obvious.
5 PERFORMANCE ON SYNTHETIC LARGE-SCALE DISEASES
In current practice, most reinforcement learning methods are still powerless when faced with large and sparse action space. However, the existing real-world datasets limited in the number of diseases and the user goals. Therefore, order to show the advantages and effects of our method on large-scale datasets, we will conduct validity experiments on Synthetic datasets.
5.1 OVERALL PERFORMANCE
As shown in Table like Flat-DQN, most reinforcement learning methods face a difficult choice when facing this problem. Due to the large difference in the distribution of diseases and symptoms in the SymCat dataset, when the reward for the symptoms is too high, the agent will continue to inquiry symptoms without judging the disease. On the contrary, the agent will directly predict the discase and give up to search the sparse symptom space (like the Flat-DQN model reported in table). Both of these situations are limited help to real-world medical practice, but the balance between the two is difficult hold and the system is extremely unstable.
Compared to baseline models, our method takes more turns to have interactions with the user so that it collect more information of their implicit symptoms. With more information about implicit symptoms, our model significantly outperforms the other baselines in the diagnosis success rate, which shows method reasonable and stable.

---

Under review as a conference paper at ICLR 2022
Overall performance on SymCat dataset. We conduct all experiments times, and the reported number is the average. denotes missing numbers, and Acc., M.R, Avg. ' are the abbre- viations of Accuracy, Match Rate and Average Turns respectively.
Table 3:
Model
Acc.
M.R.
Avg. T
Flat-DQN
0.343
0.023
2.46
HRL-pretrained
0.452
6.84
Ours
0.504
0.495
12.96
5.2 STABILITY DISEASE CLASSIFIER
Because different diseases may have the same symptoms, it is impossible task to distinguish all diseases correctly. At the same time, the same type of disease usually have similar treatment regimen. Therefore, if the disease classifier can make the judgment that is close as possible to the correct disease, it can also help doctors roughly determine the type of disease. In order to have deeper analysis of the user goals which have been informed the wrong disease by the agent, collect all the wrong informed user goals and present the error matrix in Fig 3
It shows the disease prediction result for all the 9 groups We square is darker than the others, which means the wrongly predicted disease and the correct disease in the same group. This is reasonable because diseases in the same groups usually share similar symptoms and are therefore difficult to be distinguished. On the other hand, it also proved that even the model cannot make correct predictions, can still find right type of diseases to assist in real consultations.
the color of the diagonal
true group
12
13
14
19
predicted group
Figure 3: The error analysis for the disease classifier in different groups on our HRI. model, the square with true group and predicted group means discase in group is misclassified into group j by the disease classifier, the darker the color, the higher the value.
5.3 EFFICIENCY OF DIFFERENT WORKERS
While proving the effect of the disease classifier, we also hope that the workers in each group can also play a positive role in the judgment of the model. We evaluate the performance of workers in terms of success rate, average intrinsic rewards and match rate. Match rate means the proportion of actions which have requested about the implicit symptoms that the user has. The results can be seen in Appendix[A.2 A.2 We found most of the workers patient suffered. proves that workers in different groups can learn the symptom characteristics of the group, and use the knowledge guide the consultation process.
successfully exit by querying the symptoms which
8

---

Under review as a conference paper at ICLR 2022
6 RELATED WORK
This paper is related to two major research fields, namely hierarchical reinforcement learning and RI.-based methods for disease diagnosis.
Hierarchical reinforcement learning HRL has hierarchical policy and has been proposed to solve the problems with large action space. One classic framework of HRI. is options frame- work (Sutton et al., 1999), which involves abstractions over action space. At each step, the agent chooses either a one-step "primitive" action or "multi-step" action (option). Kulkarni et al. (2018) proposed hierarchical-DOU.
which integrates deep Q-learning into HRL, to learn policies for different levels simultaneously. HRL has been successfully applied to different tasks and reached promising results. (Zhang et al.) 2019, Wang et al. 2018: Zhang et al.) 2018 Takanobu et al. 2018 Feng et al.] 2018 Guo et al.J 2018). Most existing works decompose the corresponding task into two steps manually, where the first step is finished by high level policy while the second step is fin- ished by the low level policy. The task-specific design limits the generalization of HRL to complex tasks. Florensa et al. (2017) proposed a general framework that first learns useful skills (high level policies) in an environment and then leverages the acquired skills for learning faster in downstream tasks. What's more, some methods that generate or discover goals automatically when training the policies of two levels have been proposed. (Nachum et al. 2018 Florensa et al.] 2018 Tang al., 2018).
RL--based methods for disease diagnosis There are some previous works that applies the flat RL-based method in the medical dialogue system Wei et al. (2018); Xu et al.] (2019a); [Tang et al. (2016) Kao et al. (2018): Peng et al. (2018) and generates positive results. In the work of Wei et al. (2018) and Xu et al. (2019a). both symptoms and diseases are treated as actions and flat policy is used for choosing actions. After that, the HTC Research & Healthcare team did a lot of interesting work on this field. Kao et al. (2018) present context-aware hierarchical reinforcement method, using policy gradients to make decisions based on the patient's age, gender, season, and explicit symptoms. Peng al. (018) proposed reward shaping and feature rehuilding techniques to help agent effectively learn a better strategy andC et al. (2019) introduced a new multiple action policy representation to help agent suggesting medical tests to facilitate disease diagnosis. While pursuing a higher accuracy of disease diagnosis, researchers also hope to make the inquiry process more efficient and logical. In order to encourage the agent to locate the symptoms more specifically, Xu et al. (2199) introduce the knowledge graph into their dialogue system, Xia et al.) apply GAN as policy network capture the relations between different symptoms, these two works all reach a better accuracy than previous works.
7 CONCLUSIONS AND FUTURE WORK
In this work, we formulate disease diagnosis hierarchical policy learning problem, where symp- tom acquisition and disease diagnosis are assigned to different kinds of workers in the lower level of the hierarchy. A master model designed in the higher level that is responsible for triggering models in low level. We improve the previous model by introducing disease classifier and internal critic and then train them together to reach the optimal. Besides the exist dataset, we extend the Muzhi dataset and build a synthetic dataset to evaluate our hierarchical model. To the best of our knowledge, this is the first time both kinds of datasets are used for model evaluation and we make both datasets public. The experimental results on all datasets demonstrate that our hierarchical model outperforms other RL-baased model in both disease accuracy and symptom recall.
In the future, we would like to continue our research in three directions. First, we would like to make contribution on real-world dataset construction by introducing more diseases and annotate more samples. Second, although we can get higher symptom recall, the symptom inquiry sequence is still disorderly. We can try to use some special reward setting to make it more logical. Third, we only use MLP out deep learning network in this paper, and it would be interesting to integrate our hierarchical structure with other deep learning models and training strategies in the online self- diagnosis.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Y. E. Chen, K. F. Tang, Y. S. Peng, and E. Y. Chang. Effective medical test suggestions using deep reinforcement learning. 2019.
Finale Doshi-Velez. Yaorong Ge, and Isaac Kohane. Comorbidity clusters in autism spectrum dis- orders: an electronic health record time-series analysis. Pediatries, 133(L):e54-663, 2014.
Jun Feng, Minlie Huang, Vijie Zhang, Yang Yang, and Xiaoyan Zhu. Relation mention extraction from noisy data with hierarchical reinforcement learning. arXiv preprint /rXiv:/8/ (0/::////... 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint: 4XX1:1704.0301222011
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. 2018.
Mohammad Ghavamzadeh. Hierarchical reinforcement in continuous state and multi multi-agent environ- ments. 2005.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Weijie He, Xiaohao Mao, Chao Ma, José Miguel Hermández-Lobato, and Ting Chen. Fit: a fast and accurate framework for solving medical inquiring and diagnosing tasks. arXiv preprint arXiv:2012.01065, 2020.
Siddhartha Jonnalagadda, Abhishek K Adupa, Ravi P Garg, Jessica Corona-Cox, and Sanjiv J Shah. Text mining of the electronic health record: An information extraction approach for auto- mated identification and subphenotyping of hipef patients for clinical trials. Journal of cardio- vascular translational research, 10(3):313-321) 2017.
Hao-Cheng Kao, Kai-Fu Tang, and Edward Y Chang. Context-aware symptom checking for disease diagnosis using hierarchical reinforcement learning. 2018.
Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Sacedi, and Joshua B. Tenenbaum. Hicrarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv 2018.
preprint: arXiv: arXiv:/604.000
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 5187740):522, 2015.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. arXiv preprint: arXiv:1805.0822 2018
Ronald Parr and Stuart J Russell. Reinforcement Learning with hierarchies of machines. Advances in neural information processing systems, 1998.
Yu-shao Peng, Kai-Fu Tang, Hsuan-Tien Lin, and Edward Y Chang. Refuel: Exploring sparse features in deep reinforcement learning for fast disease diagnosis. 2018.
Chaitanya Shivade, Preethi Raghavan, Eric Foster-Lussier, Peter Embi, Noemie Elhadad, Stephen B Johnson, and Albert M Lai. review of approaches to identifying patient phenotype cohorts using electronic health records. Journal the American Medical Informatics Association, 21(2):221230, 2013.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.
Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. A hierarchical framework for relation extraction with reinforcement learning. arXiv preprint arXiv: 0e11////100995 2018
10

---

Under review as a conference paper at ICLR 2022
Da Tang, Xiujun Li, Jianfeng Gao, Chong Wang, Lihong Li, and Tony Jebara. Subgoal discovery 2018.
for hierarchical dialogue policy learning. arXiv preprint
Kai-Fu Tang, Hao-Cheng Kao, Chun-Nan Chou, and Edward Y Chang. Inquire and diagnose: Neural symptom checking ensemble using deep reinforcement learning. In Proceedings of NIPS Workshop on Deep Reinforcement Learning, 2016.
Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video captioning via hierarchical reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4213-4222, 2018.
Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-Fai Wong, and Xiangying Dai. Trie-tre dialogue system for automatic diagnosis. Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), volume 2. pp. 201-2077 2018.
Yuan Xia, Jingho Zhou, Zhenhui Shi, Chao Lu, and Haifeng Huang. Generative adversarial regular- ized mutual information policy gradient framework for automatic diagnosis.
Yuan Xia, Jingbo Zhou, Zhenhui Shi, Chao Lu, and Haifeng Huang. Generative adversarial regu- larized mutual information policy gradient framework for automatic diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1062-1069, 2020.
Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang, Jianheng Tang, and Liang Lin. End-to- end knowledge-routed relational dialogue system for automatic diagnosis. arXiv preprint: arXiv:19011 10623, 2019a.
Lin Xu, Qixian Zhou, Ke Gong, Xiaodan Liang, Jianheng Tang, and Liang Lin. End-to-end knowledge-routed relational dialogue system for automatic diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 7346-7353, 2019b.
Jiaping Zhang, Tiancheng Zhao, and Zhou Yu. Multimodal hicrarchical reinforcement learning policy for task-oriented visual dialog. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pp. 140-150, 2018
Jing Zhang, Bowen Hao, Bo Chen, Cuiping Li, Hong Chen, and Jimeng Sund. Hierarchical rein- forcement learning for course recommendation in moocs. Psychology, 5-4.64))5-65,, 2019.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
The for master and all the workers are all set to 0.1. For the master, the maximal dialogue turn T is set to 22, it will receive a extrinsic reward of +1 if the master inform the right disease. Otherwise, will receive a extrinsic reward -1 if the dialogue turn reaches the maximal turn or wrong disease is informed. At non-terminal turns, the extrinsic reward is O. Moreover, the sum of the extrinsic rewards (after reward shaping) over one subtask taken by worker will be the reward for master. The maximal dialogue turn T'sh is set to 5 for each worker.
For master and all the workers, the neural network of DQN is three-layer network with two dropout layers and the size of hidden layer is 512, learning rate for the DQN network is set to 0.0005. All parameters are set empirically and settings for the datasets are the same. In addition, all the workers are trained every 10 epochs separately during the training process of master. For the disease classifier, the neural network is two-layer network with a dropout layer and the size of hidden layer is 512, learning rate for the network is set to 0.0005. Moreover, it's trained every epochs during the training process of master.
During the training process, it will take about 5000 epochs for the model to reach convergence, which takes about 18 hours given one GPU, For the best-performing model, y is set to 0.95, dis- counted factor Yhe is set to 0.99, À in reward shaping is set to +1.
A.2 THE PERFORMANCE OF DIFFERENT WORKERS IN SYMCAT DATASET.
11

---

Under review as a conference paper at ICLR 2022
Group id
Success rate
Ave intrinsic reward
Match rate
Activation times
48.6%
0.031
16.74%
0.615
4
54.6%
-0.150
5.02%
0.375
5
38.8%
-0.013
7.96%
3.252
6
48.0%
-0.036
9.58%
0.942
7
48.3%
0.057
18.57%
1.280
12
43.0%
0.021
11.26%
0.666
13
52.4%
-0.138
7.18%
0.823
14 72.2%
-0.111
3.77%
0.614
19
47.4%
0.031
22.72%
1.124
Average
50.3%
-0.041
10.49%
1.077
Table 4: The performance of different workers in in Symcat dataset.
12


Reviews:
[{"review_id": "nvAUqJrH1SM", "reviewer": "Reviewer_eBvJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The task of automatic disease diagnosis that this work focuses on is worth exploring and has a very potential beneficial impact on society. \n2. The paper is well organized and easy to understand\n\nWeaknesses:\n1. My main concern is the technical novelty of this work. This work is more like a combination of existing technologies. HRL, User Simulator, and Reward Shaping are all adopted in previous RL-based methods for disease diagnosis. The author's contribution on methodology is quite limited.\n\n2. In section 4.3 ablation studies, more components (e.g., Reward Shaping) need to be considered to make the experimental results more convincing.\n\n3. It would be helpful if the authors could provide some examples to understand how their systems behave.", "summary_of_the_paper": "This paper introduces hierarchical reinforcement learning (HRL) into automatic disease diagnosis, which reduces the action space and improves training efficiency. Besides, the authors also expand an existing public dataset and build a synthetic dataset for evaluation. The Experimental results show that their proposed hierarchical framework achieves higher accuracy and symptom recall in disease diagnosis than existing several baselines.\n", "summary_of_the_review": "This work focuses on a meaningful task and constructs datasets for better evaluation, but the limited contribution at the methodological level makes me inclined to think that it has not yet reached the bar for a top conference.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gN5kC6hklxG", "reviewer": "Reviewer_J3rx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " \n##########################################################################\n\nPros:  \n\n1. The paper takes one of the most important task, i.e., automatic disease diagnosis. However, the proposed approach only works at a semantic-level and is not an end-to-end approach. That is, it does not consider the NLU and NLg modules of the typical goal-oriented dialog systems and the paper is not clear about it. It took me a long time figure out what they are trying to do.\n\n 2. Experiments are provided to show that the proposed methods work well. However, I am concerned about whether they are actually comparing against state-of-the-art systems?\n\n3. The proposes a nice application of the hierarchical reinforcement learning.\n\n \n##########################################################################\n\nCons: \n\n1. The paper claims that in the conventional RL tasks, the action is not big, whereas the action space in disease diagnosis huge. However, there exists several works in RL where action space is huge, way bigger than the proposed work. Examples include generating natural language text, where action is equal to the vocabulary size. How does this work differ than those from the technical point of view.\n\n2. Regarding comparison with the SOTA systems, I am familiar with at least one paper (there must be many out there) from EMNLP 2020. The title of the paper is: MedDialog: Large-scale Medical Dialogue Datasets. \n\n3. From a technical point of view, I am not sure what is contribution of the paper.\n\n4. In the experiments, the paper shows that having large number of turns as a positive sign, whereas in the goal-oriented dialog systems literature, the less number of turns in a conversation is better (as it is efficient to get job done quicker). However, the open-domain dialog systems literature is different where the goal is have longer conversations. \n\n########################################################################## \n\n\n", "summary_of_the_paper": "##########################################################################\n\nSummary:\nThis works proposes dialogue policy learning by incorporating a hierarchical policy. The proposed hierarchical reinforcement learning-based approach has two models, i.e., one master model that instantiates low models. The low models have several symptoms checkers and disease classifier. Experiments are conducted on both real and synthetic datasets to demonstrate the efficacy of the proposed work. \n \n##########################################################################", "summary_of_the_review": "########################################################################## \n\nReasons for score: \n \nOverall, I vote for rejection. I like the application side of the paper. But, technically, I am unable to see significant contribution. Moreover, the paper lacks in comparison with the state-of-the-art systems in the relevant domain.\n\n\n########################################################################## \n\nQuestions for rebuttal:\nFeel free to answer to all the concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E4nHjyoDO6Y", "reviewer": "Reviewer_WycQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This paper is well organized and easy to follow.\n- The proposed method is well motivated and achieved strong empirical results in terms of disease classification accuracy and recall of symptoms.\n\nCons:\n-  I am not very familiar with RL literature so I might be wrong. But to me, the proposed method looks like a straightforward application of HRL. Therefore, the technical contribution of the work is weak.\n- Regarding the experimental results, I found that the proposed method requires significantly higher *Averaged Turns* for disease diagnosis compared to other baselines (as shown in Table 2). I don't understand why the authors bold the highest *Averaged Turns* in Table 2. But in task-oriented dialogues, accomplishing a user goal with fewer turns is better. Otherwise, the system could have checked all the possible symptoms one by one before making a final decision.\n", "summary_of_the_paper": "This paper applies Hierarchical Reinforcement Learning (HRL) to automatic disease diagnosis in task-oriented dialogues setting. The authors argue that applying RL to automatic disease diagnosis is challenging because the action space (i.e., symptoms) is very large.\nTherefore, they propose to learn a hierarchical dialogue policy where the high level policy is for categorizing patents into different groups and the low level policy is for checking symptoms and classifying diseases within a group.  Experimental results on multiple datasets show that the proposed HRL strategy achieve higher disease diagnosis accuracy compared to existing RL systems.\n\n\n\n\n", "summary_of_the_review": "Overall, this is an OK work with limited novelty. I have some concerns about experimental results (see Main Review) which need to be clarified in the author's response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DlLu9Gn9N0", "reviewer": "Reviewer_65Z2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed HRL approach achieves better performance compared to the existing baselines.\n2. New synthetic data was proposed for the task and can be useful for future study.\n\n\nWeaknesses:\n1. The idea of hierarchical RL is not particularly novel. It seems from Table 2, the existing best baseline is FIT, which is a non-RL approach, so it's not clear to me why RL is necessary to solve the automatic disease diagnosis. Since a non-RL approach achieves the best result, maybe we should develop on the non-RL approach?\n2. It's worth showing some dialogue examples for the audience to understand what automatic disease diagnosis works under a task-oriented dialogue setting. It's not clear how the language generation part is handled in both simulated setting and real-life settings.\n3. Many metrics are missing from the baselines to get a fair comparison in Table 2.\n\n\nQuestions:\n1. Is the \"disease discriminator\" the same thing as \"disease classifier\"?\n2. What does \"user goal\" mean under the task context? Are the users trying to get some treatment?\n3. What exactly is the match rate of the dialogue? Is average turns the higher the better? It seems it's better if the diagnosis can be solved in reasonable amount of time instead of the longer the better?\n\n\nTypo: \n2.1.5, \"base on\" --> \"based on\"\n", "summary_of_the_paper": "The paper tackles the problem of automatic disease diagnosis through reinforcement learning under a setting of task-oriented dialogues. The authors proposed to integrate hierarchical policies with two levels (one high-level master model, and one low-level policy) into the dialogue policy learning. Experiments on both real-life and synthetic data suggest the proposed approach is effective.", "summary_of_the_review": "Overall I think the paper addresses an important problem, but it needs a bit more justification on why RL is necessary for this type of tasks, and some details of the paper can be clarified further.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper applies a reinforcement learning (RL) approach to a medical diagnosis dialog task. Motivated by a large action space, the authors utilize a hierarchical model where the higher level model triggers a lower level model comprising of symptom checkers and disease classifiers. They evaluate their approach on real-world and synthetic data sets.

Pros
+ The application (societal relevance) and the hierarchical approach (large action space) are motivated well
+ The paper is presented relatively clearly (with caveats: see reviewer comments) and improves performance over reasonable baselines (with caveats over one metric: why longer dialog is better?)

Cons
- The novelty of the work was not entirely clear, other than the application to a new task
- Lack of examples make it difficult to gauge the complexity of the task
- Ablation studies would also have provided better insight into task and the proposed model

The reviewers have several concerns about the work described in the paper. But the authors did not provide any response unfortunately.
Decision: Reject