CoAD: Automatic Diagnosis through Symptom and Disease Collaborative Generation
Year: 2023

Abstract:
Automatic diagnosis (AD), a critical application of AI in healthcare, employs machine learning techniques to assist doctors in gathering patient symptom information for precise disease diagnosis. The Transformer-based method utilizes an input symptom sequence, predicts itself through auto-regression, and employs the hidden state of the final symptom to determine the disease. Despite its simplicity and superior performance demonstrated, a decline in disease diagnosis accuracy is observed caused by 1) a mismatch between symptoms observed during training and generation, and 2) the effect of different symptom orders on disease prediction. To address the above obstacles, we introduce the CoAD, a novel disease and symptom collaborative generation framework, which incorporates several key innovations to improve AD: 1) aligning sentence-level disease labels with multiple possible symptom inquiry steps to bridge the gap between training and generation; 2) expanding symptom labels for each sub-sequence of symptoms to enhance annotation and eliminate the effect of symptom order; 3) developing a repeated symptom input schema to effectively and efficiently learn the expanded disease and symptom labels. We evaluate the CoAD framework using four datasets, including three public and one private, and demonstrate that it achieves an average 2.3% improvement over previous state-of-theart results in automatic disease diagnosis. For reproducibility, we release the code and data at https://github.com/KwanWaiChung/coad.

heading: 1 Introduction
text: The healthcare industry worldwide is facing an acute shortage of healthcare professionals such as doctors, nurses, and other staff, which results in millions of people not receiving the care they need, particularly in low-income countries (World Health Organization, 2016). Artificial intelligence (AI)
∗Equal Contribution
has the potential to revolutionize medicine by automating tasks traditionally done by humans, reducing the time and cost of such tasks. Automatic diagnosis (AD) is a valuable application of AI in healthcare that aims to improve patient outcomes. When deployed on mobile devices, the AD agent functions as a chatbot, querying patients about their symptoms and health concerns, and directing them to the appropriate care based on the diagnosis. This allows for faster treatment decisions, prompt notification of care teams, and increased communication between providers, ultimately leading to improved patient outcomes and potentially saving lives. The process of AD can be conceptualized as a series of questions and answers. As depicted in Figure 1, the diagnosis begins with the patient reporting initial symptoms (in this example, only a headache). The AD agent then determines whether to ask for additional symptoms or provide a disease diagnosis. The agent carefully selects relevant queries to gather more information about the patient’s condition. In this illustration, the agent chose to inquire about two specific symptoms, “reduced or lost sense of taste” and “runny nose”, and received positive responses from the patient. The AD agent has two objectives during the question-and-answer process for diagnosis. Firstly, it strategically selects symptoms that provide the most information for disease discrimination. Secondly, it aims to identify the disease as accurately as possible. These goals are interdependent. If the agent fails to thoroughly gather information about the patient’s symptoms, it risks missing crucial information related to the un-
6348
derlying disease, resulting in an incorrect diagnosis. On the other hand, an accurate disease diagnosis enables the agent to cluster symptoms associated with the disease, thereby improving its ability to ask relevant symptom inquiries.
A significant group of existing methods approaches AD as a sequential decision-making problem and employs reinforcement learning (RL) to solve it (Kao et al., 2018; Peng et al., 2018; Xu et al., 2019; Zhong et al., 2022). These methods focus on enhancing the RL agent’s performance in disease diagnosis by incorporating medical knowledge, fine-tuning reward functions, and utilizing hierarchical model structures. However, they often overlook the agent’s ability to gather symptoms. In reality, it is challenging for an RL agent to simultaneously optimize for both accurate symptom inquiry and disease diagnosis. We observed in our experiments that there were fluctuations in the performance of diseases and symptoms and the RL agent frequently fell into local minima. Additionally, many RL methods use an immediate reward of “-1” to encourage shorter turns during training, which may be beneficial for other task-oriented dialogs such as ticket-booking, but is unhelpful for training the agent in symptom inquiry. As a result, most RL-based agents only ask for one or two symptoms before prematurely moving on to disease prediction. Insufficient symptom information not only leads to incorrect disease judgment but also diminishes the reliability and interpretability of the decision. To address the limitations of RL-based methods, Chen et al. (2022) proposed a generation model to acquire symptom inquiry logic and three training tasks to train the AD agent to master symptom generation rules. Their method achieved competitive results, particularly in symptom recall, which confirms the superiority of generation-based models for diagnosis modeling. However, they did not consider the benefits of disease information.
In this paper, we propose CoAD, a Transformerdecoder-based framework for Automatic Diagnosis that collaboratively generates disease and symptom information. CoAD utilizes three key techniques to enhance performance. First, it aligns disease labels with symptom inquiry steps, bridging the gap between training and generation for improved disease diagnosis. Second, it expands symptom labels to each sub-sequence with subsequent symptoms, making training more robust to varying symptom orders. Lastly, it incorporates a repeated symptom
input schema and an additional symptom attention module for simultaneous optimization.
Our main contributions include:
• A novel framework that effectively aligns disease labels with symptom steps, bridging the gap between training and generation.
• A symptom label augmentation mechanism that strengthens training signals and enhances disease diagnosis, regardless of symptom order.
• An approach that combines repeated symptom input and symptom attention schema for concurrent symptom and disease generation.

heading: 2 Related Work
text: RL-based approaches. Early work in automatic diagnosis often used the Markov decision process framework and employed reinforcement learning (RL) to solve the problem. For example, Wei et al. (2018) approached it as a task-oriented dialog task and trained an agent using deep Q-learning to collect implicit symptoms during patient interactions. To improve the poor efficiency of RL-based methods, Kao et al. (2018) added contextual information to the dialog state and applied a hierarchical RL schema by dividing diseases into different body parts. Similarly, Zhong et al. (2022) employed a hierarchical setting, using a master model to trigger a low-level model comprised of symptom checkers and a disease classifier. Peng et al. (2018) developed an RL-based diagnosis model that incorporated reward shaping to guide the search for better policies and feature rebuilding to improve the agent’s ability to learn correlations between features. They also integrated domain knowledge to make more accurate decisions. Similarly, Xu et al. (2019) demonstrated that incorporating medical knowledge through a knowledge-routed graph improved diagnosis performance. Furthermore, Xia et al. (2020) applied a GAN-based approach to the diagnosis problem, using the generator as a policy network, the discriminator as part of the reward function, and mutual information to encourage the model to select the most discriminative symptoms for diagnosis.
Generation-based approaches. Generationbased diagnosis methods have shown promise in their ability to predict symptoms with stronger performance compared to reinforcement learning
(RL)-based methods. For example, Lin et al. (2019) used a Bi-LSTM to encode symptom word sequences and trained a sequence-to-sequence model with a CRF decoder to recognize symptoms based on a symptom graph, document-level and corpus-level attentions. More recently, Chen et al. (2022) aimed to alleviate the inefficiencies of exploration and sparse rewards in RL by formulating the diagnosis process as a sequence generation problem and using a Transformer-based network to learn symptom sequences and disease classifiers through three training tasks. Their model demonstrated significant improvements over RL-based counterparts.

heading: 3 Sequence Generation based AD
text: The process of AD involves predicting a disease by asking a series of questions about potential symptoms from a patient who has provided initial symptoms (Peng et al., 2018). AD aims to optimize for two goals simultaneously: 1) asking questions to gain the most information about the patient’s condition, and 2) identifying the disease quickly and accurately. In this study, we approach the problem of AD as a combined task of generating a sequence of symptoms and classifying the disease. Let S denote the set of symptoms and D denote the set of diseases. An AD dataset considers a set of possible symptom profiles: Xs = s1E , · · · , sNE , s1I , · · · , sMI with the symptoms’ status of True, False, Uncertain, where siE , (i = 1, 2, · · · , N) are the initial reported symptoms (i.e., explicit symptoms), and N is the number of explicit symptoms; siI , (i = 1, 2, · · · ,M) are the subsequent acquired symptoms (i.e., implicit symptoms), and M is the number of implicit symptoms. In the proposed method, CoAD, we represent a symptom status as 1 if the patient confirms having that symptom, 2 if the patient does not have it, or 0 if the patient is uncertain about having it or not. During diagnosis, CoAD will inquire about a symptom s ∈ S or produce an end token, signaling the end of symptom checking and switching to predicting a disease d ∈ D.
During the diagnosis process, the AD system that interacts with a patient can be thought of as an agent. The agent’s goal is to ask questions about key symptoms that will lead to a quick and accurate disease diagnosis. Since symptoms are acquired in chronological order, it is common to factorize the joint probabilities of symptoms as the product of
conditional probabilities:
Pθ(s1:MI |s1:NE ) = M∏
K=1
Pθ(sKI |s1:K−1I , s1:NE ), (1)
where s1:NE denotes {s1E , · · · , sNE } and s1:MI denotes {s1I , · · · , sMI }. In practice, the probabilities can be approximated by a network parameterized by θ. Once enough symptoms that aid in distinguishing the disease has been acquired, the agent proceeds to make a disease prediction. Let d∗ denote the given disease label of the symptoms. The agent’s goal is to learn a strategy that can select d∗ with a high probability Pθ(d∗|s1:MI , s1:NE ).
Disease Accuracy, Symptom Recall, and Combined Score. Enhancing symptom recall may lead to a decrease in disease accuracy, meaning that the agent’s diagnostic performance could suffer when it achieves optimal symptom recall. This is because most previous models with higher symptom recall tend to have longer turns in a limitedturns setting. Longer sequences are less common in the training set, resulting in a distribution mismatch between training and testing that hinders the model’s ability to accurately identify diseases. Consequently, disease accuracy and symptom recall tend to have an inverse correlation during testing in limited-turns settings. To comprehensively measure the diagnosis performance, we introduce a combined score, Cs = 2·Rc·Ac(Rc+Ac) , which indicates the tradeoff between high disease-prediction accuracy and high symptom-acquisition recall.

heading: 4 Symptom and Disease Collaborative Generation Framework
text: Even though the generation-based method has great potential in AD (Chen et al., 2022), applying this method to the disease diagnosis task faces two challenges. First, the disease classifier is trained on a complete symptom sequence but is only tested on partial symptoms, which creates a gap between the visible symptoms in training and inference. Second, the order of symptoms in the training set can be inconsistent with the generated ones, which can lead to a wrong diagnosis if the symptoms in different order are incorrectly identified as different symptom sequences. To address the first challenge, we propose a disease and symptom collaborative generation framework, in which the disease label d∗ is expanded to multiple possible symptom steps
to fill the gap of supported symptoms between training and generation. We refer to this disease label expansion procedure as d-label alignment. To tackle the second challenge, we present a symptom label augmentation mechanism (called s-label augmentation) that enriches the annotation and eliminates the impact of the symptoms order. Furthermore, we design a repeated symptom input schema for efficient and effective learning from the expanded disease and symptom labels. Our strategies are illustrated in Figure 3, in which a Transformer decoder takes the repeated symptom embedding, symptom status embedding, and symptom mask as input. Additionally, a symptom head and a disease head are equipped to predict the slabels and d-labels, respectively. We will provide more details in the following sections.
4.1 d-label Alignment
The idea behind d-label alignment is straightforward: we assign the disease label d∗ of a symptom sequence {s1:NE , s1:MI } to each available implicit symptom sKI ∈ s1:MI . A symptom s is considered available if the sub-sequence {s1:NE , s1:KI } is not present in the training set. The number of generated symptoms is much less than the number of symptoms that a patient actually has. The key to making this work is through data augmentation, transforming one symptom sequence with one disease label into multiple symptom sequences corresponding to the same disease label. This augmentation helps to fill the gap in symptoms supporting diagnoses between training and generation. At the same time, filtering out the unavailable symptoms reduces the impact of d-label alignment on the samples with disease labels. Additionally, d-label alignment results in a disease generation task, since each symptom corresponds to a disease label if we assign a special token to unavailable symptoms that are ignored during training. This enables the joint probabilities of disease sequences over symptoms to be factored as the product of conditional probabilities:
Pθ(d∗, d1:M |s1:MI , s1:NE ) = Pθ(d∗|s1:MI , s1:NE ) M∏
K=1
Pθ(dK |s1:K−1I , s1:NE ),
(2) where d1:M are the assigned disease labels to the implicit symptoms, and d∗ is the originally given label for the piece of the sample.
4.2 s-label Augmentation
As mentioned above, the order of symptoms can potentially affect diagnosis accuracy. In this section, we investigate whether we can use s-label augmentation to provide better training signals to reduce the effects of symptom order. s-label augmentation is based on the assumption that determining a disease is independent of the order of symptoms. Taking advantage of this disorder, we perform additional data augmentation with symptom labels. A sub-sequence of symptoms s1:KI not only has the symptom label of the next symptom sK+1I , but also of the subsequent symptoms sK+1:MI . Instead of using the original single symptom label of {s1:NE , s1:MI }, we replace it with all possible implicit symptom labels and rewrite it as:
M︷ ︸︸ ︷ s1I , · · · , sMI , · · · , M−K︷ ︸︸ ︷ sK+1I , · · · , sMI , · · · , 1︷︸︸︷ sMI ,#
where # is the token ignored during training. We denote the expanded symptom labels as s̃1:M ′I , where M ′ is calculated as M ′ = (1+M)·M/2+1. Then the joint probabilities of s̃1:M ′I over the symptoms are as follows:
Pθ(s̃1:M ′I |s1:NE ) = M∏
K=1
M∏
T=K
Pθ(sTI |s1:K−1I , s1:NE ).
(3)
Similarly, the aligned disease labels can also be expanded to align with s̃1:M ′I as the following:
M︷ ︸︸ ︷ d11, · · · , d1M , · · · , M−K︷ ︸︸ ︷ dK+1K+1, · · · , dK+1M , · · · , 1︷ ︸︸ ︷ dM−1, d∗,
where dKT is given following the d-label assignment rules, i.e., it is d∗ if sTI is the available symptom of sequence {s1:NE , s1:K−1I , sTI }, else it is #. We denoted the expanded aligned disease labels as d̃1:M ′ , then the joint probabilities of d̃1:M ′ over the symptoms are given as:
Pθ(d∗, d̃1:M ′ |s1:MI , s1:NE ) = Pθ(d∗|s1:MI , s1:NE ) M∏
K=1
M∏
T=K
Pθ(dKT |s1:K−1I , s1:NE ).
(4)
Repeated Symptom Input. Now that we have obtained the expanded symptom and disease labels, we will explain how to expand the input symptoms to align with the labels in an auto-regression generation model. The solution is straightforward, as shown in Figure 2, we simply repeat the symptoms as many times as their corresponding expanded symptom or disease labels. For example, in Figure 2, the last explicit symptom Sneezing has 3 extended symptom labels Allergy,Rash and Dyspnea, thus it will be repeated 3 times. Formally, an implicit symptom sKI will be repeated M −K + 1 times. To this end, the repeated symptom inputs are represented as:
M︷ ︸︸ ︷ sNE , ··, sNE , M−1︷ ︸︸ ︷ s1I , ··, s1I , · · ·, M−K︷ ︸︸ ︷ sKI , ··, sKI , · · ·, 1︷ ︸︸ ︷ sM−1I , s M I .
We denote the repeated symptom input as s1:M ′ rept, the mappings between s1:M ′ rept and d̃1:M ′ as well as s1:M ′
rept and s̃1:M ′
I . In addition to the repeated symptom input, extra symptom attention is integrated to mask the redundant input symptoms. Specifically, in each multi-head attention of the Transformer block, each repeated symptom can only see itself, the explicit symptoms, and only one of the previous repeated symptoms. Formally, the representations of symptom tokens are updated in multi-head attention as:
(sK ′ I(rept)) l    ← MH-Attn(Q = (sK′I(rept))(l−1), KV = [(sXI(rept)) (l−1), (s1:N−1E ) (l−1)]) if K > M ← MH-Attn(Q = (sK′I(rept))(l−1), KV = [sK ′ I(rept)) (l−1), (s1:N−1E ) (l−1)]) else, (5) where Q,K, V are the query, key, and value in multi-head attention respectively. [.] represents the concatenation along the symptom sequence dimension, (sK
′ I (rept))
l indicates the lth Transformer block layer output of the K ′th repeated symptoms input, and X = { (K+1)(2·M−K)2 },K = 0, 1, · · · , such that X < K ′. Significantly, sM
′ I(rept) is anchored for the final diagnosis with only disease label d∗, its representation of the lth layer is updated in multi-head attention as MH-Attn(Q = (sM
′ I(rept)) (l−1),KV =
[(s { (K+1)(2·M−K) 2 ,M ′}
I(rept) ) (l−1), (s1:N−1E ) (l−1)]), where K = 0, · · ·,M .
The symptom attention mechanism is a key aspect of the repeated symptom input, functioning as a form of data augmentation akin to dropout noise in accomplished by Transformer training (Vaswani et al., 2017). This is achieved by placing dropout
masks on both the fully-connected feed-forward layers and the attention probabilities. During repeated symptom generation, dropout noise serves as data augmentation by independently sampling dropout masks (Gao et al., 2021). This process involves feeding the same samples to the decoder multiple times and performing data augmentation.
It is important to note that the key distinction between our approach and a simple symptom permutation method lies in the dropout noise applied to the explicit symptoms. In the permutation approach, different representations of the explicit symptoms are used as context due to varying dropouts applied to each separate augmented sample. In contrast, our method with the addition of the S-label utilizes the same representations of the explicit symptoms as context for all subsequent implicit symptoms. Maintaining the stability of the explicit symptoms’ representation is essential for the model to learn more effectively.
Collaborative Generation Loss. We follow the auto-regression framework in (Vaswani et al., 2017) and take a cross-entropy objective to learn the expanded labels d̃1:M ′ and s̃1:M ′I jointly as minimizing the following loss:
Lθ = − M∑
K=1
M∑
T=K
WK · [ logPθ(sTI |s1:K−1, sNE )+
logPθ(dKT |s1:K−1, sNE ) ] + logPθ(d∗|s1:M , sNE ) = − M ′∑
K′=1
WK ′ · [ logPθ(s̃K′I |s1:K ′−1 rept , s N E )
+ logPθ(d̃K′ |s1:K ′−1 rept , s N E ) ] ,
(6) where WK = 1M−T+1 is the weight added to normalize the ∑M T=K logPθ(sTI |s1:K−1, sNE ), and WK ′
is the weight to normalize the losses of the expanded labels in the repeated symptom input. It can be set as 1M−n−1 ,where n(2·M−n+1) 2 < K
′ ≤ (n+1)(2·M)
2 .

heading: 5 Experiments
text: Datasets. We evaluate our model on three public datasets: Dxy (Xu et al., 2019), MuZhi (Wei et al., 2018), MuZhi-2, and one in private—Ped. We only consider datasets collected from real clinical practice and exclude the Synthetic dataset (Liao et al., 2020). Additional detail of the data is given in Appendix 1.
Baselines. In order to evaluate the effectiveness of RL-based AD models, we utilize several established techniques as benchmark models including: DQN (Wei et al., 2018) agent which uses Deep QNetwork to learn an agent that chooses a symptom to inquire about or outputs the disease diagnosis; PPO (Schulman et al., 2017) agent which is a Proximal Policy Optimization (Schulman et al., 2017) based agent that learns diagnosis model with a GPT2 backbone, and Hierarchical DQN (HDQN) (Liao et al., 2020) agent which is a DQN-based hierarchical policy that has a master selecting the disease and a separate worker for each disease to inquire symptoms. Additionally, we include the current leading Transformer-based model, Diaformer (Chen et al., 2022) agent which formulates AD as a sequence generation task and learns a Transformerbased model to learn the generation of symptom inquiry and disease diagnosis.
Ablation Study. Some CoAD’s variants includes GPT-2dis agent which is based on GPT-2 (Radford et al., 2019) that generates symptoms in an autoregressive manner and predicts the disease at the end; CoADw/o s agent which is a variant of CoAD that is trained without s-label using the same hyperparameters of CoAD; CoADw/o d agent which is a variant of CoAD that is trained without dlabel using the same hyperparameters of CoAD. Meanwhile, a BERTfull (Devlin et al., 2019) agent which uses BERT as the backbone to classify disease given the full ground truth symptoms is compared.

heading: 5.1 Experiment Setup
text: Evaluation Metrics. We evaluate the performance of the models using four metrics: implicit symptom recall, disease accuracy, average inquiry turns, and a combined score. The combined score is calculated as the harmonic mean of disease accuracy and implicit symptom recall, providing an overall measure of the model’s effectiveness in both disease diagnosis and symptom inquiry. To ensure the reliability of the findings, all evaluation results are derived from the average of five distinct groups.
Training Setting. To gain insight into the behavior of the models under different conditions, we conduct experiments in both limited turn and fixed turn modes. In the fixed turn mode, the models are required to inquire about symptoms within a fixed number of turns. In the limited turn mode, there is a maximum turn limit for symptom inquiry, but
the models are allowed to stop before reaching the limit. To ensure a fair comparison with the previous state-of-the-art Diaformer (Chen et al., 2022), we use the same turn numbers (5, 10, 15, and 20) in our experiments. Additional implementation is shown in Appendix B.
Main results. Table 1 shows the evaluation results on four AD number of 20 turns. For the models DQN and H-DQN, we cite the results from (Chen et al., 2022) where only the results for Dxy and Muzhi are available. The main results indicate that the proposed CoAD model achieves the highest disease accuracy, symptom recall, and combined score on all datasets, demonstrating the effectiveness of the d-label and s-label for AD. Compared to the previous state-of-the-art model, Diaformer, CoAD shows a significant improvement in disease accuracy, for example, in the Ped dataset the gain of CoAD is 3.92%. The improvement in disease accuracy is a result of the combined effects of the d-label and s-label. Specifically, the s-label helps CoAD capture symptom relationships in different sequences, while the d-label helps CoAD generalize better to unseen symptom sequences. Furthermore, CoAD achieves a substantial improvement in symptom recall in all datasets over Diaformer with at least 10% improvement, highlighting the potential of repeated symptom input in improving the model’s ability to inquire about appropriate symptoms during diagnosis. Notably, both Muzhi-2 and Ped contain negative symptom statuses, adding complexity to the symptom sequence input and providing more challenges to learning the relationships between symptoms and the target disease. However, both d-label alignment and s-label augmentation consistently bring improvements regardless of
the setting. It is worth noting that all the reinforcement learning (RL) methods present poor performance in terms of symptom recall, which is not unexpected as they tend to inquire about a limited number of symptoms and stop at early turns. The early stop of RL-based methods can be attributed to intermediate negative rewards, which incentivize the model to end in as few turns as possible for efficiency. On the other hand, generation-based methods are generally capable of probing more symptoms compared to RL-based methods. Notably, CoAD achieves the best performance with shorter turns in comparison to other generation-based methods, demonstrating the effectiveness and efficiency of CoAD’s diagnostic logic.
Finally, the BERTfull model is trained to predict the disease based on the sequence of ground truth symptoms. Intuitively, we expected this model to provide the theoretical upper bound for disease accuracy. However, both Diaformer and CoAD outperform BERTfull in all datasets except Muzhi2. This is not surprising as there can be irrelevant symptoms that negatively impact the accuracy of decision-making for disease diagnosis. Additionally, BERTfull model lacks the ability to distinguish the symptoms that are relevant to the final diagnosed disease.
Diagnosis with smaller limited turns. Table 2 presents the results of automatic diagnosis with smaller limited turns (i.e., 5, 10, 15) on the four datasets. The results indicate that CoAD consistently outperforms Diaformer in terms of disease accuracy, symptom recall, and combined score, demonstrating the robustness of the proposed methods under different limited turn settings. Specif-
ically, CoAD shows significant improvements in disease accuracy and symptom recall compared to previous results. As the allowed turns increase, we observe a monotonic improvement in both disease accuracy and symptom recall, indicating that CoAD can provide a better-quality diagnosis with more turns.
Diagnosis with fixed turns. For fairer comparisons, we evaluate the models under a fixed number of turns and the results are presented in Figure 4 in Appendix C. We choose the fixed turns as 5, 10, and 15. The DQN-based method (Wei et al., 2018) is not considered as it uses a single action space for both disease and symptoms, making it only suitable for limited turn settings. Overall, the results indicate that CoAD achieves the best performance across all evaluation metrics on all datasets with different fixed turns (except for the disease accuracy of Muzhi with a fixed turn of 15). Notably, the improvement is more substantial in shorter turns (8.2% in 15 turns vs 2.5% in 5 turns increase on average over Diaformer), showcasing the strength of CoAD in real-life deployment scenarios where efficiency is crucial.
In terms of symptom recall, CoAD consistently outperforms the other models by a wide margin.
The largest improvement is 24% on the Muzhi dataset with five fixed turns over Diaformer. In contrast, the improvement in disease accuracy is relatively modest. The significant improvement in symptom recall aligns with expectations, as the d-label augmentation encourages CoAD to explore the relationship between intermediate symptoms and the final disease during training, allowing CoAD to inquire about the most relevant symptoms for distinguishing different diseases. As a result, CoAD is able to make correct diagnostic decisions even with insufficient symptoms. Therefore, when CoAD is forced to inquire about more symptoms in the fixed turns, the additional symptoms provide less value in diagnosis compared to other models.
Ablation Studies. To further understand the contributions of the different components of CoAD, we conduct a series of ablation studies to isolate the effects of s-label augmentation and d-label augmentation. In Table 1 and Table 2, we can observe that CoADw/o d consistently improves symptom recall across all datasets and different limited turns. These results indicate that the s-label is effective in guiding the model to inquire about informative symptoms, leading to more accurate diagnoses and, as a result, better disease accuracy in most settings.
On the other hand, CoADw/o s improves disease accuracy over GPT-2dis in most settings (e.g., an improvement of 12% in the Ped dataset), highlighting the effectiveness of d-label augmentation. Furthermore, the combination of d-label and s-label is beneficial as CoAD achieves the best disease accuracy and combined score in all settings, and better symptom recall in most cases.

heading: 6 Conclusions
text: This paper introduces CoAD, a symptom and disease co-generation framework, which significantly improves the state-of-the-art in symptom prediction and disease determination for automatic diagnosis. CoAD addresses the discrepancy between visible symptoms during training and generation through disease label alignment, mitigates the impact of symptom order on diagnosis through symptom label augmentation, and utilizes a repeated symptoms input schema to enable the model to simultaneously learn aligned diseases and expanded symptoms efficiently. CoAD presents a novel approach to data augmentation by reusing labels and text input, and it can be extended to other joint learning tasks for generation and classification.
Limitations
In this work, we have identified two key limitations of CoAD that can be further examined in future research. The first limitation is that CoAD only allows for the querying of one symptom at a time, making it unsuitable for scenarios where multiple symptoms are present. However, CoAD has superior performance in the main metrics for automatic diagnosis. To relieve this limitation, potential solutions include relaxing symptom feedback conditions and allowing the model to produce symptoms sequentially until a stop signal is encountered or querying the top K symptoms in a single turn.
Additionally, CoAD has some restrictions on input format, requiring standardized symptoms and values. To make it more applicable to end-to-end settings, an natural language understanding module (NLU) is required to parse plain text and obtain the input symptom sequence, and a natural language generation (NLG) module is needed to translate the predicted symptom or disease to text. The ultimate goal of automatic diagnosis is to support the dialogue between doctors and patients, after CoAD determines the symptom or disease, rulebased NLU and NLG modules can help to achieve
the text to text communication.
Ethics Statement
Our work adheres to the ACL Ethics Policy. This paper aims to investigate generative model-based approaches for learning automatic diagnostic logic, with the objective of reducing the burden on doctors and promoting the advancement of automatic diagnosis systems. It is crucial to emphasize that the proposed methods are designed solely for research purposes and are not suitable for direct clinical application due to the potential risks associated with the misuse of automatic diagnosis systems.
It is important to note that the introduced dataset (Ped) was sourced from genuine electronic medical records, with all patient privacy-related information meticulously eliminated. To ensure data privacy and security, we performed a comprehensive manual review of the dataset, confirming that it contains no identifiable or offensive pieces of information within the experimental dataset.

heading: Acknowledgements
text: We appreciate the constructive and insightful comments provided by the anonymous reviewers. This research work is partially supported by CUHK under Project No. 3230377.

heading: A Dataset
text: The statistics of these datasets can be found in Table 3. We evaluate our model on four datasets:
Dataset Dxy Muzhi Muzhi-2 Ped
# Disease 5 4 6 44 # Symptom 41 66 347 273
Symptom type True True True/False True/False Average length 4.7 5.7 9.9 9.6 # Training 421 568 1882 5000 # Test 104 142 165 1000
Specifically, we use the small variant of the Transformer’s decoder (Vaswani et al., 2017) (L=6, H=768, A=6) as the backbone of CoAD. The disease head and symptom head are fully connected
1https://dxy.com/ 2http://muzhi.baidu.com 3http://www.fudan-disc.com/sharedtask/imcs21/index.html
layers that map the hidden states of the decoder to the appropriate output space. In the fixed turn setting, the hidden state of the last symptom is used as input for the disease head. In the varied turn setting, the model terminates symptom inquiries by predicting the end token and proceeds with disease diagnosis immediately. The batch size and learning rate for each dataset are presented in Table 4.
C Comparison of Fixed Turns
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations.
3 A2. Did you discuss any potential risks of your work? Ethics Statement.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? The Abstract provides the link to our code.
3 B1. Did you cite the creators of artifacts you used? In appendix A dataset.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We follow the license or terms of the used artifacts.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The data we collected is sufficiently anonymized, we state the proposed model is for research purposes only in Section Ethics Statement.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data is safe and commonly used by many previous work.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? In appendix A dataset.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In appendix A dataset.
C 3 Did you run computational experiments? Section 5.1
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Implementation in Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.1 Experiment Setup
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.1 Main results.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Implementation in Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section Ethics Statement
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Limitations
In this work, we have identified two key limitations of CoAD that can be further examined in future research. The first limitation is that CoAD only allows for the querying of one symptom at a time, making it unsuitable for scenarios where multiple symptoms are present. However, CoAD has superior performance in the main metrics for automatic diagnosis. To relieve this limitation, potential solutions include relaxing symptom feedback conditions and allowing the model to produce symptoms sequentially until a stop signal is encountered or querying the top K symptoms in a single turn. Additionally, CoAD has some restrictions on input format, requiring standardized symptoms and values. To make it more applicable to end-to-end settings, an natural language understanding module (NLU) is required to parse plain text and obtain the input symptom sequence, and a natural language generation (NLG) module is needed to translate the predicted symptom or disease to text. The ultimate goal of automatic diagnosis is to support the dialogue between doctors and patients, after CoAD determines the symptom or disease, rulebased NLU and NLG modules can help to achieve
the text to text communication