Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This work proposes DDXPlus, a symptom-disease dataset of synthetic patient records containing demographic information (age, gender, geographic), each patient's disease, symptoms, antecedents and differential diagnosis.
Each patient's disease and evidences (i.e. symptoms and antecedents) are sythetically generated based on a proprietary KB and a commercial synthesizer (i.e. Synthea).
Differential diagnosis for each patient is generated by feeding all evidences to a commercial telemedicine software, which generates a set of possible diseases.
DDXPlus differs from previous datasets in several ways such as providing a differential diagnosis, size of the dataset, using symptoms that are non-binary, etc.
In the experiments, existing models that can perform automatic diagnosis was applied to DDXPlus and demonstrated good performance.
strengths: The paper has following strengths:
- Although there are similar datasets, DDXPlus is the only one that comes with a paired differential diagnosis.

- DDXPlus clearly divides evidences into to subsets (i.e. symptoms and antecedents), which could potentially be useful for other researchers.

- The dataset construction process is clearly described.

weaknesses: The paper has following weaknesses.
- The paper claims that, unlike previous datasets that only use binary symptoms, DDXPlus uses multi-class and multi-label symptoms. However, this can also be achieved with previous datasets by post-processing. Based on simple domain knowledge, individual evidences can be grouped into a higher-level evidence (e.g. chest pain, neck pain, wrist pain, ... ==> pain), which will convert a subset of evidences from the existing datasets to multi-label/multi-class evidences.

- The paper claims that it is a large-scale dataset, unlike existing datasets. However, some existing datasets (e.g. SymCAT, HPO) are theoretically unbounded in size, because a user can generate as many records as they need, because they geneyrate synthetic records by exploiting pre-defined domain knowledge (e.g. medical ontology, census data). This is actually the same for DDXPlus, which also generates purely synthetic records based on on medical ontology and census data. DDXPlus could have been a billion-size dataset if the authors chose so, and there is no justification as to why 1.3 million records were generated as the final dataset.

- Although DDXPlus is the only dataset with the ground-truth (GT) differential diagnosis (ddx), they are generated by a rule-based telemedicine software. Therefore the quality of the GT ddx is in serious question. Note that an automatic diagnosis model (e.g. AARLC) can learn to generate ddx by training on previous datasets (e.g. SymCAT, HPO, Muzhi) that do not have GT ddx. The limitation is that there is no way to evaluate if the model-generated ddx is clinically meaningful. DDXPlus can potentially fill this gap, because it claims to have GT ddx. But since that GT ddx is generated by a very questionable rule-based software (which cannot be open-sourced because it is a proprietary product), there is hardly any value in it. Moreover, if ddx can be generated by a rule-based process, why should we train machine learning models to do it?

- Although the authors claim that training the model to predict ddx instead of the final pathology is helpful, the experimental results say otherwise. According to BASD's performance in Table 1, using ddx during training only degrades the prediction performance of the true pathology, and makes negligible improvement in the model's ability to find the evidences.

Overall, it is very difficult to consider DDXPlus, which is a synthetic derivative of a census data, medical knowledge base (the quality of which is unknown) and a rule-based system (also quality unknown), a new dataset that brings significant new value to the community.

correctness: Although the overall dataset construction process is reasonable, it has several shortcomings due to the fact that the dataset is completely synthetic, and completely relies on existing medical ontology and a rule-based software. The details of the shortcomings are described in the Weaknesses section.
clarity: The paper is overall nicely structured and very well-written.
All the necessary details that are required to understand the dataset construction process is provided in a detailed manner.
Especially, the elucidation of the assumptions made during the synthesis of patient records is much appreciated.
relation to prior work: The paper makes several claims as to how DDXPlus is different from the datasets, such as the size, types of symptoms, having differential diagnosis, etc. Although a couple of them are true, the others do not seem as a solid difference. The details are described in the Strenghts and Weaknesses section.
additional feedback: Is there any future plan to open-source the proprietary medical knowledge base and the telemedicine software? It would be way easier for the research community to study them to learn new insight, rather than try to replicate their functionalities by using their synthetic derivativess.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors propose a large-scale synthetic dataset of differential diagnosis, ranking a list of plausible diseases that a patient might suffer. Unlike the previous datasets, the proposed dataset contains more pathologies, symptoms, and antecedents with non-binary questions and differential diagnosis data. The potential usage of the dataset is to assist doctors in telemedicine service.

strengths: - The paper is well-motivated and differentiates itself from existing works.
- The dataset contains a large scale of data that includes various pathologies, symptoms, and antecedents. 
- The dataset contains not only binary but also categorical and multi-choice symptoms and antecedents.
- The data generation process seems well-thought-out.
weaknesses: - It would be great to have some sample data instance either in the main paper or the supplementary material. 
correctness: The claims appear to be sound and reasonable.
clarity: The paper is well motivated and easy to follow.
relation to prior work: Related works are well discussed.
additional feedback: N/A
review_title: Review of A New Dataset For Automatic Medical Diagnosis

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work presents a new, large-scale synthetic dataset that can be used to evaluate differential diagnosis symptoms. By extracting diagnosis rules from medical literature, the authors automatically label a massive set of synthetic patients. Using these data, the authors compare two existing automatic diagnosis systems on their dataset, finding that incorporating differential diagnosis into the train improves their performance.
strengths: This paper has many strengths:
* The released dataset is extremely large
* The data include differential diagnosis information for synthetic patients, which is lacking in prior datasets
* Since the differential rules are probabilistic, it's believable that the synthetic data are generally realistic
weaknesses: This paper could be strengthened with the following considerations:
* Clearer discussion of what these data can be used for would be helpful. What would it take to adapt these data to a real case? Or if the data are primarily for evaluation, is there a benefit from the large size?
* The experiments only cover two existing systems, so it's not obvious how much room there is for improvement in existing systems according to these new data. Or if these two are representative, this should be explained further.
* The experimental results discussion should be expanded. How good should these metrics be? The F1 scores seem low, so it appears there's room for improvement on these data, but it would help to discuss the limitations of existing systems on these data to motivate how the community will use this new resource.
correctness: The claims appear correct.
clarity: The paper is generally written well.
relation to prior work: Since these data are so large, it would be helpful to list the typical scale of existing datasets, though the authors helpfully note that the prior works contain fewer features.
additional feedback: Feel free to address any comments above
review_title: Review for DDXPlus

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a large-scale synthetic dataset focusing on cough, sore throat, and breathing problems for automatic medical diagnosis. They provide a good description of the information about the dataset. The experiments of two extended methods demonstrated that the collected dataset is useful for studying automatic diagnosis systems. Their contributions are:

- 1.3 million generated patients information covering 49 pathologies, 110 symptoms, and 113 antecedents.
- Well description and analysis of the released data.
- Methods, AARLC and BASD, are evaluated to show the usefulness of the collected dataset. 
strengths: - Clear difference with existing datasets. The limitations of existing datasets and the advantages of the proposed dataset are explained.
- Well description and organization of the released data. The paper includes a detailed data generation process and summary statistics, dataset documentation is also included in the Appendix.
- They reported the experimental results of the extended existing methods. The comparison results between differential diagnosis and not are analyzed to show the potential impact of the proposed dataset.
weaknesses: - Lacks the evaluation of medical experts for the assumptions and the generated data, although the paper explains the data are generated based on medical knowledge.
- The limitation of the proposed dataset can be described. 
correctness: I think the dataset is constructed in a sound way.
clarity: The paper is well written.
relation to prior work: It is clearly discussed. The limitations of the existing dataset are described in Section 2, and the advantages of the proposed dataset are in Section 3.4. 
additional feedback: See Weaknesses.
review_title: Nice paper, I think that this paper introduces a valuable dataset for differential diagnosis

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors introduce a new dataset of synthetically generated data on patients for Automatic Symptom Detection (ASD) / Automatic Diagnosis (AD). 

The major novelties (not in order of importance) of the dataset  are: 
- being large-scale (1.3M patients)
- containing differential diagnosis, i.e. for each patient a list of plausible pathologies is provided
- introducing a distinction between "symptoms" and "antecedents" (patient history) and these being categorical and multi-choice rather than only binary.

In summary, to generate the data the authors make use of a propietary knowledge base, which provides provides probabilities for disease/symptoms/antecedents (conditioned on sex,age). These probabilities are then combined with census data and fed to a tool specifically developed to synthesize patients (Synthea) to generate the patient data. Subsequently the authors deploy a propietary ASD system to generate the differential diagnosis for each patient.

Finally the authors perform experiments with two existing (and tweaked) ASD/AD systems to asses the impact of training a system with the newly generated dataset.




strengths: Besides its size (skating over "quality vs. quantity"), the dataset introduced in the paper provides an important contribution, as it extends the spectrum of existing datasets by providing data for what should be a critical component in ASD/AD systems, namely differential diagnosis.
This dataset can have a positive impact to the community to which it is addressed and will allow to foster further research on AD/ASD systems. Being a synthetic dataset it also presents the benefits of being freely sharable, thus completely removing any barrier to its access and use by other researchers. 

weaknesses: I would like to restate that the authors provide a relevant contribution to the field with the dataset they generated, and that orchestrating all the necessary resources to create the data must have been an important effort.

However, I think that two aspects of the manuscript need further attention, namely: (a) the description of the tools used to generate the dataset and (b) experiments performed.

W.r.t. (a) I cannot help but to notice that while the KB used to generated the data is adequately described in terms of its content, little to no information is provided as to "how" it was compiled. The only reference I could find was "which was constructed by compiling over 20 000 medical papers". Since this is a primary ingredient of the pipeline, I think that further details should be provided, e.g.: exactly who compiled it? Medical data can be "tricky" to interpret, was there any measure of the quality of the KB, e.g. in terms of agreement on diseases/sympots? What kind of medical papers? Where did they come from? I am not sure whether the nature of the resource prevents it, but answers to those questions seems to me mandatory. The same line of reasoning goes to for the "rule-based ASD" system used to generate the differential diagnosis. I think that the reader should be informed at least minimally as to how the system works, e.g. how are the different inputs processed and combined to arrive at the differential diagnosis?

W.r.t. (b), since the authors introduce a large-scale synthetic dataset, my expectation was that one evaluation experiment would have been devoted at cross-dataset validation, i.e. training a system on the synthetic dataset and then validate the gain in performance on a new unseen dataset. This would be a critical testbed considering the type of applications to which the dataset is aimed.
Additionally, I think that experiment section would greatly benefit from an (even minimal) error analysis in order to assess the beneficial impact of the dataset. For instance, I was surprised to see that there wasn't a clear cut improvement in the "Positive evidence" (PE) - R/P/F1 metrics. Intuitively speaking, if a system is trained to predict a differential diagnosis (i.e. multiple diseases), shouldn't the coverage of the symptoms/antecedents be broader? While for AARLC system this seems to be the case (+$\sim$20 in PER) it seems to have no effect for BASD.  




correctness: To the best of my knowledge, my opinion is that the dataset is constructed in a sound way, however in my opinion certain aspects of the creation process necessitate further clarification.
clarity: The paper is clearly written and the reader can follow the authors' arguments without effort.
The only minor defect is that equations are not numbered, which should be fixed.

relation to prior work: To the best of my knowledge all the relevant literature is correctly cited to have a clear understanding of the contribution and the position of the paper in the broader context of its research field.
additional feedback: 
- line 109:  As a reader, I think I would like to know right away that the KB and the rule-based ASD system used to generate the data where provided by "Dialogue Health Technologies Inc" straight away, e.g. by having a footnote, rather finding this out in the acknowledgements.
- line 158-161: should mention at least a suggestion to overcome the limitation 
- line 164-168: if the KB was compiled by medical papers, the "domination" of specific pathologies is "reflecting" the true data distribution. Why change it? What is the problem of having a disease occuring twice a year, which is actually a common thing.
- line 171: is there a specific reason to choose 10%?

### Metrics
Maybe I am misunderstanding something but, if GTPA@k:
- "measures whether the differential diagnosis predicted by an agent contains the pathology p a patient was simulated from within its top-k entries"

and DDR@k: 
- "measure whether the topk pathologies $\hat{Y}_{k}$ extracted from this distribution [distribution $\hat{Y}$ over the pathologies] are present in the topk pathologies Yk",  

=> then isn't DDR@1 == GPTA@1?

### Minor remarks:
These in my opinion would help improving readability and ease comprehension:
- extensively define asap "antecedents"
- extensively define asap "Automatic Symptom Detection (ASD)" and "Automatic Diagnosis (AD)" underlying their differences
- line 35: if possible, split the references between ASD and AD rather then having a single long list
- line 97: HPO is an abbreviation but was never introduced. 
- line 97: there should be a distinction between the reference for HPO (Köhler et al.) and the one for the ASD/AD system (Guan and Baral)

Supplementary material:
- Table 7,8,9 should be merged.
review_title: Relevant dataset but paper requires further details and experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a large-scale synthetic automatic diagnosis dataset, including 1.3 million patient data. It highlights the importance of differential diagnosis during automatic symptom detection and includes underlying pathologies for each patient in the dataset. 

In addition, the authors propose that the symptoms and antecedents are of different importance from a doctor's perspective and should be clearly distinguished. Compared with existing datasets, it contains a mixture of multi-choice, categorical and binary evidence. 

To demonstrate the usefulness of incorporating the differential diagnosis, the authors implemented and extended two existing AD and ASD systems upon this dataset.
strengths: 1. The clinical scenario proposed in the paper, i.e., differential diagnosis, has been neglected in the previous works while is of great importance in practice.

2. This dataset is quite large, which is feasible for data-driven models.

3. The dataset combines medical knowledge base information and thus is richer in content than previous datasets.
weaknesses: 1. The main weakness of this dataset derives from the fact that it is synthesized based on a series of very strong assumptions, which can lead to severe problems under inappropriate usage. In particular, these assumptions include:

    - Characterizing each disease by an incidence rate and/or a prevalence rate collected from medical papers.
    - Assuming for a given disease, the evidence experienced by a patient does not depend on the geographical region (If there is any reference to support this, please cite it).
    - Assuming the distributions of age, sex, and geographical region are independent.
    - Assuming the proportion of population traveling each month is 0.25 and the distribution regarding the geographical regions of destination is uniform.
    - Caping the chance of diseases with extremely low incidence rates at a minimum of 10%.

    These assumptions raise the following potential risks:

    - The pathogenesis of human beings is quite complex, yet the above pre-settings are too idealistic.
    - The medical knowledge base used in this work was constructed by compiling medical papers. However, there is a gap between academic papers and the practical situation. Also, the distribution of clinical data varies across regions, races, time periods, etc. These factors can significantly affect the generalizability of this dataset.
    - Taking simple hypotheses to generate patient data while only increasing the prevalence of rare diseases will exacerbate the mismatch between the dataset and the real patient data. 

    Thus, the performance of ASD/AD systems is hard to soundly evaluate upon this dataset in the case of violating the actual data pattern, even if only for research purposes.

2. Even though using this dataset solely for training and evaluating the performance of data-driven models, there are also several concerns. The main contribution of this dataset is the introduction of differential diagnosis information. However, all the labels and differential diagnoses are auto-generated by a rule-based system. This incurs the problem that a model trained on this dataset will approximate the rule-based system but cannot surpass it. Therefore, it is necessary to adopt strategies to prevent models trained on this dataset from being another implementation of a rule-based system.

3. The importance of recovering differential diagnoses seems not well demonstrated in the experiment section.
    - From the experimental results in Table 3, models without considering differential diagnosis achieve better overall performance, i.e., high-quality predictions (GTPA@k) and shorter interaction length (IL). 
    - The prediction objective of Table 4 is to recover the differential diagnosis, which is somewhat confusing. First, the agent trained to predict the differential diagnosis outperforming the one not exposed to such information on this task is not surprising, even a bit tricky. In addition, it would be better to elaborate on the reasons for treating differential diagnosis as a predictive target and using it as a criterion of model performance.

4. More detailed case studies and/or discussions are expected to support the motivation of this paper.

--------------------------------------------------------------------------------------
I will be glad to raise my score if my concerns are well addressed.
correctness: As a synthetic dataset for medical purposes, it is inevitably facing the problems of data bias, low generalizability, etc. Besides, all the labels and differential diagnoses are auto-annotated without the involvement of experts, raising concerns over the usability of the dataset.

clarity: The paper is well written, and the content is clear and well-organized.
relation to prior work: In Section 2, this paper clearly discusses the difference between the presented and the existing datasets.

additional feedback: It would be better to maintain the relevant data preprocessing, model implementation, and evaluation scripts in a code repository, e.g., GitHub.
review_title: DDXPlus: A New Dataset For Automatic Medical Diagnosis

Meta-review: metareview: This paper proposes a symptom-disease dataset of synthetic patient records containing demographic information, disease, symptoms, antecedents and differential diagnosis. The disease, evidences, and differential diagnoses are synthetically generated which makes the dataset easy to share. Several reviewers pointed out important concerns in the methods used to synthesize the data which may raise ethical problems if it is assumed that performance on this dataset is indicative of performance on real patients. Nevertheless, I believe that this dataset can be of value to the community as it can help the community develop better ways to connect differential diagnosis to the AD/ASD task (while we wait for a dataset with real differential diagnoses). I urge the authors to clearly caution readers about the datasets limitations!
confidence: 3: The area chair is somewhat confident
Decision: Accept