PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications

Authors: dingkang  yang
jinjie  wei
dongling  xiao
shunli  wang
tong  wu
gang  li
mingcheng  li
shuaibing  wang
jiawei  chen
yue  jiang
qingyao  xu
ke  li
§  peng zhai
lihua  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 16:21:08

Abstract: Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures.
To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline.
In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase,
a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. The project and data will be released at https://github.com/ydk122024/PediatricsGPT.

Main paper: 
 introduction: 
 As an essential component of medicine, pediatrics plays an indispensable role in ensuring children's health growth [22, 23] . The unbalanced distribution of healthcare resources [36] has resulted in a massive shortage of pediatricians, especially in populous countries led by China [37, 19] . With the models [5, 33] equipped with certain internal knowledge regarding biomedicine, they are incompetent in real-world medical applications due to the absence of domain-specific disciplines. In this context, several efforts [45, 50, 15, 30] attempt to construct medically tailored LLMs from multiple perspectives. For instance, ChatDoctor [30] uses patient-doctor conversation data based on LLaMA [43] to enhance the language model's accuracy in healthcare. DoctorGLM [50] proves that a healthcarepurpose LLM can be implemented with affordable overhead by fine-tuning ChatGLM-6B [20] .After that, more Chinese medical LLMs [51, 8, 58, 14, 56] are progressively presented to generate doctor-like robust responses, such as HuatuoGPT [58] , DISC-MedLLM [8] , and Zhongjing [56] . Despite advances in general medical knowledge, current models are suboptimal for pressing pediatric applications. In comparison, our sophisticated training procedure and high-quality instruction datasets inject new insights and prospects for developing specialized LLMs with pediatric expertise. 
imaging studies: 
 (1) Wrist joint X-ray, AP view: Measuring bone age to understand child growth and development further.(2) Head CT: Craniopharyngiomas in the sellar region can grow in various directions with significant individual variation. Radiologically, they appear round or lobulated. The tumors may be cystic, solid, or mixed. Cystic fluid typically shows low density on CT. Solid components appear as uneven, iso-, or slightly hyperdense. Typical craniopharyngiomas on CT may show "eggshell" calcifications.(3) Head MRI: Due to varying components, MRI signals vary. Cystic lesions mostly appear as high signal on T2WI, some as low signal (due to keratin or calcium salts), and T1WI signals vary depending on components, appearing low (contains deoxyhemoglobin) or high (high protein content); post-enhancement, the solid parts may show varying degrees of enhancement.(4) Other imaging studies such as CTA, MRA, Digital Subtraction Angiography (DSA), etc., to evaluate the relationship between the tumor and blood vessels.Clinically may manifest as visual impairment, but typically shows as low-density lesions on CT without clear calcifications; on MRI, T1WI is iso-or hyperintense, T2WI is hyperintense, and enhancement is usually uniform after contrast administration; can be distinguished from craniopharyngiomas.</USER>
 [OUTPUT]<ASSISTANT> { "Q1": "What are the key laboratory tests required for diagnosing childhood craniopharyngioma, and how do specific hormone levels impact the evaluation?", "Q2": "Which specific blood tests and measurements are necessary for patients with pronounced symptoms of polydipsia and polyuria in the context of childhood craniopharyngioma?", "Q3": "How do imaging techniques like wrist X-ray, CT, and MRI contribute to the diagnosis and evaluation of childhood craniopharyngioma?", "Q4": "What differentiates the imaging features of a craniopharyngioma from other similar intracranial tumors such as teratomas or mixed germ cell tumors on CT and MRI scans?", "Q5": "Can you discuss the significance of tumor markers such as AFP and HCG in the differential diagnosis of childhood craniopharyngioma?", "Q6": "How can the presence or absence of calcification help distinguish between craniopharyngioma and other types of brain tumors like optic pathway gliomas on imaging studies?" }</ASSISTANT>[/OUTPUT] Figure 7 : Illustration of the GPT-4 prompt template for the "inquirer" role. The original Chinese content is translated into the corresponding English version to facilitate intuitive understanding.
<SYS>Please act as an experienced pediatrician and fulfill the following requirements: 1. For each inquiry, provide comprehensive responses, taking into account the intricacies of pediatric healthcare and the needs of patients, ensuring accuracy and humanistic care in the answers. 2. Answers should be based on the provided pediatric corpus, focusing on pediatric healthcare to ensure professionalism and relevance. 3. Answers should be logically coherent and semantically clear. 4. Generated answers should be formatted as: {"A1":"","A2":"","A3":"","An":""}, where each "An" represents an independent question.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We fully discuss the limitations in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 This section describes the proposed PedCorpus dataset and the sequential pipeline for developing PediatricsGPT. Figure 1 illustrates the comprehensive method workflow.To endow the model with versatile diagnostic proficiency, PedCorpus is constructed through the multi-dimensional corpus across three application-oriented medical tasks, including Knowledge Question-Answer (MedKQ&A), Evidence-based Diagnosis (EviDiag), and Treatment Recommendation (TreRecom). Table 1 shows the detailed statistical information from different data sources. We explain the three patterns of PedCorpus construction below.Specialized Pediatric Data. Extracting pediatric data from textbooks, guidelines, and knowledge graphs ensures knowledge professionalism. Specifically, we automatically extract standard medical definitions and descriptions from physical textbooks covering 131 disease types in 11 broad categories. Over 500 corresponding disease guidelines are collected, including diagnostic protocols and treatment consensus. Additionally, extensive knowledge entities are sampled from ternary instances in the knowledge graphs. Based on these resources, we introduce a role-playing-driven instruction building rule via GPT-4 API that produces well-organized instructions to enable accurate and humanistic model responses. The detailed building procedure is shown in Appendix A.1.Real Doctor-patient Conversations. To avoid the model collapse dilemma [42] , we incorporate authentic doctor-patient dialogues from online treatment platforms and voice transcriptions during medical consultations. The single-/multi-turn instructions are jointly considered to equip the model with healthcare interrogation and contextual understanding. Original responses from real doctors are usually terse and noisy, potentially worsening the generation quality [58] . To this end, we craft 100 high-quality examples to guide the advanced language model by the in-context learning to regularize vanilla conversations in the self-instruct pattern [17, 46] . This approach ensures doctor-like and patient-friendly model responses. More regularization details are shown in Appendix A.2.Distilled Medical Datasets. Integrating general medical knowledge from existing datasets [29, 26, 60] is a common practice in previous efforts [15, 50, 51, 8] . However, we find numerous unclear and incomplete representations in the instruction instances from public benchmarks due to the absence of careful calibration, potentially triggering hallucinated outputs. Consequently, we manually sample Preference Learning The sequential pipeline for developing PediatricsGPT. We begin by injecting intensive medical and world knowledge into the foundation model through the hybrid instruction mechanism in CPT phase. Then, full-parameter SFT is implemented to improve the model's instruction-following capabilities regarding medical generalists. After that, we introduce the direct following preference optimization to control the model behaviour to align with human preference. In the parameter-efficient SFT phase, the LoRA-based mixture of universal-specific experts is devised to mitigate conflicts across downstream tasks and competition between pediatric expertise and general mastery.107,177 knowledge-intensive instructions from three mainstream benchmarks (i.e., Huatuo-26M [29] , MedDialog [26] , and CMeKG [10] ), adhering to the philosophy of quality over quantity [62] . After that, a progressive instruction reconstruction rule is proposed to distill the sampled instructions to ensure informative and logical model responses. The rule process can be found in Appendix A.3.Continuous Pre-Training (CPT) is essential in developing domain-specific models [14, 49, 56] since it can break the scaling law [24] to a certain extent. For this purpose, we introduce the PedCorpus-CPT dataset to ensure a high-quality pre-training corpus. From Table 1 , PedCorpus-CPT consists of threepart data components. (i) We integrate plain texts from vanilla pediatric textbooks, guidelines, and knowledge graphs. (ii) The filtered Chinese Wikipedia [3] is also considered to achieve the model's trade-off for medical-general knowledge memory capacity. (iii) In practice, we observe that CPT leads to catastrophic forgetting of the models at follow-up due to different data distribution and format discrepancies compared to the original pre-training and SFT. Thus, we introduce a hybrid instruction pre-training mechanism to bridge these discrepancies. The core philosophy is to assemble instruction data from PedCorpus with Input-Output forms into Completion forms, which are then assimilated into plain texts to provide multi-task and complementary information. This mechanism effectively mitigates inconsistencies between the internal-injected medical knowledge of the foundation model while reinforcing medical domain adaptation. Moreover, we take PedCorpus as the seed instructions to improve multiple-department corpus density and breadth via knowledge-enhanced prompts. The prompt template is shown in Appendix B.We pre-train the foundation model to follow the causal language modelling paradigm. Given any input token sequence t = (t 0 , t 1 , t 2 , ...) ∈ D cpt from the above multi-channel corpus D cpt , the next token t i is autoregressively predicted by minimizing the negative log-likelihood:EQUATIONwhere θ is the model parameter and the input context consists of t 0 , t 1 , ..., t i-1 .During this phase, we activate the model's ability to follow medical instructions by the Full-parameter Supervised Fine-tuning (FSFT). The full-parameter pattern enables a fuller invocation of the intensive knowledge in CPT and promotes comprehension and logical reasoning about diverse structured instructions. The training data at this phase is composed of the following three aspects. (i) We utilize the multi-department medical data in the PedCorpus dataset to develop the medical generalist. (ii)Chinese instruction data (i.e., Alpaca dataset [35] and ShareGPT [4] ) from general domains are selectively integrated to avoid the potential overfitting risk. (iii) Providing safety measures is vital for LLM assistants yet overlooked by prior methods [62] . In contrast, we write 200 training instructions with some degree of maliciousness, hallucinations, and counterfactuals. Correspondingly, the refusal responses with detailed explanations for disobedience are carefully crafted. We also include 300 examples related to self-cognition content. These data significantly improve the robustness and security of the model against unfriendly commands.Given any input instruction x = (x 0 , x 1 , x 2 , ...) ∈ D f sf t and corresponding target response y = (y 0 , y 1 , y 2 , ...) ∈ D f sf t from the above-integrated fine-tuning dataset D f sf t , the optimization objective can be formulated as follows:EQUATIONAligning human intention preferences facilitates the model to generate harmless responses. To this end, we introduce PedCorpus-DFPO D df po , a preference dataset to guide the model in learning human preference behaviours. PedCorpus-DFPO contains the input instruction set x = (x 0 , x 1 , x 2 , ...) ∈ D df po , which is selectively sampled from vanilla PedCorpus. On the one hand, we perform a humanistic stylistic rephrasing of the outputs to generate preferred responses y w = (y w 0 , y w 1 , y w 2 , ...) ∈ D df po . On the other hand, the corresponding low responses y l = (y l 0 , y l 1 , y l 2 , ...) ∈ D df po are generated from the feedback of a low-capability medical assistant [45] to maintain domain consistency.Despite impressive improvements achieved by RLHF-based approaches [56, 58] , challenges remain due to unstable reward modelling and significant computational costs [41, 61] . Inspired by singlestage preference learning [38] , we propose a stable and lightweight method for domain-specific LLMs called Direct Following Preference Optimization (DFPO). DFPO utilizes variable changes to formulate the preference loss as a policy function that efficiently optimizes the policy with a simple binary cross-entropy objective. Meanwhile, our method directly regularizes model behaviour boundaries in an instruction-following paradigm on medical demonstrations of preferred responses, facilitating robustness and smoothing of the preference learning.Theoretically, the observed probability of a particular preference pair usually follows the Bradley-Terry model [9] , and y w is preferred over y l (denoted y w ≻ y l ):EQUATIONwhere γ(x, y w/l ) means the parameterized reward function and σ(•) is the sigmoid activation. In this case, the overall optimization objective is expressed as:EQUATIONwhere π θ and π r are the desired optimal policy and the reference policy, respectively. β is the control parameter reflecting the deviation from the basic π r . For the fine-tuning regularization term Φ(x, y w ) with the scaling coefficient µ, the implementation process is equivalent to maximizing the log probability p(y w | x) regarding the preferred responses y w given the input instructions x:EQUATIONThis phase aims to reinforce the model performance for various pediatric applications through the LoRA-based Parameter-efficient SFT (PSFT). The used dataset D psf t is derived from the pediatric department in PedCorpus and partial general medical/world data. In practice, we observe that competition across different pediatric tasks and the conflicts between medical generalization and specialized knowledge deteriorate instruction-following abilities. Accordingly, we propose a mixture of universal-specific experts strategy to address these challenges. Formally, LoRA adapters [27] act as experts to replace the linear layers in the Feed-Forward Neural (FFN) networks of LLMs, providing (6) W g and W n are the learnable weights. S(φ(xW n ) is the noise term for regularizing the expert utilization balance, where S(•) and φ(•) represent the Standard Normal distribution sampling and Softplus function, respectively. Moreover, we consistently activate a universal expert E u across all training data to prevent general knowledge forgetting and mitigate competency conflict. The parameterized output z of all the experts in the forward process can be mathematized as follows:EQUATIONwhere r is the rank value and α is a hyper-parameter for approximating the learning rate.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
datasets and implementation details: 
 Extensive experiments are conducted on three application-oriented benchmarks to assess the model's pediatric medical abilities, including Knowledge Question-Answer (MedKQ&A), Evidence-based Diagnosis (EviDiag), and Treatment Recommendation (TreRecom). Each benchmark contains 300 held-out samples to reject data leakage during training. In addition, we select two publicly available Chinese medical benchmarks to validate the model's generalizability in general healthcare. Specifically, we sample 50 challenging instances of diagnostic queries from each department from the webMedQA [25] and CMD [1] benchmarks, respectively, leading to testing sets with 300 samples.Our PediatricsGPT is developed upon the Baichuan2-Base [52] and DeepSpeed packages using eight Nvidia A800 GPUs. The ZeRO strategy [39] is employed to alleviate the memory overhead during full parameter training. The AdamW optimizer [31] is adopted for network optimization, and the bf16 data accuracy is chosen. More detailed hyper-parameter configurations for different stages are shown in Appendix C.We compare a series of LLMs for comprehensive evaluations. Concretely, Baichuan2-7B/13B (Chat) models [52] are trained on 2.6 trillion tokens as the baselines, which have excellent abilities in different domains. Meditron-7B [16] is a 7 billion parameters model adapted to the medical domain from Llama2-7B through continued pre-training on a comprehensively curated medical corpus. Llama3.1-8B [21] is a robust multilingual large language model through systematic training. For reproducible Chinese medical works, DISC-MedLLM (13B) [8] is fine-tuned through reconstructed medical dialogues and behavioural preference instructions. HuatuoGPT (13B) [58] performs SFT based on mixed instruction data and introduce human feedback in RLHF. HuatuoGPT-II (13B) [14] enhances the medical-specific domain adaptation of LLMs through one-stage unified training. Zhongjing (13B) [56] implements a complete pipeline based on Ziya-LLaMA-13B to enhance the model's multiturn medical conversation abilities. ChatGPT [33] and GPT-4 [5] have impressive performance in general medical fields as closed-source models developed by OpenAI.Metrics-based Evaluation. In Table 2 , we present the comparison results of different models on three pediatric benchmarks through multifaceted metrics, including ROUGE-1/2/L, BLEU-1/2/3/4, GLEU, and Distinct-1/2. The key observations are listed below. (i) PediatricsGPT-13B significantly outperforms the baselines and SOTA medical models on the vast majority of metrics across all benchmarks, demonstrating excellent pediatric expertise. (ii) Our 7B version also achieves competitive results compared to the 13B models. For instance, PediatricsGPT-7B yields absolute improvements of 3.53% and 4.44% on metrics ROUGE-L and GLEU in the EviDiag task compared to HuatuoGPT-II, respectively, generating more accurate and informative content. (iii) By contrast to Zhongjing and HuatuoGPT-II with massive training corpora, our method confirms that the training data quality outweighs quantity for performance gains. (iv) The worst results at baselines emphasize that targetoriented fine-tuning is an effective strategy for improving domain-specific abilities.Automated GPT-4 Evaluation. Measuring model performance from multiple aspects is essential in the pediatric medical domain. To this end, we consider four dimensions to holistically assess response quality, including usefulness, correctness, consistency, and smoothness. Advanced GPT-4 [5] is prompted to select the winning response between pairwise models based on these dimensions. The dimension explanations and the prompt template for GPT-4 can be found in Appendix D. (i) As Figure 2 shows, PediatricsGPT-13B wins all LLMs by large margins in the MedKQ&A task, implying the necessity of implementing the knowledge-intensive CPT. (ii) The favourable win rates on the TreRecom and EviDiag tasks compared to medical LLMs show the superiority of our model in both single-turn treatment recommendations and multi-turn medical diagnostics. For example, our model beats Zhongjing via the 59% win rate on the EviDiag, which specializes in multi-round consultations.Manual Doctor Evaluation. Doctor approval of LLM assistants is a vital step toward realistic applications. We invite three doctors (each paid $300) to determine the winner of pairwise models by the majority voting rule. The evaluation requires simultaneous consideration of the responses' professionalism, factuality, and safety. (i) Excluding ChatGPT, the dominance of our model in Figure 3 shows the effectiveness of considering safety measure data while incorporating specialized pediatric knowledge. (ii) The proposed direct following preference optimization makes PediatricsGPT-13B more favoured by human preferences compared to other behavioural alignment efforts [8, 56, 58] .(iii) The competitive performance of ChatGPT when human judgments indicate that the scaling law still holds, stemming from the high agreement between its behaviours and human intentions.Generalization Ability Evaluation. We show the GLEU metric-based scores of different models on the Chinese medical benchmarks in Figure 4 for CMD and Figure 5 for webMedQA. (i) PediatricsGPT-13B achieves impressive results across diverse medical departments (including pediatrics), exhibiting medical generalist and pediatric competency mastery. (ii) The 7B counterpart similarly outperforms most 13B Chinese medical LLMs and exceeds ChatGPT in some departments.For instance, PediatricsGPT-7B brings relative gains of 18.8% and 7.1% compared to ChatGPT in the Gynecology and Oncology tasks on the CMD benchmark. These findings confirm the robust generalization of our model and its ability to capture the multifaceted medical dialogue distributions.We perform thorough ablation studies on five medical benchmarks to investigate the effects of different modelling components. Following [58] , we compare the responses from each of the proposed model variants with ChatGPT, and then calculate the win rate (%) of our model in pairwise responses by GPT-4 and doctor evaluations. Table 3 shows the following observations.Importance of Continuous Pre-training. Firstly, we remove the complete continuous pre-training phase to observe performance variations. (i) The significantly deteriorated win rates reveal that Necessity of Supervised Fine-tuning.(i) We observe consistent performance gaps when removing the Fullparameter SFT (FSFT) and Parameterefficient SFT (PSFT) phases, respectively. This makes sense since SFTs are necessary to activate the model's healthcare instruction-following capabilities. (ii) Moreover, PSFT is more critical for three pediatric applications because it facilitates pediatric-related knowledge accumulation, while FSFT focuses on consolidating general medical semantic representations. (iii) Then, we replace the proposed Mixture of Universal-specific Experts (MUE) version with the vanilla single LoRA. The reduced performance on pediatric EviDiag and TreRecom benchmarks verifies that it is essential to introduce multiple LoRAs that act as specific experts on different tasks. A reasonable explanation is that the single-LoRA model suffers from the task competition between learning the knowledge questionanswer and mastering the diagnostic recommendation abilities. (iv) Furthermore, we find that the universal LoRA expert significantly improves the results on the general medical benchmarks (i.e., CMD and webMedQA), proving that it mitigates the competency conflict between general medical and pediatric knowledge.Effectiveness of Preference Alignment. (i) When the Direct Following Preference Optimization (DFPO) phase is removed, the model exhibits significant performance drops in doctor evaluations compared to the full version. This observation proves that DFPO effectively helps the model to align human preferences, reducing harmful content while generating doctor-like output. (ii) As two candidates, the vanilla DPO and RLHF methods are inferior to the proposed DFPO, suggesting that our strategy can more safely control model behaviour, leading to more favoured humanistic responses.Effect of Specific Expert Numbers. As a complement to the ablation of LoRA experts, Figure 6 (a) explores the gain effects of varying the number of specific experts while maintaining the universal expert. (i) Noticeably, our MCE strategy with three specific experts achieves a reasonable performance trade-off across the three tasks with only 0.95% trainable parameters. (ii) Conversely, excessively introducing LoRA experts does not result in appreciable gains but increases the training overhead.Analysis of Expert Utilization. To confirm the duties of specific LoRA experts in the routing process, we visualize the normalized weights assigned by the routing gating when encountering data from different downstream tasks. CMD and webMedQA data are merged to compose general healthcare data. From Figure 6 (b), (i) Experts 2 and 3 are emphatically activated on the TreRecom and MedKQ&A tasks, respectively, implying their focal ability to tackle medical knowledge interpretations and treatment recommendations. (ii) In contrast, Expert 1 is more proficient at learning multi-turn diagnosis semantics in the EviDiag task, which is different from the other tasks of instruction content. (iii) Additionally, there is no clear difference in the specific expert utilization on general healthcare, implying that the general task is handled by the consistently universal expert. The above observations demonstrate the effectiveness and necessity of the proposed MCE strategy.To intuitively compare the output quality of medical LLMs, we show the responses of different models for each of the three types of medical inquiries in Figures 13&14&15 from Appendix E .From the results, Zhongjing offers insufficient information due to limited output content. Although HuatuoGPT-II gives well-organized responses, it lacks accuracy and informativeness. In comparison, our model can provide more specialized and detailed medical knowledge and diagnostic guidance in extended response contexts, confirming its application potential in diverse healthcare services. Ethical Issues. We fully recognize the critical importance of privacy and data protection. All data used has been meticulously de-identified, with all sensitive information removed, and this process has been verified by the partnering medical institutions. For the public databases, we strictly follow specific license agreements for use and adaptation. For the constructed corpus, we underwent an internal ethical review by the ethical review board of the partnering medical institutions with license and approval. We will release relevant resources to the extent that they are controlled and permitted. 
conclusion and discussion: 
 We provide more discussions of the future work in Appendix F. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We fully discuss the potential social impacts in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: To ensure the safe release of data, we describe measures for dataset construction in detail in Section 3.1. To ensure that the responses of the proposed model are harmless and safe, we provide adversarial instructions to control the model behaviour in Section 3.3. Meanwhile, we perform human preference optimization for the model in Section 3.4, which further strengthens the safety and robustness of the model.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We provide reasonable references for the datasets and models used in Sections 4.1 and 4.2, respectively.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide the relevant documentation in the Appendix.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: We offer $300 each to participating experts. Expert doctors are asked to evaluate the quality of the models' responses. The relevant descriptions can be found in Section 4.3.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes] Justification: The research in the paper has no risks to be faced by participants who are only used as evaluators. Moreover, we underwent an internal ethical review by the ethical review board of the partnering medical institutions with license and approval. The relevant descriptions can be found in Section 5.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Role-playing-driven Instruction Building RuleAfter integrating pediatric textbooks, guidelines, and knowledge graphs into a consolidated textual database, the content is segmented according to individual diseases. Subsequently, two instances of the GPT-4 model are deployed, designated as the "inquirer" and the "expert pediatrician" respectively. Disease-specific segments are then fed into the "inquirer" GPT-4, tasked with formulating a series of relevant and scholarly pediatric inquiries. Following this, the original disease segments and the formulated inquiries are fed into the "expert pediatrician" GPT-4 to generate precise responses for each inquiry, leveraging the segmented text as the contextual reference. We show the prompt templates for the "inquirer" and "expert pediatrician" in Figures 7 and 8 , respectively.We guide GPT-4 to regularize concise and noisy doctor responses in authentic doctor-patient consultations by the context learning strategy. Specifically, we manually craft 100 instruction examples with high-quality content to allow GPT-4 to learn doctor-like and patient-friendly behavioral styles.In each round of regularization, we randomly sample 10 out of 100 examples to perform 10-shot context prompts. Immediately after that, vanilla dialogues are fed to GPT-4 as seed instructions to optimize instructions according to user requirements. Constrained by the space, we show the prompt case with one example in Figure 9 .Medical knowledge integration from existing datasets is common but frequently imprecise, resulting in unclear or incomplete instructions and potentially inaccurate outputs. Consequently, 107,177 instructions are selected from three significant benchmarks, prioritizing quality over quantity. In this case, we design a progressive instruction reconstruction rule to refine these instructions, ensuring informative and logical model responses.As shown in Figure 10 , we first prompt GPT-4 to take the perspective of the experienced doctor to complete Tasks 1 and 2 in the given instruction and answer scenarios. Task 1 focuses on bridging the gaps in the vanilla instructions and reinforcing the completeness, professionalism, and medical relevance. Based on the refined instructions, Task 2 requires the GPT-4 to make further targeted improvements to the answer parts. In practice, this progressive reconstruction rule can activate better instruction following capabilities in advanced language models.To enrich the density and breadth of the multiple-department CPT corpus, we transform the structured instruction data from the vanilla PedCorpus dataset into comprehensive medical knowledge texts using knowledge-enhanced prompts. The medical knowledge texts are integrated as complementary content to construct the PedCorpus-CPT dataset. The prompt template is shown in Figure 11 .In this section, we list in detail the hyper-parameter configurations for the different training phases.Continuous Pre-training. During this procedure, we train each model for just a single epoch, setting the learning rate at 1e-6 and the batch size at 128. We adopt a maximum cutoff length of 4096, enabling the model to process extensive text sequences in one batch. This significantly enhances the model's contextual understanding and coherence.Full-parameter Supervised Fine-tuning. In this configuration, we train all models for three epochs with a learning rate adjusted to 5e-5 and a batch size of 64, capping the maximum sequence length at 2048. We introduce a warmup_steps setting at 200 to gradually ramp up the learning rate from an initial lower value, aiding the optimizer in adapting to gradient changes. This approach boosts stability and performance and guides the model towards a better convergence path. Also, we specify eval_steps at 100 and save the best-performing weights on the validation set to ensure optimal results.Human Preference Alignment. In this setup, we train five epochs with the learning rate set to 1e-6 and the batch size maintained at 64. To enhance the robustness and smoothness of preference learning, we adjust the control parameter β to 0.1 and the scaling coefficient µ to 1.0. We specify eval_steps at 100, selecting the best-performing weights on the validation set.LoRA-based Parameter-efficient SFT. Here, we train three epochs with a learning rate of 1e-6 and adjust the batch size to 32. We configure the LoRA parameters by setting the rank r to 8, α to 16, and the Dropout rate to 0.05, targeting all modules. The default number of LoRA adapters is set to 4, including one constant universal expert and three specific experts. Ultimately, we select the adapters that perform best on the validation set.We consider four complementary dimensions in the automated evaluation to guide GPT-4 in judging the quality of model responses from a comprehensive perspective. The full definitions of these dimensions are shown as follows.Usefulness: measures the extent to which the model response has pediatric expertise and relevance to the instruction intention.Correctness: measures the extent to which harmful, misleading, and inaccurate information is present in the model response.Consistency: measures the degree to which the model response is logically self-contradictory and the information is coherent in context.Smoothness: measures whether the response content is fluent, natural, and conforms to the language expression style of human habits.In this case, we present GPT-4 with paired responses from different models, assessing various criteria such as pediatric expertise in the responses, presence of harmful, misleading, or inaccurate information, logical consistency, and the fluency and naturalness of the language, which should conform to human linguistic habits. GPT-4 assesses these responses on their merits and selects the superior one. To maintain fairness and mitigate potential position bias, the order of the responses is randomised. This methodology is supported by recent studies demonstrating GPT-4's strong agreement with human judgment in evaluating responses. Figure 12 demonstrates the prompt template used to evaluate the quality of paired model responses.In this section, we visualize the responses of the proposed PediatricsGPT-13B and two SOTA Chinese medical LLMs across three tasks from the same medical inquiries to provide intuitive qualitative comparisons. Specifically, Figures 13 and 15 illustrate the medical knowledge question-answer and treatment recommendation tasks, respectively, which follow a single-turn dialogue pattern. The multi-turn conversation pattern is considered in the evidence-based diagnosis task from Figure 14 .We list future work below to provide potential optimization directions.Enhancing Security Against Model Manipulation. To mitigate the security risks associated with online deployment, our future strategy involves implementing multi-layered security measures for the proposed PediatricsGPT model. This will include advanced input validation techniques to detect and neutralize potentially malicious inputs that could manipulate model outputs. Continuous updates and patches will also be prioritized to address emerging security threats and vulnerabilities.Expanding Language Support. To overcome the challenge of incomplete language coverage, we are committed to broadening the linguistic capabilities of PediatricsGPT. This expansion will involve training the model on a more diverse dataset that includes a broader range of languages and dialects, particularly those prevalent in underserved regions. By doing so, we aim to make the model more accessible and useful to a global audience, ensuring that non-Chinese speakers also benefit from reliable and localized medical information.
<SYS>Please act as an inquirer with a broad reserve of pediatric knowledge and complete the following requirements： 1. Based on the rich corpus of pediatric knowledge, carefully formulate a series of valuable, logical and inspiring questions； 2. Ensure that the uniqueness of each question is designed to comprehensively cover the needs of medical applications such as pediatric knowledge and answers, the consultation process, and advice on diagnosis and treatment of diseases, and to avoid simplicity or repetitiveness of questions； 3. All questions should be strictly limited to the scope of language processing, does not involve pictures, audio and other nonverbal form of the question, and shall not contain any sensitive or private information that may be involved in the real world; 4. Output the generated questions in the following format: {"Q1": "", "Q2": "", ..., "Qn": ""}； Please generate the questions directly.</SYS>
1. Laboratory Tests (1) Measurement of anterior pituitary hormone levels: Cortisol (F), Adrenocorticotropic Hormone (ACTH), thyroid function [Free Triiodothyronine (FT3), Free Thyroxine (FT4), Thyroid Stimulating Hormone (TSH), etc.], Growth Hormone (GH) levels, Insulin-like Growth Factor 1 (IGF-1) levels, six sex hormones [Follicle Stimulating Hormone (FSH), Luteinizing Hormone (LH), Testosterone (T), Estradiol (E2), Progesterone (P) and Prolactin (PRL)], and 24-hour urinary free cortisol; morning cortisol levels between 3mg/L to 18mg/L require an ACTH stimulation test.(2) Patients with significant polydipsia and polyuria: Monitoring of blood ion levels (potassium, sodium, chloride), plasma osmolarity, 24-hour urine volume, 24-hour urinary free cortisol, urine specific gravity, urine osmolarity, and urinary electrolytes. For patients with difficult-to-diagnose central diabetes insipidus, a vasopressin test should be performed to determine the presence of central diabetes insipidus.(3) Blood tests (and cerebrospinal fluid when necessary) for tumor markers such as Alpha-fetoprotein (AFP) and Human Chorionic Gonadotropin (HCG) for differential diagnosis.Teratomas, mixed germ cell tumors etc., also show calcifications on CT. Mature teratomas have negative tumor markers like AFP, HCG, but often contain tissues such as fat that provide mixed densities for differential diagnosis; mixed germ cell tumors have varying marker levels depending on the proportion of components, which can be distinguished from craniopharyngiomas.
<USER> { "Q1": "What are the key laboratory tests required for diagnosing childhood craniopharyngioma, and how do specific hormone levels impact the evaluation?", "Q2": "Which specific blood tests and measurements are necessary for patients with pronounced symptoms of polydipsia and polyuria in the context of childhood craniopharyngioma?", "Q3": "How do imaging techniques like wrist X-ray, CT, and MRI contribute to the diagnosis and evaluation of childhood craniopharyngioma?", "Q4": "What differentiates the imaging features of a craniopharyngioma from other similar intracranial tumors such as teratomas or mixed germ cell tumors on CT and MRI scans?", "Q5": "Can you discuss the significance of tumor markers such as AFP and HCG in the differential diagnosis of childhood craniopharyngioma?", "Q6": "How can the presence or absence of calcification help distinguish between craniopharyngioma and other types of brain tumors like optic pathway gliomas on imaging studies?"[OUTPUT]<ASSISTANT>{ "A1": "Diagnosing childhood craniopharyngioma involves various key laboratory tests, notably hormonal evaluations, to assess endocrine function. Specific tests include measurements of serum cortisol, ACTH, thyroid-stimulating hormone (TSH), free T4, prolactin, and growth hormone levels. Low levels of these hormones can indicate hypothalamic or pituitary dysfunction, common in craniopharyngioma patients due to the tumor's proximity to these structures. Anterior pituitary hormone deficiencies are a hallmark, and monitoring them helps in assessing the extent of the tumor's impact on the endocrine system.", "A2": "For patients with pronounced symptoms of polydipsia and polyuria, specific blood tests include serum sodium, osmolarity, and blood glucose levels to rule out diabetes mellitus. Additionally, measuring serum antidiuretic hormone (ADH) levels is crucial to differentiate between central and nephrogenic diabetes insipidus. Urine osmolality and a water deprivation test followed by desmopressin (ADH analog) administration can help confirm central diabetes insipidus, which is often associated with craniopharyngioma.", "A3": "Imaging techniques such as wrist X-ray, CT, and MRI are pivotal in diagnosing and evaluating childhood craniopharyngioma. Wrist X-rays can reveal delayed bone age, indicative of growth hormone deficiency. CT scans provide detailed images of bone structures and can detect calcifications commonly seen in craniopharyngiomas. MRI is the most informative, offering high-resolution images of soft tissues, facilitating the identification of cystic components, solid tumors, and involvement of adjacent structures like the hypothalamus and optic chiasm.", "A4": "Craniopharyngiomas often present with a mixture of cystic and solid components and frequent calcifications visible on CT scans. In contrast, teratomas may also show calcifications but often have fat and different tissue types typical of germ cell origin. Mixed germ cell tumors may present with hemorrhagic and necrotic areas and elevate tumor markers like AFP and HCG, which are not elevated in craniopharyngiomas. MRI helps distinguish these tumors based on their unique signal characteristics and anatomical relationships.", "A5": "Tumor markers such as alpha-fetoprotein (AFP) and beta-human chorionic gonadotropin (HCG) are significant in differentiating craniopharyngiomas from germ cell tumors, which often elevate these markers. Craniopharyngiomas typically do not alter AFP or HCG levels, so normal levels of these markers can help exclude germ cell tumors and support the diagnosis of craniopharyngioma.", "A6": "Calcification is a prominent feature of craniopharyngiomas and is readily detected on CT scans. In contrast, optic pathway gliomas, typically associated with conditions like neurofibromatosis type 1, usually lack calcification and present as fusiform, enhancing lesions along the optic nerves or chiasm on MRI. Thus, the presence of calcification strongly suggests craniopharyngioma over optic pathway gliomas." }</ASSISTANT>[/OUTPUT] Figure 8 : Illustration of the GPT-4 prompt template for the "expert pediatrician" role. The original Chinese content is translated into the corresponding English version to facilitate intuitive understanding.
<SYS>Please act as an AI medical assistant and complete the multi-round medical dialogue refinement task by referring to the already refined honest medical-patient dialogue Med-D. The specific requirements are as follows: 1. Keep the patient's questions from the original dialogue and only refine the doctor's responses. Mimic the response style of the doctor in D, ensuring medical professionalism while making the doctor's responses both professional and patient-friendly. 2. For questions posed by the patient involving images, audio, or other non-verbal forms, clearly respond: "I apologise, but as an AI assistant, I cannot receive or process any files or images. I can only communicate and provide information through text." 3. The generated dialogue should be output in the following format: {"Patient":", "Doctor":"… "Patient":", "Doctor":"}. 4. The refined medical-patient dialogue Med-D is ${ "Patient": "Kidney disease. There is slight swelling in the lower legs; ultrasound shows a 0.6 cm kidney stone in the left kidney, multiple kidney cysts, slight hydronephrosis, diffuse changes in the renal parenchyma, and a shrunken left kidney (8 cm). Kidney function tests show urea at 7.26, others normal, and kidney function urine alpha-1 microglobulin at 10.13. Urinalysis shows red blood cells at 31.45, occult blood positive, and other items normal. Taking a quarter of an antihypertensive pill daily, blood pressure has been normal for the past one or two years." "Doctor": "Your kidney issues include a left kidney stone, multiple cysts, slight hydronephrosis, changes in the renal parenchyma, and a shrunken left kidney. The high red blood cell count in your urine suggests minor bleeding, and elevated alpha-1 microglobulin may indicate kidney damage. Here's a summary:1. Stones and Cysts: The stone might obstruct urine flow, and the cysts could impact kidney function. Shrinkage in your left kidney may result from these structural changes.2. Renal Changes and Hydronephrosis: These signs suggest tissue damage and reduced kidney function.3. Tests: High red blood cells and positive occult blood in urine tests point to slight bleeding. Your urea levels are normal, but further detailed checks are necessary.4. Management: Regularly monitor kidney function, maintain blood pressure with antihypertensive medication, and make dietary and lifestyle changes to reduce kidney stress, like limiting sodium and staying hydrated. Avoid kidney-damaging medications." "Patient": "Thank you, what medication should I take?" "Doctor": "I cannot provide personalized medical advice, but I can offer general information and suggestions:1. Blood Pressure Control: Continue taking your prescribed antihypertensive medications, as managing blood pressure is crucial for reducing kidney stress.2. Prevention of Stones: Depending on their type, your doctor may recommend medications like potassium citrate or hydrochlorothiazide (HCTZ) to manage calcium and uric acid levels in your urine.3. Kidney Function Protection: Medications such as angiotensin-converting enzyme inhibitors (ACEIs) or angiotensin receptor blockers (ARBs) may be recommended to help protect your kidney function. Urinary Tract Health: For urinary tract infections, antibiotics may be necessary. Use NSAIDs cautiously as they can worsen kidney damage.4. Hydration: Adequate hydration can help prevent stone formation and flush the urinary tract.5. Diet: Maintain a low-salt, lowprotein diet, limit meat and other protein-rich foods, increase fruit and vegetable intake, manage calcium intake properly, and avoid highoxalate foods like spinach and strawberries."}
<USER>{ "Patient": "I have muscle soreness in my lower limbs and red patches. Tripterygium wilfordii was very effective before, but I can't take it due to stomach ulcers. The symptoms recurred after I stopped taking it." "Doctor": "I'm very sorry to hear about your discomfort. Based on your description of muscle soreness in the lower limbs accompanied by red patches, it could be related to various diseases. I suggest you visit a hospital for further examination. " "Patient": "The doctor I saw said it's vasculitis." "Doctor": "Understood. Vasculitis is a possibility, but there are many types of vasculitis. To confirm the diagnosis, I recommend you visit a dermatology department for further examination. "[OUTPUT]<ASSISTANT>{ "Patient": "I have muscle soreness in my lower limbs and red patches. Tripterygium wilfordii was very effective before, but I can't take it due to stomach ulcers. The symptoms recurred after I stopped taking it.", "Doctor": "Since Tripterygium wilfordii isn't suitable due to your stomach ulcers, you'll need alternative ways to manage your symptoms. Consulting with your healthcare provider is crucial to tailor a treatment plan, especially as your symptoms suggest multiple potential underlying conditions. Here are some non-medical strategies to help manage your muscle soreness and skin symptoms:1. Warm Compresses: Apply warmth to sore muscles to increase blood flow and relieve pain. Warm compresses can also soothe inflamed red patches, depending on their cause.2. Gentle Exercise and Stretching: Light movement can alleviate muscle soreness. Be gentle to avoid worsening the pain.3. Over-the-Counter Pain Relief: Consider using acetaminophen or topical NSAIDs (if approved by your doctor) to manage pain and inflammation. Use oral NSAIDs cautiously because of your stomach.4. Hydration and Nutrition: Keep hydrated and maintain a balanced diet to support muscle and skin health.5. Skin Care: Adopt gentle skin care routines, avoid irritants, and consider hypoallergenic products, depending on your specific skin condition." "Patient": "The doctor I saw said it's vasculitis.", "Doctor": "Thanks for sharing your situation. Managing vasculitis can be tricky, especially with stomach ulcers. Since Tripterygium wilfordii isn't an option, consider these alternatives with your doctor:1. Corticosteroids and Immunosuppressants: These reduce inflammation but must be used carefully due to your ulcers.2. Stomach-Protecting Medications: You might combine anti-inflammatory treatments with medications like proton pump inhibitors or H2 blockers to protect your stomach. Medical knowledge integration from existing datasets is common but frequently imprecise, resulting in unclear or incomplete instructions and potentially inaccurate outputs. Consequently, 107,177 instructions were selected from three significant benchmarks(Huatuo-26M, MedDialog, and CMeKG), prioritising quality over quantity. A progressive reconstruction rule was applied to refine these instructions, ensuring informative and logical model responses.
<SYS>Please assume the role of an experienced doctor, as you are now provided with a pair of medical instruction data, which includes the fields 'instruction' and 'answer'. Please complete Task 1 and Task 2 according to the following guidelines: Task 1: Refine and expand the 'instruction' field. Requirements:1. Utilize your professional medical knowledge to expand 'instruction', turning it into a medical question of depth and relevance. 2. The question should be specific, clear, and capable of eliciting detailed medical discussions. 'Instruction' should contain sufficient background information to facilitate understanding and response. Task 2: Based on the refined 'instruction' from Task 1, write the 'output' field. Requirements:1. 'Output' should integrate the information provided in 'answer', enriching the content while maintaining medical accuracy.2. The output should be logically coherent and fluently expressed, and must strictly eliminate any potential references to hospital and personal privacy information. The final response should be formatted as follows:{"instruction":"","output":""}</SYS>

<USER>{ "instruction": "What tests are needed for the Vertigo", "answer": "The purpose of testing for vertigo is to identify the cause of the symptoms, as vertigo can be caused by a variety of issues including inner ear problems, cervical spine issues, and brain diseases. Common tests include hearing tests, balance assessments, eye movement tests, and intracranial CT scans. However, the specific testing plan should be determined based on the patient's specific circumstances (such as the duration of symptoms, frequency of symptoms, accompanying symptoms, etc.). If you can provide more information, I will be better able to guide your treatment plan. Additionally, to alleviate symptoms, it is recommended that you rise slowly when getting up or changing positions, avoid excessive mental fatigue, and engage in appropriate physical activity.[OUTPUT]<ASSISTANT>{ "instruction": "What detailed diagnostic tests should be performed to confirm vertigo, and what are their purposes and specific contexts in which they are used? Please specify whether certain underlying causes, such as inner ear problems, cervical issues, or brain diseases, need to be considered and how to choose appropriate tests based on the patient's clinical symptoms. Are there any general recommendations that could help alleviate patient symptoms?" , "output":"The purpose of diagnostic tests for vertigo is to identify the underlying cause, as vertigo can be triggered by various issues, including inner ear problems, cervical spine issues, and brain diseases. To accurately diagnose vertigo, the following tests are commonly performed: 1. Hearing Test (Audiometry): This helps detect issues related to the inner ear or auditory system, as conditions like Meniere's disease or acoustic neuroma often cause vertigo. 2. Balance Tests: These assess the vestibular system's function by evaluating the patient's balance. Standard balance tests include the Vestibular Evoked Myogenic Potentials (VEMP) and Electronystagmography (ENG). 3. Eye Movement Tests: Electronystagmography (ENG) involves observing and recording eye movements to identify nystagmus and other abnormalities, which often indicate issues with the vestibular system. 4. Cranial CT Scan or MRI: These imaging tests are used to rule out brain disorders or intracranial pathologies, such as strokes, brain tumors, or multiple sclerosis, which can also lead to vertigo symptoms. The specific combination of tests should be determined based on the patient's clinical presentation, including the duration of symptoms, frequency, and accompanying signs. For instance, hearing and balance tests may be prioritized if a patient also experiences tinnitus or hearing loss. Conversely, imaging studies like a CT scan or MRI may be more warranted if the patient has headaches or other neurological signs. In addition to these diagnostic tests, some general recommendations can help alleviate vertigo symptoms: patients should rise slowly from bed or when changing positions, avoid overexertion or prolonged periods of mental fatigue, and engage in appropriate physical activities to promote overall circulation. These measures can help somewhat mitigate vertigo symptoms' severity." }</ASSISTANT>[/OUTPUT] 
<SYS>Please assume the role of an experienced data worker. Your task is to create high-quality medical knowledge texts based on the given instruction data. The specific requirements are as follows: 1. Drawing on the provided dialogue and your extensive knowledge base, compile an informative medical knowledge texts from the perspective of writing an encyclopedia. 2. Ensure that the generated content is objectively accurate. 3. The language should be smooth and concise.</SYS>

<USER> { "instruction": "What is hemophilia?", "output": "Hemophilia is a group of genetic bleeding disorders caused by the deficiency of certain clotting factors in the blood, leading to severe coagulation disorders. It can affect both males and females, but the vast majority of patients are male. This includes Hemophilia A, Hemophilia B, and Factor XI deficiency (formerly known as Hemophilia C). The first two are X-linked recessive inherited, while the latter is autosomally inherited as an incomplete recessive trait. Hemophilia is the most common congenital bleeding disorder, with bleeding being the primary clinical manifestation of the disease."Hemophilia is a rare genetic disorder characterized primarily by the inability to properly form blood clots, leading to prolonged bleeding. This condition is primarily due to the deficiency or absence of specific clotting factors, proteins essential for blood clot formation. The most common types of this disorder are Hemophilia A and Hemophilia B, which involve deficiencies in factor VIII and factor IX, respectively. Hemophilia A is about four times more common than Hemophilia B. The disorder is generally inherited in an X-linked recessive pattern, which explains its higher prevalence among males. Females can be carriers of the condition, and while they usually do not exhibit severe symptoms, they can still experience bleeding episodes. Another form, known as Hemophilia C or factor XI deficiency, follows an autosomal recessive inheritance pattern, affecting both males and females equally, and generally leads to milder symptoms. Individuals with hemophilia may experience spontaneous bleeding events or prolonged bleeding after injuries or surgery. Common complications include bleeding into joints and muscles, leading to pain and, over time, potentially severe joint damage. Modern treatment primarily involves replacing the missing clotting factor, either regularly as a preventive measure or to manage bleeding episodes when they occur. Diagnosis typically occurs in childhood and is based on blood tests that measure clotting factor levels. Advances in medical science, particularly the development of recombinant clotting factors and gene therapy, continue to improve the quality of life and life expectancy for those affected with hemophilia. Instruction-1: 我家孩子情况如下：男，因"生后呼吸困难10小时余"就诊。请问是什么原因呢？ (My child's situation is as follows: male, presented due to "difficulty breathing for over 10 hours" after birth. What could be the cause? ) Instruction-2: 孩子辅助检查结果如下：血常规：WBC 24.82×10^9/L, RBC 8.23×10^12/L，Hb 198 g/L, PLT 378×10^9/L, CRP 25mg/L。血气分 析：pH 7.20，PaO2 52mmHg，PaCO2 56mmHg，cHCO3-17.2mmol/L。血培养：阴性。胸片：两肺野透亮度减低，弥漫细颗 粒状影，外周型支气管充气征。心超：动脉导管未闭 3.3mm，卵圆孔未闭 1.8mm，三尖瓣返流压差 58mmHg。 (The child's auxiliary examination results: Complete blood count: WBC 24.82×10^9/L, RBC 8.23×10^12/L, Hb 198 g/L, PLT 378×10^9/L, CRP 25mg/L. Blood gas analysis: pH 7.20, PaO2 52mmHg, PaCO2 56mmHg, cHCO3-17.2mmol/L. Blood culture: negative. Chest X-ray: Reduced transparency in both lung fields, diffuse fine granular shadows, peripheral signs of bronchial inflation. Echocardiogram: Patent ductus arteriosus 3.3mm, patent foramen ovale 1.8mm, tricuspid regurgitation pressure gradient 58mmHg.)Response-1: 你好，请问有什么可以帮助你的吗？ 很高兴为您服务，请问有什么可以帮助您的吗？ (Hello, may I help you with anything? I'm happy to assist you. Is there anything I can help you with? ) Response-2: 根据孩子的辅助检查结果，血常规各个指标都在正常范围。血气分析正常，但是cHCO3-偏低，说明孩子可能存在酸中毒。胸 片显示两肺野透亮度减低，可见弥漫细颗粒状影，可能是因为呼吸系统疾病引起的。心超显示动脉导管未闭、卵圆孔未闭、 三尖瓣返流压差较高，需要进一步评估和治疗。 (Based on the child's auxiliary examination results, all parameters in the complete blood count are within normal ranges. Blood gas analysis is normal, but a low cHCO3-suggests the child may have acidosis. The chest X-ray shows reduced transparency in both lung fields with visible diffuse fine granular shadows, likely due to a respiratory system disease. The echocardiogram reveals a patent ductus arteriosus, a patent foramen ovale, and a high tricuspid regurgitation pressure gradient, necessitating further evaluation and treatment.) (Based on the provided information, the child's medical evaluations indicate complex health issues across multiple systems: 1. Complete Blood Count reveals elevated WBC at 24.82×10^9/L suggesting infection or inflammation, high RBC and Hb possibly indicating polycythemia, normal PLT, and elevated CRP pointing to inflammation. 2. Blood Gas Analysis shows a pH of 7.20, elevated PaCO2, low cHCO3-suggesting metabolic acidosis with respiratory compensation, and low PaO2 indicating hypoxemia. 3. Chest X-ray reveals decreased transparency and diffuse granular shadows, suggesting infection or pulmonary edema, with peripheral bronchial inflation signs hinting at chronic lung conditions. 4. Echocardiography shows a patent ductus arteriosus and foramen ovale with right-to-left shunting, and a tricuspid regurgitation pressure gradient of 58 mmHg indicating significant pulmonary hypertension. These findings suggest lung infection or pulmonary edema, metabolic acidosis and respiratory compensation. Patent ductus arteriosus and foramen ovale could cause hemodynamic issues, including pulmonary hypertension. Immediate measures should include monitoring vital signs and oxygen levels, respiratory support, antimicrobial therapy, and acid-base correction. Continuous cardiac monitoring and potential cardiac support are necessary. Coordination with a multidisciplinary team including pediatric cardiology and respiratory specialists is crucial for an integrated treatment approach.) Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.† Equal first contributions. § Equal second contributions. * Corresponding authors. Project lead. 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper presents PediatricsGPT, the Chinese pediatric Large Language Model assistant, which demonstrates expertise in the specialized domain of pediatrics. The authors propose a novel domain adaptation paradigm and training pipeline, including continuous pre-training, full-parameter supervised fine-tuning, and preference optimization.

The reviewers generally acknowledge the significance of this work as a milestone in Chinese pediatric LLMs and appreciate the innovative training approach. However, several concerns were raised, primarily regarding ethical issues, data privacy, and evaluation methodology.

The main ethical concern was the lack of explicit details about data usage agreements and IRB approval for human subject research. In their rebuttal, the authors clarified that they underwent an internal ethical review by a Chinese healthcare institution, which complies with NeurIPS 2024 guidelines. They also explained that all data sources have appropriate licenses and have been anonymized and desensitized to protect privacy.

Regarding the evaluation datasets, reviewers expressed concerns about their accessibility for further research and verification. The authors addressed this by committing to open-source both the model and datasets under the MIT license, ensuring that the research community can access and verify their work.

The use of the GPT-4 API for instruction data generation raised privacy concerns. The authors provided a detailed explanation of their data handling process, emphasizing that only non-sensitive medical knowledge was used and that strict measures were taken to prevent privacy leakage.

Some reviewers noted the absence of comparisons with leading medical LLMs like Meditron, Me-LLaMA, and GPT-4. The authors acknowledged this limitation and committed to incorporating these baselines in their revised evaluation.

After considering the reviews, rebuttal, and overall contribution of the work, I recommend accepting this paper. The authors have adequately addressed the main concerns raised by reviewers, and the work represents a significant contribution to the field of pediatric LLMs in Chinese. However, I strongly advise the authors to include the promised additional baselines and provide more detailed information about data usage agreements and ethical approvals in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes the first Chinese pediatric Large Language Model (LLM) assistant, designed through a robust training pipeline including continuous pre-training, full-parameter supervised fine-tuning, and direct following preference optimization. The model is shown to outperform existing Chinese medical LLMs in pediatric tasks, and the authors intend to open-source both the model and dataset for community development.
strength_weakness: strength_weakness: - Ethical Issues of the Data: The paper lacks explicit details about the usage agreements for the data sources, including license information and open-source details. Given the data leakage issues inherent in LLMs, clarifying these aspects is crucial for the ethical release and use of both the model and dataset.

- Evaluation Datasets: The three main evaluated datasets appear to be private, raising concerns about their accessibility for further research and verification. The paper should clarify whether these datasets can be openly released and if there are any ethic issues of releasing these datasets.

- Privacy Concerns with GPT-4 API: The use of the GPT-4 API to build instruction data introduces potential privacy concerns, particularly regarding whether data from guidelines and knowledge bases can be legally and ethically fed into a commercialized API like GPT-4.

- Missing Representative Baselines: The paper does not include comparisons with leading medical LLMs such as Meditron, Me-LLaMA, and other advanced models like GPT-4, LLaMA3. Including these baselines would provide a more comprehensive evaluation of PediatricsGPT’s performance.
questions: questions: N/A
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents PediatricsGPT, a specialized large language model (LLM) designed to assist in pediatric medical consultations in China, where there is a significant shortage of healthcare resources. PediatricsGPT is built upon PedCorpus, a high-quality dataset containing over 300,000 instructions from pediatric textbooks, guidelines, and knowledge graphs. The model undergoes a systematic training process that includes continuous pre-training, full-parameter supervised fine-tuning, human preference alignment, and parameter-efficient secondary supervised fine-tuning. This comprehensive approach aims to address the inadequacies of existing Chinese medical LLMs in pediatric applications.
strength_weakness: strength_weakness: 1) Overall I believe the developed model with the curated datasets will be a useful resource. But I think the authors can probably better highlight the novelty in this paper and how the techniques employed in this paper differs from other works.
2) As one of the main contribution of this paper lie in the creation of the dataset, I am not sure if this paper is more suitable for the D&B track in NeurIPS. However, this does not affect my overall rating. 
3) The improvements attributed to the universal expert appear marginal, raising questions about whether the gains justify the additional time and memory costs.
questions: questions: See above.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a Chinese pediatric LLM assistant, PediatricsGPT. It follows a standard pretraining and SFT pipeline to incorporate the general medical knowledge schema into the models. Specifically, they optimize the response to enhance the generation of pediatrician-like humanistic responses.
strength_weakness: strength_weakness: 1. The authors claim that they crafted 100 high-quality examples to guide the advanced language model, using in-context learning to regularize vanilla conversations in the self-instruct pattern, ensuring doctor-like and patient-friendly model responses. I would like to see a rigorous evaluation of this. 

2. According to Table 3, MUE, Universal Expert and RLHF seems to result in a limited performance gain.
questions: questions: 1. When designing such an LLM for pediatrics, is there any specific design/technique regarding such a target group, compared with designing a general medical LLM? I understand the preference optimization for the generation of pediatrician-like humanistic responses is one of such designs; however, is there any other specific design in the training/SFT/knowledge integration phase?

2. In Figures 2&3, how consistent is it for each question in GPT and doctor evaluation?
limitations: limitations: See weakness
paper_summary: paper_summary: This paper tried to build an AI-powered pediatric consultation system. Their motivation is current Chinese conversation LLMs for healthcare underperform in pediatric applications due to insufficient instruction data and bad training procedures. To tackle this problem, the authors proposed PedCorpus, a high-quality dataset with over 300K multi-task instructions, and PediatricsGPT, the first pediatric LLM assistant in Chinese. The PediatricsGPT baseline models, enhanced with a hybrid instruction pre-training mechanism, supervised fine-tuning, preference optimization, and a strategy to balance general and pediatric expertise, consistently outperforms previous Chinese conversation LLMs.
strength_weakness: strength_weakness: - "Direct Following Preference Optimization (DFPO) in human intention alignment is devised to enhance response robustness and align human preferences." - where is the evidence for this? It would be better to visualize some cases other than using ablation study as we can not see a huge performance difference.

- "ternary instances in the knowledge graphs" How to build the ternary instance for your task.

- "complementary resources" What is a complementary resource?

- Not sure how to protect patient privacy in your Real Doctor-patient Conversations subset.     

- As a pediatrician, I don't think the current model has "competence in different pediatric and general healthcare service scenarios". I review your data in the supplementary, it seems most of data do not contain real pediatrics AI challenge. To prove the generalization of the proposed model or prove the value of the dataset in Pediatrics, you should consider building datasets with more specialized pediatric diseases such as autism, cerebral palsy...     

- It is also important to test the performance of proposed model in other Out-of-Domain dataset, such as other medical QA dataset for adult.     

- It is confusing in your dataset building process. Why choose “expert pediatrician” GPT-4 but not real pediatrician to help you construct the dataset?

- "doctor-like and patient-friendly" - how to define doctor-like and patient-friendly. Do you mean 'professional' here?

- The scale of user study is limited. You state "We invite three doctors (each paid $300) to determine the winner of pairwise models by the majority voting rule." However, it would be better to conduct a wide user study from pediatricians as three people can not represent the human preference of the whole group.    

- Institutional Review Board (IRB) Approval is needed as this paper has human user study and the released data includes real doctor-patient conversations.    

- Some minor issues:
"Chinese medicine" can be replaced by healthcare service in Chinese as its clinical meaning can be found here: https://www.hopkinsmedicine.org/health/wellness-and-prevention/chinese-medicine
questions: questions: You can check the Weaknesses part and try to answer the question proposed in each item.      

As a physician, I agree this paper has enough novelty and can serve as a useful foundation model in the future. Thus I tend to give Borderline to this paper before reading the rebuttal. However, there are still plenty of issues that need the author to solve them.      

During the rebuttal, the author needs to solve the lacking issue of IRB Approval first. If IRB approval is not attached, the dataset and model may contain ethical issues and can not be released.       

I am also curious how ChatGPT achieves good medical and pediatric QA performance (only slightly lower than the proposed PediatricsGPT model) with in-context learning.
limitations: limitations: The main limitation is lacking IRB Approval. And most of the data do not contain real pediatrics AI challenges. This dataset can only be considered as a supplement of current released medical QA data for adults in China such as DISC-MedLLM.
paper_summary: paper_summary: - This paper introduces PediatricsGPT, a Chinese AI assistant for paediatrics
- They created a large dataset (PedCorpus) with 300k+ medical instructions
- The training process is pretty involved - includes pre-training, fine-tuning, and preference alignment
- They came up with some new techniques, like hybrid instruction pre-training
- Evaluation was thorough - used metrics, GPT-4, and even had real doctors test it
- Results show it outperforms other Chinese medical AI models in various health tasks, as well as ChatGPT
- The authors discuss its potential to support doctors and improve pediatric care
strength_weakness: strength_weakness: Overall the paper would benefit from more information on dataset construction, for each specific task and in general. In many cases, it is not clear what is meant by manual sampling, automatic extraction, dataset collection and similar. 

It would be great to see the standard deviation for the reported results.

It would be great to see a comparison with human experts, but this is too much to ask for now, so future work.
questions: questions: Line 12: Immediately, the... Please rephrase, a bit hard to understand

Line 29: Please either reference your own results, or cite a resource that supports your claim.

Line 101: Please add information on how was this done, how were the books selected and how was the knowledge automatically extracted. 

Line 109: Given that real patient conversations were used, it would be beneficial to know from where exactly was this data obtained, and were the appropriate ethical guideliens followed. 

Line 107: Manually sampled - does this mean a human went through everything and selected the appropriate data.

Line 164-168
- PedCorpus-DFPO is selectively sampled from vanilla PedCorpus - What does it mean "selectively sampled", how was this done? Were there any biases? What parts of PedCorpus were sampled and in which proportion?
- Humanistic stylistic rephrasing - what exactly is this? Who performed the rephrasing (humans or AI, or a combination)? What guidelines were used? How was the quality and consistency of the rephrasing ensured? 
- Low-capability medical assistant - please add a bit more info on why was the HuaTuo model chosen, and why is it low capability (how was this verified).

Line 208: How were the 208 difficult tasks sampled, and how was difficulty determined?

The paper would benefit (especially the first couple of pages) from a review of the grammar and sentence clarity.

Note: most of the datasets are in Chinese, maybe some information was obvious, but because of my lack of understanding of the language I was unable to determine that.
limitations: limitations: A very limited validation on real-world use cases, this is more of an initial research paper that can be built upon and further tested in the real world. Would be great to address this in the main part of the paper and not an appendix.
paper_summary: paper_summary: The authors of this paper built PedCorpus, a Chinese pediatric dataset, and PedistrcsGPT, the first Chinese pediatric LLM assistant. Their model was built via continuous pertaining, full-parameter supervised fine-tuning (SFT), direct following preference optimization, and parameter-efficient secondary SFT. Performance evaluation results show that their model outperforms other models.
strength_weakness: strength_weakness: 1. The techniques in this paper are not that novel, as it applies LLM to another field with main techniques either not original or adapted from others (e.g., DFPO). 
2. The paper focuses on the application level. I believe the model could be important if it is deployed. However, there is no evaluation from real users (patients or patients' parents). Doctor's evaluations could be different from the patients' evaluations, especially since this model will be used for children. This weakens this paper's contributions. I think the authors should at least discuss their plan of deployment or intervention as future work.
questions: questions: 1. Do you think DFPO performs better only on your specific task or can generally perform better than RHLF and vanilla DPO? If it can generally perform better, then this can be a great contribution.
2. How will you deploy your model in practice to complement the current shortage of pediatric healthcare resources?
limitations: limitations: The authors list two limitations, which make sense. I wonder whether your model can be 100% accurate. If it generates errors or hallucinations, it should be important to find them and avoid showing them to patients and their parents. As your model will be used for medicine and children, this should be more important than the LLM applied to other fields. This again goes back to my question before: how will you deploy your model in practice?

Comments: title: Discussion with Ethics Reviewer
It's reviewer discussion time
Official Comment by reviewer eZYU
Response to reviewer eZYU
Response to reviewer eZYU
Response to reviewer eZYU
Comment on Ethics Review
And about the Ethic reviews
About DISC-MedLLM
Thanks for the comment and we will make further clarifications about  the practical needs.
Thanks for the comment and we will make further clarifications about  the model and dataset.
Thanks for the comment and we will make further clarifications about the primary contribution.
Thanks for the comment and we will make further clarifications about similar solutions.
Thanks for the comment and we will make further clarifications about humanistic responses.
Thanks for your rebuttal and I will keep my score.
Response to Reviewer MG71
Response to Reviewer DSSd
Acknowledgement
comment: Dear Ethics Reviewer pxJq:

Thanks for your help taking time to check the ethic issue. Could you discuss with us based on authors' reply? Thanks. It is important for us to make the final decision based on the ethic review. 

Best,
Reviewer eZYU
Dear all reviewers,
This is reviewer discussion time. Your peer reviewers' reviews are in, as well as the author responses. 
As I understand the process now is when we go in for discussion and aim to form a consensus about this paper.

Can I ask each of you to do the following:
1. Read over each other's reviews
2. Review your own review
3. Look over the author's rebuttal and reviewer's feedback.
4. If you feel like any changes or improvements are needed to your own review -- please update accordingly.
5. Post as a comment to confirm that you've taken a look (just so we know who has/hasn't had a chance to respond yet).

Thanks!
Thank you for your response. I will consider adjusting the score after further discussion with the ethics reviewer. However, I believe it’s important to think about whether the ML/AI community is truly addressing the needs of the Medical/Pediatrics community. While AI/ML researchers often prioritize model novelty or the introduction of new datasets, those in the medical field are more concerned with addressing real clinical needs. This balance between innovation and practical application deserves careful consideration. Though LLM + X is a very good direction for us to publish papers, datasets and models that are out of touch with actual needs will cause many potential problems.
Dear Program Chairs:

We looked forward to addressing potential concerns during the extended discussions. However, we have found that the overall discussion seems to be unworkable. We have done our best to respond to comments to avoid misunderstandings and concerns based on multiple responses. We'd like to know if there's anything the authors can do now to.

Best regards,

Authors
Dear Ethics Reviewer pxJq:

We have clarified your concerns regarding ethical issues. We would like to know if there is anything else you would like us to clarify so that we can respond as soon as possible. We would like to address these potential concerns before the discussion is concluded to avoid subsequent misunderstandings due to the author's inability to participate in the discussion. We sincerely look forward to your response.

Best,

Authors
Dear reviewer eZYU: 

Once this work is accepted, the dataset and model will be released immediately. Of course, we can provide the required documents immediately if the policy allows it. We would like to emphasize that the internal ethical review we went through has the authors' names on the approval, and it's possible to violate the double-blind policy.

Best,

Authors
Dear reviewer eZYU:

As the authors are unable to reply under your comment in openreview at this point in time, we are replying to you in the comments under public rebuttal. We appreciate your time and effort as a professional. We very much agree with your discussion and suggestions on the dataset from your professional perspective as a pediatrician.

As an effort that seeks to explore and develop both in the pediatric and generic domains, we would like to try to build an initial bridge between the machine learning and pediatric communities by providing a new set of training paradigms for large language models as well as initial data benchmarks. Our original intention is to provide some insights into subsequent research, both at the data and modeling level, through our current efforts. The purpose of doing so is to avoid the serious disconnect you mentioned. We believe that someone has to take the first step in order to drive the future of large language model technology to more valuable applications in real-world clinical practice.

We hope that the reviewers will see the valuable and noteworthy contributions of this work, which will be the greatest encouragement for us to continue to work for community development.

Best,

Authors
Dear reviewer eZYU:

As the authors are unable to reply under your comment in openreview at this point in time, we are replying to you in the comments under public rebuttal. We appreciate your time and effort as a professional. We very much agree with your discussion and suggestions on the dataset from your professional perspective as a pediatrician.

As an effort that seeks to explore and develop both in the pediatric and generic domains, we would like to try to build an initial bridge between the machine learning and pediatric communities by providing a new set of training paradigms for large language models as well as initial data benchmarks. Our original intention is to provide some insights into subsequent research, both at the data and modeling level, through our current efforts. The purpose of doing so is to avoid the serious disconnect you mentioned. We believe that someone has to take the first step in order to drive the future of large language model technology to more valuable applications in real-world clinical practice.

We hope that the reviewers will see the valuable and noteworthy contributions of this work, which will be the greatest encouragement for us to continue to work for community development.

Best,

Authors
Dear Ethics Reviewer pxJq:

Thanks for your help taking time to check the ethic issue.
Could you reply to the authors? Thanks. It is important for us to give the final score based on the ethic review. And we really hope you can discuss them with the authors.

Best,    
Reviewer eZYU
I think it is also important to discuss related information about the IRB, dataset released timeline with the ethic reviewers, because the dataset is the biggest contribution of your paper. Can you provide them related documents? And we also need the ACs to encourage the ethic reviewers paticipating the discussion.

Best,     
Reviewer eZYU
Dear Author of Paper 385,

I believe there may be some misunderstanding regarding the reason I referenced DISC-MedLLM in my earlier comments. My intention was to highlight that while there are several datasets for medical LLMs with similar formats to yours, the key contribution, as you have stated, should be the quality of the dataset, particularly in the specialized field of Pediatrics.

Upon reviewing the data examples provided in both your paper’s appendix and the rebuttal document, I’m concerned that the dataset may not align with what pediatric practitioners would find most valuable for their domain. This is my areas of expertise. DISC-MedLLM encounters similar issues, which is why there’s resistance from healthcare professionals toward over-generalized declarations made by these models.

For instance, the data point in your appendix:
"My child fell while playing a few days ago and complained of severe pain in the lower back. After visiting the hospital, the diagnosis was confirmed as lumbar disc herniation."

The model's response:
"Lumbar disc herniation is rare in children, but treatment still requires caution. For pediatric lumbar disc herniation, pharmacological treatment mainly includes: 1. Nonsteroidal anti-inflammatory drugs (NSAIDs), such as ibuprofen or acetaminophen, can relieve pain and reduce inflammation; ... "

Similarly, in the rebuttal pdf:
"I'm glad to help! Pediatric cerebral palsy is a condition caused by a non-progressive brain injury that affects a child's movement and posture. The goal of treatment is to maximize the child's physical functioning and quality of life. Treatment is usually a multidisciplinary approach that includes the following: 1. Physical therapy: Specific exercises and activities are used to build muscle strength and flexibility and to help the child learn correct posture. 2. Medication: Childhood doses of medications are used to help control symptoms, including anti-spasticity medications such as baclofen, dantrolene, and botulinum toxin injections...... 3. Surgery: Selective posterior spinal rhizotomy or muscle tendon lengthening surgery is required for some children with severe muscle spasticity...... Each child with cerebral palsy is different, and the treatment plan should be individualized and tailored to the child's specific needs and abilities."

These responses resemble textbook excerpts or clinical notes, which may not be practical for real-world pediatric practice. This could lead to a significant disconnect between the machine learning and pediatrics communities, something we should aim to avoid.

To address this gap, I strongly recommend consulting with pediatric specialists (your clinical collaborators) to review your dataset. Their insights would be invaluable in ensuring that the data reflects the requirements of real-world clinical practice, rather than just verifying the accuracy of the model's reply.

Best,      
Reviewer eZYU
Dear Program Chairs:

Many thanks for your time and effort. We are always ready to discuss for necessary clarifications.

Best regards,

Authors
Dear Ethical Reviewer:

We look forward to receiving your reply if there is any further clarification required in response to the ethics review.

Best regards,

Authors
Hi all, the author-reviewer discussion for this paper is extended to Aug 16 11:59pm ET for authors to reply to ethics reviews. -- PCs
Dear Ethical Reviewer:

We appreciate your time and effort. The reason we wrote N/A when referring to the IRB in the checklist at the end of the paper is that research involving human subjects in our region of China usually undergoes internal ethical review by the collaborating organization. In other words, we underwent the internal ethical review by the corresponding Chinese healthcare institution, which is fully compliant with the ethical guidelines of NeurIPS 2024. The corresponding guidelines are described below: "In cases when no formal process exists, they can undergo an equivalent informal process (e.g., via their peers or an internal ethics review)."
Dear Reviewer eZYU: 

We appreciate the time and effort you have put into reviewing our work. We value the hard-earned time for discussion to address your concerns and make necessary clarifications. We are ready to discuss at any time.

Best regards,

Authors
**Q5**: Different pediatric departments were consulted to meet the practical needs of the researcher.



**A5**: We agree with the reviewer's suggestion. Considering the multifaceted tasks and diagnostic needs in real medical applications, we believe that there is no method currently available that ensures complete generalization to address the challenges in all applications. However, we have the ability to do a good capacity building of models for practical needs on several specific tasks.
With the suggestions and needs of collaborating healthcare institutions, our work focuses on three applications in pediatric healthcare, including knowledge question-answer, evidence-based diagnosis, and treatment recommendation. Correspondingly, we constructed novel MUE-based SFT processes through task-specific data for learning and mastering relevant knowledge. Also, we incorporates 131 pediatric diseases from 11 broad categories from textbooks to provide the pediatric knowledge needed for these tasks.

From Table 2 and Figures 2\&3, the experiments on the three task-specific testing benchmarks clearly demonstrate the necessity of the model in solving these pediatric tasks. More importantly, the doctor evaluations in Figure 3 also reflect the users' favor and approval of the model responses. Further, we provide more participants in Figure 1 of the **global response** to demonstrate the model effectiveness on more user studies. In addition, we also test the model's generalizability on publicly available healthcare benchmarks webMedQA and CMD with multiple departments, and provide corresponding insights and analyses.

As an exploratory research study, we believe that the current contributions and work are noteworthy and valuable. We are committed to collaborating with more pediatric departments and meeting further needs and tasks in our future plan.
**Q4**:  The model and dataset rely on previous work such as DISC-MedLLM [1].

**A4**: We clarify the differences between our approach and previous work in terms of both the model and the dataset.

**Model Level**: Current Chinese medical LLMs are constructed based on open-source foundation models for secondary development. In this case, we design a systematic training pipeline that incorporates the new direct following preference optimization (DFPO) and the mixture of universal-specific experts (MUE) structures, both of which are absent from previous Chinese healthcare models. Our DFPO facilitates the control of model behavioral boundaries, mitigating potentially harmful and unfaithful outputs. Our MUE structure effectively resolves conflicts between different pediatric tasks and the general medical knowledge, facilitating better knowledge learning and instruction following. In comparison, **DISC-MedLLM [1] does not have extra innovations in model structure and simply uses Baichuan-13B-Base to execute SFT procedures to develop medical competencies**.

**Dataset level**: We elaborate on our PedCorpus  and DISC-Med-SFT from DISC-MedLLM to emphasize that the two are completely different. 

As described in Section 3.1\&Table 1 of the manuscript, and the **global response**, the proposed PedCorpus consists of three parts, including pediatric data, real doctor-patient conversations, and distilled medical datasets. 

* The pediatric data comes from instruction pairs that we extracted from scratch from specialized textbooks, guidelines, and knowledge graphs, which are key to building models about task-specific pediatric applications.Among these, the knowledge graphs were constructed from scratch by ourselves from the pediatric domain. In contrast, DISC-Med-SFT uses a publicly available knowledge graph, CMeKG, which focuses on general medical knowledge and differs from our data sources and modeling philosophy.


* Our doctor-patient conversation data come from two components: voice transcriptions collected at our partnering healthcare institution and the publicly accessible dataset cMedQA2. For the former, we have clarified the relevant ethical issues in the global response and given privacy protection and implementation schemes that match the NeurIPS guidelines. For the latter, we have regularized the vanilla doctor response using a context-learning strategy, which leads to more organized and fluent responses compared to the vanilla instructions. In contrast, DISC-Med-SFT uses the publicly available MedDialog and cMedQA2, which adapts the entire dialogue through prompt engineering. We would like to emphasize that incorporating cMedQA2 in the construction of medical LLMs is a public practice rather than a particular work-specific design. However, the different paradigms for processing vanilla instructions by the different works resulted in completely different final SFT data. These observations and measures imply that doctor-patient conversation data in PedCorpus and DISC-Med-SFT are significantly different.


* For other datasets, we found numerous unclear and incomplete representations in the instruction instances from public benchmarks due to the absence of careful calibration, potentially triggering hallucinated outputs. To this end, we sampled a small set of knowledge-intensive instructions following the principle of quality over quantity. We introduced a progressive instruction reconstruction rule to distill the input and output parts of instruction pairs separately, resulting in completely different data compared to vanilla instructions. In contrast, DISC-Med-SFT primarily uses publicly available data from MedMCQA, moss-sft-003, and Alpaca-GPT4.


Depending on the citation format we provide, ACs and reviewers can readily access DISC-MedLLM to explore our statements and clarifications.



[1] Bao, Z., Chen, W., Xiao, S., Ren, K., Wu, J., Zhong, C., ... & Wei, Z. (2023). Disc-medllm: Bridging general large language models and real-world medical consultation. arXiv preprint arXiv:2308.14346.
**Q2**: Given that your model and data are positioned as the first Chinese pediatric LLM assistant, the primary contribution of this paper should be the quality of the dataset.


**A2**: As stated in lines 33-43, this paper addresses the shortcomings at the dataset and framework level of the LLM construction in the Chinese healthcare and pediatric domains. We propose a high-quality dataset PedCorpus through a series of new instruction-building mechanisms and a systematic training pipeline through new strategies to improve the different challenges in multi-stage training. Our technical novelty is distributed among these mechanisms and strategies.


For the dataset dimension, existing instruction data typically involve vanilla rephrasing of the general medical corpus or aggregation of doctor-like dialogues, which loses the specialization and focus in pediatric applications. More importantly, the current straightforward different round instruction construction paradigms fail to accommodate multi-task healthcare services in real-world scenarios, limiting the model generalization.

In comparison, the novelties of our PedCorpus derive from three characteristics.
* Task Diversity: besides containing generalist healthcare data, three application-oriented tasks are considered, including medical question-answer, evidence-based diagnosis, and treatment recommendation.
* Source Richness: distinct pediatric textbooks, guidelines, and knowledge graph resources provide solid assurance of medical knowledge's accuracy.
* Instruction Extensibility: vanilla instructions can be readily extended to seed instructions for generating specialized corpora to serve different training phases.



For the technique dimension, prior methods relied on SFT to compensate for medical instruction following capabilities, ignoring the discrepancies between inherent and externally absorbed knowledge within the models. This single pattern causes secondary LLMs to lapse into excessive role-playing rather than understanding. Despite a few attempts in the reinforcement learning from human feedback phases, their performance is restricted by actor-critic instability and online sampling bias.

In addition to the new techniques in the training process mentioned in **A1**, we propose three effective methods in ensuring the quality of the dataset. Specifically,


* As stated in lines 106-107, we present a role-playing-driven instruction building mechanism to convert the consolidated textual data into well-designed pediatric instructions. The proposed approach endows the advanced language model with the professional role of an expert pediatrician to generate accurate instruction knowledge for the target model training.
* As stated in lines 113-155, We introduce a in-context learning paradigm to regularize and refine vanilla conversations through advanced language models with the self-instruct pattern。
* As stated in lines 123-124, we design a progressive instruction reconstruction mechanism to distill the sampled instructions to ensure informative pediatric model responses. Unlike the traditional self-instruct pattern of using APIs, our mechanism guides APIs to take the perspective of the experienced pediatrician to complete progressive refinement tasks in the given instruction and answer scenarios.


Based on the above analysis and explanations, our main contributions are distributed among dataset construction and framework structure, and the different mechanisms and strategies in these two aspects help our work perform better on task-specific pediatric domains and on general purpose medical applications.
Dear Reviewer eZYU: 

We deeply appreciate the reviewer's comments. Based on your response and original comments, our rebuttal appears to have addressed the vast majority of your concerns. We would like to clarify the remaining concerns as follows.

**Q1**: The reviewer recognizes and appreciates the thorough evaluation and novel training pipeline. However, the reviewer affirms that there are similar solutions in the generalized domain.

**A1**:
We are concerns that we do not fully understand your insights into similar solutions. It would be helpful if the reviewer could present specific examples for reference that would help explore the interpretation and provide evidence for this statement. Here, we go through the content to help the reviewer address the concern.

We start by listing some of the other reviewers' comments to help you come to a consensus with them, as you hope to do in your response. As described by the **reviewer oou4**, our innovative training process, which includes continuous pre-training, full-parameter supervised fine-tuning, preference optimization, and mixture of universal-specific experts-based supervised fine-tuning, effectively adapts the model to the pediatric domain. The **reviewer jQHJ** explicitly noted the multi-stage training process of this work helped enhance the model's ability to produce good responses and adapt to the complexity of pediatric consultations. In addition, The **reviewers eEmB** and **MG71** pointed out multiple domain-specific innovations in our systematic training pipeline in the strengths, including the new preference optimization strategy DFPO and the hybrid instruction pre-training mechanism. Of course, these new contributions we have also accomplished explicit clarifications in the manuscript.  Specifically,

* As stated in lines 131-138, we propose a hybrid instruction pre-training mechanism in Continuous Pre-Training (CPT) to bridge the capability weakening due to corpus format discrepancies between the internal and injected pediatric knowledge of foundation models, facilitating knowledge accumulation and extension. Our mechanism better facilitates the adaptation of model capabilities in pediatric healthcare.
* As stated in lines 169-176, we present a stable method for domain-specific LLMs called Direct Following Preference Optimization (DFPO). DFPO utilizes variable changes to formulate the preference loss as a policy function that efficiently optimizes the policy with a simple binary cross-entropy objective. Meanwhile, our method directly regularizes model behavior boundaries in an instruction-following paradigm on medical demonstrations of preferred responses, facilitating robustness and smoothing of the preference learning.
* As stated in lines 186-199, we devise a mixture of universal-specific experts strategy to tackle the competency conflict between medical generalist and pediatric expertise in secondary SFT, which strengthens the model's adaptability to distinct downstream tasks. Specific experts master different pediatric tasks by soft routing gating control with noise. Also, a universal expert is consistently activated to prevent general knowledge forgetting and mitigate competency conflict. Our strategy enables our models to perform more comprehensively and competitively in pediatric medical applications.


We believe that the above contributions can significantly differentiate and strengthen the necessity of our work compared to techniques in the generalized domain.
**Q3**：Whether the conversational responses in the dataset support the assertion that the model produces humanistic responses.


**A3**: We would like to clarify that Appendix A.1 and Figures 7\&8 show how valid instruction pairs can be extracted from streamed text from specialized pediatric data.  At this stage, we expect the role-play driven API to build instructions with as much guaranteed knowledge completeness and accuracy as possible, rather than contextually adapted answers. Adapted answers will largely suffer from the API's own hallucination problem, leading to potentially noisy and non-factual information. After this stage, we feed the built initial instructions back to the pediatricians to help us with humanistic stylistic rephrasing, referring to making the responses as caring and well-presented as the pediatricians’.

We promise to supplement that the humanistic style comes from the physicians' rewrites after rather than extracting the preliminary instructions in the revision.
Dear Author of Paper 385,

Thank you for your rebuttal and the additional clarifications provided. After reviewing your comments, the rebuttal document, and revisiting the appendix of your original submission, I still have several concerns regarding your paper.

The main issue is not with the model architecture itself. I commend you for thoroughly testing various baseline models on your dataset and proposing a new training pipeline. However, it’s important to note that similar solutions have been proposed for general-domain LLMs. Given that your model and data are positioned as the first Chinese pediatric LLM assistant, the primary contribution of this paper should be the quality of the dataset.

Your paper makes a strong assertion that LLMs can be effectively used to answer pediatric questions and generate humanistic responses. However, the conversational responses in the dataset provided do not seem to fully support this claim, as they often appear to be direct extracts from textbooks rather than original, contextually adapted answers. Furthermore, it appears that your model, including the dataset, heavily relies on prior work such as DISC-MedLLM. While the concept of developing an LLM specifically for pediatrics is compelling, the current implementation may not adequately address real-world challenges.

I suggest consulting with researchers from different pediatric departments before proceeding further with this dataset to ensure that it meets the practical needs of the field.

Additionally, we require more time to discuss the ethical implications of your paper with the ethics reviewers once they submit their evaluations. Therefore, I will be maintaining my current score at this time. However, I will make every effort to reach a consensus with the other reviewers in the ML domain, and it is possible that the final score may be adjusted.

Best regards,
Reviewer eZYU
Dear Reviewer eZYU:

As the discussion period is closing, we sincerely look forward to your feedback. We deeply appreciate your valuable time and efforts. It would be very much appreciated if you could once again help review our responses and let us know if these address or partially address your concerns and if our explanations are heading in the right direction. Please also let us know if there are further questions or comments about this paper. We strive to improve the paper consistently, and it is our pleasure to have your feedback!

Best regards,

Authors
Dear Reviewer eZYU:

We would like to thank the reviewer for taking the time to review our paper and for the comments.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us.

Best regards,

Authors
Thank you for the detailed response. I will keep my score.
Dear Reviewer oou4:

As the discussion period is closing, we sincerely look forward to your feedback. We deeply appreciate your valuable time and efforts. It would be very much appreciated if you could once again help review our responses and let us know if these address or partially address your concerns and if our explanations are heading in the right direction. Please also let us know if there are further questions or comments about this paper. We strive to improve the paper consistently, and it is our pleasure to have your feedback!

Best regards,

Authors
Dear Reviewer eZYU:

As the discussion period is closing, we sincerely look forward to your feedback. We deeply appreciate your valuable time and efforts. It would be very much appreciated if you could once again help review our responses and let us know if these address or partially address your concerns and if our explanations are heading in the right direction. Please also let us know if there are further questions or comments about this paper. We strive to improve the paper consistently, and it is our pleasure to have your feedback!

Best regards,

Authors
Dear Reviewer eEmB:

We would like to thank the reviewer for taking the time to review our paper and for the comments.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us.

Best regards,

Authors
Dear Reviewer jQHJ:

We would like to thank the reviewer for taking the time to review our paper and for the comments.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us.

Best regards,

Authors
Dear Reviewer eZYU:

We would like to thank the reviewer for taking the time to review our paper and for the comments.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us.

Best regards,

Authors
Dear Reviewer oou4:

We would like to thank the reviewer for taking the time to review our paper and for the comments.

Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us.

Best regards,

Authors
We thank the reviewer for the meticulous advice!
Thanks for all your work and clarification. My concerns are addressed.
Many thanks to the reviewer for the overall constructive and insightful comments!
Thank you for your response. Most of my concerns have been addressed and I will keep my score.