HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

Authors: junying  chen
xidong  wang
ke  ji
anningzhe  gao
feng  jiang
shunian  chen
hongbo  zhang
dingjie  song
wenya  xie
chuyi  kong
jianquan  li
xiang  wan
haizhou  li
benyou  wang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 13:27:40

Abstract: Adapting a language model (LM) into a specific domain, *a.k.a* domain adaption, is a common practice when specialized knowledge, e.g. medicine, is not encapsulated in a general language model like Llama2. This typically involves a two-stage process including *continued pre-training* and *supervised fine-tuning*. Implementing a pipeline solution with these two stages not only introduces complexities (necessitating dual meticulous tuning) but also leads to two occurrences of data distribution shifts, exacerbating catastrophic forgetting. To mitigate these, we propose a one-stage domain adaption protocol where heterogeneous data from both the traditional pre-training and supervised stages are unified into a simple instruction-output pair format to achieve efficient knowledge injection. Subsequently, a data priority sampling strategy is introduced to adaptively adjust data mixture during training. Following this protocol, we train HuatuoGPT-II, a specialized LLM for the medical domain in Chinese. HuatuoGPT-II achieve competitive performance with GPT4 across multiple benchmarks, which especially shows the state-of-the-art (SOTA) performance in multiple Chinese medical benchmarks and the newest pharmacist licensure examinations. Furthermore, we explore the phenomenon of one-stage protocols, and the experiments reflect that the simplicity of the proposed protocol improves training stability and domain generalization.

Main paper: 
 introduction: 
 Currently, general large language model (LLM), such as the Llama series (Touvron et al., 2023) , are developing rapidly. Simultaneously, in some vertical domains 1 , some researchers (Cui et al., 2023; Yang et al., 2023c; Li et al., 2023a) attempt to develop specialized models. Specialized models have the potential to yield results comparable to those of larger models by utilizing a medium-sized model through the exclusion of certain general knowledge (Wang et al., 2024; Chen et al., 2023b; Yang et al., 2023c) . For instance, financial knowledge may not be sufficiently usefully in the medical field and can be therefore omitted in moderately-sized medical LLMs, thereby freeing up more capacity for memorizing medical knowledge.Additionally, in a spectrum of evaluation methods, the one-stage protocol of HuatuoGPT-II prove more effective than the traditional two-stage training paradigm.The key contributions are:• A unified protocol for domain adaption. The paper introduces a simplified one-stage domain adaption protocol for training, streamlining the traditionally complex pipeline process.• A advanced Chinese medical LLM. Our developed model, HuatuoGPT-II, leverages this protocol to achieve exceptional performance in Chinese healthcare domains, particularly in Traditional Chinese Medicine.• A novel generalization Test. A novel benchmark using the fresh 2023 Chinese National Pharmacist Licensure Examination provides a robust assessment of HuatuoGPT-II, addressing test data leakage concerns and demonstrating superior performance of HuatuoGPT-II. 
data collection: 
 Domain data is typically divided into two parts: pre-training corpora and fine-tuning instructions.Medical Pre-training Corpus Domain corpus is pivotal for augmenting domain-specific expertise. We collect 1.1TB of both Chinese and English corpus, sourced from encyclopedias, books, literature, and web corpus, blending general corpora like C4 and specialized corpora such as PubMed. All the corpora are publicly accessible, detailed in Appendix C. Then, a meticulous collection pipeline are established for curating high-quality domain data, detailed in the Appendix C. As a result, we obtained 5,252,894 premium medical documents for the pre-training corpus.Medical Fine-Tuning Instructions For the fine-tuning instruction, We acquire 142K real-world medical questions as instructions from Huatuo-26M (Li et al., 2023c) , and had GPT-4 respond to them as outputs. The fine-tuning instruction is utilized to generalize the model's capability to interact with users within the domain. The first method is suitable for detection within the same language, but it cannot handle cases where the English text is translated into a Chinese response. Therefore, we rely on a more robust method. We have people annotate whether an answer deviates and use this data to train a large language model for response content detection.Based on these detection methods, if a generated response fails the tests, we instruct the Large Language Model (LLM) to regenerate it. 
one-stage adaption: 
 One-stage Adaption strategy aims to unify the conventional Two-stage Adaption process (continued pre-training and supervised fine-tuning) into a single stage, as shown in Figure 1 . The adaption process can be executed in two succinct steps: 1) Data unification and 2) One-stage training.Subsequently, we will detail our method for adapting to the Chinese healthcare sector and developing and the development of HuatuoGPT-II.Domain pre-training corpus is pivotal for augmenting domain-specific expertise. However, it faces challenges such as diverse languages and genres, punctuation errors, and ethical concerns in its pre-training corpus. Particularly, there's a noticeable discrepancy between its unsupervised training and the supervised instruction learning in Supervised Fine-Tuning (SFT). Data Unification aims to unify this data into a consistent format, aligning it with SFT data. Traditional methods like those in Cheng et al., 2023 fall short due to the variation in language and genre. Hence, we leverage Large Language Models to achieve effective data unification.Our method of data unification is straightforward yet effective. We generate instructions based on the text of the corpus, and then we generate responses based on both the corpus and the instructions.The prompt for generating instructions is shown in Figure 2 .Please create a <question> that closely aligns with the provided <text>. Ensure that the <question> is formulated in [target language] and does not explicitly reference the text. You may incorporate specific scenarios or contexts in the <question>, allowing the <text> to serve as a comprehensive and precise answer.<text>: [domain-specific corpus] <question>:Figure 2 : The prompt for question generation.[target language] is Chinese, and [domain-specific corpus] refers to a corpus in the domain-specific pre-training corpora.After obtaining questions from the corpus text, we use the prompt, shown in Figrue 3, to let a Large Language Model (LLM) generate responses based on the questions and the corpus. Therefore, all pre-training corpora are converted into a instruction-output format, identical to our single-turn SFT data. An example of our final SFT data is shown in Table 7 .Here, we use ChatGPT as the LLM for data unification, converting all the medical corpus into instructions of the same language and genre. As detailed in Appendix F, Data Unification can be achieved independently of external LLMs. In Appendix E, we verified that the selection of LLM for data unification is inconsequential. This strategy also mitigates potential ethical concerns inherent in the corpus. Additionally, we also use statistical and semantic recognition to keep the model-generated answers close to the data's original content, as explained in Appendix H.In the one-stage training process, we integrate data from the Medical Pre-training Instruction and Medical Fine-tuning Instruction datasets to form dataset D. follow. It exacerbates the catastrophic forgetting of the knowledge acquired in the first stage (Bhat et al., 2022) . Referring to Touvron et al., 2023 , simply mixing all data could hinder the ability of Large Language Models (LLMs). To address this, we introduce a priority sampling strategy in this study. This strategy starts with domain knowledge, gradually transitioning to fine-tuning data while reducing the sampling ratio of previously learned data over time.In the priority sampling strategy, the sampling probability of each data x ∈ D changes over the course of training. The sampling probability of data x at step t during training is:P t (x) = π(x) ∑ y∈D-S t π(y)Here, π(x) denotes the priority of element x, and S t represents the sampled data before step t. The priority setting and data sampling distribution are delineated in Figure 4 . Notably, the priority π(x) is static, whereas the sampling probabilities of each data source dynamically changes. More precisely, consider data sources D t 1 , D t 2 , . . . , D t n ⊆ D -S t at time t, each with an assigned priority Published as a conference paper at COLM 2024π(x ∈ D t i ) = β K i .The probability of selecting an element from D t i is given by:P t (x ∈ D t i ) = |D t i |β K i ∑ n j=1 |D t j |β K jAfter an element is selected from D t i , the size ofD t+1 i becomes |D t+1 i | = |D t i | -1, resulting in: P t+1 (x ∈ D t+1 i ) < P t (x ∈ D t i ).Therefore, each selection from D i slightly decreases its sampling probability, leading to a dynamic update. The parameter β plays a crucial role in adjusting the sampling intensity among high-priority sources, with higher β values favoring sequential sampling, while lower values encourage mixed sampling.To enable the model to first learn domain capabilities and then gradually shift to instruction interaction learning, we assign a higher priority to pre-training instruction data. Furthermore, to facilitate the model's transition from low to high knowledge density learning, we assign different priorities to the four types of data sources. Appendix I shows details of how we determine sampling priority values for different types of data.Raw Corpus (1.1 TB) foundational corpus. We identified four primary data types for extensive collection: encyclopedias, books, academic literature, and web content. Focusing on these categories, we amassed 1.1TB of Chinese and English data, as shown in Table 9 .The domain data collection pipeline is an essential part of ensuring the quality of domain language corpora, designed to extract high-quality and diverse domain corpora from large-scale corpora.The methodology encompasses four primary steps:1. Extract Medical Corpus: This process aims to remove irrelevant domain corpora, serving as the first step in filtering massive corpora. We employ a dictionary-based approach, obtaining dictionaries containing medical vocabulary from THUOCL 5 and The SPECIALIST Lexicon 6 from the Unified Medical Language System. We strive to exclude non-medical terms to form a domain-specific dictionary. For each text segment, we evaluate whether it's domain-specific by assessing the density of matched domain words from the dictionary. This dictionary method is an effective and efficient way to extract domain text.2. Segmentation: Since we need to convert the corpora into instructions and do the data cleaning and de-duplication, it's necessary to segment all corpora into fragments. We divide each text into sentences, then use a moving window to turn the corpus into segments with a length limit, ensuring no loss of information by including sentences before and after the window.We notice a significant proportion of medical corpora is related to medical advertising, despite appearing fluent, contains little domain knowledge and introduces bias. To filter out medical advertisement texts and low-quality texts, we utilize ChatGPT to select problematic texts and train a corpus quality classification model to clean corpora. 
experimental settings: 
 Base Model & Setup HuatuoGPT-II, tailored for Chinese healthcare, builds upon the foundations o f the Baichuan2-7B-Base and Baichuan2-13B-Base models. Since all data consists solely of singleround format instruction, we enrich the Medical Fine-tuning Instruction dataset by integrating ShareGPT data3 . This allows HuatuoGPT-II to support multi-round dialogues while maintaining its general domain capabilities. For training details, please see the Appendix J.We compare HuatuoGPT-II with several leading general large language models known for their excellent general chat capabilities in Chinese. These models are Baichuan2-7B/13B-Chat (Yang et al., 2023a) , Qwen-7B/14B-Chat (Bai et al., 2023) and ChatGLM3-6B (Zeng et al., 2023) . Additionally, for the Chinese medical context, we carefully select DISC-MedLLM (Bao et al., 2023) and HuatuoGPT (Zhang et al., 2023) based on an experimental experiment detailed in Appendix R. We also consider leading proprietary models, including ERNIE Bot ( 文心一言) (Sun et al., 2021) , ChatGPT(OpenAI, 2022), and GPT-44 (OpenAI, 2023), noted for their extensive parameters and superior performance. For the details of these models, please refer to Appendix M.In this section, we evaluate the medical capabilities of HuatuoGPT-II on popular benchmarks. We focus on four medical benchmarks (MedQA, MedMCQA, CMB, CMExam) and two general benchmarks (C-Eval, CMMLU), focusing specifically on their medical components. See Appendix K for more details on these benchmarks. To ensure fairness, we use the most straightforward prompt for all the models, as shown in Appendix K. The Fresh Medical Exams Tackling data contamination in LLM training is challenging (Huang et al., 2023) , especially with extensive and intricate training data. To counter this, we selected the 2023 Chinese National Pharmacist Licensure Examination, held on October 21, 2023, as our benchmarks. This date is crucially before both our data collection cut-off (October 7, 2023) and the release of our assessment models. The annual uniqueness of the exam questions ensures a reliable safeguard against contamination risks. The results in Table 2 show that HuatuoGPT-II ranked second after GPT-4 in the Pharmacy track. However, in the Traditional Chinese Medicine track, HuatuoGPT-II led with 51.6 points. HuatuoGPT-II demonstrated superior performance in average scores across both tracks, highlighting its proficiency in the medical field.To evaluate the model's performance in real-world medical scenarios, we utilize real-wrold questions in both single-round and multi-round formats, sourced respectively from KUAKE-QIC (Zhang et al., 2021) and Med-dialog (Zeng et al., 2020) , following the approach of HuatuoGPT (Zhang et al., 2023) . The assessment details are provided in the Appendix L. We utilize GPT-4 to evaluate which of the two models generated better outputs. The results, as indicated in Table 3 (for more compared baselines, see Table 13 ), show that HuatuoGPT-II has a higher win rate compared to other models. Notably, HuatuoGPT-II achieve a higher win rate in comparison with GPT-4. Although its fine-tuning data originated from GPT-4, its extensive medical corpus provided it with more medical knowledge, as shown in Table 2 .Expert Evaluation For further evaluating the quality of the model outputs, we invite four licensed physicians to score them, with the detailed criteria available in the Appendix L.2. Due to the high cost of expert evaluation, we select four models for comparison with HuatuoGPT-II. Results, as outlined in Tables 3, indicate that HuatuoGPT-II consistently outperforms its peers, aligning with automatic evaluation. This consensus between expert opinions and GPT-4's evaluations underscores HuatuoGPT-II's efficacy in medical response generation. The case study on the model's response can be found in Appendix S. To validate the superiority of One-stage Adaption, we conduct a traditional Two-stage Adaption to fine-tune the the Baichuan2-7B-Base on the same medical data.Figure 5 shows the training loss of these two training methods. The loss of Two-stage Adaption shows instability, marked by pronounced fluctuations and loss spikes. This instability arises from the varied content and styles within the medical pre-training corpus, which includes four distinct data types as detailed in Table 9 . The variation is further amplified by the differences between Chinese and English datasets. Our one-stage Adaption unifies the diverse contents and styles of the pre-training corpus, improving training stability and reducing loss fluctuation. Moreover, due to data discrepancies between the two stages, there is a noticeable loss divergence between the pre-training and fine-tuning stages in Two-stage Adaption. In contrast, One-stage Adaption effectively resolves this issue by simplifying the pre-training corpus into a unified language and style, aligning with SFT data, making a more stable and smooth training process.The results of the previously mentioned experiments also demonstrate that One-stage Adaption achieves better performance than other methods, as shown in Figure 6 . It can be seen that on all six datasets, our One-stage Adaption performance is significantly better than the Two-stage Adaption performance (from 5.3% to 23%), especially in single-round Q&A and multi-round conversation tasks. This superiority is likely due to two-stage unification and more effective knowledge generalization in One-stage Adaption. In evaluating the priority sampling strategy, we tested it across different β settings using the same HuatuoGPT-II (7B) configuration. Our results show that model performance drastically drops when β = 0 (mixed sampling) or when β is excessively high (Sequential Sampling), highlighting the importance of priority sampling. The best performance is achieved when β is around 2.Moreover, we also conducted an ablation study on data unification in appendix G. The experiment indicates that data unification is crucial for the performance improvement of the one-stage training.De-duplication is a crucial step in corpus processing, as domain knowledge often has significant redundancy. We use a sentence embedding model to convert corpora into embeddings and then employ dense retrieval methods to remove semantically similar texts.Our data sources consist of four categories: (1) Web Corpus, which includes C4 (Raffel et al., 2020) and Wudao (Yuan et al., 2021) ; ( 2) Books, primarily comprising Textbook and the_pile_books3 (Gao et al., 2020) ; (3) Encyclopedia, encompassing Chinese Medical Encyclopedia and Wikipedia; (4) Medical Literature, which consists of PubMed and Chinese literature. Following the aforementioned steps, the corpus is converted into 5,252K of instruction data. 
conclusion: 
 In this work, we propose a one-stage domain adaption method, simplifying the conventional two-stage domain adaption process and mitigating its associated challenges. This approach is straightforward, involving the use of LLM capabilities to align domain corpus with SFT data and adopting a priority sampling strategy to enhance domain adaption. Based on this method, we develop a Chinese medical language model, HuatuoGPT-II. In the experiment, HuatuoGPT-II demonstrates state-of-the-art performance in the Chinese medicine domain across various benchmarks. It even surpasses proprietary models like ChatGPT and GPT-4 in some aspects, particularly in Traditional Chinese Medicine. The experimental results also confirm the reliability of the one-stage domain adaption method, which shows a significant improvement over the traditional two-stage performance. This One-stage Adaption promises to offer valuable insights and a methodological framework for future LLM domain adaption works. 
 
 Appendix: The development of HuatuoGPT-II, a specialized language model for Chinese healthcare, raises several potential risks.While HuatuoGPT-II has shown promising results in the domain of Chinese healthcare, it's crucial to underscore that at this stage, it should not be used to provide any medical advice. This caution stems from the inherent limitations of large language models, including their capacity for generating plausible yet inaccurate or misleading information.The ethical handling of data is paramount, especially in the sensitive field of healthcare. The primary data sources for HuatuoGPT-II include medical texts, such as textbooks and literature, ensuring that patient-specific data is not utilized. This approach aligns with ethical guidelines and privacy regulations, ensuring that individual patient information is not compromised. Another significant aspect of our methodology is the 'data unification' process, which aims to address potential ethical issues in the training data. By employing large language models to rewrite the medical corpora, we aim to eliminate any ethically questionable content, thereby ensuring that the training process and the model align with ethical standards. ZDSYS20230626091302006), and Shenzhen Stability Science Program 2023, Shenzhen Key Lab of Multi-Modal Cognitive Computing.Table 4 : The results of Chinese National Medical Licensing Examinations. The year represents the actual examination year. Note that the exam here may not be complete, and the blue fonts indicates the number of questions.To evaluate HuatuoGPT-II's medical competence, we utilize a dataset comprising medical examination questions from China and the United States. This approach aims to measure the model's adaptability and proficiency across varied medical knowledge frameworks and terminologies. The results of the Chinese National Medical Licensing Examination are outlined in Table 4 . In the results, HuatuoGPT-II (13B) not only surpassed all open-source models but also closely approached the performance of the leading proprietary model, ERNIE Bot. This notable outcome reflects the model's advanced understanding of Chinese medical principles and its adeptness in applying this knowledge in complex examination contexts. The 13B variant's elevated scores signify enhanced analytical capabilities and deeper comprehension of the nuances inherent in Chinese medical practice.CMB-Clin CMB-Clin is a dataset designed to evaluate the Clinical Diagnostic capabilities of LLMs, based on 74 classical complex and real-world cases derived from textbooks. Distinct from the response quality evaluations, CMB-Clin provides standard answers for reference and scores each model individually. We follow the same evaluation strategy from the original paper which utilizes GPT-4 as the evaluator. Results, as delineated in Table 5 , indicate that HuatuoGPT-II outperforms its counterparts, excluding GPT-4. Intriguingly, the 7B version of HuatuoGPT-II demonstrates superior efficacy over its 13B variant, a phenomenon potentially attributable to foundational capacity variances, as evidenced by Baichuan2-7B-Chat's superior performance compared to Baichuan2-13B-Chat. Meidcal Books (cn)Table 7 illustrates the process of generating instruction and response based on documents in Data Unification.Raw Data (from English Textbook)It is one of the most frequent mental illnesses, making up a large proportion of persons in assisted living and skilled nursing facilities. Although Alzheimer disease has been described at every period of adult life, the majority of patients are in their sixties or older; a relatively small number have been in their late fifties or younger....什么是阿尔茨海默病的主要临床和病理特征？( Translation: What are the main clinical and pathological features of Alzheimer's disease?)Answer of Data Unification 阿尔茨海默病是一种最常见的精神疾病，占据了较大比例的辅助生活和专业护理设施的人群。虽然阿尔茨 海默病在成年期的各个阶段都可能出现，但大多数患者在六十岁或以上；只有一小部分在五十多岁甚至更年 轻...(Translation: Alzheimer's disease is one of the most common mental illnesses, occupying a larger percentage of the population in assisted living and skilled nursing facilities. Although Alzheimer's disease can occur at all stages of adulthood, most patients are in their sixties or older; only a small percentage are in their fifties or even younger...) Table 7 : An example of our final SFT data.As shown in Section 3.1, we use external LLMs (like ChatGPT) for data unification for two main purposes:1. Data Unification using ChatGPT: The primary data source at this stage is raw medical corpora, not ChatGPT. ChatGPT is merely utilized for data refinement and back translation. Its function is similar to rule-based data polishing software or a fine-tuned T5 model, where medical expertise is not essential. We will subsequently present a case study using 'Baichuan2-Chat-7B' as an alternative for data unification.2. Fine-tuning Datasets using GPT-4 and ChatGPT: Our fine-tuning data and ShareGPT are used to train our model to follow instructions and engage in conversation. However, they don't significantly contribute to the medical knowledge of our model. This aligns with the 'Superficial Alignment Hypothesis', suggesting that a model's knowledge and capabilities are primarily derived during its pre-training phase. Instruction fine-tuning mainly guides the model to apply specific formats and styles during user interaction, rather than substantially enhancing its knowledge base.To illustrate, we conduct experiments to deeply analyze how using external large models affects HautuoGPT-II's medical knowledge. We sample 1/40 of our pre-training and fine-tuning data to assess their impact on the model's medical capability: The comparison between Experiments #4 and #5 shows that augmenting pre-training data is more effective than adding an equivalent amount of instruction tuning data. This suggests that the enhancement in medical knowledge benchmarks is primarily due to the raw data itself. ChatGPT play a role in transforming this raw data into formats that are more digestible for the trained LLMs, leading to our model outperforming ChatGPT and GPT-4 in medical benchmarks.Exper. No. # Fine-Furthermore, as outlined in Appendix M, leading Chinese medical models like DISC-MedLLM also employ ChatGPT and GPT-4 for their training data construction, yet our model surpasses them in domain benchmarks.To eliminate the performance improvement of our method caused by relying on external large language models, we experimente with adapting Baichuan2-7B-Chat in the medical domain using our method, without the assistance of other LLMs. We randomly select 5% of the pre-training corpus and medical queries, using Baichuan2-7B-Chat itself for data unification and direct response to the SFT-data's queries as fine-tuning data, a process we term as self-data-unification. The experiment is carried out as shown in Table F .The results show that when using self-data-unification to perform self-data-unification, our method named One-stage Adaptation still exceeds Two-stage Adaptation by an average of 6.6 points. This Table 9 : Adaptation gaps between Two-stage and On-stage adaptation.Moreover, we find even Baichuan2-7B-Chat demonstrates a good ability to perform data-unification, as shown in the Table 10 :Description content Raw Data(from PubMed)The functional and structural alterations of the striatum in chronic spontaneous urticaria. The brain has long been known to be the regulation center of itch, but the neuropathology of chronic itch, su ch as chronic spontaneous urticaria (CSU)... To validate the importance of Data Unification, we conducted an ablation study on Data Unification using a reduced dataset (5% due to time constraints) as shown in figure G.In the data unification phase, we instruct LLM to refer to the text content to provide a detailed response. The responses are expected to mainly contain information from the text. However, the ability of the language models to follow instructions can't be fully guaranteed, and there might be instances where the model answers a question without referring to the text. To ensure that the response contains text knowledge, we adopt the following two methods to detect deviations from the original text:1. Statistical Method: We convert both the text and the response into sets of 1-grams, and then use the Jaccard Similarity Coefficient to calculate the similarity between these two sets to determine the similarity of content between the text and the response. We set a threshold to detect content deviations.In this section, we first explain how we handle ordering between pre-training and SFT data, and then give details of ordering within pre-training dataOrder between pre-training and SFT data We adopt a sequential training approach, prioritizing Pre-training Data followed by Fine-tuning Data (Pre-training Data→Fine-tuning Data). This method mirrors human learning processes: akin to initially learning from textbooks before attempting exercises. The rationale is straightforward comprehensive foundational knowledge, gained from pre-training data, should precede the more specific applications found in fine-tuning data. This sequence prevents a scenario akin to attempting exercises without sufficient textbook study, which could lead to random guesses and, in the context of our model, the propensity to generate inaccurate or 'hallucinated' information. Additionally, the fine-tuning data, often structured in a question-andanswer format, closely aligns with real-world user interactions. By introducing this type of data later in the training process, we ensure that the model is better attuned to actual user scenarios, enhancing its practical applicability and effectiveness.Order within pre-training data Data quality is gauged using the data sampling epochs from the LLaMA (Touvron et al., 2023 ) study, as is shown in Table 12 .Initially, the significance of different data sources, as determined by sampling epochs in LLaMA, was ranked as "Web Corpus→Literature→Books→Encyclopedia". Nevertheless, after consulting with medical experts, it became clear that Books should be accorded greater importance. Consequently, we have adjusted our training sequence to: Web Corpus→Literature→Encyclopedia→Books. Lastly, we set the relative priority, a hyper-parameter in priority sampling.Translation:Please answer the following multiple choice questions.Of little significance in assessing the prognosis of a patient with cirrhosis is We utilize the single-round and multi-round evaluation data from HuatuoGPT (Wang et al., 2023b) to evaluate the model's medical response capability. Sources from these datasets include KUAKE-QIC (Zhang et al., 2021) for single-round questions and Med-dialog (Zeng et al., 2020) for multiround cases. We slightly modify these evaluations for fairer assessment, focusing on the information in the answers rather than the tone of a doctor's response.This evaluation is designed to evaluate the response capabilities of large-scale language models in medical scenarios. It includes two types of evaluations. The first type assesses the singleround answer capability, primarily comprising real patient questions sourced from KUAKE-QIC.The second type is a multi-round diagnostic evaluation, with data containing real patient case information, sourced from Med-dialog. In the single-round evaluation, we have the models directly answer medical questions. In the multi-round evaluation, we simulate a patient asking questions to a doctor using ChatGPT, based on the patient's medical record information. The simulated patient's prompt is shown as below( [Patient Case Information] is patient case in multi-round data of HuatuoEval.). The models then engage in dialogue with the simulated patient to generate conversations. For a fair comparison, we have the ChatGPT-simulated patient continuously asking questions, and each model must respond twice. • Llama2-7B/13B-Chat (Touvron et al., 2023) Llama2 series are successors models of Llama trained on 2 trillion tokens with diverse datasets. Their chat model has robust performance. Unlike the first three Chinese-supported models, Llama2 series are English models.Additionally, we select two representative and strong large language models for Chinese medical scenario. (we conducted an experimental experiment to select these two models, see Appendix R for details). These models are:• DISC-MedLLM (Bao et al., 2023 ) DISC-MedLLM is fine-tuned over 470K medical data and 34k general domain conversation and instruction samples.• HuatuoGPT (Zhang et al., 2023) HuatuoGPT is trained using real-world data and distilled data from ChatGPT, adopting RLMF (a method combining ChatGPT and doctor preferences) to leverage the advantages of mixed data.Proprietary Baselines Furthermore, we compare three representative proprietary models, which have larger parameters and stronger performance:• ERNIE Bot ( 文心一言) (Sun et al., 2021) ERNIE bot is a closed-source large predictive model developed by Baidu. It is one of the strongest Chinese language models to date, with an API interface and a web version available for use.• ChatGPT(OpenAI, 2022) ChatGPT is a large language model released by OpenAI, possessing significant influence and currently holding an excellent standard among large models.• GPT-4(OpenAI, 2023) GPT-4 is a language model published by OpenAI following ChatGPT.It is undoubtedly one of the most powerful and widely used large language models currently available.We sampled 100 questions from the pre-training instructions and manually categorized their type as follows:Yes The effect differences of HuatuoGPT2 with different model parameters: We conducted experiments with different LLMs to generate SFT data, using 5% of the data, to verify the importance of SFT data. The results are shown in the table.It shows minimal impact with the other LLM, which indicates that SFT data does not provide primary knowledge. The proposed one-stage training protocol is applicable beyond the Chinese medical domain.To address the reviewer's question, we tested it on the MedQA-English (Jin et al., 2021) and CaseHOLD (Zheng et al., 2021) datasets, which represent the English medical and law domains, respectively. For MedQA-English, we used MedQA's English textbooks as the pre-training corpus and its training set for Supervised Fine-Tuning (SFT). For CaseHOLD, we sampled 4% of the corpora from FreeLaw Opinions (Biderman et al., 2022) When selecting baselines for Chinese medical applications, we also tested the performance of other Chinese medical large models in the Chinese National Medical Licensing Examination. The results, as shown in Table 20 , indicate that based on their superior performance, HuatuoGPT and DISC-MedLLM were chosen as the baselines for comparison.In our study, we observed that many models, including GPT-4, experience significant hallucinations in Chinese medical contexts. These hallucinations arise from two factors: 1) The model itself lacks specific medical knowledge; 2) Misconceptions arise in Chinese. Tables 21 and 22 present two examples of simple Chinese medical questions about pharmaceuticals."Vertical domains" refer to specialized fields such as medicine, law, and finance, where training focuses on deep, domain-specific knowledge and disregards other unnecessary domains. This contrasts with horizontally trained models, which possess broader but shallower general knowledge.This refers to using the Chinese language for healthcare, rather than Traditional Chinese Medicine.https://huggingface.co/datasets/philschmid/sharegpt-rawThe versions are gpt-3.5-turbo-0613 and gpt-4-0613. The same version we use in data unification.https://github.com/thunlp/THUOCLhttps://www.nlm.nih.gov/research/umls/new_users /online_learning/LEX_001.htmlThe results emphasize the importance of Data Unification. Without it, the single-stage method falls short of the two-stage one. This could be due to the considerable discrepancies between the two data stages. Simply merging them doesn't yield effective results. 


Decision: The paper on HuatuoGPT-II introduces a novel one-stage domain adaptation protocol that combines pre-training and supervised fine-tuning into a unified instruction-output pair format. The reviewers are happy with the paper for its clarity and the strong performance of HuatuoGPT-II on various Chinese medical benchmarks; they also express some concerns about the clarity of certain methodological aspects and some concerns about using GPT-4 for data generation and evaluation. The paper was rated 6 (marginally above the acceptance threshold), and the authors are asked to provide additional clarifications and address the raised concerns.

Comments from PC: Please include the additional results from the rebuttal in the revision, as they significantly help clarify the questions raised by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper introduces a one-stage domain-adaptation protocol for language models and produces a medical domain language model HuatuoGPT-II. Different from a two-stage process (pre-training, supervised fine-tuning), the paper proposes a one-stage approach to merge these two stages. The key technique here is to unify different format of data into instruction tuning pairs, leveraged by auto-labeling ability of large models (i.e., GPT-4). The paper also proposes a priority sampling strategy for training. The models are shown to perform better compared to other models on different Chinese medical benchmarks.
strength_weakness: strength_weakness: - The connection between training on pre-training corpora and training on instruction data generated from pre-training corpora is not clear to me. But it seems to be a core inspiration of the methodology. I think providing more and deep insights will be useful.
- The baseline approaches are a bit weak as they are mostly general-domain models.
- Given the paper uses GPT-4 to generate data, uses GPT-4 to evaluate results, and compares the proposed model with GPT-4, it will be tricky to interpret the results, especially when comparing to GPT-4.
questions: questions: - What types of questions are generated on the pre-training corpus? 
- The paper makes an assumption that training with instruction data (which is generated on pre-training corpus) has the same functioning of training on pre-training corpus. What is the intuition behind it?
paper_summary: paper_summary: The authors tackle the problem of domain adaptation of large language models (LLMs) by proposing a unified, one-stage adaptation process in the place of the more common two-stage approach of continued pretraining and supervised fine tuning. Their approach transforms the domain-specific pretraining corpus into a form that more closely resembles the instruction tuning data used during fine tuning. The authors select a filtered dataset of over 5 million documents in the medical domain to demonstrate this transformation process and how this data is used to adapt a general-domain LLM to the medical domain. The resulting model and competing baseline models are graded against a recent, previously unseen Chinese National Pharmacist Licensure Examination, with the hope that the contents of this test are not present in some form in either the general or domain-specific training data used by the models. Both automatic and human-expert evaluations are performed. Results show the authors' models generally outperforming the baseline models on Chinese-language tasks, including commercial offerings such as GPT-4.
strength_weakness: strength_weakness: While there has been great care to note that the licensing exam has been chosen for novelty (and was published after the data collection cut-off date), there may questions present in earlier published study guides that were then incorporated as training or tuning data. It may be worth a quick examination on scores from previous editions of the licensing exams for comparison.
questions: questions: - Can you define what is meant by 'vertical domains' in Section 1?
- A footnote for a definition of 'Philosophy of Parsimony' may be useful
- "unpopular languages" may be better represented as "lower-resourced languages" in Section 1
- Section 3.1, corpus is -> corpora are
paper_summary: paper_summary: This work presents a one-stage domain adaptation pipeline that relies on instruction tuning in place of the more common two-stage pipeline comprising continual pre-training and supervised finetuning. This approach aims to offer several advantages: a form of simplification, a means of reducing catastrophic forgetting, and mechanism to adaptively adjust data source during training. To test the efficacy of this model, there are numerous experiments (and a rich appendix) to substantiate claims.
strength_weakness: strength_weakness: There are some considerations that limit impact, namely:
- Prior work (Peng et al., https://arxiv.org/pdf/2304.03277) demonstrated the efficacy of instruction tuning with GPT-4, which would be useful to contextualize the present work. Further, it would be useful to add discussion around whether this setup might suggest that GPT-4 offers an upper found on performance for these tasks were it similarly finetuned. On the one hand, the experimental results suggest HuatuoGPT-II has a superior win rate, but it's not quite an equal comparison.
- Adding clarity around the underlying distribution shifts that are present would be useful because there is both a significant shift in language and general vs medical domain present in this work. In particular, the evaluation chosen relies on resources like MedQA which are primarily clinical questions (from medical exams) whereas the training data is drawn from biomedical (as opposed to clinical) domain.
- Focus the discussion of the results to emphasize the key takeaways. Currently there is a rich appendix that substantiates several claims made throughout the text but the main paper is difficult to read in absence of the appendix because key details have been relegated there, some of which could arguably be considered more important than those that appear directly in the results, such as Appendix N (Other Medical Models), which provides context for what performance means in this space.
questions: questions: Suggestions:
- Carefully consider whether there are results that appear in the appendix that would be better included in the main text.
- This is stylistic, but consider removing some parentheticals where not necessary, e.g. "(necessitating dual meticulous training)".

Questions:
- What are future directions can be highlighted in discussion that are revealed by differences between Huatuo and Huatuo-II?
- If a different LLM is used in place of GPT-4 for generating instruction tuning data, how significantly does this impact the quality of the resulting model?
- How is performance affected with a very small (e.g. 1B) model or much larger model?
paper_summary: paper_summary: This paper introduces a one-stage domain adaptation technique that combines diverse data from pre-training and supervised stages into a concise instruction-output pair format for efficient knowledge injection. The proposed mechanism, according to the authors, improves both training stability and domain generalization. Extensive experiments show the effectiveness of the method in four medical benchmarks (MedQA, MedMCQA, CMB, CMExam) and two general benchmarks (C-Eval, CMMLU).
strength_weakness: strength_weakness: - The model, HuatuoGPT-II, specically developed only for the medical domain in Chinese. I have reservation with the claims of training stability and domain generalization. Please refer to 'Questions To Authors' for more details.
- The quality of the writing is a little sloppy. For instance, the overuse of "Camel Case," which is superfluous, detracted from the quality of writing. Furthermore, it is wrong to say '... let **an** Large Language Model (LLM) generate...' and unusual in academic writing to say '... and close to the data’s original content...'. I would like the authors to carefully review these minor issues throughout the manuscript.
questions: questions: The authors claim that the proposed protocol improves training stability and domain generalization. What potential effects might it have on medical fields other than Chinese or other domains (e.g., law, finance and so on)?

Comments: title: Acknowledgement of rebuttal
Acknowledgement of Rebuttal
Acknowledgment on rebuttal
comment: Thanks for responding to my review. I will keep my score.
Thank you for providing additional information and clarifications for your work, in particular regarding performance on previous years exams. At this time, I will concur with the other reviewers and maintain the current review score.
Thank you for answering these questions. In particular, the answers to Q5 and Q6 are quite interesting and I would encourage the inclusion of these results, whether in the work itself or an appendix if space doesn't permit. For Q2, please do try to add clarity around this point in revision but it's not necessary to do so here.
Thanks for your efforts in preparing the rebuttal. I have read the authors' feedback and other reviews. I would prefer to maintain my initial assessment.