HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs
Authors: Junying Chen
Xidong Wang
Ke Ji
Anningzhe Gao
Feng Jiang
Shunian Chen
Hongbo Zhang
Song Dingjie
Wenya Xie
Chuyi Kong
Jianquan Li
Xiang Wan
Haizhou Li
Benyou Wang
Venue: COLM (2024)
Keywords: domain adaption, one-stage training, data sampling strategy, specialized LLM
TL;DR: A unified one-stage protocol for domain adaptation and a strong Chinese medical LLM.

Abstract: Adapting a language model (LM) into a specific domain, *a.k.a* domain adaption, is a common practice when specialized knowledge, e.g. medicine, is not encapsulated in a general language model like Llama2. This typically involves a two-stage process including *continued pre-training* and *supervised fine-tuning*. Implementing a pipeline solution with these two stages not only introduces complexities (necessitating dual meticulous tuning) but also leads to two occurrences of data distribution shifts, exacerbating catastrophic forgetting. To mitigate these, we propose a one-stage domain adaption protocol where heterogeneous data from both the traditional pre-training and supervised stages are unified into a simple instruction-output pair format to achieve efficient knowledge injection. Subsequently, a data priority sampling strategy is introduced to adaptively adjust data mixture during training. Following this protocol, we train HuatuoGPT-II, a specialized LLM for the medical domain in Chinese. HuatuoGPT-II achieve competitive performance with GPT4 across multiple benchmarks, which especially shows the state-of-the-art (SOTA) performance in multiple Chinese medical benchmarks and the newest pharmacist licensure examinations. Furthermore, we explore the phenomenon of one-stage protocols, and the experiments reflect that the simplicity of the proposed protocol improves training stability and domain generalization.

Published as conference paper at COLM 2024
HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs
Junying Chen 1.2, Xidong Wang 122 Ke ji Anningzhe Gao Feng Jiang', Shunian Chen 1., Hongbo Zhangl2 DDinjjie Songl?, Wenya Xie Chuyi Kong! Jianquan Li?,
Xiang Wan Haizhou Li Benyou Wang
Shenzhen Research Institute of Big Data
The Chinese University of Hong Kong, Shenzhen
junying. chen. cs@gmail. com, gaoamningzhe@sriee en, wangbenyou@cuhkked cn
Abstract
Adapting a language model (LM) into a specific domain, a.k.a 'domain adaption', is common practice when specialized knowledge, e.g. medicine, not encapsu- lated in a general language model like Llama2. This typically involves two-stage process including continued pre-training and supervised fine-tuning. Implement- ing pipeline solution with these two stages not only introduces complexities (nccessitating dual meticulous tuning) but also leads to two occurrences data dis- tribution shifts, exacerbating catastrophic forgetting. To mitigate these, we propose a one-stage domain adaption protocol where heterogeneous data from both the traditional pre-training and supervised stages unified into simple instruction- output pair format to achieve efficient knowledge injection. Subsequently, a data priority sampling strategy is introduced to adaptively adjust data mixture during training. Following this protocol, we train HuatuoGPI-LI, specialized LLM for the medical domain in Chinese. HuatuoGPT-II achieve competitive performance with GPT4 across multiple benchmarks, which especially shows the state-of-the-are (SOTA) performance multiple Chinese medical benchmarks and the newest phar- macist licensure examinations. Furthermore, we explore the phenomenon of one- stage protocols, and the experiments reflect that the simplicity of the proposed pro- tocol improves training stability and domain generalization. Our code, data, and http:///eeeeeeeoeeeeeeeeeeee/tte
models are available https:
1 Introduction
Currently, general large language model (LLM), such as the Llama series (Touvron et al., 2023), are developing rapidly. Simultaneously, in some vertical domains' some researchers (Cui et al., 2023; Yang al., 2023c; Li et al., 2023a) attempt to develop specialized models. Specialized models have the potential to yield results comparable to those of larger models by utilizing a medium-sized model through the exclusion of certain general knowledge (Wang et al., 2024; Chen al., 2023b; Yang et al., 2023c). For instance, financial knowledge may not be sufficiently usefully in the medical field and can be therefore omitted in moderately-sinne medical LLMs, thereby freeing up more capacity for memorizing medical knowledge.
"Benyou and Anningzhe the corresponding authors.
Verrtical domains" refer to specialized fields such as medicine, law, and finance, where training
focuses on deep, domain-specific knowledge and disregards other unnecessary domains. This contrasts with horizontally trained models, which possess broader but shallower general knowledge.
1

---

Published as a conference paper at COLM 2024
The two-stage protocol Adaption of general large language models in vertical domains typically involves two stages: continued pre-training and supervised fine-tuning (SFT) (Wang et al., 2023a). For this adaption in medicine domain, continued pre-training aims to inject specialized knowledge, such as medical expertise, while supervised fine-tuning seeks to activate the ability to follow medical instruction, as stated in Zhou et al., 2023.
Issues of the two-stage protocol In the two-stage adaption, LLMs face two key challenge: 1) the difference in continual pre-training and supervised fine-tuning optimization objectives, and 2) the inherent difference in data distribution from general pre-training to continued pre-training. Therefore, the two-stage process suggests that the LLM experiences dual shifts in data distribution. As stated in Goodfellow et al., 2013, catastrophic forgetting occurs when neural networks learn multiple sequential tasks in a pipeline, resulting in the loss of knowledge from previously learned tasks. This problem worsens when the two-stage training involves significantly divergent data distributions and objectives (Bhat et al., 2022). Specifically, Cheng et al., 2023 argues continued pre-training on domain-specific corpora reduces the LLM's prompting capabilities. Secondly, the two-stage training pipeline adds complexity due to the interdependence of its stages. This intricacy not only complicates the optimization process but also limits scalability and adaptability. Each stage possesses distinct hyperparameters such as batch size, learning rate, and warmup procedures. These parameters necessitate careful manual selection through rigorous experimentation.
The proposed one-stage protocol Following the philosophy of Parsimony, this work proposes a simpler protocol of domain adaption that unifies the two stages (continued pre-training and SFT) into a single stage. The core recipe is to transform domain-specific pre-training data into a unified format similar to fine-tuning data: a straightforward (instruction, oufput) pairs. This strategy diverges from the conventional dependence on unsupervised learning in continued pre- rre-training, opting instead for a focused learning goal that emphasizes eliciting knowledge- driven responses to given instructions. The reformulated data is subsequently merged with fine-tuning data to facilitate one-stage domain adaption. In this process, we introduce a data priority sampling strategy aimed at initially learning domain knowledge from pre-training data and then progressively shifting focus to downstream fine-tuning data. This approach enhances the model's 's capability to utilize domain knowledge effectively.
Verification for the new protocol To verify the one-stage protocol, we experiment on Chinese healthcare 2 where ChatGPT and GPT-4 perform relatively poorly (Wang et al., 2023a). Leveraging the proposed protocol, we train a Chinese medical language model, HuatuoGPT-II Inspired by back-translation (Li et al., 2023d), we employ the prowess of LLMs for data unification, where all diverse and multilingual pre-training data are converted to Chinese instructions with a consistent style. This stage bridges the gap between two-stage data, especially for training LLMs in unpopular languages, where English data is overwhelmingly more abundant and high-quality. Subsequently, priority sampling strategy is used to integrate pre-training and SFT instructions for domain adaption. We believe this unified domain adaptation protocol can be similarly effective in other specialized areas such as finance and law, well as in different languages.
Experimental Results Experimental results demonstrate that HuatuoGPT-II, a new Chinese medical language model, outperforms other open-source models and rivals proprietary ones like GPT-4 and ChatGPT in some Chinese medical Benchmarks. In expert manual evaluations, HuatuoGPT-II shows remarkable win rate of 38% and tie in another 38% against GPT-4. Moreover, in recent and untouched Chinese National Pharmacist Licensure Examination (2023), HuatuoGPT- II's superiority over other models, highlighting its specialized effectiveness in the medical field.
2hhis refers to using the Chinese language for healthcare, rather than Traditional Chinese Medicine.
2

---

Published as conference paper at COLM 2024
Additionally, in a spectrum of evaluation methods, the one-stage protocol of HuatuoGPI-II prove more effective than the traditional two-stage training paradigm.
Contributions
The key contributions are:
A unified protocol for domain adaption. The paper introduces a simplified one-stage domain adaption protocol for training, streamlining the traditionally complex pipeline process.
A advanced Chinese medical LLM. Our developed model, HuatuoGPT-IL, leverages this protocol to achieve exceptional performance in Chinese healthcare domains, particularly in Traditional Chinese Medicine.
A novel generalization Test. A novel benchmark using the fresh 2023 Chinese National Pharmacist Licensure Examination provides robust assessment of HuatuoGPT-II, address- ing test data leakage concerns and demonstrating superior performance of HuatuoGPT-II.
2 Data Collection
Domain data is typically divided into two parts: pre-training corpora and fine-tuning instructions.
Medical Pre-training Corpus Domain corpus is pivotal for augmenting domain-specific expertise. We collect 1.1TB of both Chinese and English corpus, sourced from encyclopedias, books, literature, and web corpus, blending general corpora like C4 and specialized corpora such as PubMed. All the corpora are publicly accessible, detailed in Appendix C. Then, meticulous collection pipeline are established for curating high-quality domain data, detailed in the Appendix C. As a result, we obtained 5,252,894 premium medical documents for the pre-training corpus.
Medical Fine-Tuning Instructions For the fine-tuning instruction, We acquire 142K real-world medical questions as instructions from Huatuo-26M (Li et al., 2023c), and had GPT-4 respond to them as outputs. The fine-tuning instruction is utilized to generalize the model's capability to interact with users within the domain.
Chinese & English Corpus (5252K Documents)
Medical Pre-training Instruction
WWW
(5252K Items)
Data Unification (Via LTM)
Bookss
Web Corpus
W
Encyclopodia
Adaptation
One-stage
HuatuoGPT-2
Training
GPT4
Medical Question (142K Items)
Medicul Fine-tuning Instruction
(142K Items)
Figure 1: Schematic of One-stage Adaption of HuatuoGPT-II.
3

---

Published as conference paper at COLM 2024
3 One-stage Adaption
One-stage Adaption strategy aims to unify the conventional Two-stage Adaption process (continued pre-training and supervised fine-tuning) into a single stage, as shown in Figure 1. The adaption process can be executed in two succinct steps: 1) Data unification and 2) One-stage training.
Subsequently, we will detail our method for adapting to the Chinese healthcare sector and develop- ing and the development of HuatuoGPT-II.
3.1 Data Unification
Domain pre-training corpus pivotal for augmenting domain-specific expertise. However, it faces challenges such as diverse languages and genres, punctuation errors, and ethical concerns in its pre-training corpus. Particularly, there's noticeable discrepancy between its unsupervised training and the supervised instruction learning in Supervised Fine-Tuning (SFT). Data Unification aims to unify this data into consistent format, aligning with SFT data. Traditional methods like those in Cheng et al., 2023 fall short due to the variation in language and genre. Hence, we leverage Large Language Models to achieve effective data unification.
Our method of data unification is straightforward yet effective. We generate instructions based on the text of the corpus, and then we generate responses based on both the corpus and the instructions. The prompt for generating instructions is shown in Figure 2.
Please create a <question> that closely aligns with the provided <text>, Ensure that the <question> is formulated in [target language] and does explicitly reference the text. You may incurpurate specific scenarios contexts in the <question>, allowing the <text> serve as comprehensive and precise answer.
<texts:
(dpcain-specific corpus]
<question>:
Figure 2:
The prompt for question generation.
[target languagel is Chinese, and
[domain-specific corpus] refers to corpus in the domain-specific pre-training corpora.
After obtaining questions from the corpus text, we use the prompt, shown in Figrue to let a Large Language Model (LLM) generate responses based on the questions and the corpus. Therefore, all pre-training corpora are converted into a instruction-output format, identical our single-turn SFT data. An example of our final SFT data is shown in Table 7.
Here, we use ChatGPT the LLM for data unification, converting all the medical corpus into instructions of the same language and genre. As detailed in Appendix F, Data Unification can be achieved independently of external LLMs. In Appendix E, we verified that the selection of LLM for data unification is inconsequential. This strategy also mitigates potential ethical concerns inherent the corpus. Additionally, we also use statistical and semantic recognition to keep the model-generated answers close to the data's original content, explained in Appendix H.
3.2 One-stage Training
In the one-stage training process, we integrate data from the Medical Pre-training Instruction and Medical Fine-tuning Instruction datasets to form dataset D. The traditional two-stage pipeline adopts completely separate form, first learning knowledge and then learning instructions to
4

---

Published as conference paper at COLM 2024
You are [model name] equipped with in-depth knowledge in [domain] Your task is to directly answer the user's <question> in
[target language] formulating your response, you thoughtfully reference <reference text>; ensuring your reply does
not disclose your reliance on creference texts, Aim to provide comprehensive and informative response, incorporating relevant insights from creference text> to best assist user. Please be cautious to avoid including any content that might raise ethical concerns.
<question>:
[question generated by LLM]
creference texts:
[domain- corpus)
<reply>:
Figure 3: Prompt for answer generation. [model name] refers to HuatuoGPT-II,
[domain]
medicine, and [question generated by LLM]] the previously text-derived query.
Length
Sampling Priority
Instruction Type
Podding
Medea Corpacjna"
Madeal Eecyckoudia ioutt
Uhen Savephig Nele
0.8
Media Enodepada gat"
Booka
Matea Boaks (ca"*
Medial
Medical Web Corpus
Pre-training Instruction
65
Mideal Wek Oi Carpasjee"
Sampler
pro-imining
Medical Literature
84
Medical Encyclopedia
B3
Medical Books
Fine-tuning Instruction
(B²)
Fine-tuning
Pre-training Instruction
60
10
Treking Daratier
Figure One-stage training process and sampling priority. The diagram (left) illustrates the one- stage training methodology. The table (right) details the sampling priority for various instructional data types, with the f (set as 2) dictating the relative priority. Higher B values indicate more sequential sampling approach, whereas lower values suggest blended strategy.
follow. It exacerbates the catastrophic forgetting of the knowledge acquired in the first stage (Bhat al., 2022). Referring Touvron al., 2023, simply mixing all data could hinder the ability of Large Language Models (LLMs). To address this, we introduce priority sampling strategy in this study. This strategy starts with domain knowledge, gradually transitioning to fine-tuning data while reducing the sampling ratio of previously learned data over time.
In the priority sampling strategy, the sampling probability of each data E D changes over the course of training. The sampling probability of data at step ! during training is:
xx())
Px(x)
EyeD-s, T(y)
Here, xxx) denotes the priority of element x, and S, represents the sampled data before step f. The priority setting and data sampling distribution are delineated in Figure 4. Notably, the priority (xx is static, whereas the sampling probabilities of each data source dynamically changes. More precisely, consider data sources DJ, D',...... D C - St at time t, each with assigned priority
5

---

Published as conference paper at COLM 2024
π(x D/) BK, The probability of selecting an element from DJ is given by:
|
Pa D/)
Ej=1 1D
After element is selected from D the size of D becomes |D
|D|| 1, resulting in:
(x+1(x E D < Px(x E Dj). Therefore, each selection from D; slightly decreases its sampling probability, leading to a dynamic update. The parameter & plays crucial role in adjusting the sampling intensity among high-priority sources, with higher & values favoring sequential sampling, while lower values encourage mixed sampling.
To enable the model to first learn domain capabilities and then gradually shift to instruction interaction learning, we assign a higher priority to pre-training instruction data. Furthermore, to facilitate the model's 's transition from low to high knowledge density learning, we assign different priorities to the four types of data sources. Appendix I shows details of how we determine sampling priority values for different types of data.
4 Experiments
4.1
Experimental settings
Base Model & Setup HuatuoGPT-II, tailored for Chinese healthcare, builds upon the foundations of the Baichuan2-7B-Base and Baichuan2-13B-Base models. Since all data consists solely of single- round format instruction, we enrich the Medical Fine-tuning Instruction dataset by integrating ShareGPT data 9, This allows HuatuoGPTT-II to support multi-round dialogues while maintaining its general domain capabilities. For training details, please see the Appendix
Baselines We compare HuatuoGPT-III with several leading general large language models known for their excellent general chat capabilities in Chinese. These models are Baichuan2-7B/13B-
Chat(Yang al., 2023a), Bree-7BBi4B-Chattee Additionally, for the Chinese medical context, we carefully select DISC-MedLLM((aa et al., 2023) and HuatuoGPT et al., 2023) based on an experimental experiment detailed in Appendix R. We also consider leading proprietary models, including ERNIE Bot (X-- ž) (Sun al., 2021),
et al., 2023) and ChatGLM3-6BB(Zeng et al., 2023).
ChaeGPP(OOeenA,, 2022), and (OpenAA,,, 2023), noted for their extensive parameters and superior performance. For the details of these models, please refer to Appendix M.
4.2 Medical Benchmark
In this section, we evaluate the medical capabilities of HuatuoGPT-II on popular benchmarks. We focus on four medical benchmarks (MedQA, MedMCQA, CMB, CMExam) and two general benchmarks (C-Eval, CMMLU), focusing specifically on their medical components. See Appendix K for more details on these benchmarks. To ensure fairness, we use the most straightforward prompt for all the models, as shown in Appendix K.
Medical Benchmark shown in Table 1, the benchmark results highlight HuatuoGP'l-II's impres- sive proficiency in the medical domain. Its exceptional performance in Chinese medical benchmarks
3https: htap:::://uggiggaaccc ct wwt.attsts// phissspppppppp wehhisshhiiddshrrreeeeee "The versions gpt-3.5 --11----rurb--6613 and gpt-4-0613. The same version use in data unification.
6

---

Published as conference paper at COLM 2024
English
Chinese
Model
MedQA
MedMCQA
CMB
CMExam
CMMLUMat
C. Evalytat
Average
DISC-MedLLM
28.67
32.47
36.62
HuatuuGPT
25.77
31.20
28.81
31.07
33.23
36.53
31
C3atGL.M3-6B
28.75
35.91
39.81
43.21
46.97
48.80
40.73
Baichuan2-7B-CHa
33.31
38.90
46.33
50.48
50.74
51.47
45.39
Baichuan2-155 1BB-Cht
50.87
54.90
52.95
Qwen-7B-Chor
33.46
41.36
49.39
53.33
54.65
52.80
47:50
Owen-14B-Chat
42.81
60.28
63.57
64.55
65.07
57.16
ChatGPT (API)
52.24
53.60
43.26
46.51
50.37
48.80
48.51
HuatuoGPT-II (7B)
41.13
41.87
60.39
65.81
39.08
62.40
55.13
HuatuoGPI-II (13B)
45.68
47.41
63.34
68.98
61.45
64.00
58.47
Table 1: Medical benchmark results. Med. signifies extraction of only medical-related questions. '-' indicate that the model cannot follow the question and make a choice. Due to the too large size of these benchmarks, exclude the testing of GPT-4 and ERNIE Bot here.
like CMB and CMExam reflects its deep understanding of medical concepts in Chinese context. Moreover, HuatuoGPT-II maintains high level on English benchmarks, demonstrating its medical capabilities. To further evaluate the medical expertise, national medical exams. The results, displayed in Table 4, reveal that HuatuoGPT-II surpass all open-source models and closely rivaled the proprietary model, ERNIE Bot. In Appendix B, present the performance of HuatuoGPT-II on additional medical benchmarks.
utilize dataset with various Chinese
2023 Pharmacist Licensure Examination (Pharmacy) Optimal Choice
2023 Pharmacist Licensure Examination (TCM)
Model
Matched Selection
Integrated Analysis
Multiple Choice
Total Score
Optimal Choice
Matched Selection
Integrated Analysis
Multiple Choice
Total Score
Avg.
DISC-MedLLM
22.2
23.3
24.4
24.9
HuatuoGPT
25.6
25,5
23.3
2.6
23.4
24.1
26.8
31.6
7.5
24.9
242
ChaiGLMB-6B
39.5
34.1
10.5
34.6
31.8
38.2
25.0
20.0
32.9
33.8
Qwen-7B-chat
43.8
46.8
33.3
18.4
41.9
40.0
43.2
33.3
17.5
38.8
40.4
Qwen-14B-chut
56.2
58.6
41.7
21.1
52.7
51.3
51.0
27.5
41.7
47.9
50.3
Baichuan2-7B-CCI
50,9
Baichuan2-13B-CT 130-Chat
43.8
52.7
36.7
7.9
44.2
41.3
46.4
43.3
15.0
41.7
43.0
ERNIE (API)
45.0
60.9
36.7
23.7
49.6
38.3
51.5
50.6
ChatGPT (API)
45.6
44.1
36.7
13.2
41.2
34.4
32.3
30,0
15.0
31.2
36.2
GPT-4 (API)
65.1
59.6
46.7
15.8
57.3
40.6
42.7
33.3
17.5
38.8
48.1
HustuoGPT-M1/2
41.9
61.0
35.0
15.7
47.7
52.5
51.4
41.7
15.0
47.5
47.6
HustuoGPT-IK1135
Table 2: Results of the 2023 Chinese National Pharmacist Licensure Examination. It consists of two tracks of Pharmacy track and Traditional Chinese Medicine (TCM) Pharmacy track.
The Fresh Medical Exams Tackling data contamination in LLM training is challenging (Huang al., 2023), especially with extensive and intricate training data. To counter this, we selected the 2023 Chinese National Pharmacist Licensure Examination, held on October 21, 2023, as our benchmarks. This date crucially before both our data collection cut-off (October 7, 2023) and the release of our assessment models. The annual uniqueness of the exam questions ensures reliable safeguard against contamination risks. The results in Table 2 show that HuatuoGPT-II ranked second after GPT-4 in the Pharmacy track. However, in the Traditional Chinese Medicine track, HuatuoGPT-II led with 51.6 points. HuatuoGPT-II demonstrated superior performance in average scores across both tracks, highlighting its proficiency in the medical field.
7

---

Published as conference paper at COLM 2024
4.3 Medical Response Quality
To evaluate the model's performance in real-world medical scenarios, we utilize real-wrold ques- tions in both single-round and multi-round formats, sourced respectively from KUAKE-QIC (Zhang al., 2021) and Med-dialog (Zeng et al., 2020), following the approach of HuatuoGPT (Zhang et al., 2023). The assessment details are provided in the Appendix L.
Single-round QA Tie
Multi-round Dialogue Fail
Average Win/Tie
HuatuoGPT-H17B)
Win
Fail
Tie
Automated Evaluation Using GPT-4
HuatwGGT-IICP vs GPT-4
58
21
21
15
23
78.0%
HuatoCPT-I1(7) vs ChatGPT
62
18
20
69
14
17
81.5%
HuatuoGPT-II/77) Brichuan2-13B-Chat
14
22
11
14
82.0%
HuatuoGPT-I1(7)) vs HuatuoGPT
87
7
67
15
18
88.0%
Human Expert Evaluation
HuatmCPT-11(78) vs GPT-4
38
38
24
53
17
30
73%
HuatuoGPTT-I(7T) ChatGPT
33
11
33
76%
HuatuoGPT-I117B) vs Baichuan2-13B-Chat
63
19
18
63
19
18
82%
HuatuoGPT- (-))/B)) HuatuoGIT
83%
Table Results of the Automated Evaluation Using GPT-4 and Expert Evaluation.
Automatic Evaluation We utilize GPT-4 to evaluate which of the two models generated better outputs. The results, as indicated in Table (for more compared baselines, see Table 13), show that HuatuoGPTTII has a higher win rate compared to other models. Notably, HuatuoGPT-I achieve higher win rate in comparison with GPT-4. Although its fine-tuning data originated from GPT-4, its extensive medical corpus provided it with more medical knowledge, shown in Table 2.
Expert Evaluation For further evaluating the quality of the model outputs, we invite four licensed physicians to score them, with the detailed criteria available in the Appendix L.2. Due the high cost of expert evaluation, we select four models for comparison with HuatuoGPT-II. Results, as outlined in Tables 3, indicate that HuatuoGPT-II consistently outperforms peers, aligning with automatic evaluation. This consensus between expert opinions and GPT-4's evaluations underscores HuatuoGPT-II's efficacy in medical response generation. The case study on the model's response can be found in Appendix S.
4.4
One-stage Vs. Two-stage Adaption
2.00
175
18
1.2
1.0
0.75
Stage [luatunGPT-H (7B)
1st Stage: Pre-traiuing
2nd Stuge: Fine-tuning
5000
10601
15500
3000
20000
10000
19900
Step
Step
Figure 5: Comparison the loss outcomes between proposed One-stage Adaption and conventional Two-stage Adaption using the same SFT data and medical corpus.
8

---

Published as conference paper at COLM 2024
One-Slage Adapation lion
Twe-Stage Adapation
Only Fine-tuning
5142
40.23
47.43
CMB
CMMLUO
2023 Bussination
2023 Eustitation
(vs. ChIGPP
(s. ChurGPT) Mill-mmmd Didagae
(Pbarmasa))
(TCM)
Singleroond
Figure 6: The comparison results of One-stage Adaption and Two-stage Adaption. "Only Fine- tuning" refers the model that only fine-tunes the backbone directly. Win Rate the result scored by automatic evaluation using GPT-4.
To validate the superiority of One-stage Adaption, we conduct traditional Two-stage Adaption to fine-tune the the Baichuan2-7B-Ban on the same medical data.
Figure shows the training loss of these two training methods. The loss of Two-stage Adaption shows instability, marked by pronounced fluctuations and loss spikes. This instability arises from the varied content and styles within the medical pre-training corpus, which includes four distinct data types detailed in Table 9. The variation is further amplified by the differences between Chinese and English datasets. Our one-stage Adaption unifies the diverse contents and styles of the pre-training corpus, improving training stability and reducing loss fluctuation. Moreover, due to data discrepancies between the two stages, there is a noticeable loss divergence between the pre-training and fine-tuning stages in Two-stage Adaption. In contrast, One-stage Adaption effectively resolves this issue by simplifying the pre-training corpus into a unified language and style, aligning with SFT data, making a more stable and smooth training process.
The results of the previously mentioned experiments also demonstrate that One-stage Adaption achieves better performance than other methods, as shown in Figure 6. It can be seen that on all six datasets, our One-stage Adaption performance is significantly better than the Two-stage Adaption performance (from 5.3% to 23%), especially single-round Q&A and multi-round conversation tasks. This superiority is likely due to two-stage unification and more effective knowledge generalization in One-stage Adaption.
4.5
The Relative Priority for Sampling
Examimation S
Mive Sertping
46.7%
.7.
47.25
Sequentill Sampling
Appmsimaz
5.
45.25
E:
Relative Priority
Figure 7: Comparison of model performance under different relative priority settings.
9

---

Published as conference paper at COLM 2024
In evaluating the priority sampling strategy, we tested it across different & settings using the same HuaIuoGPT-II (7B) configuration. Our results show that model performance drastically drops when 5 0 (mixed sampling) or when ß is excessively high (Sequential Sampling), highlighting the importance of priority sampling. The best performance is achieved when B is around 2.
Moreover, we also conducted an ablation study on data unification in appendix G. The experiment indicates that data unification is crucial for the performance improvement of the one-stage training.
5 Conclusion
In this work, we propose one-stage domain adaption method, simplifying the conventional two-stage domain adaption process and mitigating its associated challenges. This approach straightforward, involving the use of LLM capabilities to align domain corpus with SFT data and adopting a priority sampling strategy to enhance domain adaption. Based on this method, we de- velop Chinese medical language model, HuatuoGPT-II. In the experiment, HuatuoGP'T-I demon- strates state-of-the-an performance in the Chinese medicine domain across various benchmarks. It even surpasses proprietary models like ChatGPT and GPT-4 in some aspects, particularly in Traditional Chinese Medicine. The experimental results also confirm the reliability of the one-stage domain adaption method, which shows significant improvement over the traditional two-stage performance. This One-stage Adaption promises to offer valuable insights and methodological framework for future LLM domain adaption works.
Ethic Statement
The development of HuatuoGPT-II, specialized language model for Chinese healthcare, raises several potential risks.
Accuracy of Medical Advice While HuatuoGPT-II has shown promising results in the domain of Chinese healthcare, it's crucial to underscore that at this stage, it should not be used to provide any medical advice. This caution stems from the inherent limitations of large language models, including their capacity for generating plausible yet inaccurate or misleading information.
Data Privacy and Ethics The ethical handling of data is paramount, especially in the sensitive field of healthcare. The primary data sources for HuatuoGPT-II include medical texts, such as textbooks and literature, ensuring that patient-specific data is not utilized. This approach aligns with ethical guidelines and privacy regulations, ensuring that individual patient information not compromised. Another significant aspect of our methodology is the 'data unification' process, which aims to address potential ethical issues in the training data. By employing large language models to rewrite the medical corpora, we aim to eliminate any ethically questionable content, thereby ensuring that the training process and the model align with ethical standards.
Acknowledgement
This work was supported by the Shenzhen Science and Technology Program [CV]20220818100000000 Shenzhen Doctoral Startup Funding (RCI52022100809333066) Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608), Shenzhen Key Laboratory of Cross-Modal Cognitive Computing (grant number
10

---

Published as conference paper at COLM 2024
ZDSYS20223622691132000), and Shenzhen Stability Science Program 2023, Shenzhen Key Lab of Multi-Modal Cognitive Computing.
References
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. Xiv preprint arXio:2309.1600, 2023.
Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, and Zhongyu Wei. Disc-medllm: Bridging general large language models and real-world medical consultation, 2023.
Prashant Shivaram Bhat, Bahram Zonooz, and Elahe Arani. Consistency is the key to further mitigating catastrophic forgetting in continual learning. In Conference on Lifelong Learning Agents, PP. 1195-1212. PMLR, 2022.
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arv preprint ar XrX1:22001.03111, 2022.
Yirong Chen, Zhenyu Wang, Xiaofen Xing, huimin zheng, Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, and Xiangmin Xu. Bianque: Balancing the questioning and suggestion ability of health iims with multi-turn health conversations polished by chatgpt, 2023a.
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint aXXiv:2311.16079, 2023b.
Zhihong Chen, Feng Jiang, Junying Chen, Tiannan Wang, Fei Yu, Guiming Chen, Hongbo Zhang, Juhao Liang, Chen Zhang, Zhiyi Zhang, et al. Phoenix: Democratizing chatgpt across languages. ar arxiv preprint ar Xrx::2304.10453 2023c.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading comprehension. arvio preprint ar arXii:23309.0555 09.33, 2023.
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint ar Xiv::3366.1699. 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset diverse text for language modeling. ar Xiv preprint ar XrXi::2001.000 2020.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXivv13122.2211, 2013.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. Xiu preprint ar Xiv:2333 100006..16444 11644, 2023.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Junlee Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. C-eval: multi-level multi-discipline chinese evaluation suite for foundation models. arxiv preprint arXin:2305.08322 2023.
11

---

Published as a conference paper at COLM 2024
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14))6421, 2021.
Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang. Large language model as a user simulator. Xiv preprint drXiv:2308.11533 2023.
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, et al. Chatharuhi: Reviving anime character in reality via large language model. arXio preprint ar Xrr::2300.0885 09597, 2023a.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint ar orXiv:2366.99212, 2023b.
Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. Huatuo-26m, a large-scale chinese medical qa dataset. arviv preprint ar arXit:2305.01522 2023c.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXto:2308.06259 2023d.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 2023e.
Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Xiv preprint ar Xiv::2306.0000 03030, 2023.
OpenAL, Introducing chatgpt. https:/ hatpss///opnnaae /ppp//pppnaamom////// //////ll////// /csp//hhtttpttc 2022.
OpenAL. Gpt-4 technical report, 2023.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on PMLR, 2022.
Health, Inference, and Leaningg pp.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):44855553 2020.
Yu Sun, Shuohuan Wang. Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint ar Xiix2210.002137 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprint ar Xix:2307.009255 2023.
Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. Pre- trained language models in biomedical domain: A systematic survey. ACM Computing Surveys, 56(3):1-52, 2023a.
12

---

Published as conference paper at COLM 2024
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge, 2023b.
Haochun Wang, Chi Liu, Sendong Zhao, Bing Qin, and Ting Liu. chatglm-med: #T #ichatglm i di H.. https://githubl com http//ccm////LLi// /SIIR-IIIIIIHSSSSS /Madd Cheeth,a,,LM, 2023c.
Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, al. Cmb: comprehensive medical benchmark in chinese. Xiv preprint ar MxKio:2308.08833 2023d.
Xidong Wang, Nuo Chen, Junyin Chen, Yan Hu, Yidong Wang, Xiangbo Wu, Anningzhe Gao, Xiang Wan, Haizhou Li, and Benyou Wang. Apollo: Lightweight multilingual medical llms towards democratizing medical ai to 6b people. aiv preprint ar xriv:2403.0066/) 2024.
Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Qian Wang, and Dinggang Shen. Doctorglm: Fine-tuning your chinese doctor not a herculean task. aiv preprint ar arXin:334.001997, 202..
Ming Xu. Medicalgpt: Training medical gpt model. 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al. Baichuan 2: Open large-scale language models. arXiv preprint arX:::2309. 10305, 2023a.
Songhua Yang, Hanjia Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. arv preprint arXie:2308.03559, 2023b.
Yi Yang, Yixuan Tang, and Kar Yan Tam. Investim: A large language model for investment using financial domain instruction tuning. arr preprint ar arXiv:2309.13064 2023c.
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Al Open, 2:65-68, 68, 2021.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, et al. Meddialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9241-9250, 2020.
Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor. Xiu preprint ar arXio:2305.1505, 2023.
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, et al. Cblue: A chinese biomedical language understanding evaluation benchmark. ar Xio preprint ar 2106:02106..8887, 2021.
13

---

Published as conference paper at COLM 2024
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law, pp. 159-168, 2021.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. aiv preprint arXiv:235..11206, 2023.
Wei Zhu and Xiaoling Wang. Chatmed: A chinese medical large language model. https:/ /::::://httttt tht....... wwwphhhhhhwwwhhhhhhhee
2023.
14

---

Published as conference paper at COLM 2024
A Related Work
A.1 Domain adaption
Recent research has also indicated that such domain adaption using further training Cheng et al., 2023 leads to a drastic drop despite still benefiting fine-tuning evaluation and knowledge probing tests. This inspires us to design different protocol for domain adaption. Also, Gunasekar et al. (2023); Li et al. (2023e); Chen et al. (2023c); Kong et al. (2023) show possibility that the 10B well-selected dataset could achieve comparable performance to much larger model.
A.2 Medical LLMs
The rapid advancement of large language models (LLMs) in Chinese medical field is driven by the release of open-source Chinese LLMs, notably trained via instruction fine-tuning. Doctor- GLM (Xiong et al., 2023) and MedicalGPT (Xu, 2023) are fine-tuned on various Chinese and English medical dialogue datasets. Another Chinese medical LLM, Ben'isao (Wang al., 2023b) fine- tuned on distilled data derived from knowledge graphs. Bianque-2 (Chen et al., 2023a) includes multiple rounds of medical expansions, encompassing drug instructions and encyclopedia knowl- edge instructions. ChatMed-Conse consultation data and ChatGPT responses. DISC-MedLLM (Bao et al., 2023) is fine-tuned on more than 470,000 medical data including doctor-patiens dialogues and knowledge QA pairs.
(Zhu & Wang, 2023) is fine-tuned on both Chinese online
By integrating the reinforcement learning, HuatuoGPT (Zhang et al., 2023) is fine-tuned on a 220,000 medical dataset consisting of ChatGPT-distilled instructions and dialogues alongside real doctor- patient single-turn Q&As and multi-turn dialogues. Zhongling (Yang et al., 2023b) undergoes comprehensive three-stage training process: continuous pre-training on various medical data, instruction fine-tuning on single-turn and multi-turn dialogue data as well as medical NLP tasks data, and reinforcement learning adjudicated by experts to ensure reliability and safety.
B Supplementary Experiment
Traditional Chinese Medicine
Clinical
Assislant
Physician
Pharmacist
Assistant
Physician
Avg.
Model
2015
2016
2017
2012
2013
2016
2017
2018
2019
2018
2019
2020
2018
2019
2020
(300)
(220)
(116)
(60D)
(600)
(430)
(168)
(167)
(223)
(234)
(244)
(244)
(449)
(476)
(436)
HuatuoGPT
26.0
30.9
32.8
31.3
26.8
30.2
18.4
27.5
25.1
32.5
33.6
27.5
32.7
28.6
30.1
28.9
DISC-MedLLM
38.3
26.7
35.1
25.0
22.2
41.8
36.6
35.8
35.1
ChatGi
45.7
45.0
50.0
46.3
45.8
46.5
42.3
28.1
35.4
50.9
48.8
43.9
41.7
43.9
43.6
43.9
Baichuan2-7B-CC
57.3
57.3
58.6
57.9
41.7
41.9
45.7
55.3
51.0
53.6
50.0
53.4
Beichuan2-138-Char
64.7
58.2
62.9
61.7
61.5
63.3
54.2
38.9
48.4
66.2
64.8
63.1
65.9
58.8
61.5
59.6
Owen-7B-Chat
54.7
55.9
56.0
52.7
53.5
54.4
44.0
33.5
43.0
68.8
63.9
57.8
60.6
57.6
54.1
54.0
Owen-14B-Chat
65.3
63.2
67.2
63.3
54.8
49.1
32.5
74.8
75.0
73.7
69.7
68.8
65.3
ERNIE But (API)
73.3
66.3
73.3
20.0
71.8
66.7
55.9
50.3
60.0
78.2
77.0
77.5
66.6
20.8
74.1
68.8
ChatCPT (API)
46.0
41.4
40.5
32.1
30.0
63.3
53.7
53.7
52.5
51.8
44.2
GPT-4 (API)
47.3
48.2
53.5
50.3
53.7
54.2
41.1
43.7
48.0
79.9
72.5
70.9
74.8
73.1
68.4
58.6
HuatuoGPT-II (7B)
67.1
65.2
67.5
67.9
67.4
64.9
53.0
46.7
51.0
70.9
73.2
69.4
68.8
66.5
67.7
64.5
HuatuoGPT-II (13B)
20.3
70.0
71.6
71.0
69.2
70.2
56.5
52.1
34.7
73.
76.6
70.1
72.8
68.9
72.2
68.0
Table 4: The results of Chinese National Medical Licensing Examinations. The year represents the actual examination year. Note that the exam here may not be complete, and the blue fonts indicates the number of questions.
15

---

Published as conference paper at COLM 2024
Chinese Medical Licensing Examination To evaluate HuatuoGPT--I's medical competence, we utilize a dataset comprising medical examination questions from China and the United States. This approach aims to measure the model's s adaptability and proficiency across varied medical knowledge frameworks and terminologies. The results of the Chinese National Medical Licensing Examination are outlined in Table 4. In the results, HuatuoGPT-II (13B) not only surpassed all open-source models but also closely approached the performance of the leading proprietary model, ERNIE Bot. This notable outcome reflects the model's advanced understanding of Chinese medical principles and its adeptness in applying this knowledge in complex examination contexts. The 13B variant's elevated scores signify enhanced analytical capabilities and deeper comprehension of the nuances inherent in Chinese medical practice.
CMB-Clin CMB-Clin is a dataset designed to evaluate the Clinical Diagnostic capabilities of LLMs, based on 74 classical complex and real-world cases derived from textbooks. Distinct from the response quality evaluations, CMB-Clin provides standard answers for reference and scores each model individually. We follow the same evaluation strategy from the original paper which utilizes GPT-4 as the evaluator. Results, delineated in Table indicate that HuatuoGPT-II outperforms its counterparts, excluding GPT-4. Intriguingly, the 7B version HuatuoGPT-II demonstrates superior efficacy over its 13B variant, phenomenon potentially attributable to foundational capacity variances, as evidenced by Baichuan2-7B-Char superior performance compared to Baichuan2-13B Chat.
Model
Fluency
Relevance
Completeness
Proficiency
Avg. 1
GPT-4
4.95
4.71
4.35
4.66
4.67
HuatuoGPT-II (7B)
4.94
4.56
4.24
4.46
4.55
ERNIE Bot
4.92
4.53
4.16
4.55
4.54
ChatGPT
4.97
4.49
4.12
4.53
4.53
HualuoGPT-II (13B)
4.38
4.00
4.40
4.43
Baichuan2-7B-Chal
4.93
4.41
4.03
4.36
4.43
Qwen-14B-Chal
4.90
4.35
3,93
4.48
4.41
Qwen-7B-Chat
4.94
4.17
3.67
4.33
4.28
Baichuan2-13B-Chr
4.88
4.18
3.78
4.27
4.28
ChatGLM3-6B
4.92
4.11
3.74
4.23
4.25
HuatuoGPT
4.89
3.76
3.38
3.86
3.97
DISC-MedLLM 4.82
3.24
2.75
3.51
3.58
Table Results of CMB-Clin on Automatic Evaluation using GPT-4.
The United States Medical Licensing Examination (USMLE) outcomes, delineated in Table 6, reveal that HuatuoGPT-II (13B) outperforms comparative open-source models. Despite a discernible disparity with ChatGPT, it is important to note that the USMLE's English-centric nature imposes constraints on HuatuoGPTTIlI, primarily designed for Chinese medical contexts. However, its commendable performance in the USMLE underscores its proficiency in employing medical knowledge across diverse scenarios, effectively addressing the range of challenges posed by the USMLEE.
USMLE
C Domain Corpus Collection
Domain corpus is pivotal for augmenting domain-spediie expertise. Domain data necessitates a comprehensive collection of high-quality and domain-specifee content, surpassing the scope of the
16

---

Published as conference paper at COLM 2024
Stagel
Stage2&3 (5148)
Avg.
Models
(6308)
(11456)
HuatuoGPT
28.68
28.38
28.54
ChatGLM3-6B
33.39
32.49
32.98
Baichuan2-7B-Chaà
38.11
37.32
37.76
Baichuan2-13B-Chal
44.99
45.57
45.25
Qwen-7B-Chat
40.90
36.09
38.73
Qwen-14B-Chat
48.73
42.45
45.90
ChatGPT (API)
57.04
56.27
56.69
HuatuoGPT-I1(7B)
45.72
44.45
45.15
HuatuoGPT-II(33B
47.34
49.37
48.25
Table 6: The results of The United States Medical Licensing Examination (USMLE) from MedQA. The blue fonts indicate the number of questions.
Raw Corpus (1.1 TB)
Meddaal Web Corpras (en)
Type of Data
Doc
Source
2.6%
Mendall Raceeleualin (mn)
Mediaa Books (eu)
Chinese
Meidral Liberafure (en)
Meddaal
(ca)
Web Corpus
640,621
Wudao
Encyckpedia
Books
1,802,517
Textbook
Medioal Litenature (so)
Encyclopedia
411,183
Online Encyclopedia
Literature
177,261
Chinese Literature
Extract medical corpus
-eggmentation clean
1.5%
English
Web Corpus
394,490
C4 Textbook,
3. De-duplication
Books
801,522
the pile books3
Encyclopedia
147,059
Wikipida PubMed
0.7%
Literature
878,241
Total
5,252,894
Figure 9: Summary of the Medical Pre-training Corpus.
Figure 8: Domain Data collection.
foundational corpus. We identified four primary data types for extensive collection: encyclopedias, books, academic literature, and web content. Focusing on these categories, we amassed 1.1TB of Chinese and English data, shown in Table 9.
The domain data collection pipeline is an essential part of ensuring the quality of domain language corpora, designed to extract high-quality and diverse domain corpora from large-scale corpora. The methodology encompasses four primary steps:
1. Extract Medical Corpus:
This process aims to remove irrelevant domain corpora, serving as the first step in filtering massive corpora. We employ dictionary-based approach, obtaining dictionaries containing medical vocabulary from THUOCL and The SPECIALIST Lexicon from the Unified Medical Language System. We strive to exclude non-medical
Ohtps://wwwwallmnnigggo/rrrreee rreeeearr.o//eerh.hh.c...a. httheearrrccrrls//eeeeeee umls/new http//ew__useeess /online / LLEnngg///XX__eee LEX_001.Xhiett htm.
17

---

Published as conference paper at COLM 2024
terms to form domain-specific dictionary. For each text segment, we evaluate whether it's domain-specific by assessing the density of matched domain words from the dictionary. This dictionary method is an effective and efficient way to extract domain text.
2. Segmentation: Since we need convert the corpora into instructions and do the data cleaning and de-duplication, it's necessary to segment all corpora into fragments. We divide each text into sentences, then use a moving window to turn the corpus into segments with length limit, ensuring no loss of information by including sentences before and after the window.
3. Cleaning: We notice a significant proportion of medical corpora related to medical advertising, despite appearing fluent, contains little domain knowledge and introduces bias. To filter out medical advertisement texts and low-quality texts, we utilize ChatGPT to select problematic texts and train corpus quality classification model to clean corpora.
4. De-duplication:
De-duplication is crucial step in corpus processing, domain knowl- edge often has significant redundancy. We use a sentence embedding model to convert corpora into embeddings and then employ dense retrieval methods to remove semantically similar texts.
Our data sources consist of four categories: (1) Web Corpus, which includes C4 (Raffel et al., 2020) and Wudao (Yuan et al., 2021); (2) Books, primarily comprising Textbook and the pile books3 et al., 2020); (3) Encyclopedia, encompassing Chinese Medical Encyclopedia and Wikipedia; (4) Medical Literature, which consists of PubMed and Chinese literature. Following the aforementioned steps, the corpus is converted into 5,252K of instruction data.
D Example of Data Unification
Table 7 illustrates the process of generating instruction and response based on documents in Data Unification.
Description
content
Raw Data (from En- glish Textbook)
It one frequent mental illnesses, making up large propertion of persons assisted living and skilled nursing facilities. Although Alzheimer disease has been described at every period of adult life, the fifties
majority of patients their sixties older; relatively small number have younger....
Query of Data Unifica- tion
A P of Alzheimer's s discase?)
#24 Au? (Translation: What main clinical and pathological features
Answer of Data Unifi- cation
24 fl. da & á A
&
2 M. # it Wh क
* (Taanssassit:::: Alzheimer's disease one the common mental illnesses, occupying larger percentage of the pupulation in assisted living and skilled nursing facilities. Although Alcheimer's 's discuse can oceur at all stages adulthood, most patients are in their sixties or older; only small percentage are their fifties or even younger..)
Table 7: An example of our final SFT data.
Pre-training Instruction: Knowledge Provision
As shown in Section 3.1, we use external LLMs (like ChatGPT) for data unification for two main purposes:
18

---

Published as conference paper at COLM 2024
1. Data Unification using ChatGPT: The primary data source this stage raw medical corpora, not ChatGPT. ChatGPT is merely utilized for data refinement and back translation. Its function is similar to rule-based data polishing software or a fine-tuned T5 model, where medical expertise is not essential. We will subsequently present case study using BBiichan2-Chat-7B as an alternative for data unification.
2. Fine-tuning Datasets using GPT-4 and ChatGPT: Our fine-tuning data and ShareGPT are used to train our model to follow instructions and engage in conversation. However, they don't significantly contribute to the medical knowledge of our model. This aligns with the 'Superficial Alignment Hypothesis', suggesting that a model's knowledge and capabilities primarily derived during its pre-trat pre-training phase. Instruction fine-tuning mainly guides the model to apply specific formats and styles during user interaction, rather than substantially enhancing its knowledge base.
To illustrate, we conduct experiments to deeply analyze how using external large models affects HautuoGPT-II' medical knowledge. We sample 1/40 our pre-training and fine-tuning data to assess their impact on the model's s medical capability:
Exper. No.
Fine-Tuning Data
# Pre-Training Data
2023 Exam (Pharmacist)
2023 Exam (TCM)
#1
3.6K
131.3K
38.9
37.7
#2
3.6K
60K
131.3K
37.7
38.1
#3
3.6K
131.3K + 60K
37.9
40.0
#4
3.6K
120K
131.3K
37.3
35.7
#5
3.6K
131.3K + 120K
39.6
40.2
Table 8: Experiment on data scale.
The comparison between Experiments #4 and #5 shows that augmenting pre-training data is more effective than adding an equivalent amount of instruction tuning data. This suggests that the enhancement in medical knowledge benchmarks is primarily due to the raw data itself. ChatGPT play role in transforming this raw data into formats that are more digestible for the trained LLMs, leading to our model outperforming ChatGPT and GPT-4 in medical benchmarks.
Furthermore, as outlined in Appendix M, leading Chinese medical models like DISC-MedLLM also employ ChatGPT and GPT-4 for their training data construction, yet our model surpasses them in domain benchmarks.
F
Data Unification without External LLMs
To eliminate the performance improvement our method caused by relying on external large language models, we experimente with adapting Baichuan2-7B-Chat in the medical domain using our method, without the assistance of other LLMs. We randomly select 5% of the pre-training corpus and medical queries, using Baichuan2-7B- 7hthhat itself for data unification and direct response to the SFT-data's 's queries as fine-tuning data, a process we term as carried out as shown in Table F.
The experiment
The results show that when using self-data-unifican to perform self-data-unffiattio, our method named One-stage Adaptation still exceeds Two-stage Adaptation by an average of 6.6 points. This
19

---

Published as conference paper at COLM 2024
demonstrates our method's ability to function independently of other LLMs, effectively enhancing the model's domain adaptation capabilities and outperforming traditional continuous pre-training.
CMExam 2023 Exam (Pharmacist)
2023 Exam (TCM)
Two-stage Adaptation
on Baichuan2-Chat-7B (Without Data-Unification)
40.5
33.5
34.8
Our One-stage Adaptation on Baichuan2-Chat-7B (With Data- Unification)
48.3
40.0
40.3
Table 9: Adaptation gaps between Two-stage and On-stage adaptation.
Moreover, we find even Baichuan2-7B-Chh demonstrates good ability to perform data-unification, as shown in the Table 10:
Description
content
(from
The functional and structural alterations known to be the regulation center of itch, but the neuropathology of chronic itch, ch chronic spontaneous urticaria (CSU)...
chronic spontaneous urticaria.
long been
PubMed)
Query of Data Unifi- cation (Generated by
è 12 A # # (CSU) A Rid x 1 Aa
(Translation: Are there alter-
activity and structure regions associated with climical symptoms chronic spontancous urticaria
Baichuan2-7B-Chat)
(CSU)?)
Answer Uni-
& # (CSU)
#
fication (Generated by
* (striatum) & ग & # de le & M] 4 8- de f2rt & #t #K
(CSU) (Translation. In
Baichuan2-788 2BB-Chat)
chronic spontaneous urticaria (CSU), alterations in the activity and structure brain regions associated with clinical sympturss do exist, Studies have shown that the brain, especially the striatum (striatum), plays key role the regulation of itch. For chronic itch, such as chromic spontameous urticaria (CSU)...
Table 10: An example of QA data generated by Baichuan2-7B-Chat
G
Ablation Study of Data Unification
Setting
CMExam
2023 Exam (Pharmacist)
2023 Exam (TCM)
Two-stage, Without Data Unification One-stage, Without Data Unification
50.5
38.8
38.3
49.3
37.2
35.6
One-stage, With Data Unification
53.4
39.7
40.2
Table 11: Performance comparison: with vs. without Data Unification.
To validate the importance of Data Unification, conducted an ablation study on Data Unification using a reduced dataset (5% due to time constraints) as shown in figure G.
The results emphasize the importance of Data Unification. Without it, the single-stage method falls short of the two-stage one. This could be due to the considerable discrepancies between the two data stages. Simply merging them doesn't yield effective results.
20

---

Published as conference paper at COLM 2024
H Deviate Detection
In the data unification phase, we instruct LLM to refer to the text content to provide a detailed response. The responses are expected to mainly contain information from the text. However, the ability of the language models to follow instructions can't be fully guaranteed, and there might be instances where the model answers question without referring to the text. To ensure that the response contains text knowledge, we adopt the following two methods to detect deviations from the original text:
1. Statistical Method: We convert both the text and the response into sets of 1-grams, and then use the Jaccard Similarity Coefficient to calculate the similarity between these two sets to determine the similarity of content between the text and the response. We set threshold to detect content deviations.
2. Model Detection Method: The first method is suitable for detection within the same language, but it cannot handle cases where the English text is translated into a Chinese response. Therefore, we rely on a more robust method. We have people annotate whether an answer deviates and use this data to train large language model for response content detection.
Based on these detection methods, if generated response fails the tests, we instruct the Large Language Model (LLM) to regenerate it.
I Priority Sampling Strategy
In this section, we first explain how we handle ordering between pre-training and SFT data, and then give details of ordering within pre-training data
Order between pre-training and SFT data We adopt sequential training approach, prioritizing Pre-training Data followed by Fine-tuning Data (Pre-training Data- Data). This method mirrors human learning processes: akin initially learning from textbooks before attempting exercises. The rationale is straightforward comprehensive foundational knowledge, gained from pre-training data, should precede the more specific applications found in fine-tuning data. This sequence prevents scenario akin to attempting exercises without sufficient textbook study, which could lead to random guesses and, in the context our model, the propensity to generate inaccurate 'hallucinated" information. Additionally, the fine-tuning data, often structured in a question-and- answer format, closely aligns with real-world user interactions. By introducing this type of data later in the training process, we ensure that the model is better attuned to actual user scenarios, enhancing its practical applicability and effectiveness.
Order within pre-training data Data quality is gauged using the data sampling epochs from the LLaMA (Touvron et al., 2023) study, as is shown in Table 12.
Initially, the significance of different data sources, determined by sampling epochs in LLaMA, was ranked as "Web
LLiterature Sonntturre-Bookee BBooks "inckks-ppddllddiia Nevertheless, after consulting with medical experts, it became clear that Books should be accorded greater importance. Consequently, we have adjusted our training sequence to: Web Corpus-~Litratuun -Litrrature dutterature-Enccclopeell -ooooppeda-BBoos..... Lastly, set the relative priority, hyper-parameter in priority sampling.
21

---

Published as conference paper at COLM 2024
Dataset
Data Type
LLaMA Sampling Epochs
C4
Web Corpus Literature
1.06
ArXiv Books
1.06
Book
2.23
Wikipedia
Encyclopedia
2.45
Table 12: Sampling Epochs in LLaMA (Touvron 2023).
J Training Detail
For HuatuoGPT-II, was set to 2. In other settings, set the sampling epoch for Medical Fine- tuning Instruction and for Medical Pre-training Instruction to 3. Additionally, we unified the data encoding. Sampled instructions were concatenated as much possible to form fixed fixed-length token sequence of 4096 for training. The training was conducted with a batch size of 128 and a learning rate of le-4. As all the data instructional, we only optimize the output loss and do not learn from the input loss. Our model is implemented in Py Tyrch using the Accelerate and leverage ZeRO algorithm to distribute the model. Training was conducted using Nvidia A100 cards, and the HuatuoGPT-II (7B) training time was about 4 days.
K Benchmark Details
We evaluate the medical capabilities of HuatuoGPT-II on popular benchmarks. Here, we select four medical benchmarks and three general benchmarks, noting that we only evaluate the medical part of the general benchmarks. The medical benchmarks include: MedQA (Jin et al., 2021), which collected from professional medical board exams in various languages. We used its English test set for evaluation. MedMCQA (Pal et al., 2022), amassed from Indian medical entrance exams, and we evaluate using the development set. CMB (Wang et al., 2023d), comprehensive medical benchmark in Chinese, where we specifically used the CMB-Exam for evaluation. CMExam (Liu al., 2023), a comprehensive Chinese medical exam dataset. The general benchmarks include: C-Eval (Huang et al., 2023), an all-encompassing Chinese evaluation framework. CMMLU (Li et al., 2023b), designed to critically appraise the knowledge and reasoning prowess of large language models in Chinese.
For the general benchmarks, we only use their medically related evaluation content. For CMMLU, the evaluation sections used are 'clinical knowledge', 'agronomy', 'college medicine', 'genetics', 'nutrition', 'Traditional Chinese Medicine', and 'virology'. For C-Eval, we use the 'clinical medicine' and 'basic medicine' parts.
Since all evaluations in our experiments are for Chat models, we uniformly adopted a Zero-shot setting, using consistent form, shown as below:
# #
A. *
B.
#
-.0 A
D. #
22

---

Published as conference paper at COLM 2024
Translation:
Please answer the following multiple choice questions.
Of little significance in assessing the prognosis of patient with cirrhosis is
A. ascites
B. albumin
C. blood electrolytes
D. prothrombin time
L Details of HuatuoEval
We utilize the single-round and multi-round evaluation data from HuatuoGPT (Wang et al., 2023b) to evaluate the model's s medical response capability. Sources from these datasets include KUAKE- QIC (Zhang et al., 2021) for single-round questions and Med-dialog (Zeng al., 2020) for multi- round cases. We slightly modify these evaluations for fairer assessment, focusing on the information the answers rather than the tone of a doctor's response.
This evaluation is designed to evaluate the response capabilities of large-scale language models in medical scenarios. includes two types of evaluations. The first type assesses the single- round answer capability, primarily comprising real patient questions sourced from KUAKE-QIC. The second type is a multi-round diagnostic evaluation, with data containing real patient case information, sourced from Med-dialog. In the single-round evaluation, we have the models directly answer medical questions. the multi-round evaluation, we simulate a patient asking questions to doctor using ChatGPT, based on the patient's medical record information. The simulated patient's prompt is shown as below( [Patient Case Information] is patient case in multi-round data of HvaluoEval.). The models then engage in dialogue with the simulated patient to generate conversations. For fair comparison, we have the ChatGPT-simulated patient continuously asking questions, and each model must respond twice.
Single-mound QA
Multi-rondd Dialogue
Average Win/Tie Rate
HustwoGPT-H1(B) vs Other Model
Win
Tie
Fail
Win
Tie
Fail
HuatuoGPT-I1(B) vs HuatuoGPT-I((33)
39
22
39
41
13
46
57.5%
HuatwoGPT-H177) ERNIE
75.0%
HuatuoGPT:11(77) vs ChatG1MM-6B
75
11
14
76
10
14
86.0%
HuatuoGPT-I1(7)) vs Baichuan2-7B-Chan
18
84
86.5%
HuatuoGPT:1177) DISC-MedILLL
12
73
88.0%
HuatuoGPT-I1770 vs Qwen-14B-Chan
82
6
79
12
91.0%
HuatuoGPT-H1/7) Qwet-7B-Chat
91.0%
Table 13: Results the Automated Evaluation Using GPT-4 in Chinese medical scenarios.
- T $
# A H # 1 # 1 # de 46
3
i R
PPatient Case Information]
Translation:
23

---

Published as conference paper at COLM 2024
You are patient, here is your condition and you are asking the doctor questions related your condition, note that this is multi-round questioning process, remember not to let the dialogue end, but continue ask the doctor questions related to your condition.
[Patient Case Information]
L.1 Automatic Evaluation
During automatic evaluation, we compare model responses in pairs. We present the dialogue or Q&A content of two models to GPT-4, which then judges which model's response is better. The prompts for single-round and multi-round automatic evaluations are shown in Table 14. To mitigate potential position bias in GPT-4 as a judge, each data point is evaluated for interaction position twice.
The Prompt Single-Round Automatic Evaluation:
[Question]
[Question]
[End of Questton]
[Assistant
[The Response Model 1]
[End Assistant
|Assistant
[The Response of Model 2]
[Fnd Assistant 2]
[System]
would request your feedback the assistants response user question displayed Requinements: The response should be to the point and adress the problem of user. The description of symptoms should be comprehensive accurate,
most reasonable inference based and possibilities. The treatment recommendations should be effective and reliable, taking into account the severity stages of the illness. The prescriptions should be effective and reliable, considering indications, contraindications and dosages. Please compare the performance their responses. You should tell whether Assistant 1 is 'better than', 'worse than', 'equal to' Assistant 2.
provided diagnosis should
relevant factors
Please first compare their respunse and analyze which
line with the given requireents..
In the last line, please output single line containing only a single label selecting from 'Assistant is better than Assistant 2', "Assistant worse than Assistant and 'Assistant equal to Assistant
24

---

Published as conference paper at COLM 2024
The Prompt Multi-Round Automatic Evaluation:
|Assistant II
[The Conversation Model
[End of Assistant
[Assistant
[The Conversation from Model 2]
[End Assistant 2]
[System]
would like request your feedback multi-tum conversations between Al assistant and displayed above. Requirements: The response should be to the point and adress the prublem of user. The description of symptomns should be comprehensive accurate,
provided diagnosis should the most reasonable inference based and possibilities. The treatment recommendations should be effective and reliable, taking into account the severity stages of the illness. The prescriptions should be effective and reliable, considering indications, contraindications and dosages. Please compare the performance the Al assistant each conversation. You should tell whether Assistant 'better than', 'worse tham', or 'equal to' Assistant Please first compare their responses and analyze which In the last line, please output single line containing only a single label selecting frum 'Assistant 1 is better than Assistant 2', "Assistant warse than Assistant 2', and 'Assistant equal to Assistant
relevant factors-
line with the given requirements.
Table 14: The prompt for automatic evaluation on single-round and multi-round setting, Note
that [.] is what needs to be filled in. 'Model and 'Model Indicates two models to compared.
L.2 Expert Evaluation
For manual evaluation, we provide licensed medical doctors with evaluation criteria shown as below. We then offer platform for experts to conduct evaluations. Experts choose which response (from a pair of model responses) is better. The selection interface is shown in Figure 10. All model information is anonymized and interaction positions are randomized to ensure fairness.
R 7 *
2. * 3
4. ANDAASSTARTT
Translation:
1. The response should be orientated towards the user's problem and provide solution.
2. consider the richness and logical clarity of the model response.
3. consider the professionalism and accuracy of the model.
4. the model response should be humanistic.
M Baselines
Open-Source Baselines We compare to the most representative general large language mod- els, which possess excellent chat capabilities and are adaptable to various scenarios, including healthcare. They are follows:
25

---

Published as conference paper at COLM 2024
Huatuo Human Evaluation q
FREE BY
20 BEREEERR---RRRRRRR 78 AEL ( BE. BUTT
FUUH 1E
1E (Tyndinemia)) - SE
MA (-19 (LL66)))) h1183881 28:4 aa 51 1758
IE 15 www.
(rraa-)) (
18 05
IE 1 AATES
11 (TAC-2)
dehydrogenase) BNB (2
1e 17 www.eeeeeee aa Mim (21561 Ai.tt
2 A
5E 18
H-S W0 3E... (883840100111 IS R.
BRAEE
4E
Figure 10: The interface of the expert evaluation.
Biichuan277113B--hh(((a et al., 2023a) The Baichuan2-7B and Baichuan2-13B models are trained on 2.6 trillion tokens. Sharing the same backbone as ours, their chat-version models are well adapted to the base, fully leveraging its capabilities.
al., 2023) The Qwen series are trained on trillion tokens of diverse texts and codes. The chat models are fine-tuned carefully on a dataset related to different tasks. RLHF is also applied to generate human-preferred responses.
(rren-7B/44B-hht((//
ChatGL.M3-6(ZZeng et al., 2023) ChaIGLM2-6B is a model trained with 1.4 trillion bilingual tokens. Based ChatGLM2, the C3atGLM3 model has more diverse training dataset, increased training steps, and a more reasonable training strategy.
Llama2-7B/13B-Chm (Touvron et al., 2023) Llama2 series are successors models of Llama trained on trillion tokens with diverse datasets. Their chat model has robust performance. Unlike the first three Chinese-supported models, Llama2 series English models.
Additionally, we select two representative and strong large language models for Chinese medical scenario. (we conducted experimental experiment to select these two models, see Appendix R for details). These models are:
DISC-MedLLM(Bao et al., 2023) DISC-MedILM is fine-tuned over 470K medical data and 34k general domain conversation and instruction samples.
HuatuoGPT et al., 2023) HuatuoGPT is trained using real-world data and dis- tilled data from ChatGPT, adopting RLMF (a method combining ChatGPT and doctor preferences) to leverage the advantages of mixed data.
26

---

Published as conference paper at COLM 2024
Proprietary Baselines Furthermore, we compare three representative proprietary models, which have larger parameters and stronger performance:
ERNIE Bot -)) (Sun et al., 2021) ERNIE bot is closed-source large predictive model developed by Baidu. It is one of the strongest Chinese language models to date, with an API interface and web version available for use.
ChatGPT(OOee 2022) ChatGPT is large language model released by OpenAT, pos- sessing significant influence and currently holding an excellent standard among large models.
GPT-4(OpenAA, 2023) GPT-4 is a language model published by OpenAl following ChatGPT. It is undoubtedly one of the most powerful and widely used large language models currently available.
N Distribution of Synthesized Questions
We sampled 100 questions from the pre-training instructions and manually categorized their type follows:
Yes/ No Question
Open-ended Question Comparison Question
Causal Question
Question
Table 15: Distribution of synthesized questions.
O Performance on Different Sizes of Models
The effect differences of HuatuoGPT2 with different model parameters:
Model Size
Backbone
CMExam
2023 Pharmacist
2023 TCM
7B
Baichuan2-78
65.8
47.7
47.5
13B
Baichuan2-133 1-BBBB
69.0
52.9
51.6
34B
Yi-34B
77.2
65.5
62.4
Table 16: Performance on different sizes of models.
P Other SFT Data
We conducted experiments with different LLMs to generate SFT data, using 5% of the data, to verify the importance of SFT data. The results are shown in the table.
It shows minimal impact with the other LLM, which indicates that SFT data does not provide primary knowledge.
27

---

Published as conference paper at COLM 2024
CMExam
2023 Pharmacist
2023 TCM
One Stage, with SFT data generated by GPT-4 One Stage, with SFT data generated by ChatGPT
53.4
39.7
40.2
52.6
40.0
39.8
Table 17: Results different SFT data
One Stage Training for Other Domains
The proposed one-stage training protocol is applicable beyond the Chinese medical domain.
To address the reviewer question, we tested it on the MedQA- Englissh (Jin et al., 2021) and CaseHOLD (Zheng al., 2021) datasets, which represent the English medical and law domains, respectively. For MedQA-Einglshh, we used MedQA's English textbooks as the pre-training corpus and its training set for Supervised Finn-Tuning (SFT). For CaseHOLD, we sampled 4% of the corpora from FreeLaw Opinions (Biderman et al., 2022) as the pre-training corpus and used its training set for SFT. The data characteristics are as follows:
Dataset
Pre-training documents SFT data
Test data
Domain
MedQA-en CaseHOLD
156,960
10,178 45,000
1,273 3,900
English Medical Domain
141,342
Law Domain
Table 18: Dataset characteristics in other domains.
We trained on Baichuan2-7B-Base and compared one-stage training with traditional two-stage training:
Method
MedQA-en (English Medical Domain)
CaseHOLD (Law Domain)
Two-stage training
46.2
41.8
One-stage training
48.6
43.6
Table 19: Performance comparison of training methods across domains
The results indicate that one-stage training can enhance performance across domains.
R Other Medical Models
When selecting baselines for Chinese medical applications, we also tested the performance of other Chinese medical large models the Chinese National Medical Licensing Examination. The results, as shown in Table 20, indicate that based on their superior performance, HuatuoGPT and DISC-MedLLM were chosen as the baselines for comparison.
S
Case Study
In our study, we observed that many models, including GPT-4, experience significant hallucinations
in Chinese medical contexts. These hallucinations arise from two factors: 1) The model itself lacks specific medical knowledge; 2) Misconceptions arise in Chinese. Tables 21 and 22 present two examples of simple Chinese medical questions about pharmaceuticals.
28

---

Published as conference paper at COLM 2024
Traditional Chinese Medicine
Clinical (122k)
( (5)
Avg.
Assistant W JU
Physician (E
Pharmacist (E5)l)
Assistant 15: 10
Physician (
Model
J0)
2015
2016
2017
2012
2013
2016
2017
2018
2019
2018
2019
2020
2018
2019
2020
DoctorGLM Xiong et al. (2023)
1.4
3.5
1.8
1.8
2.1
2.4
2.2
2.6
3.3
1.8
2.2
BianQue-2 (Chen et al,, 2023a)
3.7
2.3
3.5
4.2
4.5
4.0
7.1
4.2
9.0
4.7
5.3
1.6
3.8
5.9
3.7
4.5
ChatMed-Consult (Wang
2023c)
20.0
14.7
16.7
17.5
23.8
21.4
BenTsao (Wang et al, 2023b)
23.3
26.8
17.2
19.0
19.5
22.1
20.2
20.4
18.8
18.4
24.6
27.5
21.6
18.9
18.1
21.1
ChatGLM-Med (Wang et al,, 2023c)
20.7
23.2
20.7
21.8
21.8
22.6
16.7
21.0
15.3
30.3
23.0
29.9
18.5
20.4
24.8
22.0
MedicalGPT (Xu, 2023)
25.0
33.4
32.1
HuatuoCPT (Selected)
26.0
30.9
32.8
31.3
26.8
30.2
18.4
27.5
25.1
32.5
33.6
27.5
32.7
28.6
30.1
28.9
DISC-MedLLM (Selected)
38.3
41.8
26.7
36.2
36.7
35.1
25.0
22.2
22.0
49.2
36.1
41.8
414
36.6
35.8
35.1
HuatuoGPT-II (7B)
67.1
65.2
67.5
67.9
67.4
64.9
53.0
46.7
51.0
70.9
73.2
69.4
68.8
66.5
67.7
64.5
Table 20: The results of the Chinese National Medical Licensing Examination.
As shown in Table 21, GPT-4 seems to misunderstand the drug compound 'Methoxyphenamine: essentially providing irrelevant responses. Baichuan2-7B-Chhe appears to comprehend this drug but only answers correctly in part, also exhibiting significant hallucinations. In contrast, HuatuoGPT-II accurately and comprehensively addresses the drug's 's details.
Query:
Translation: Introduce compound methoxyphenamine)
GFT-4:
1. # : i - H Fi Hi Mist
3
h T
2. ZZ Hi C X 74 JL 10
HT 15
म #8 #9...
Compound Methoxyphenaminn is a medicine used to relieve flu and cold symptoms. It usually contains the following ingredients:
Translation:
1. Methoxyphenam
This is an antihistamine used to relieve allergy symptoms such as sneezing,
runny nose and itchy eyes. compounded methenamine, it helps relieve nasal congestion and other related symptoms caused by cold flu.
2. Acetaminophen (Paracetamol): This is common analgesic and antipyretic used to relieve pain and reduce fever caused by colds or flu...)
Baichuan2-7B-Chatt
E # EH
J #
. A #
SE (- 12
Compound Methoxyphenamine prescription medication used to treat variety inflammatory diseases, such as bronchitis, asthma, allergic rhinitis. It is combination four ingredients: Methoxyphenanine (a bronchodilator) chlorpheniramine (an antihistamine), aluminum hydroxide (a gastric acid inhibitor), and pyridinium methylsulfate (an over-the-counter antipyretic and analgesic)..)
TTranslation:
29

---

Published as conference paper at COLM 2024
HuatuoGPT-I((BB
-
- fayT
# プ HAR AT - FL FZ M ( AT 11 BET
Translation: Compound Methoxyphenamine is compounded preparation consisting methoxyphenamine hydrochloride, noscapine, aminophyliine and chlorpheniramine maleate. It is mainly used for the treatment of respiratory related diseases such as bronchial asthma and wheezing bronchitis.
Some adverse reactions may occur when using Compound Methocarbamol. Examples occasionally include rash, redness of the skin, itchy cancer, nausea, vomiting, loss of appetite, dizziness, palpita- tions and difficulty in urination. If these not mouth, good reactions occur, should stop the drug and consult doctor...)
Table 21: Chinese medical hallucinations case. The green font indicates correct information, and the brown font indicates incorrect information.
Another example, as illustrated in Table 22, involves a direct question in Chinese about 'OxyCon- tin'. GPT-4 erroncously associates it with 'Oscillococcinum' providing misleading information. Meanwhile, Baichuan2-7B-Chon experienced more severe hallucinations, referring to non-existent drug named 'Oscilloclasm'. HuatuoGPT-II, however, correctly understood this as 'Oxycodone' and provided accurate information.
Query:
Translation: What is the medication OxyContin?)
GPT-4:
3 (Oscillcoocciumm
87 SAAAEEEEEEEETATHAAAATTATASTTTTTE iff *
MA 3
# A
OxyContin (Oscillococciuum) is a homeopathic medicine used for the treatment of flu symptoms. It is usually used for mild or initial flu symptoms such as fever, chills, fatigue and headache. The main ingredient in Oscillococcinum a substance extracted from the liver and heart ducks, and this extract is highly diluted to comply with homeopathic principles.... )
(Translation:
Baichuan2-7B-Chat
L (Oscilloclasm)
: S EMK
IIGHI71CIH 318
(Translation: OxyContin (Oscilloclasm) is an antiepileptic drug that is primarily used to treat various types of seizures. Its chemical name is hwtpiinhhoobbeeooo/amoooo//ooooo acid", structural formula is C16H17NO4, and its molecular weight is 285.35.)
30

---

Published as conference paper at COLM 2024
HuatuoGPT-I((7B)
A (Oxycodone)
#66 12 ANTUU IT
SEnt Ho...
OxyContin (Oxycodone) is a potent opioid analgesic used primarily to treat moderate to severe pain. Opioids reduce pain primarily by affecting opioid receptors in the brain, which are found mainly in the brain and spinal cord.
(Translation:
OxyContin has wide range applications the medical field, including for post-operative pain, cancer pain, and chronic pain. However, due to its potent analgesic effects, OxyContin also has the potential to be abused, leading to addiction issues...)
Table 22: Chinese medical hallucinations case. The green font indicates correct information, and the brown font indicates incorrect information.
These instances highlight GPT-4's limitations in the Chinese medical domain. We believe there is critical need to enhance domain-specific capabilities, especially for sensitive topics like healthcare, and HuatuoGPT-II appears to be more adept at this during its current developmental phase.
31


Reviews:
[{"review_id": "NgsnGOU7iT", "reviewer": "Reviewer_sbnB", "summary": "This paper introduces a one-stage domain-adaptation protocol for language models and produces a medical domain language model HuatuoGPT-II. Different from a two-stage process (pre-training, supervised fine-tuning), the paper proposes a one-stage approach to merge these two stages. The key technique here is to unify different format of data into instruction tuning pairs, leveraged by auto-labeling ability of large models (i.e., GPT-4). The paper also proposes a priority sampling strategy for training. The models are shown to perform better compared to other models on different Chinese medical benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. Below, we address your concerns.\n\n> Q1: The Connection between Pre-Training Corpora and Pre-Training Instructions\n\nBoth learning approaches aim for autoregressive goals, yet the differences between them are both subtle and significant:\n\n- **Vanilla Pre-Training** focuses on learning the probability of a sequence $ p(y_i \\mid y_0, \\ldots, y_{i-1})$ , primarily targeting next-token prediction. This method enables the model to learn *correlations* between words, with the expectation that it can generalize to downstream scenarios to handle various types of questions.\n\n- In contrast, **HuatuoGPT-II** aims to learn $p(y_i \\mid X, y_0, \\ldots, y_{i-1}) $, integrating the input $X$. This method learns a direct mapping from $ X $ to $ Y $, establishing a more high-level relationship (e.g. *causal*)  rather than merely *correlations*, which aligns more closely with knowledge understanding and the SFT objective.\n\n**Validation Experiment** As shown in *Appendix G*, training in a QA format  outperforms vanilla pre-training:\n\n|   | CMExam | 2023 Pharmacist | 2023 TCM |\n|-|-|-|-|\n| Training on *pre-training corpora* | 49.3 |37.2|35.6|\n| Training on *pre-training instruction* |53.4|39.7|40.2|\n\n> Q2: The use of baselines.\n\nWe extra conduct a comparison of **eight** medical baselines in *Appendix N* for a more comprehensive domain-specific analysis.\n\n> Q3: Usage of GPT-4 in data generation and evaluation.\n\n**GPT-4 for Evaluation**\nIn addition to using GPT-4 for evaluation, we also employed medical expert evaluations and the latest medical exams to ensure fairness.\n\n**GPT-4 in Data Generation**\nGPT-4 is used to generate SFT data for instruction-following. Moreover, *Appendix E* demonstrates that SFT data is not essential for domain expertise but rather for pre-training instruction.\n\n> Q4: Types of questions generated.\n\nWe sampled 100 questions from the pre-training instructions and manually categorized their type as follows:\n\n| | Yes/No Question| Open-ended Question| Comparison Question| Causal Question| \n|-|-|-|-|-|\n| # Question |12| 47 | 9 | 32 | \n\n> Q5: Intuition behind training with the pre-training instruction.\n\nAs outlined in response to  `Q1`, our intuition is that pre-training instruction facilitates a more direct method to learn explicit causal relationships, as opposed to mere word correlations in the pre-training. We appreciate this valuable inquiry and will clarify this further in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a simple training protocol, which can indeed help reduce the efforts needed in a traditional two-stage training. \n- The paper is easy to understand.\n- HuatuoGPT-II shows strong performance on different benchmarks\n- The paper also introduces a new Chinese Pharmacist Licensure Examination to avoid the test data leakage concerns for existing benchmarks.", "reasons_to_reject": "- The connection between training on pre-training corpora and training on instruction data generated from pre-training corpora is not clear to me. But it seems to be a core inspiration of the methodology. I think providing more and deep insights will be useful.\n- The baseline approaches are a bit weak as they are mostly general-domain models.\n- Given the paper uses GPT-4 to generate data, uses GPT-4 to evaluate results, and compares the proposed model with GPT-4, it will be tricky to interpret the results, especially when comparing to GPT-4.", "questions_to_authors": "- What types of questions are generated on the pre-training corpus? \n- The paper makes an assumption that training with instruction data (which is generated on pre-training corpus) has the same functioning of training on pre-training corpus. What is the intuition behind it?", "extra_scores": {}}, {"review_id": "12zIEIQG5y", "reviewer": "Reviewer_FV3D", "summary": "The authors tackle the problem of domain adaptation of large language models (LLMs) by proposing a unified, one-stage adaptation process in the place of the more common two-stage approach of continued pretraining and supervised fine tuning. Their approach transforms the domain-specific pretraining corpus into a form that more closely resembles the instruction tuning data used during fine tuning. The authors select a filtered dataset of over 5 million documents in the medical domain to demonstrate this transformation process and how this data is used to adapt a general-domain LLM to the medical domain. The resulting model and competing baseline models are graded against a recent, previously unseen Chinese National Pharmacist Licensure Examination, with the hope that the contents of this test are not present in some form in either the general or domain-specific training data used by the models. Both automatic and human-expert evaluations are performed. Results show the authors' models generally outperforming the baseline models on Chinese-language tasks, including commercial offerings such as GPT-4.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. Below, we provide responses to your concerns.\n\n> Q1: Concern about the latest exams.\n\nThank you for raising a valid point regarding the potential overlaps with prior materials. We will add the following clarification to our paper:\n\n- **Uniqueness of the Exam:** According to the [official exam website](http://www.cqlp.org), the questions for the National Pharmacist Licensing Exam are strictly designed and newly formulated. This significantly reduces the risk of data leakage and ensures greater fairness.\n\n- **Comparison with Previous Exams**\nIt is quite worthwhile to examine the scores from previous exams! In *Appendix B*, we provide the results of the pharmacist exams from the past three years for comparison:\n\n|                    | 2017 Exam | 2018 Exam | 2019 Exam | 2023 Exam (Latest) |\n| ------------------ | --------- | --------- | --------- | ----------------- |\n| Baichuan2-13B-Chat | 54.2      | 38.9      | 48.4      | 43.0              |\n| GPT-4              | 41.1      | 43.7      | 48.0      | 48.1              |\n| ERNIE Bot          | 55.9      | 50.3      | 60.0      | 50.6              |\n| HuatuoGPT-II (13B) | 56.3      | 52.1      | 54.7      | 52.3              |\n\nNote that the exam questions from 2017-2019 are incomplete, with details available in *Appendix B*.\n\nNonetheless, we observe that: **(i)** HuatuoGPT-II scores are relatively close to those of previous exams, and **(ii)** some models (such as ERNIE Bot and Baichuan2-13B-Chat) scored notably higher in previous years compared to the 2023 exam.\n\n> Q2: \"Vertical Domains\" & \"Philosophy of Parsimony\"\n\n*“Vertical domains”* refer to specialized fields such as medicine, law, and finance, where training focuses on deep, domain-specific knowledge  and disregards other unnecessary domains. This contrasts with horizontally trained models, which possess broader but shallower general knowledge.\n\nWe appreciate the reviewer's suggestion and will add a footnote explaining this *\"Philosophy of Parsimony\"*.\n\n> Q3: Terminology and grammatical corrections\n\nSpecial thanks to the reviewer for contributing to the improvement of our manuscript. We commit to correct these linguistic errors and thoroughly review our paper to enhance its quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The unified one-stage approach for tuning a LLM to a specific domain is interesting, especially where the pretraining data is transformed into instruction-tuning-like data via interactions with a separate LLM (the authors note there is no strong reliance on a single model in this transformation step.)", "reasons_to_reject": "While there has been great care to note that the licensing exam has been chosen for novelty (and was published after the data collection cut-off date), there may questions present in earlier published study guides that were then incorporated as training or tuning data. It may be worth a quick examination on scores from previous editions of the licensing exams for comparison.", "questions_to_authors": "- Can you define what is meant by 'vertical domains' in Section 1?\n- A footnote for a definition of 'Philosophy of Parsimony' may be useful\n- \"unpopular languages\" may be better represented as \"lower-resourced languages\" in Section 1\n- Section 3.1, corpus is -> corpora are", "extra_scores": {}}, {"review_id": "2zWZtIzQyu", "reviewer": "Reviewer_j9Xe", "summary": "This work presents a one-stage domain adaptation pipeline that relies on instruction tuning in place of the more common two-stage pipeline comprising continual pre-training and supervised finetuning. This approach aims to offer several advantages: a form of simplification, a means of reducing catastrophic forgetting, and mechanism to adaptively adjust data source during training. To test the efficacy of this model, there are numerous experiments (and a rich appendix) to substantiate claims.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. Below, we provide responses to your concerns.\n\n> Q1: The role of GPT-4 and fairness.\n\nWe appreciate your suggestions. We will expand the discussion as follows:\n\n**Upper Bound** GPT-4 is not the upper bound of our models in the Chinese medical domain, as they learn from domain-specific corpora rather than relying solely on GPT-4.  *Appendix E* shows that the SFT data (generated by GPT-4) is not essential for domain expertise. \n\n**Fairness of Comparison** We strive to enhance the fairness of evaluations by utilizing expert assessments and the latest exams.\n\n\n> Q2: Clarify the distribution shifts.\n\nOur training data is rich and varied, containing not only biomedical content but also clinical content, such as web corpora and textbooks. We value your suggestion and will clarify the distribution shifts from general to medical.\n\n\n> Q3: The suggestion about key appendices.\n\nThank you for your advice and feedback! In the next version, we will endeavor to move important appendices to the forefront to enhance readability.\n\n> Q4: Future directions about HuatuoGPT.\n\nThe evolution from Huatuo to Huatuo-II highlights our focus on two critical areas:\n- **Enhancing Trustworthiness**: Minimizing hallucinatory outputs is crucial in the medical domain. We are exploring retrieval-augmented generation and reliability scoring for trustworthiness.\n- **Enhancing Domain Capability**: We are advancing their abilities to analyze medical images, such as CT and MRI scans, and improve decision-making abilities in diagnostic support.\n\n\n> Q5: Other LLMs to generate SFT data.\n\nIn response to your concern, we conducted experiments with different LLMs to generate SFT data, using 5% of the data due to time constraints. The results are as follows:\n\n|| CMExam | 2023 Pharmacist | 2023 TCM |\n|-|-|-|-|\n| One Stage, with SFT data generated by *GPT-4* | 53.4| 39.7| 40.2|\n| One Stage, with SFT data generated by *ChatGPT* | 52.6 | 40.0 | 39.8 |\nIt shows minimal impact with the other LLM (ChatGPT).\n\n> Q6: Performance on smaller or larger models.\n\nWe will train a tiny HuatuoGPT-II to assess the smaller models. In fact, there is a 34B version of HuatuoGPT-II (based on Yi-34B), and its performance compared is as follows:\n| Model Size | CMExam | 2023 Pharmacist | 2023 TCM |\n|-|-|-|-|\n| 7B | 65.8 | 47.7 | 47.5 |\n| 13B | 69.0 | 52.9 | 51.6 |\n| 34B | 77.2 | 65.5 | 62.4 |\nThe results demonstrate that larger model parameters can significantly enhance performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work demonstrates several strengths:\n- There are many experiments that aim to substantiate the claims (and a significant appendix) that helps piece together several parts of the story. \n- The notion of simplification and adding a \"knob\" to more easily adjust data mixing are both quite useful in the context of model adaptation and could potentially help mitigate multiple distribution shifts using a consistent method.\n- The use of other tools, e.g. GPT, to construct instructions introduce an additional source of information that can be scaled more easily than the traditional data generation process.", "reasons_to_reject": "There are some considerations that limit impact, namely:\n- Prior work (Peng et al., https://arxiv.org/pdf/2304.03277) demonstrated the efficacy of instruction tuning with GPT-4, which would be useful to contextualize the present work. Further, it would be useful to add discussion around whether this setup might suggest that GPT-4 offers an upper found on performance for these tasks were it similarly finetuned. On the one hand, the experimental results suggest HuatuoGPT-II has a superior win rate, but it's not quite an equal comparison.\n- Adding clarity around the underlying distribution shifts that are present would be useful because there is both a significant shift in language and general vs medical domain present in this work. In particular, the evaluation chosen relies on resources like MedQA which are primarily clinical questions (from medical exams) whereas the training data is drawn from biomedical (as opposed to clinical) domain.\n- Focus the discussion of the results to emphasize the key takeaways. Currently there is a rich appendix that substantiates several claims made throughout the text but the main paper is difficult to read in absence of the appendix because key details have been relegated there, some of which could arguably be considered more important than those that appear directly in the results, such as Appendix N (Other Medical Models), which provides context for what performance means in this space.", "questions_to_authors": "Suggestions:\n- Carefully consider whether there are results that appear in the appendix that would be better included in the main text.\n- This is stylistic, but consider removing some parentheticals where not necessary, e.g. \"(necessitating dual meticulous training)\".\n\nQuestions:\n- What are future directions can be highlighted in discussion that are revealed by differences between Huatuo and Huatuo-II?\n- If a different LLM is used in place of GPT-4 for generating instruction tuning data, how significantly does this impact the quality of the resulting model?\n- How is performance affected with a very small (e.g. 1B) model or much larger model?", "extra_scores": {}}, {"review_id": "0iLFkANqHU", "reviewer": "Reviewer_F1mt", "summary": "This paper introduces a one-stage domain adaptation technique that combines diverse data from pre-training and supervised stages into a concise instruction-output pair format for efficient knowledge injection. The proposed mechanism, according to the authors, improves both training stability and domain generalization. Extensive experiments show the effectiveness of the method in four medical benchmarks (MedQA, MedMCQA, CMB, CMExam) and two general benchmarks (C-Eval, CMMLU).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Please see our responses below.\n\n> Q1: The claim and the effects on other domains.\n\n### Training Stability and Domain Generalization\n**Training stability** involves reducing data variability by transforming chaotic pre-training data into consistent QA format that aligns closely with SFT data. This results in a steadily decrease in training loss, as depicted in Figure 5.\n\nRegarding **domain generalization**, we realize our explanation was vague. We aimed to convey that one-stage training can better generalize domain-specific knowledge to downstream tasks. We will revise this statement to avoid confusion.\n\n### Effects on Other Domains\n\nOur one-stage training protocol is applicable beyond the Chinese medical domain.\n\nTo address the reviewer’s question, we test it on the *MedQA-English* (Jin et al., 2020)  and *CaseHOLD* (Zheng et al., 2021)  datasets, representing the English medical and law domains, respectively. For *MedQA-English*, we used *MedQA*'s English textbooks as the pre-training corpus and its training set for SFT. For *CaseHOLD*, we sample 4% corpora of *FreeLaw Opinions* (Biderman et al., 2022) as the pre-training corpus and used its training set for SFT. Data characteristics are as follows:\n\n|  | # Pre-training documents | # SFT data | # Test data | Domain                 |\n| - | -| - | - | -|\n| MedQA-en   | 156,960                  | 10,178     | 1,273       | English Medical Domain |\n| CaseHOLD   | 141,342                 | 45,000     | 3,900       | Law Domain |\n\nWe trained on Baichuan2-7B-Base and compared one-stage training with traditional two-stage training:\n\n| Method         | MedQA-en (English Medical Domain) | CaseHOLD (Law Domain)\n| -- | -- | - |\n| Two-stage training (Continued Pre-training + SFT) | 46.2   |  41.8  |\n| One-stage training (Ours)      | 48.6              | 43.6 |\n\nThe results indicate that one-stage training can enhance performance across domains.\n\n> Q2. Please address the issues in writing quality noted, such as the misuse of \"Camel Case\" and other linguistic inaccuracies.\n\nWe acknowledge and appreciate your feedback on the writing quality of our manuscript, including the misuse of \"Camel Case\" and several grammatical errors. We commit to a thorough revision to address these issues and to ensure that our manuscript meets the high standards of academic writing. Your feedback is appreciated and will certainly help improve the quality of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The suggested approach seems straightforward and might be used for other domains as well.\n- Attempted to solve a critial issue of catastrophic forgetting in language model.\n- The paper performed detailed experiments and ablation study to validate the effectiveness of the proposed method.", "reasons_to_reject": "- The model, HuatuoGPT-II, specically developed only for the medical domain in Chinese. I have reservation with the claims of training stability and domain generalization. Please refer to 'Questions To Authors' for more details.\n- The quality of the writing is a little sloppy. For instance, the overuse of \"Camel Case,\" which is superfluous, detracted from the quality of writing. Furthermore, it is wrong to say '... let **an** Large Language Model (LLM) generate...' and unusual in academic writing to say '... and close to the data’s original content...'. I would like the authors to carefully review these minor issues throughout the manuscript.", "questions_to_authors": "The authors claim that the proposed protocol improves training stability and domain generalization. What potential effects might it have on medical fields other than Chinese or other domains (e.g., law, finance and so on)?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper on HuatuoGPT-II introduces a novel one-stage domain adaptation protocol that combines pre-training and supervised fine-tuning into a unified instruction-output pair format. The reviewers are happy with the paper for its clarity and the strong performance of HuatuoGPT-II on various Chinese medical benchmarks; they also express some concerns about the clarity of certain methodological aspects and some concerns about using GPT-4 for data generation and evaluation. The paper was rated 6 (marginally above the acceptance threshold), and the authors are asked to provide additional clarifications and address the raised concerns.

Comments from PC: Please include the additional results from the rebuttal in the revision, as they significantly help clarify the questions raised by the reviewers.
Decision: Accept