An Empirical Study of Instruction-tuning Large Language Models in Chinese
Authors: Qingyi Si
Tong Wang
Zheng Lin
Xu Zhang
Yanan Cao
Weiping Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: large language models
instruction fine-tune

Abstract: The success of ChatGPT validates the potential of large language models (LLMs) in  artificial general intelligence (AGI). Subsequently, the release of LLMs has sparked the open-source community's interest  in instruction-tuning, which is deemed to accelerate ChatGPT's replication process. However, research on instruction-tuning LLMs in Chinese, the world's most spoken language, is still in its early stages. Therefore, this paper makes an in-depth empirical study of instruction-tuning LLMs in Chinese, which can serve as a cookbook that provides valuable findings for effectively customizing LLMs that can better respond to Chinese instructions. Specifically, we systematically explore the impact of LLM bases, parameter-efficient methods, instruction data types, which are the three most important elements for instruction-tuning. Besides, we also conduct experiment to study the impact of other factors, e.g.,  chain-of-thought data and human-value alignment. We hope that this empirical study can make a modest contribution to the open Chinese version of ChatGPT. This paper  will release a powerful Chinese LLM that is comparable to ChatGLM. The code and data are available at https: //github.com/PhoebusSi/Alpaca-CoT.

An Empirical Study of Instruction-tunngg Large Language Models in Chinese
Qingyi Tong Wangl.2, Zheng Lin
Xu Zhang", Yanan Cao Weiipng Wang'
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Sccooo of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
APUS AiLMe Lab
(siqingyi, wangtong1 997, linzheng, caoyanan, wangweipingg@eiie cc. zhangxu@apusai. com
Bagggingy,wwngtonggee
Abstract
stantly emerging, which stimulate the specific ca- pabilities of frozen LLMs (e.g., Chain-of- oo-Thhtght (Wei et al., 2023; Wang et al., 2023; Kojima et al., 2023)) or guide them to complete specific tasks (Yang et al., 2022; Shen et al., 2023), by calling OpenAl interfaces and carefully designing prompts without model training.
The success of ChatGPT validates the poten- tial of large language models (LLMs) in artifi- cial general intelligence (AGI). Subsequently, the release of LLMs has sparked the open- source community's interest in instruction- tuning, which is deemed to accelerate Chat- GPT's replication process. However, research instruction-uuming L.LMs in Chinese, the world's most spoken language, is still in its early stages. Therefore, this paper makes an in-depth empirical study of instruction-tuninn tuning LLMs in Chinese, which can serve as a cook- book that provides valuable findings for effec- tively customizing LLMs that better re- spond to Chinese instructions. Specifically, we systematically explore the impact of LLM bases, parameter-efficient methods, instruction data types, which are the three most impor- elements for instruction-tuming. Besides, we also conduct experiment to study the im- pact of other factors, e.g., chain-of-thought data and ouman-value alignment. We hope that this empirical study can make modest contribution the open Chinese version of ChatGPT. This paper will release powerful Chinese LLM that is comparable to ChatGLM. The code and data available https: //gi thub. comm/Poobbusii/Alpccccooo
The unexpected disclosure of the pre-trained LLaMA (Touvron et al., 2023) model changes this situation, and has sparked a surge of excitement in the LLM research community. This is the first open LLM with competitive performance. Recently, Al- paca (Taori al., 2023) uses self-instruct (Ouyang et al., 2022) and ChatGPT to generate 52K instruc- tions, which can enable LLaMA to respond to var- ious human instructions like ChatGPT. This open project verifies the important role of instruction- tuning (Wei et al., 2022; Chung et al., 2022) open LLMs in replicating the ChatGPT process.
Given the open LLM LLaMA and Alpaca's high-quality instruction data, there is still a chal- lenge for researchers: even the instruction-tuning of the 7B model still requires high computational resources. To address this problem, Alpaca-LoRA extends the parameter-efficien method LoRA LLaMA, which further reduces the computing cost of instruction-tuning. tensive research in the open-source community on instraction-tuning for LLMs. On this basis, more LLMs (c.g, Bloom (Workshop, 2023), GPT-J (Wang and Komatsuzaki, 2021)) are shown to have significant improvements in instruction-followin performance with instruction-tuning. On the other hand, more instruction data is constantly being pro- posed, e.g., Belle (Ji al., 2023) constructs Chi- nese instructions in the same way, and ShareGPT collects a large number of real human-ChatGPT conversations.
further sparks ex-
1 Introduction
The emergence of ChatGPT gives humanity a real sense of hope for AGI for the first time, and inspires researchers to realize the importance of LLM re- search. However, the closed source of LLMs (e.g., GPT-3 (Brown al., 2020) and PaLM (Chowdh- ery al., 2022)) coupled with the requirement for massive computing resources to build the exclusive LLM has deterred researchers from reaching the LLM training stage. Subsequently, a series of "API research" based on GPT-3 and ChatGPT are con-
However, research on Intrucction-tning LLMSs in Chinese, the world's most spoken language, is still in its early stages. LLM bases, parameter-
"Equal contribution.
Corresponding author: Zheng Lin.

---

base LLMs
Tokens
Language
Size
LLaMA
IT
Mainly in English
6.7B, 33B, 65B
Bloom
341B
languages
7.1B, 176B
moss base
700B
Chinese and English
16.1B
ChatGLM*
IT
Chinese and English
6B
sft LLMS
Basc
Instruction data
Vicuna
LLaMA
70k Human-ChatGiP conversations
Bloomz & Bloomz-mt
Bloom
13-crosslingualltas mixture xP3 & xP3mt
moss-sft
moss-base
moss--022sff-data
ChatGLM
C
wrknowable
Table 1: Pre-training details of the popular open base LLMs (upper). Supervised fine-tuning (sft) details of the open sft LLMs. "*" denotes closed source, which denotes that ChatGLM only releases the supervised fine-tuned version. More details of each open LLM can be found Appendix A.2.
efficient methods, and instruction data are three es- sential elements for customizing Chinese ChatGPT- like LLMs. There are no tutorials in the academic community them yet. Some important ques- tions have not yet been explored and answered: "Which open LLM is more suitable as a foun- dation for Chinese instruction-tuningg"" 2) "How do parameter-effcient methods other than LoRA affect LLMs?" and 3) "What is the impact of vari- ous types of instruction datasets?" To answer these questions, we collect range of LLMs, parameter- efficient methods, and instruction datasets. Besides, consider the AGI cnttructtoo-followinn capa- bility and professional knowledge reserve (human exams) of models, and correspondingly select two benchmarks Belle-eval (Ji et al., 2023) and MMCU (Zeng, 2023) for comprehensively evaluation.
Instruction-tuning Triplets
2.1 Preliminaries
Problem Formulation. LLM bases E M, parameter-efficient methods p P and instruc- tion datasets d E D are the three crucial elements in instruction-tuinng This section examines the impact of each element in the instruction-tuning triplet (m, d) on final performance. We traverse the target element to thoroughly explore its impact, while fixing two elements in the triplet to control the variable. For example, we analyze the impact of different types of instruction datasets by comparing the performance of {(m, d.))
We select two evaluation bench- marks, Belle-eval and MMCU, to comprehensively evaluate LLM competencies in Chinese. Bellc-eval is constructed by self instruct with ChatGPT, which has 1,000 diverse instructions that involve 10 cate- gories covering common NLP tasks (e.g.. QA) and challenging tasks (e.g.. code and math). We ChatGPT to rate the model responses based on the golden answers. This benchmark is considered be as the assessment of AGI (instruction (osslooinnllllwin)) capability. MMCU is collection of Chinese mul- tiple choice questions in four professional disci- plines of medicine, law, psychology and education (e.g., Gaokao examination). It allows LLMs take exams in human society in multiple-choice test manner, making it suitable for evaluating the breadth and depth of knowledge of LLMs across multiple disciplines. More statistics and details are shown in Appendix A.
Benchmarks.
We also conduct experiments to explore several other factors that may affect the final performance. Specifically, we find that tuning with Chain-of- Thought (CoT) data can improve the ability to respond to complex reasoning questions. Differ- ent LLMS may be suitable for different language prompts (excluding instruction parts) in instruction- tuning. Human-value alignment results into slight performance drop. On the basis of the above find- ings, this paper carefully instruction-tune tunes power- ful Chinese LLMs that is comparable to ChatGLM.
The contributions can be summarized as fol- lows: (1) We are the first to systematically study instruction-tuning in Chinese through adequate experiments, which can serve cookbook that provides valuable findings for customizing Chinese version of ChatGPT. (2) We relcase a powerful Chinese LLM that is comparable to ChatGLM.
2.2 Open Large Language Models
To answer "Which open LLM more suitable as foundation for Chinese ceorracttonttuninn we

---

Open
Brain
Close
LLMs
Code
Clf.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
Storm
QA
LLaMA
45.0
6.6
17.9
41.3
16.8
40.2
42.5
61.2
28.8
27.6
32.8
base
Bloom
51.1
15.4
41.8
56.4
26.0
53.7
63.3
74.1
42.5
58.0
48.2
moss-base base
40.3
5.8
52.9
20.4
13.1
51.4
32.8
47.0
4.0
17.6
28.5
Vicuna
62.6
17.8
84.6
48.4
34.7
85.0
59.7
77.2
39,4
40.5
55.0
Bloomz
49.7
15.5
54.4
52.2
15.5
60.9
37.5
71.0
43.8
38.4
43,9
sft
Bloomz-mt
46.8
13.2
58.5
49.8
15.1
59.1
45.0
72.4
40,8
33.8
43.7
moss-sft
63.9
25.7
78.5
33.4
15.1
78.4
46.0
58.5
22.7
22.4
44.5
ChatGLM
64.7
39.9
91.8
53.2
46.5
91.0
61.9
82.4
48.8
53.8
63.4
Ours
72.4
41.4
91.5
64.7
36.1
92.3
62.5
85.8
45.6
38.9
63.1
ChatGPT
84.3
54.9
93.0
74.6
88.2
94.4
64.0
87.2
66.9
58.1
76.6
Table 2:
Performance of open LLMs Belle-eval. "Ours" is carefully designed instruction tuutee LLM, which is discussed in detail in Section 4. The best scores are bold, the second best scores are underlined. The results of ChatGPT are only for display and will not be compared.
LLMs
Med.
Psyc.
Law
Edu.
Avg.
Chinese tokens and HFRL. 3) The Open QA, Math, CloseQA and Extract categories are still very chal- lenging for existing open LLMs. 4) Vicuna and moss-sft have clear improvements compared their bases, LLaMA and moss-base, respectively,
LLaMA
2.66
3.75
1.14
1.95
2.38
Bloom
4.29
4.50
5.68
1.41
3.97
moss-base
7.70
7.70
7.98
8.47
7.96
Vicuna
10.50
10.20
8.44
13.12
10.57
Bloomz
35.15
32.30
17.29
36.87
30.40
The gain brought by Vicuna is more significant (22.2%) because instruction data, collected by ShareGPT, are the real conversations between hu- mans and ChatGPT, with higher quality. 5) In con- trast, the performance of sft models, Bloomz and Bloomz-mt, is reduced compared to the base model Bloom, because they tend to generate shorter sponse (refer to Appendix C.I). Unfortunately, ChatGPT often scores lower for short responses. The reason for this phenomenon is that Bloomz and Bloomz-mt -nt are fine-tuned from xP3, which is built from the NLP task collection where many tasks have brief annotations.
Bloomz-mt
33.77
31.40
15.59
35.12
28.97
moss-sft
16.78
14,45
9.47
15.43
14.03
ChatGLM
31.04
28.65
15.86
29.96
26.38
Ours
27.88
23.60
13.86
25.73
22.77
ChatGT
50.90
43.50
23.98
45.72
41.03
Table 3: Performance of open L.LMs on MMCU.
collect and evaluate the most widely used open LLMs' available in the open source community, as shown in Table
2.2.1 Evaluation of Existing LLMs
Performance on Belle-eval. Table shows the scores of open LLMs on Belle-eval. The upper part shows base LLMs while the lower part shows the the supervised fine-tuned (sft) LLMs. We can derive several observations: 1) For base LLMs. Bloom performs the best because its ROOTS (Workshop, 2023) pre-training dataset has large proportion in Chinese (2615 B), second only to En- glish. Although moss-base an LLM specifically proposed for Chinese, its performance is poor as is obtained through further pre- vve-training based on the CodeGen (Nijkamp et al., 2023) model and has only seen 100B Chinese data. 2) For sft LLMs, ChatGLM outperforms others by large margins, thanks to the fact that it is trained with the most
Performance on MMCU. Table 3 shows the ac- curacy of LLMs on MMCU, we find that: 1) All base LLMs perform poorly because it is almost difficult to generate content in the specified format before fine-tuning, c.g., outputting option numbers. 2) All sft LLMs outperform their corresponding base LLMs, respectively.
particular, Bloomz performs the best (even beats ChatGLM) because can generate option number directly as required without generating other irrelevant content (refer to Appendix C.2), which is also due to the data characteristics of its supervised fine-tuning dataset xP3. 3) Among the four disciplines, law is the most challenging for LLMs.
The I_MSs' performance on MMCU is much lower than that of Belle-eval because MMCU re- quires higher professional knowledge. All open
We mainly explore the version around 7b of them (except for 16b of Moss), which strike balance between performance and computation resource requirements.

---

LLMs
Param.
Layer
Belle
MMCU
56.2
57.8
AdaLoRA
5.6M
each
51.9
14.28
51.8
LoRA
15M
each
58.1
19.07
43.7
+136
+7.3
prompt
0.08M
embed
46.8
4.49
$00.9
p-tuning
1.IM
embed
46.0
15.50
prefix
30.8M
each
$1.6
16.18
SadapterP
7.5M
each
56.9
17.24
2.4
SadapterH
15M
each
58.7
20.23
Nama
bloom
Vituma
Bloomz
Bloomrentt
moxe-m
chelgim
P-adapter
15M
each
15.47
29.34
SadapterP-1
60M
each
55.0
16.10
20
18.07
18.82
+2.9
SadapterH-I
120M
each
54.7
18.60
15.20
24.40
P-adapter-1
120M
each
56.3
19.40
10
2,82
1244
8.12
10.57
10:44
Table 4:
Comparison of parameter-efficien meth- ods. "Param." denotes the trainable parameter quantity. "Layer" denotes the layers adapters are added. "-I" de- notes the version with large number of parameter. More details of each parameter-efficee methods can be found in Appendix A.3.
-2228
11.15
-00
Abma
Bloom
mods base
vizona
Bloomz
bloomp-mt
moss-oft
chalgit
Figure 1: Performance gains (denoted by origin bars) of open LLMs on Belle-eval (Upper) and MMCU (Lower)
from instruction-tuning The instruction tuned perfor- mance is denoted by blue bars and red numbers.
loss
LLMs still have significant room for improvement compared to ChatGPT.
2.2.2 Instruction-tuning Different LLMs
To determine the appropriateness of different LLMs as foundation for instruction-tuning in Chinese, we fine-tune all the open LLMs with the same parameter-efficient method LoRA and the same instruction dataset Alpaca-GPT4. The results are shown in Figure 12, where find that: 1) On Belle-eval, the performance improvement of sft LLMs brought by instruction- caaructiionttunnng is not as sig- nificant as that of base LLMs, except for sft Bloomz and Bloomz-mt. This is because the instructions of xP3 used for their supervised fine-tuning are not diverse enough. 2) Vicuna and ChatGLM encounter performance drops after instruction- tuning, because Vieuna is trained from real human- ChatGPT conversations, with better quality than Alpaca-GPT4. ChatGLM adopts HFRL (Ouyang et al., 2022), which may be no longer suitable for further instruction-tuning 3) On MMCU, most LLMs achieve performance boosts after instruction- tuning, with the exception of Bloomz and Bloomz- mt, which have unexpectedly significantly de- creased performance. This is because that original Bloomz and Bloomz-mt excel in multiple choice questions, but after further instruction-tuning they suffer catastrophic forgetting.
steps
AdaloRRA Prompt ". P-tuning -Prefix
LoRA Saclapert Sadaptori P-adapter
Figure 2: Training loss over steps for different parameter- methods.
After instruction-tuning. Bloom has signifi- cant improvements and performs well on both benchmarks. Although ChatGLM beats Bloom consistently, it suffers performance drop during instruction-tuning. Therefore, among all open LLMs, Bloom is most suitable as foundation model in the subsequent experiments for Chinese instruction instruction-tuning exploration.
2.3 Parameter-efficient Methods
For most researchers, parameter-efficient methods are essential for instruction-tuningg tuming due to limita- tions in computing resources. These methods tend to freeze the pre-trained model weights and in- jects trainable weights (adapters), which greatly reduces the number of trainable parameters. To answer "How do parameter-efticient methods other than LoRA affect LLMs?". we collect a range of crrameter--fffciien methods to instruction-
Fuul results are shown in Table 10 and 11 in App. B.I.

---

Datasets
Num
Con
Type diverse instructions
Source
Aplaca-GPTT
49K
SI
GPT-4
Belle
1.54M
SI
diverse instructions
text-davinci-003
ShareGPT-zhh
158K
MIX
Tuman-ChatGPT conversations diverse instructions
human & ChatGPT
moss-sft-data
1.76M
SI
text-davinci-003
instinwild
52K
SI
diverse instructions
text-davinci-003
firefly
1655M
COL
23 NLP tasks (1.15M) & Belle (0.5M)
human
IIC3
40K
MIX
QA dataset collection
human & ChatGPT
xP3/zh
1.07M
COL
16 NI.P tasks
human
COIG-ceme
68K
MIX
LLM-LLM role-playing chats based on knowledge graph dataset
COIG-trans
66K
COL
2000+ NLP tasks
translated
COIG-exam
64K
COL
examinations in China
human
PCLUE
1.2M
COL
9 NLP tasks
human
Table 5: The details of existing Chinese instruction datasets. "Con" column shows the dataset construction methods. "Type" column shows the data types. "Source" column shows the source where the data was generated. "SI" and "COL" denotes the self-instruct methods and the collection of existing datasets, respectively. "MIX" denotes the joint construction of humans and machines. "translated" denotes a translation from non- non-Chinese instructions. We filtered all datasets to remove incomplete instructions. More details each dataset can be found in Appendix A.4.
Bloom on Apaca-GPTT dataset.
quickly converge on training data and fit it well.
Comparison of Parameter--fficient Methods. From Table 4, several observations can be de- rived: 1) SadapterH performs the best among all parameter-efffiient methods, which can be used as an alternative to LoRA. 2) P-tuning and prompt- tuming underperform others by large margins, in- dicating that only adding trainable layers in the embedding layer are not enough to support LLMs for generation tasks. 3) Although AdaLoRA is improvement of LoRA, its performance has clear drop, possibly because the LoRA's trainable parameters for LLMs are not suitable for further re- duction. 4) Comparing the upper and lower parts, can be seen that increasing the number of trainable parameters for sequential adapters (i.e., SadapterP and SadapterH) does not bring gain, while the oppo- site phenomenon is observed for parallel adapters (i.e., P-adapter). This may provide inspiration for the design of adapters for L.M.. Since LoRA is cur- rently the most popular parameter-efficient method, if not otherwise specified, we adopt LoRA by de- fault in the experiments.
2.4 Chinese Instructions Datasets
Alpaca (Taori et al., 2023) inspires researchers further explore instruction data. To systematically explore "What is the impact of various types of instruction datasets?", we gather popular open Chi- nese instructions (as shown in Table 5) to fine-tune Bloom with LoRA.
Performance on Belle-eval. As shown in Ta- ble 6 upper part, it can be seen that: 1) the in- struction data constructed by ChatGPT (e.g., using self-instruction methods or collecting real human- ChatGPT conversations) consistently enhances the firrucioon-folowingg ability with 3.1 ~ 11-pt score increases. 2) Among these datasets, Belle has the best performance due to the largest amount of instruction data. However, the performance of models trained on moss-sft-data, containing more data built in similar way, is unsatisfactory. This because poss--ft-data's instructions sacrifice the diversity to achieve the goals of helpfulness, honey, and harmlessness. 3) The performance brought by the Alpaca-GPT4 instructions is the second best, with only 49K being comparable to the 1.54M Belle. This is because Alpaca-GPaa GPT4 uses the GPT- 4 engine while Belle uses the text-avinci-003 en- gine, which further illustrates that improving data quality can reduce the demand for data volumes. 4) Instinwild brings the least performance gains among them because the seed instructions it crawls from Tweet ("in wild") are not as comprehensive as those (like Alpaca) carefully designed by hu-
Training Loss. Figure 2 shows the training loss of different parameter-efficient methods. We find that: 1) commpt-tuning and P-tuning converge the slowest and has the highest losses after conver- gence. This shows that cmbedding-only adapters are suitable for instruction-tuning LLMs. 2) The initial loss of AdaLoRA is very high because it requires simultaneous learning of parameter bud- get allocation, which makes the model unable to fit the training data well. 3) The other methods can

---

Open
Brain
Close
Datasets
Code
Clf.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
Storm
QA
51.1
15.4
41.8
56.4
26.0
53.7
63.3
74.1
42.5
58.0
48.2
Alpaca-GPT4
53.7
35.0
88.2
50.5
36.8
89.5
60.3
82.9
40.0
44.1
58.1
Belle
64.3
37.3
86.0
66.2
22.3
88.6
62.6
83.5
43.3
38.1
59.2
ShareGPT-zhh
53.7
25.4
75.6
47.8
34.0
81.2
62.8
80.8
33.5
41.6
53.6
moss doss-aft-data
55.4
21.7
81.6
51.7
24.3
81.8
65.4
78.1
37.6
45.1
54.3
instinwild
55,5
24.5
70.3
40.8
26.9
79.6
61.8
78.4
37.7
37.0
51.3
firefly
61.4
31.8
79.8
53.2
26.5
84.4
62.5
83.1
36.7
47.8
56.7
HC3
46.8
21.3
24.6
33.9
44.4
30.3
47.8
36.5
19.7
36.4
xP3/2h
32.1
13.3
17.0
39.4
12.1
24.1
21.5
66.5
40.0
28.1
29.4
COIG-trans
34.7
15.5
58.9
47.9
24.4
62.1
51.5
80.0
37.7
44.6
45.7
COIG-ceme
39.2
15.0
44.0
16.5
21.2
22.3
23.3
31.6
19.8
19.3
25.2
COIG-exam
24.2
15.4
53.7
33.3
17.9
60.2
44.0
69.6
27.1
24.9
37.0
PCLUE
19.7
19.1
53.5
34.3
36.9
50.5
34.0
68.5
44.4
39.3
40.0
Table 6: Belle-eval performance of models Instruction-tungg from Bloom on different instruction datasets.
Datasets
Med.
Psyc.
Law
Edu.
Avg.
tasks with a wide variety of task instructions. In contrast, xP3 and COIG-ceme have the worst neg- 3,97
4.29
4.50
5.68
1.41
Alpaca-GPT4
27.70
17.35
17.59
13.63
19.07
ative impact on model performance. Both of them only cover few types of tasks (translation and QA for the former, counterfactual correction conversa- tions for the latter), which hardly cover the popular instructions and tasks for humans.
Belle
21.57
19.05
15.13
15.28
17.76
ShareGPT-zh
8.83
8.75
12.18
12.04
10.45
moss-sft-data
16.60
17.75
11.72
17.29
15.84
instinwild
14.90
17.45
11.50
14.17
14.51
firefly
22.49
18.30
9.69
20.50
17.75
HC3
9.15
14.10
8.01
7.24
9.63
Performance on MMCU. Table 7 compares the performance MMCU brought by different in- struction datasets. 1) Instruction-tuning on each dataset can always result in performance improve- ment. 2) Among the ChaiGPT-based data shown in the upper part, ShareGPT-zh underperforms oth- ers by large margins. This may due to the fact that real users rarely ask multiple choice ques- tions about academic topics. 3) Among the dataset- collection data shown in the lower part, HC3 and COIG-ceme results in the lowest accuracy because that the unique questions of HC3 is only 13K, and the task format of COIG-ceme is significantly dif- ferent with MMCU, 4) COIG-exam' brings the greatest accuracy improvement, benefiting from the similar task format as MMCU.
xP3/zh
20.43
19.50
15.62
19.60
18.79
COIG-trans
18.62
17.9
11.31
16.51
16.09
COIG-ceme
7.31
10.15
8.12
7.63
8.30
COIG-exam
32.56
26.90
16.18
PCLUE
20.29
25.40
13.91
27.80
21.85
Table 7: MMCU performance of models instruction- tuned from Bloom on different instruction datasets.
mans. 5) These ChatGPT-based data mainly have significant improvement effect on open generation tasks such as Brain Storm and Generation, while there is significant decrease tasks that require high reading comprehension skills, such as Close QA and Extract, which require completing tasks based on given materials. This inspires researchers to consider the mondiin-comppehenniion ability for building more comprehensive instruction datasets.
Other Important Factors
The lower part of Table shows the results of models trained on dataset-based data, which is mainly constructed by collecting NLP or exami- nation datasets. These instruction datasets cause damage to the model's instruction-follwwee ity, because the form and intent of each NLP or examination dataset are unitary, which can easily be overfitted. Among them, COIG-trans performs the best because it involves over 2000 different
Problem Formulation. In addition to the essen- tial three elements (m, p. d) discussed above, there are many factors worth exploring. e.g., COT. If not otherwise specified, we use Bloom as the LLM base, LoRA as the parameter-efficien method, and Alpaca-GPa GPT: as the instruction data. On this basis,
abil-
Chinese data due large quantity. "Due to the overlap hetween COIG-exam and MMCU- Edu., the accuracy on Edu. discipline will not be reported.
"We use

---

Belle-eval
MMCU
55.6
Data
Code
Math
Avg.
Edu.
Avg.
51.2
Rama
Alpaca-GPT4
53.7
36.8
58.1
13.63
19.07
43.7
Rama- Dooma-vo--pae
Nama voc
Alpaca-GPT4-CoT
60.8
41.7
57.9
21.56
19.85
Hame-voc-ppee-
Apaca-GPT4+CoTT
62,9
39.5
57.2
22.07
21.56
21.72
18.92
Table 8: The impact ofain-of-thought data on com- plex tasks requiring reasoning. "*" denotes that using prompt & # ("think step by step" in
3.9
2.82
4.17
MMCU
Chinese) during inference.
Figure Performance comparison of LLaMA and expanded vocabulary versions. "llama-voc,, "Ilama-voc- pre" and (lamm-voc-pre-I1 denotes instruction-tuni tuning the models obtained by further pre-training LLaMA with expanded vocabulary OB, 20B and 100B Chi- nese tokens, respectively.
we explore its impact by observing the performance changes after considering the target factor.
Chain-of-Though Data. Chain-of- Thoought is a hot topic in LLM research. Existing works find that adding rationales explanations to the inference prompts (Wei et al., 2023; Wang et al., 2023; Ko- jima al., 2023) (based APIs of GPT-3 and ChatGPT) or training corpus (Wei et al., 2022; Chung al., 2022; Zhang et al., 2023c) (based on normal language models, e.g. T5(Raffel et al., 2020) and FLAN-T5(Wei et al., 2022)) can enhance the model's s reasoning ability, which is useful for solving complex problems. However, extending CoT into Open LLM has not yet been thoroughly explored. Alpaca-Cor C (Qingyi Si, 2023) uses sev- eral qualitative examples demonstrate the effec- tiveness of CoT in reasoning. A systematic eval- uation is still necessary. To this end, this paper conducts experiments to analyze the impact of CoT data for LLMs.
the model to respond to instructions based on the chain-of-thouuht. As shown in the line of AAlpaca- GPT4+CoT*" the simple sentence can further im- prove the performance of reasoning tasks Code and Education, while the Math performance is slightly inferior to "Alpaca-GPT44CO0 This may require us to further explore more robust prompts.
Expansion of Chinese Vocabulary. Intuitively, the number of Chinese tokens the tokenizer's vocabulary affects LLMs' ability to express Chi- nese. For example, if a Chinese character is in the vocabulary, it can be represented by single token, otherwise may require multiple tokens represent it. Because Bloom adopts vocabulary of 250k tokens which cover most Chinese charac- ters, we mainly conduct experiments on LLaMA, which uses SentencePicce (Sennrich et al., 2016; Kudo and Richardson, 2018) (32K vocabulary size) covering few Chinese characters.
We collect 9 CoT datasets and their prompts from FLAN (Wei al., 2022), and then translates them into Chinese using Google Translate. We compare the performance before and after adding CoT data during instruction-tuna tuning in Table 8. outperforms "AlpacaGPT4" in the Code and Math tasks that require strong rea- soning ability. Besides, there is also significant improvement in MMCU education task, which is derived from the questions of Gaokao, involving range of subjects, c.g., math, physics, history. The accuracy improvement across all subjects il- lustrates that the CoT reasoning ability is gener- ally required in various subjects. However, CoT training data cannot continue to bring benefits to all tasks, and on the contrary, it will cause slight performance degradation more tasks. The full results can be found in Appendix B.3.
AApacc--GP44-CG
As shown in Figure 3, we find that the perfor- mance of "Ilama-voc" severely inferior to "llama" on Belle-eval, and is almost unable to respond cor- rectly to MMCU's instruction. This indicates that is not feasible to perform instruction-tung -tnninnnnng with- out pre-training vast data. This is because the embedding corresponding to the newly added Chi- nese token are random and meaningless, which results in the model being unable to understand the meaning of the instructions.
To make the newly added Chinese token mean- ingful, Cui et al. uses 20B and 100B token Chi- nese corpus to further pre-train LLaMA and obtain "llama-occ-n voc-pre" and "Ilama-voc-pred" models. We use Alpaca-GPT4 to instruction- tunne these models, and find that, pre-training on more Chinese cor-
Inspired by (Kojima al., 2023), we add a sen- tence" " &. $ * " ("think step by step" in Chinese) at the end each instruction, to induce

---

45.4
prmppr Dighit
porprr Sught
M/b Auman
Awraoo valve
18.07
15.66
6.27
2.87
Belle-eeval
(a) LLaMA
MMCU
Belle-eval
Bloom
MMCU
Belle-eval
MMCU
Figure Performance comparison of instruction -uning with prompts in English and Chinese. The specific prompts used in our experiments can found in Appendix D.2.
Figure 5: Performance comparison of instruction -uniing with and without human-value alignment.
pus with expansion of Chinese vocabulary con- sistently helpful for instructionnfoolooiing ability. Counterintuitively, "Ilama-voc-pre-"" is inferior to "llama-voc-pre" on MMCU shows that pre-training on more data may not necessarily lcad higher performance for academic exams.
4 Towards Better Chinese LLM
Problem Formulation. to find a optimal triplet (m, p. d) that maximizes the comprehensive capabilities:
The goal of this section
max m.p.d
Σ (aa ECT
(1)
where & denotes the evaluation of every gener- ative ability from both Belle-eval and MMCU T. fa p) denotes the model obtained by intructiin-tuning frozen LLM m with paramcter- efficient method p on instruction dataset d.
The Languages of Prompts. The popular open instruction-tuned LLMs, e-g., Alpaca and Vicuna, tend uses prompts in English. One intuitive ques- Chinese more suit-
tion is, Is instruction-tuning able for using Chinese prompts? Figure shows the results of using Chinese and English prompts based on LLaMA and Bloom. When instruction- tuning LLaMA, using Chinese prompts can im- prove the performance on both benchmarks com- pared to English prompts, while observe the opposite phenomenon on Bloom. This demon- strates that using Chinese prompts for models with weaker Chinese abilities (e.g.. LLaMA) can effec- tively help respond Chinese, while for models with good Chinese abilities (e.g., Bloom), using prompts in English (the language they are better at) can better guide the model to understand the process of fine-tuning with instructions.
Our Instruction-tuned LLM. On the basis of the findings above, we carefully design the instruction-tuning process and publicly release a Bloom-based high-performance LLM, which is comparable to ChatGLM and far surpassing Moss. In particular, we select a dataset combination with significant gains on Belle-eval or MMCU to im- prove model's comprehensive ability. Besides, carefully design suitable prompt to induce model for better-quality generation. The implemen- tation details can found in Appendix D.1.
As shown Table 2, our model is superior or comparable to ChatGLM in most categories on Belle-eval, except for the challenging Math and Extract tasks. Besides, our model slightly under- performs ChatGLM on MMCU and outperforms other LLMs that do well in Belle-eval by clear mar- gins. worth emphasizing that our model has much fewer trainable parameters (16M) based on LoRA than that of ChatGLM adopting full parame- ter fine-tuning (6B).
Human Value Alignment. To avoid LLMs gen- erating toxic content, aligning them with human values a crucial issue. We add the human-value alignment data built by COIG (see App. A.4 for details) into the instruction-tune tunnon to explore its im- pact. Figure 5 compares the results of instruction- tuning with and without human-value alignment, which shows that the human-value alignment re- sults into a slight performance drop. How bal- ance the harmlessness and performance of LLMs is a research direction worth exploring in the future.
Conclusion
This paper is the first to conduct a thorough empiri- cal study on instruction instruction-tuning open large language models Chinese, with a detail discussion of a

---

range of large language models, parameter-efffcient methods, and Chinese instruction datasets. In ad- dition, we explore several other important factors, including CoT, vocabulary, language of prompts and human-value alignment. Based on the empir- ical exploration, we publicly release LLM, that is rival to ChatGLM, with detailed implementation details.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai. Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mad- eling with pathways.
Limitations
Most experimental results are based on parameter- efficient methods, which may differ from the re- sults of full parameter fine-tuning. However, believe that the findings and conclusions in this paper are still applicable for full parameter fine- tuning. In addition, instruction-tuning based on parameter-efficien methods has broader applica- tion and research scenarios.
Ethics Statement
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro- Ros. Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models.
The open LLMs used in this paper may be driven by certain biases in their training data, and pose a risk of toxic generation. There may also exist harmful stereotypes in the open instruction datasets we are discussing. There is still long way to explore the safety of LLMss.
Acknowledgement
This work was supported by National Natural Sci- ence Foundation of China (No. 61976207) and National Social Science Foundation of China (No. 21 21AZD145).
Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca.
Biyang Guo, Xin Zhang, Ziyuan Wang. Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? compar- ison corpus, evaluation, and detection. arXiv preprint anxvv2301.07500
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig, 2021. Towards unified view of parameter-effiiient transfer learning. arXiv preprint
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optina large language models. arXiv preprint arXiv:2203. 15556.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ton Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing 4 with 90%* chatgpt quality.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea

---

Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, Parameter-efficien transfer learning for nlp. In In- ternational Conference Machine Learning, pages 2790-2799. PMLR.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebas- tian Ruder. 2020, Mad-x: An adapter-base frame- work for multi-task cross-lingual transfer. arXiv preprint arxiv:2055.0055.
Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation large language models.
Tong Wang Qingyi Si. 2023.
Alpaca-cot: An
instruction- -tunngg platform with unified interface of instruction collection, parameter- methods, and large language models. https://g/ //gitub......... thub. com/ PhoebusSi/ aaccaaCCo..
Yunjie Ji, Yan Gong Yong Deng. Qiang Niu Yip- ing Peng, Baochang Ma, and Xiangang Li. 2023. Belle: Be everyone's s large language model engine. https: /ggi thub. comm/LianjiiTTech/BELLEE
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units.
Taku Kudo and John Richardson. 2018. Sentencepiece: simple and language independent subword tok- enizer and detokenizer for neural text processing.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li. Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face.
Brian Lester. Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- Republic. Association for Computational Lin- guistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followee followwig Ilama model. https: gi wtt.hb..ootttttuuuu saaattsullabbaae
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P- tuning v2: Prompt tuning can be comparable to fine- tuning universally across scales and tasks.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanie. Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. https: //gi::///ttii htt/g/tttubu... wtt.ub.com/kinggfooll1/./ mess-trranfforme--iis
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Fd Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Yizhong Wang, Swaroop Mishra, Pegah Alipoor- molabashi, Yeganeh Kordi, Amirreza Mirzaei, Aujana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Is- han Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Ku- mar Pal, Mehrad Moradshahi, Mihir Parmar, Mi- rali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint orXiv:2304.03277

---

Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannanch Hajishirzi, and Daniel Khashabi. 2022. Supper-naturllinsstuctionss Generalization via declarative instructions on 1600+ nlp tasks.
Belle-eval
MMCU
Code
Open QA
Medicine
38
285
2819
Brainstorm
Classification
Psychology
179
65
2000
Math
Generation
Law
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022. Finetuned language models zero-shot learners.
75
98
3695
Summary
Rewrite
Education 3331
40
131
Close QA
Extract
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2023, Chain-of-thought prompting elic- its reasoning in large language models.
$2
37
Total
1000
11845
Table 9: Data statistics of Belle-eval and MMCU.
BigScience Workshop. 2023.
Bloom: 176b- parameter open-access multilingual language model.
A More Details about the Work Involved
Bo Xu, Yong Xu, Jiaqing Liang, Chenhao Xie, Bin Liang, Wanyun Cui, and Yanghua Xiao. 2017. Cn- dbpedia: A never-ending chinese knowledge extrac- tion system. In Advances Artificial Intelligence: From Theory to Practice, pages 428-438, Cham. Springer International Publishing.
A.1 Benchmarks
Belle-eval During this evaluation, ChatGPT is used to rate (from 0 to 1) the model response based on the ground truth answer. A score of O indicates that the model response is completely unacceptable, while score of 1 indicates that the response per- fectly solves the input instruction. The prompts and instructions for samples in each category are rich and varied. We consider the capability examined in this dataset to be AGI (instrcution-followiii capability.
Fuzhao Xue, Kabir Jain, Mahir Hitesh Shah, Zangwei Zheng, and Yang You. 2023. Instruction in the wild: user-based instruction dataset. attps://g/ thub. htw.///ufuzhha/nIsstttttttnonoo
Jianxin Yang. 2023. Firefly. https://github... yangjianxinn/Fireflyy
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge- based vqa.
MMCU MMCU (Zeng, 2023) is collected from online public resources, covering 11845 multiple choice questions in four professional disciplines. There are several subtasks under education and medicine disciplines. The average accuracy of all subtasks is considered the discipline score. Only when a generated answer and the annotated ground truth option number or option content completely match, is the answer considered correct. This eval- uation is relatively rigid for expected outputs. We consider the capability examined in this dataset as the reserve of professional knowledge (to deal with human examinations).
Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-1300 130b: An open bilingual pre-trained model. arXiv preprint arX0v:2210.02414.
Hui Zeng. 2023. Measuring massive multitask chinese understanding.
Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu. 2023a. Chinese open instruction generalist: A preliminary release.
These two assessments complement each other some extent. Table 9 shows the data statistics of these two benchmarks.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b. Adaptive budget allocation for parameter-efficient fine-tuning, In The Eleventh In- ternational Conference Learning Representations.
A.2 Open Large Language Models
A.2.1 Base LLMs
LLaMA LLaMA (Touvron al., 2023) is decoder-only language model based on the Trans- former (Vaswani al., 2017) architecture, and is trained on more tokens (IT, 1.4T) than what is typ- ically used (Hoffmann et al., 2022). It ranges from
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023c. Multi- modal Mhain-of-thoughh reasoning in language mod- els.

---

7B 65B parameters and outperforms existing LLMs (e.g. GPT-3 (Brown al., 2020) and PaLM (Chowdhery et al., 2022)) with fewer parameter magnitudes. However, the vocabulary of its tok- enizer contains fewer Chinese characters, which affects its expressive power in Chinese
IT tokens in Chinese and English, and then adds supervised fine-tuning and human feedback rein- forcement learning (HFRL) (Ouyang et al., 2022) processes to force model to follow instructions.
A.3 Parameter-efficien Methods
LoRA Low-Rank Adaptation (LoRA) (Hu al., 2021) injects trainable rank decomposition matri- ces into each attention layer of the Transformer architecture.
Bloom Bloom (Workshop, 2023) is multilin- gual language model trained on dataset ROOTs, in- volving 46 natural and 13 programming languages. The proportion of Chinese corpus in pre-training data is second only to English corpus. The maxi- mum version of Bloom has 175B parameters, while the most popular version is 7B.
AdaLoRA AdaLoRA (Zhang et al., 2023b) al- locates the parameter budget adaptively to each layer's LoRA module according to their importance score. Specifically, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition, which allows it to prune the singu- lar values of unimportant updates and reduce their parameter budget.
Moss-moon-003-baal (base model of MOSS series, moss-base for short) is initialized with CodeGen (Nijkamp et al.,
mooss-oon-0003-een
2023) and further self-supervised pre-trained on high-quality Chinese (100B) and English (20B) corpus. All the pre-training data contains about 700B words. It has 16B parameters.
Prefix-tuning Inspired by discrete prompts for language models, prefix-tuning (Li and Liang, 2021) adds sequence of continuous "virtual to- kens" soft prompt (namely prefix) before the original sequences of each transformer layer. Dur- ing training, prefix weights are trainable while other model parameters are frozen.
A.2.2 Supervised Fine-tuned LLMs
Vicuna Vicuna (Chiang al., 2023) is fine-tuned from LLaMA on 70K user-shared ChatGPT con- versations gathered by ShareGPT5 Vicuna claims to have achieved 90% performance of ChatGPT on a preliminary evaluation using GPT-4 as a judge, making it the most popular open source LLM. How- ever, further rigorous evaluation is needed, espe- cially in Chinese scenarios.
P-tuning Unlike prefix-tuning, p-tuning (Liu et al., 2022) injects trainable continuous tokens into only the embedding layer instead of each layer, resulting in fewer parameters being updated. Dur- ing training, freezes partial model parameters.
Bloomz & Bloomz-mt Bloomz and Bloomz-mt are fine-tuned from Bloom on crosslingual task mixture xP3 (Muennighoff et al., 2023) and xP3mt, which contain 13 training tasks in 46 language with prompts in English and in 20 languages, re- spectively, This supervised fine-tune process aims to further boosts the performance of multilingual tasks.
Prompt-tuning Similar p-tuning, prompt- tuning (Lester et al., 2021) also involves only train- ing the input prompt embeddings. Differently, it freezes all pre-trained weights.
For each transformer layer, Series Adapter methods add adapter layers af- ter both attention layers and MLP layers (i.e., S- adapterH (Houlsby et al., 2019)), or after MLP layers only (i.c., S-adapterP (Pfeiffer et al., 2020)).
Sequential Adapter
0003-st Moss-moon-003-- (moss- sft for short) fine-tuned from moss-moon-O00
base on moss-002-sft-data, which contains 0.57M English and 0.59M Chinese dialogues generated by text-davinci-003, and O. real user instructions (the corresponding response generated by gpt- 3.5-turbo) collected during internal test.
Parallel Adapter Parallel Adapter, namely P- adapter (He et al., 2021) adds adapter layers parallel with attention layers or MLP layers for each transformer layer.
ChatGLM ChatGLM-6B (Zeng et al., 2022) is an open bilingual LLM, supporting both Chinese and English. It first completes pre- pre-training on about
A.4 Chinese Instruction Datasets
A AlpacaGPT4 AlpacaGPT4 (Peng al., 2023) is deemed as an optimized version of Alpaca (Taori et al., 2023) dataset. It uses ChatGPT to translate
"htnps//sharegpp.com/

---

Alpaca's prompts into Chinese first, and then re- generale these foollooi data by GPT-4, instead of text-davinci-003.
descriptions in Super-Vaturallnssruciioon (Wang et al., 2022) along with single instance for each of them, 175 instructions of seed tasks in Self- Instruct, and 66K instructions from Unnatural In- structions (Honovich et al., 2022). CCMC, namely Counterfactual Correction Multi-round Chat, con- tains about 68K rounds of conversations between students and teachers. This dataset is built by prompting two LLMss to generate conversations based on the entities of knowledge graph dataset CN-DBpedia (Xu et al., 2017) to alleviate the hal- lucination and factual inconsistency. Exam In- structions contains 63K questions from the main Chinese commonsense tests, e.g., Gaokao, Civil Servant Examination. These questions cover six main subjects: Chinese, English, Politics, Biol- ogy, History, and Geology. Exam Instructions contains 34K Chinese samples that present shared human values in the Chinese-speateee world (3K) and regional-culture human values. Table shows some examples in this dataset.
Belle Belle (Ji et al., 2023) uses the same method Alpaca (Taori et al., 2023) to generate instruction data by rext-davinci-003. except that Belle only generates Chinese intructtoo-followinng data and artificially filters low-quality data. It contains about 1.5M instruction-following data.
0oss-002-stt-data This is multi-turn conver- sation dataset covering helpfulness, honesty, and harmlessness, which is also generated by self- instruct (Ouyang et al., 2022). We select the 0.59M Chinese conversations among them for the follow- ing experiments.
firefly Firefly (Yang, 2023) collects 23 Chinese datasets and manually writes several instruction templates for each dataset. It contains total of 1.65M training samples, covering couplet, poem, essay, and other generation tasks for the perfor- mance of traditional literature, and 0.5M Belle data for instruction diversity.
PCLUE PCLUE collects 9 Chinese tasks with a total of 73 different prompts and 1.2M samples. These tasks include Chinese tasks e.g., news clas- sification, natural language reasoning, semantic matching, keyword recognition, reading compre- hension, etc.
xP3 xP3 (Muennighoff et al., 2023) is a collec- tion of 16 natural language process datasets across 46 languages with prompts. We select the 3M Chi- nese instances among them.
Table 25 and Table 26 show representative ex- amples of the above datasets.
instinwild Instruction in the Wild (instinwild) (Xue et al., 2023) no longer manually sets initial seed instructions like Alpaca, but crawls and fil- ters 429 instructions from Twitter the seed in- structions, to avoid human involvement and cover more topics. Following self-instruet, it uses the seed instructions to generate more instructions and corresponding responses by text-davinci-003, The Chinese instructions in this dataset are about 52K.
B
More Experimental Results
B.1
Full results of different LLMs after instruction-tuning:
Table 10 and show the full Belle-eval and MMCU results after instruction-tuning with LoRA on Iipaca-GPT4 dataset, respectively.
HC3 HC3 (Guo et al.) is a corpus of Human- ChatGPT comparisons that aims to investigate how close ChatGPT is to Human Experts. To this end, it collect about questions from various public ques- tion answering datasets (e.g., medicine, law, fi- nance QA) and the corresponding human answers and ChatGPT answers. The Chinese samples in HC3 contain 13K questions, 22K human answers, and 17K Chatgpt answers.
B.2 Full results of different parameter-efficient methods.
Table 12 and Table 13 shows the full Belle-eval and MMCU results of instruction-tuning with different parameter-efffcient methods, respectively.
B.3 Full results of inttructtoo-tunnnng with CoT data.
Table 14 and 15 show the full Belle-eval and MMCU results of the models instruction-tunet without or with CoT data, respectively. Table 16 shows the detailed results on all subjects educa- tion discipline of MMCU.
COIG COIG (Zhang et al., 2023a) is Chi- nese instruction collection, consisting of: Trans- lated instructions contains about 67K instructions which are translated from three datasets: 1.6K task

---

Open
Brain
Close
Code
Math
Gen.
Rewrite
Extract
Avg.
QA
Storm
QA
I.LaMA
53.7
11.9
66.7
36.3
27.3
65.4
49.5
62.7
24.8
38.6
43.7
Bloom
53.7
35.0
88.2
50.5
36.8
89.5
60.3
82.9
40.0
44.1
58.1
moss-base
58.8
25.1
82.9
33.4
27.9
83.8
35.2
15.2
19.2
43.1
Vicuna
64.2
19.7
50.0
35.2
79.6
42.3
54.9
Bloomz
58.4
33.8
88.4
50.5
32.9
92.6
56.0
82.3
38.3
29.1
56.2
Bloomz-mt
62.9
35.1
85.6
49.6
31.7
90.5
55.1
82.4
44.2
36.2
57.3
moss-sft
67.4
30.7
88.5
46.5
33.6
88.6
47.5
72.1
20.2
22.7
51.8
ChatGLM
61.7
34.0
85.4
50.8
50.4
89.1
64.3
81.8
46.9
45.7
61.0
Table 10: Performance of open LLMs on Belle-eval after instruction-tuning with LoRA on Alpaca-GPT4 dataset.
Med.
Psyc.
Law
Edu.
Avg.
LLaMA
3.26
3.20
1.49
3.33
2.82
Bloom
27.70
17.35
17.59
13.63
19.07
moss-base
12.49
11.35
7.01
10.72
10.39
Vicuna
18.98
19.75
14.70
21.83
18.82
Bloomz
7.56
6.65
9.61
8.65
8.12
Bloomz-mt
15.61
18.65
11.58
15.19
15.26
moss-sft
16.35
14.55
17.29
14.40
ChatGLM
34.09
29.85
18.94
34.49
29.34
Table 11: MMCU Performance of open LLMs after instruction-tuning with LoRA on Alpaca-GPT4 dataset.
Open QA
Brain Storm
Close
Code
Clf.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
AdaLoRA
44.7
24.1
70.9
47.2
29.2
71.5
65.5
75.3
48.8
41,4
51.9
LoRA
53.7
35.0
88.2
50.5
36.8
89.5
60.3
82.9
40.0
44.1
58.1
prompt
47.4
11.9
39.1
43.3
42.3
46.7
60.0
49.0
61.6
46.8
p-tuning
53.9
18.8
63.2
40.0
30.7
63.4
54.7
65.7
35.8
34.1
46.0
prefix
51.6
31.8
80.5
42.8
31.5
79.3
46.3
76.4
39.6
36.2
51.6
SadapterP
58.7
33.6
48.2
34.5
87.4
61.3
82.9
34.2
44.1
56.9
SadapterH
64.7
37.5
86.3
47.7
34.5
88.3
56.5
85.6
41.7
43.8
58.7
P-adapter
53,9
33.9
83.5
48.8
31.1
86.7
61.0
81.8
37.9
38.4
55.7
SadapterP-1
62.9
33.5
85.1
48.4
36.8
85.4
51.6
77.2
35.0
34.1
55.0
SadapterH-I
35.2
84.7
50.6
36.5
86.9
53.5
40.2
25.4
54.7
P-adapter-1
65.3
37.1
85.5
46.5
34.7
86.5
56.2
82.8
38.8
29.2
56.3
Table 12: Full Belle-eval results of Bloom after instruction-tun with different parameter-efficient methods on Alpaca-GPP GPT4 dataset. "-I" denotes the version with large number of parameter.
Med.
Psyc.
Law
Edu.
Avg.
AdaLoRA
16.32
15.10
9.23
16.48
14.28
LoRA
27.67
16.60
17.35
12.43
18.51
tuning
15.43
17.30
10.64
18.61
15.50
prompt
4.68
6.30
5.20
1.77
4.49
prefix
19.33
13.70
15.13
16.54
16.18
SadapterP
21.21
16.20
13.67
17.89
17.24
SadapterH
26.85
17.40
15.24
21.44
20.23
P-adapter
17.56
13.50
14.26
16.54
15.47
SadapterP-1
18.02
16.55
12.21
17.62
16.10
SadapterH--1
21.64
16.20
14.29
22.28
18.60
P-adapter-1
26.96
15.30
16.13
19.21
19.40
Table 13: Full MMCU results Bloom after Alpaca-GPT4 dataset. "-1" denotes the version with large number of parameter.
with different parameter-effiiien methods on

---

Open
Brain
Close
Code
Cif.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
Storm
QA
Alpaca-GPTI
53.7
35.0
88.2
36.8
60.3
82.9
40.0
44.1
Alpaca-GPP GPT4-CoT
60.8
34.9
49.5
41.7
88.7
80.4
37.7
41.6
57.9
Alpaca-GPT4-HH GPT4+CoT*
62.9
36.0
85.0
53.5
39.5
86.1
49.0
83.7
35.0
41.5
57.2
Table 14:
Belle results of Bloom instruction-tune tunde with and without CoT data.
Med.
Psyc.
Law
Edu.
Avg.
Alpaca-GiP
27.70
17.35
17.59
13.63
19.07
22.35
23.05
12.45
21.56
19.85
AP
26.92
24.55
12.69
22.07
21.56
Table 15: MMCU results of Bloom instruction-tuneds with and without CoT data.
B.4 Full results of LLaMA and its expanded vocabulary versions.
C.3
Comparison samples from different instruction datasets.
Table 17 and Table 18 shows the full Belle-eval and MMCU results of LLaMA and its expanded vocabulary versions, respectively.
In Table 25 and Table 26, we select a representa- tive sample for each instruction dataset to better understand their respective characteristics.
B.5 Full results of the comparison of using English and Chinese prompts.
C.4 Comparison of the responses from models instruction-tuned on different instruction datasets.
Table 19 and Table 20 shows the full Belle-eval and MMCU results of using Chinese and English prompts based on LLaMA and Bloom.
To compare the characteristics of models trained on different instruction datasets more intuitively, we present in Table 27 the responses of models instruction tuntod on different datasets for the same question.
B.6 Full results with human-value alignment
Table 21 and Table 22 shows the full results of the models instruction-tuned with and without human- value alignment data.
Qualitative Examples
C.5 Comparison of the responses from LLaMA and its expanded vocabulary versions.
C.1 Comparison of responses of Bloom and Bloomz & Bloomz-mt Belle-eval.
As shown in Table 23, Bloomz & Bloomz-mt tend to generate shorter responses than that of Bloom. Accordingly, ChatGPT rates Bloom higher than Bloomz and Bloomz-mt. We conduct a statistical analysis and find that the average length of Bloom's response is 481 words, while that of Bloomz and Bloomz-mt are 83 and 58 words.
We present examples of responses from LLaMA and its expanded vocabulary versions in Table 28, The response from "Ilama-voc" is clearly not under- standing the meaning of the instruction. Therefore, after expanding the vocabulary, pre-training should be conducted on the vast Chinese corpus before fine-tuning instructions.
C.2 Comparison of different LLMs' responses on MMCU.
C.6 Examples from human-value alignment dataset.
As shown in Table 24, all base LLMs fails to gener- ate content in the specified format, i.e., outputting option numbers. Bloomz & Bloomz-mt rectly generate option numbers. Although the gen- eration of ChatGLM mentions the correct answer, but fails to provide the corresponding answer num- ber.
The samples of human-valuc alignment dataset, built by COIG, shown in Table 29. These sam- ples are often related to topics such "online vi- olence" and "gender discrimination", which are designed to ensure that the model has the correct values when facing relevant topics.
di-

---

Chinese
Math
Physics
Chemistry
Politics
History
Geography
Biology
Avg.
Alpaca-GPT 13.18
16.72
12.50
10.00
16.88
10.07
12.77
16.95
13.63
Alpuca-GPT4+CO
18.99
14.88
18.00
24.89
23.73
23.97
19.61
20.32
Alpaca-GPT4+Co1"
19.38
20.00
14.29
26.00
21.10
23.15
22.70
23.95
21.32
Table 16:
Results of all subjects in MMCU's education discipline of Bloom instruction-tuned with and without
CoT data.
Open
Brain Storm
Close
Code
Clf.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
QA
Numa
53.7
11.9
66.7
36,3
27.3
65.4
49.5
62.7
24.8
38.6
43.7
Nama-voc
24.7
2.7
14.1
10.8
6.9
10.1
3.7
17.3
8.7
0.0
9.9
lama-voc-pre pre
56.3
23.6
78.3
52.1
31.1
79.8
46.5
81.4
30.8
31.6
51.2
Mlaa-voc--pres
53.9
31.1
83.0
60.5
34.3
84.2
57.0
75.0
37.7
38.9
55.6
Table 17: Full Belle-eval results of LLaMA and its expanded vocabulary versions. "Ilama-voc", "Ilama-vor pre" and coonnn-voc-pr--ee denotes instruction tuning the models obtained by further pre-training LLaMA with a expanded vocabulary OB, 20B and 100B Chinese tokens, respectively.
Med.
Psyc.
Law
Edu.
Avg.
Ilama
3.26
3.20
1.49
3.33
2.82
llama-voc
3.12
4.65
6.2
2.7
4.17
llama-voc-pre
26.68
20.50
16.32
23.36
21.72
llama-voc-pre-1
23.63
16.65
14.61
21.04
18.98
Table 18: Full MMCU results of LLaMA and its expanded vocabulary versions. "llama-voe,, "Ilama voc-aa-vooeeeeee and "llama-voc-pre- denotes instruction-tumat the models obtained by further pre-training LLaMA with a expanded vocabulary OB, 20B and 100B Chinese tokens, respectively.
Open
Brain
Close
Code
Clf.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
Storm
QA
L.LaMA-en
53.7
11.9
66.7
36.3
27.3
65.4
49.5
62.7
24.8
38.6
43.7
LLaMA-zh
63.0
14.4
70.8
38.9
31.7
65.9
50.5
60.5
20.8
34,9
45.1
Bloom-en
53.7
35.0
88.2
50.5
36.8
89.5
60.3
82.9
40.0
44.1
58.1
Bloom-zh
61.8
37.2
87.5
53.9
34.3
88.0
57.0
81.4
37.1
34.1
57.2
Table 19:
Belle results of using Chinese (denoted by "-zh") and English (denoted by "-en") based LLaMA and Bloom.
Med.
Psyc.
Law
Edu.
Avg.
LLaMA-en
3,26
3.20
1.49
3.33
2.82
LLaMA-zh
8.12
8.70
1.79
6.45
6.27
Bloom-en
27.70
17.35
17.59
13.63
19.07
Bloom-zh
20.93
13.45
15.32
14.47
16.04
Table 20: MMCU results of using Chinese (denoted by "-zh") and English (denoted by "-en") based on LLaMA and Bloom.
Open
Brain
Close
Code
Clf.
Math
Gen.
Sum.
Rewrite
Extract
Avg.
QA
Storm
QA
Alpaca-GPT4
53.7
35.0
88.2
50.5
36.8
89.5
60.3
82.9
40.0
44.1
58.1
+human-vulue alignment
55.8
36.0
88.7
51.8
36.4
90.3
52.8
83.9
43.5
33.2
57.2
Table 21:
Belle results of Bloom instruction-tuned with and without human-value alignment dataset.
Med.
Psyc.
Law
Edu.
Avg.
Alpaca-GPT4
27.70
17.35
17.59
13.63
19.07
+human-value alignment
22.88
14.85
9.45
15.46
15.66
Table 22: MMCU results Bloom instruction-tunel with and without human-value alignment dataset.

---

Class
Closed QA of Belle-eval
Instruction
214 224
23..fL
2355.
Answer
#
8
M22 3/21
915.9155411--0096-243333333
#y 44
32 fi
*
Bloom
Response:
i8 11.41 214
20.7 #2 2344999
Rating of ChatGPT: #5 0.59
# fy #Y 1 M MA
#
Bloomz
Response:
T2..f..
Rating of ChatGPT: 19 3: 03
RR #
$ if 5 / H 12 3 
63 8 #
Bloomz-nt mt
Response:
3 7
Rating of ChatGPT: 55-5:: 09
An 9 to 16
Table 23: Comparison of the responses and corresponding scores of Bloom, Bloomz and Bloomz-mt under the same instruction.
Class
Psychology of MMCU
Question
Option
A: H de # AnC Mnn: sb #8
Answer
B
I.L
1. Aln2. Bln3. Cln4. Din 5.A1n6. Bln6. C...
Bloom
1. Aln2. Bln3. Cln4. D'n 5.A'm6. Bln7. C...
moss-buse
Please read the following choices and write an answer. Then, explain why you are the correct choice. AnlnA stability is personality characteristic of person's 's unique persomality that allows him her maintain u particular pattern of thinking. behaving, and behaving. \nB individuality is persomality characteristic of a person's 's unique personality that allows him or her to maintain a particular pattern of thinking, behaving and responding emotions. \nC Integration is personality characteristic of a person's 's unique personality that allows him or her to maintain particular pattern of thinking, behaving and responding to emotions. \nD functionality is personality characteristic of person's 's unique characteristic that allows him or her to maintain particular pattern of thinking, behaving, and responding to emotions.
Vicuna
Bloomz
B
Bloomz-mt
B
moss-sft
C
ChatGLM
#
A R
E
Table 24: Qualitative examples of different LLM responses to the same question of MMCU.

---

Datasets
Examples
Alpaca- GPT4
instruction:
input: output: #. +2 $1 (9.8m/s)) 1= k |1
(N)
Belle
instruction: imput: 7
output: 3agn
B
# B,04444463cK        3     3   44
X
ShareGPT-zh
input:
moss-sft-data
instruction: 33
input:
6 S
instinwild
instruction:
3/ 3
input:
output:
1
$ + # THH ,U HHHHHHH   HHHHHHHHHA3AAA3 SH
0 la (10) S5
firefly
#n
input: output: 1
Table 25: Representative samples of different instruction datasets discussed in the experiments.

---

Datasets
Examples
HC3
instruction: . 2
imput:
7
- , # Darkroom Cookbook BLS
35 i5, A B3
xP3/zh
E 2 25 1 VnSentence i i2 #5 F (e At Eugenio Scalfari by 1 &. 1
instruction: Sentence 1: 4 15
fi Ne Eugenio
VnQuestinn: Does Sentence paraphrase Sentence 2? Yes or
No?
input:
output: Yes
COIG-emee
instruction:
#5? A
imput:
31 -
#L
#6 2 S
$
COIG-trans
instruction:
Tonga,/AAAANNNSSSSSS332332433430 $
$ 2 35
0,
COIG-exan
instruction: ()
$ A+
#
PCLUE
instruction:
66 TT #5 \
input:
output:
TH
CoT' Chinese
instruction:
- 35,.-.3
input:
ト 1 $ 14
Table 26:
Representative samples of different instruction datasets discussed in the experiments. The last row is CoT dataset sample, which has problem-sol ideas and explanations.

---

Class
Responses the Following Instruction
Instruction
*
Belle
$
ShareGPT-zh
moss-sft-dtta
it
instinwild
カI$..
firefly
B
HC3
. 43333331111111333333333333333333
xP3/zh
COIG-oeme
k 2 # XrT
COIG-trans
COIG-exam
1. 200
PCLUE
18 H
Table 27: Qualitative examples of the responses from models instruction-tune on different instruction datasets.
Class
Responses the Following Instruction
Instruction
Ilama-buse
1. 4
122 SU W.
A SGN # 25. 34 Begn
llama-sft
1 bba Ab Seakmnahheett     e              k
- 
Ilama-voc
A 28 PinPoss
AirPods M5 battery 65
lifespan hattery #5
llama-voc-pre
3 33
22
R #
llama-voc-pre-plus
*. 23
Table 28: Qualitative examples of the responses of LLaMA and its expanded vocabulary versions to the same
instruction.

---

Class
Examples
$ 3
instruction
response
M AE
_ 3
M AAAAEHAAKKASSSNSSSSSSSSSSAHA AA
instruction
response
instruction
H433 # #
response
*
Table 29: Examples of the human-vale value alignment dataset built by COIG.
D Experimental details
determine the predicted answer by matching the op- tion content that appears in the response. 2) If the model generates the option number, considering that sometimes the model will analyze the content of each option: all option numbers appear in the response (with different occurrences), we will re- move the option numbers that only appear once. Otherwise, we direct use all the option numbers that appears as their final answer.
D.1 Experimental Settings.
Our code is modified from library transformers and peft. We will release the codes publicly.
During the training phase, we train the models with 8 A A100-80G. A good set of hyperparameters was discovered through experiments: We train mod- els with a warempp learning rate of 5e-4 and batch-size of 122 for epoch (for datasets with less than 100K samples) or epoch (for datasets with over 10K samples). Each sample is truncated to 512 tokens. To improve training efficiency, we load models with 8-bit quantization, except for Chat- GLM. We divide 2000 samples into a validation set to observe changes in losses and determine the final checkpoint used.
D.2 Prompts Design.
For the prompts English, we direct use the same prompt as Alpaca:
"Below is an instruction that describes a task. Write a response that appropriately completes the request. \n \n### Instruction: in m/instruction) \n### input: /i innutnnt//// ni n#### Response:"
During the inference phase, we set the max length of the generations to 512, and set temper- ature=1.0, top_p=0.9, top top_k=40, num bum_beems=10, no mo_repeat_ngram o, We set float16 precision for inference.
For the prompts in Chinese, we design it as fol-
lows:
"I
-
E 6/ _ X,
$ $ #
2
N Vnin ### # /n/instruction/ in n####
During the evaluation phase, use ChatGPT with GPT-3.5-turbo-O33. engine score the mod- els' generation for Belle-eval. When extracting answers for models' generation on MMCU, we fol- low the following steps: 1) If the model does not generate an option number (i.e., A, B, C, D), we
$ A
cimput/lyninttieee [)] 2
For the inference MMCU, we revise the ori- gin prompt as:
"Below is an instruction that describes a task. Write a response that appropriately completes the request. \n nHf#H Instruction: |n 18 i A

---

F Th IE A
+ An'n ### input: /n/input/ JE 15
Σ
\n'nH# Response:"
D.3 Implementation Details of LLM.
Our LLM is trained from Bloom with LoRA. We select a combination of datasets with significant gains on Belle or MMCU, including 10 datasets: Alpaca-GPT4, Belle. ShareGPT-zh, moss-sft-data, installwild, firefly, COIG-trans, PCLUE, and CoT data. To balance the capabilities of our model, only select 1/3 of moss-sft-data and 1/5 of firefly and PCLUE. The model perform the best with 1.3 epoch instruction-tuninn we add a sentence "M % & iu if im # ("An- swer as detailed and specific as possible" in Chi- nese) at the end of the orginal prompts.
For the specific prompt,


Reviews:
[{"review_id": "cFCmAN3a9M", "reviewer": "Reviewer_L1A9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the agreement, thorough review and constructive comments.  \n\n+  **Q1**: the \"best\" combination of factors is only locally optimal.\"\n> We fully understand the concern of \"locally optimal\", since we did not traverse all the three-factor combinations (LLMs, data, parameter-efficient methods), but instead controlled two variables to traverse all possibilities of the current variable. We will respond to this consideration from two aspects:\n> 1.  If we simultaneously traverse all combinations of the three factors, it will result in hundreds of combinations, which requires high computational costs, and we do not think that there will be more valuable findings besides searching the \"best\" combination.\n> 2. The main focus of this paper is to conduct a thorough analysis of each factor, such as the impact of different instruction construction methods, and the Chinese capabilities of different LLMs. These findings may have persistent reference value for follow-ups to customize their own Chinese LLMs, which is more urgently needed for Open-source community than simply searching a \"best\" combination.\n\n+ **Q2**: \"The more similar the data distribution between instruction datasets and the benchmark, the higher performance the model will obtain.\" & Reasons to select the two benchmark. \n> We fully agree with this concern, but in fact, any field or task will face the same problem of \"training on datasets closer to the distribution of the test set can achieve better results\".  It is precisely with this in mind that our paper chose the two benchmarks, which have significant distribution differences to avoid the impact of data distribution.   \n>  Specifically, most of the former are general instructions that require LLM to generate open and **longer** replies. The latter are mostly instructions in the form of multiple choice questions, requiring LLMs output (**shorter**) options. The two represent the capabilities of **open** generation and **closed** set selection, and it is difficult for any dataset to simultaneously approximate these two vastly different benchmarks. Therefore, we believe that our evaluation is persuasive.  \n>  In addition, more experiments in this paper are not affected by this concern, such as comparing models, vocabulary, prompt languages, etc., **under the same instruction dataset**. These experiments also have high reference value and broader applicable insights.\n\n+ **Q3**: Some typos and presentation improvements: \n> The typos have be properly addressed as we reply. Besides, we have optimized the Fig. 1 and tables to make them more reader friendly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is the first to systematically study on instruction-tuning in Chinese. The authors have conducted intensive experiments to find the best combination of LLM bases, parameter-efficient methods and instruction datasets, as well as other factors. I think these findings are very valuable for researchers to develop better Chinese LLMs.", "reasons_to_reject": "1. Because of the variables controlling method used in the paper, the \"best\" combination of factors is only locally optimal. And the results depend largely on the initially selected parameter-efficient methods and instruction datasets.\n2. The reasons that selecting Belle-eval and MMCU as two benchmarks are not very convincing. And one point that is very confusing to me is that the choice of instruction datasets may have a great impact on the evaluation results. That is, the more similar the data distribution between instruction datasets and the benchmark, the higher performance the model will obtain.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Chinese LLMs are still in the early stage. In order to provide valuable findings for effectively customizing LLMs to better respond to Chinese instructions, this work makes an in-depth empirical study of instruction-tuning LLMs in Chinese from three most important aspects - LLM bases, parameter-efficient methods, and instruction data types, as well as other factors like COT and human-value alignment. Moreover, this paper will release a powerful Chinese LLMs comparable to ChatGLM.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. line 025:  \"a powerful Chinese LLMs\" should be \" a powerful Chinese LLM\"\n2. line 074: \"huamn-ChatGPT\" should be \"human-ChatGPT\"\n3. Figure 1 is a little difficult to understand.\n4. line 217-223: point 3) is omitted.\n5. line 235: \"instruction-uning\" shoud be \"instruction-tuning\"\n6. The best results in each column in Table 6, 7, and 8 should be better in bold."}}, {"review_id": "7qxCy3qGap", "reviewer": "Reviewer_3VDv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "+ **Q1**: \"The knowledge that remains once these models and datasets become obsolete may be limited. \" & Chapter2's insights?\n> First of all, we fully agree with the suggestion of 'developing more attention to discussions that can yield broader insights applicable'. This is also the goal of our Chapter 3. However, we have different opinions on 'these discussions (in our Chapter 2) on existing datasets and models results in few generalizable insights', as follows:  \n> 1. The success of ChatGPT is promoting the gradual transformation of solutions for various NLP tasks in the (Chinese) industry towards LLMs. However, the vast majority of companies are unable to train their own pre-trained models and instead customize their own models based on models and data from the open source community. At this time of transformation, a thorough empirical study to systematically explore the available models and data  is meaningful and highly anticipated by the open source community. Their need is not for 'novelty', but for an \"guide book\" that can serve as a useful reference for them to help them know what choices are available and provide guidance on these (models, data, etc.) choices when fine-tuning LLM with Chinese instructions. As you mentioned , '... would be valuable for readers who plan to utilize these models and datasets.', therefore, our focus is not only pursuing the \"insights\" and \"novelties\".  \n>  2. Even though these datasets and models will eventually become obsolete, we believe that our work is still meaningful. Because our analysis focuses more on the technology behind them rather than the data or models themselves, e.g., analyzing the reasons of poor performance of Moss,  comparing the instruction data of self-instruct and nlp-dataset collections, and etc.     \n>  3. The feasibility of verifying some existing works in Chinese scenarios is still \"new insight\". For example, the role of CoT datasets in reasoning  tasks in Chinese scenarios, although there is no technological innovation, has yet to be confirmed. After all, there are significant differences in the basic expression logic between Chinese and English.   \n\n+ **Q2**: \"While the overall content and results are readily comprehensible, there are many deficiencies in the description of existing models and datasets\". \n> As you know, our paper is dominated by a large number of experiments and analysis, resulting in a serious shortage of space. We repeatedly streamline and compress space as much as possible without losing information, which leads into the situation that \"much information is only written in table captions\". We will further improve the presentation of the paper in the next version.  \n\n+ **Q3**: \"All evaluations are conducted with ChatGPT\" ? \n> We believe this is a serious misunderstanding and factual error, which may  affect your assessment of our soundness score. \n> 1. Firstly, **not all evaluations in our paper are conducted with ChatGPT**. We only use ChatGPT to evaluate Belle-eval, but for multiple-choice questions (i.e., benchmark  MMCU), scores can be directly computed automatically.   \n> 2. Secondly, two benchmarks used in our paper focuses on open generation and closed multi-choice selection, respectively. The former is indeed more suitable for human evaluation. However, it is hard to conduct human evaluation for the hundreds-of-group experiments and millions of samples in our paper. Although ChatGPT has certain biases, it is still the most reliable open generation evaluation method currently.  We think that compared to human evaluation that can only conducted to a small number of samples, ChatGPT, which can be extended to more test samples, may provide a more comprehensive evaluation.  \n> 3.  Finally, a very common English benchmark for instruct-tuning LLMs is MMLU. Inspired by this, in the Chinese scenario, we naturally chose an MMCU that is close to MMLU's positioning and construction procedure. We believe this benchmark can also evaluate LLM capabilities reliably. Therefore, based on the comprehensive analysis of the results of both Belle-eval and MMCU, we believe that our assessment is credible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides abundant quantitative comparisons from numerous experimental perspectives, which would be valuable for readers who plan to utilize these models and datasets.\n- Although there are minor typos, the content is clearly written and can be easily understood by the reader.\n- The application of English LLMs to Chinese provides insights that could be potentially applicable to multiple languages, likely attracting a wide range of readers.", "reasons_to_reject": "- The majority of the paper is a listing of evaluations using existing datasets and models, with little in the way of new insights. While each experiment includes a brief discussion, many statements lack substantial evidence, resulting in few generalizable insights. Consequently, the knowledge that remains once these models and datasets become obsolete may be limited. The authors should devote more attention to discussions that can yield insights applicable beyond this study, such as the application of English LLMs to Chinese discussed in Chapter 3. This could help to generate broader, more universally applicable insights.\n- While the overall content and results are readily comprehensible, there are many deficiencies in the description of existing models and datasets. Much information is only written in table captions when it should be comprehensible from the text alone. Similarly, it seems inappropriate to have statements that rely on results from the appendix.\n- All evaluations are conducted with ChatGPT, without any human evaluation. Although the ChatGPT evaluation is somewhat reliable, it has biases, such as assigning low scores to short sentences, as the authors themselves noted. Therefore, there are doubts about whether the results obtained from these experiments should be taken at face value.", "questions_to_authors": "", "paper_topic_and_main_contributions": "- This paper compares and contrasts a range of open language learning models (LLMs), datasets, and learning methods used for instruction-tuning large language models (LLMs) in Chinese, through extensive experimentation.\n- It also explores the effects of additional training when applying LLMs, primarily trained in English, to the Chinese language.\n- The authors plan to release a Chinese LLM with performance comparable to ChatGLM.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5LiWXQKtM0", "reviewer": "Reviewer_CCaJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "+ Thanks for the agreement and constructive comments. \n+ Given that you did not provide any reasons for rejecting this paper, but only scored 3 points for \"excitement\",  we respectfully speculate that you are concerned about our \"novelty\" . However, we call for an inclusive mindset towards the contribution and novelty of our work, which is different from regular papers that focus on individual techniques. We clarify our contribution from three points:\n>  1. For the (Chinese) industry, at this time of transition from traditional NLP solutions to LLMs, a thorough empirical study to systematically explore the available models, data and efficient methods  is highly anticipated.  When customizing their own LLM models, their need is not for \"novelty\", but for a \"guide book\" (findings in **Chapter 2**) that can serve as a useful reference for them to help them know what choices are available in open source community and provide guidance on these (models, data, etc.) choices.  Therefore, our focus is not only pursuing the \"insights\" and \"novelties\". Nevertheless, we try our best to analyze the technology behind each choice for detailed reasons.  \n>  2. The feasibility of verifying some existing works in Chinese scenarios is still \"new insight\".  For example, the role of CoT datasets in reasoning  tasks in Chinese scenarios, although there is no technological innovation, has yet to be confirmed. After all, there are significant differences in the basic logic of expression between Chinese and English.  These findings ( in **Chapter 3**) could help generate broader, more universally applicable insights.\n>  3. From the technical perspective, we customized an LLM  that is comparable to chatglm, with only a few parameters trainable. This can help follow-ups customize strong Chinese LLMs in resource-constrained situations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new LLM has been created and will be made publicly available. The research here is on the Chinese language.\n\nThe paper is well-written and understandable.", "reasons_to_reject": "I cannot think of any.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The present research deals with the use and adaptation of LLMs for Chinese language. Different LLMs are evaluated, as well as methods for improvement.\n\nThe evaluation is based on two evaluation benchmarks with NLP tasks, but also programming and mathematics tasks. The performance of different LLMs with Chinese language is compared on these benchmarks. \n\nEspecially the LLM \"Bloom\" can benefit from \"instruction tuning\". The LoRA method leads to the best results. Instruction datasets with Chinese instructions are used for training. Other factors that are investigated: Vocabulary (especially in the tokenizer), language of the prompt (English or Chinese), human values to avoid generating toxic content.\n\nThe resulting LLM will be made publicly available.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In the bibliography, it is not useful to write all names of all contributors, if it is a very long list."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts extensive experiments to compare a range of LLMs, datasets, and learning methods used for instruction-tuning LLMs in Chinese and investigates other factors including vocabulary and prompt language. The reviewers in general think the paper is well-written and is valuable for practitioners who plan to utilize these models and datasets, but feel less excited about it. I'd recommend accepting it into Findings.
Decision: Accept-Findings