Chinese Tiny LLM: Pretraining a Chinese-Centered Large Language Model
Authors: Xeron Du
Zhouliang Yu
Songyang Gao
Ding Pan
Cheng Yuyang
Ziyang Ma
Ruibin Yuan
Xingwei Qu
Jiaheng Liu
Tianyu Zheng
Xinchen Luo
Guorui Zhou
Wenhu Chen
Ge Zhang
Venue: COLM (2024)
Keywords: Chinese LLM, pretrain, alignment
TL;DR: CT-LLM, a groundbreaking 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in the development of LLMs.

Abstract: In this study, we introduce $\textbf{CT-LLM}$, a groundbreaking 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in the development of LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens and 400 billion English tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques including supervised fine-tuning (SFT) and direct preference optimization (DPO). Demonstrating remarkable performance on the ChineseHardCase Benchmark, CT-LLM not only excels in Chinese language tasks but also showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing full-process of CT-LLM, we aim to foster further exploration and innovation within both the academic and industrial spheres, paving the way for more inclusive and versatile language models in the future.

Published as a conference paper at COLM 2024
cL
Chinese Tiny LLM:
Pretraining Chinese-Centric Large Language Model
Xinrun Du Zhouliang Yu Songyang G Ding Pan?, Yuyang Cheng', Ziyang Ma², Ruibin Yuan Xingwei Qu', Jiaheng Liu 19 Tianyu Zheng', Xinchen Luo?, Guorui Zhou', Wenhu Chen 28 Ge Zhangl 128+t
'Mulimodaa Art Projection Research Community, Uniiveriity of Waterloo, PPeking University, 'Shanghai Jiaotong University, FFudan University, 'HKUST, 7 Kuaishou.Ine, Vector Institute, " Nanjjnng University
duxinrun20000gmail.comm zyubn@connect.ust.u gaosongyang@ojlab.orgg org.cn, gezhang@umich.edd
https://chhines--tin--11 github.oo/
Abstract
In this study, we introduce Chinese Tiny LLM (CT-LLM), a 2B large lan- guage model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incor- porating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's s ex- ceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remark- able performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size CT-LLM, we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.
1 Introduction
In the burgeoning field of linguistic intelligence, large language models (LLMs) emerge cornerstone of natural language processing (NLP), demonstrating remarkable capabilities in understanding and generating human language. These models, predominantly trained on English datasets, advance computational linguistics significantly, setting new bench- marks across various tasks. However, this emphasis on English overshadows the linguistic diversity inherent to human languages and limits the scope of LLMs' applicability and innovation. The development of LLMs grounded in non-English languages, particularly those that incorporate the complexities and nuances of such languages from inception, remains relatively uncharted domain.
This study introduces the Chinese Tiny LLM (CT-LLM), a pioneering endeavor to redefine the landscape of LLMSs by shifting towards prioritizing the Chinese language. CT-LLM,
Equal Technical Contributions. Corresponding Authors.
1

---

Published as a conference paper at COLM 2024
with its 2 billion parameters, diverges from traditional approaches by being meticulously pre-trained on a comprehensive corpus comprising 1,200 billion tokens. This corpus, distinct in its composition, includes extensive collection of 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. Our careful data processing procedures offer the Massive Appropriate Pretraining Chinese Corpus (MAP-CC), enhancing the quality of Chinese web corpora and setting new standard for dataset preparation in the field. The strategic inclusion of diverse and substantial amount of Chinese textual data enables CT- LLM to achieve exceptional proficiency in processing and understanding Chinese, setting new precedent for LLM capabilities.
Our approach further refines the model's competencies through supervised fine-tuning(SFT). The SFT not only bolsters the model's adeptness in Chinese language tasks but also enhances its versatility in comprehending and generating English text, showcasing its multi-lingual prowess. We also utilize preference optimization techniques to align CT-LLM with human preferences, to enhance its harmlessness and helpfulness. Furthermore, Chinese Hard Case Benchmark (CHC-Bench) with multidisciplinary is established to measure instruction understanding and following ability in Chinese, where CT-LLM demonstrates remark- able performance. By challenging the prevailing norms of training LLMs primarily on English corpora, CT-LLM expands the horizons of language model training, offering fresh perspectives on the potentialities of non- non-English- English--enttricc LLMs.
Central to our research is the open-sourcing of the entire training process for CT-LLM, including the meticulous data processing procedures undertaken to curate the Massive Appropriate Preetrining Chinese Corpus (MAP-CC) and the establishment of the multidisci- plinary Chinese Hard Case Benchmark (CHC-Bench). Through the dissemination of our methodologies and findings, we aim foster more inclusive and diverse landscape for future LLM developments, encouraging the exploration of models that better reflect the vast array of human languages and cultures. Our contributions are threefold:
MAP-CC An open-source Chinese pretraining dataset with scale of 800 billion tokens, along with a detailed suite of procedures for cleaning Chinese web corpora, offering the NLP community high-quality Chinese pretraining data and an effective methodology for data preparation.
CHC-Bench A well-chosen multidisciplinary Chinese hard cases instruction understanding and following benchmark.
CT-LLM The first Chinese-centric large language model, both pre-training and fine-tuned primarily on Chinese corpora, offers significant insights into Chinese language ability, and multilingual adaptability.
2 Related Works
2.1 LLM with Chinese Language Ability
In the field of LLMs, the advancement of technologies has catalyzed the development of an array of open-source models exhibiting remarkable linguistic capabilities. Notably, models such as LLaMA (Touvron et al., 2023a:b), Phi (Li et al., 2023b; Gunasekar et al., 2023), Mistral (Jiang et al., 2023), and Gemma (Team et al., 2024) have emerged as frontrunners, underscoring the technological strides made in this arena. Amidst globalized context, there's a rising demand for models proficient in bilingual or multilingual functionalities, particularly those accommodating the vast spectrum of Chinese language applications. This demand stems from the desire for localized solutions and the necessity to bridge linguistic divides worldwide. To address this need, several strategies have been employed to enhance the multilingual capabilities of LLMs, with significant emphasis incorporating higher proportion of Chinese tokens during the pretraining phase or employing techniques such as supervised fine-tuning (SFT) to activate Chinese language functionalities (Zeng et al., 2023; Bai et al., 2023; Yang et al., 2023; Team, 2023; Young et al., 2024; Bi et al., 2024). An early example in this endeavor is ChatGLM (Zeng et al., 2023), which pioneered the use of
2

---

Published as a conference paper at COLM 2024
equal distribution of Chinese and English tokens during its pretraining phase, culminating in a proficient bilingual model. Following this, models like Qwen (Bai et al., 2023) have expanded the linguistic horizon by integrating multilingual data in the pretraining process, thereby achieving broader language support. Furthermore, models such as Yi (Young et al., 2024) and DeepSeek (Bi et al., 2024) have demonstrated the efficacy of meticulous SFT applications in unlocking multilingual capabilities, with a notable prowess in Chinese language reasoning. However, despite these advancements, the existence of Chinese- centric LLM that primarily leverages Chinese as its primary language remains uncertain. This gap highlights a critical area of interest for developing localized, open-source Chinese models, underscoring the significance of tailored approaches in the evolution of language technologies.
2.2 Chinese Corpora for Pretraining and Alignment
Pretraining data is essential in developing language models, providing the base for these models to learn and comprehend human languages. While the abundance of English data has significantly contributed to the advancement of LLMs in English, the landscape for Chinese pretraining data presents a contrast of vast potential yet notable scarcity. Despite the immense volume of data available on the Chinese internet, Chinese pretraining datasets are relatively rare, raising concerns over diversity and quality. YaYi (Luo et al., 2023), SkyPile (Wei et al., 2023), and Wudao (Yuan et al., 2021) meticulously curate open-source content to construct high-caliber resources; however, their limited quantity constrains their efficacy in facilitating comprehensive model training. Conversely, Wudao boasts extensive Chinese training resources, albeit afflicted by significant variability in data quality and disregard for line breaks in formatting, thereby posing challenges for practical imple- mentation. WhineseWeb Text strikes superior balance between data quality and quantity, making it preferable for current pre-training endeavors. Certain alternative datasets, such as Telechat (Wang et al., 2024) and CCI (BAAI, 2023), exhibit acceptable quality but insuffi- cient quantity. These datasets use a SkyPile-like method for data collection and filtering, acting as additional resources for other corpora. Furthermore, although COIG series (Zhang et 2023; Zheng et al., 2024b) is categorized as SFT data, it holds promise for large-scale pre-training applications due to its vast volume. Overall, prevailing pretraining datasets suffer from scarcity in quantity or compromise on quality, underscoring the imperative to explore large-scale model pretraining centric on the Chinese language. Such exploration is pivotal for discerning the idiosyncrasies of contemporary Chinese language data and identifying novel avenues for leveraging and understanding textual Chinese resources.
2.3 Emergence of Multilingual Capacity
The prevailing paradigm in developing LLMs has largely favored English-centric pretrain- ing methodologies. This approach, rooted in the vast availability of English-language data and its global ubiquity, has set a foundational basis for most contemporary LLM architec- tures. Subsequently, strategies such as continuing pretraining, supervised fine-tuning, and instruction fine-tuning (IFT) have been employed to extend these models' linguistic reach, enabling the activation of multilingual capacities (Zeng et al., 2023; Bai et al., 2023; Yang et al., 2023; Team, 2023; Young et al., 2024; Bi et al., 2024). These methodologies have proven effective, showcasing the adaptability of LLMs to accommodate linguistic diversity beyond their initial English-centric training, with representative examples Chinese- Yao, 2024) and
(Cui & (Cui & Yao, 2024). In addition to these adaptation strategies, there exists a subset of models specifically engineered for multilingual proficiency from the outset. Models like BLOOM (Le Scao et al., 2022) and Aya (Üstün et al., 2024) exemplify this approach, incorporating multitude of languages throughout both their pretraining and fine-tuning phases. Despite these efforts to integrate linguistic diversity, English invariably remains the dominant language within these models (Zhao et al., 2024).
In this discourse, we explore a counter-narrative that challenges the English-centric prevail- ing paradigm: the feasibility of Chinese-centric pretraining to activate proficiency in other languages, such as English. By considering Chinese the primary language for pretraining, we investigate whether such a model can effectively acquire and demonstrate capabilities in additional languages. The success of a Chinese-centric approach could significantly
3

---

Published as a conference paper at COLM 2024
democratize language technologies, providing insights into creating inclusive models that reflect global linguistic diversity.
Pretraining
3.1 Data
Previous research (Hoffmann et al., 2022) has established that the magnitude of the dataset significantly influences the performance of large language models. Simultaneously, the diversity and comprehensiveness of the dataset are crucial for training a large language model for general domain. Guided by the aforementioned principles and our emphasis on utilizing Chinese corpora for model training, we have developed a dataset encompassing 1,254.68 billion tokens. This dataset integrates Chinese, English, and code data, consisting of 840.48 billion Chinese tokens, 314.88 billion English tokens, and 99.3 billion code tokens. The dataset aggregates content from diverse sources, such as web documents from Common Crawl, scholarly articles, encyclopedias, and books. The precise distribution is detailed in the Figure.1. Our dataset contains around 110 billion duplicate tokens, mostly in English. Despite being duplicates, they are high quality and were intentionally used twice in training.
Heuristic Rules We designed heuristic rules to conduct data filtering, which removes data of low quality. These rules represent an integrated framework of filtering strate- gies, inspired by methodologies derived from several datasets and models, notably RefinedWeb (Penedo et al., 2023) and CC- Net (Wenzek et al., 2020), along with some rules that are applied while training other language models, such as Gopher (Rae al., 2022) and T5 (Raffel et al., 2020). We also developed a set of rules tailored to address characteristics inherent to our dataset.
zh-books
zh-oc
zh-others
zh papere
en
zh-encyc.
code
7.91%
4.77%
0.2%
25.1%
56.75%
2.82% 2.45%
is worth mentioning that existing rules mainly aim at English data filtering. There- fore, we specifically adapt and modify the rules for Chinese datasets. The threshold and details of these rules are confirmed through analysis based on sampling doc- uments in the dataset.
Figure 1: Pretraining data distribution, where "zh" represents Chinese data, "en" represents English "cta,"cc" stands for Common Crawl, including publicly available web documents, etc., and 'eccyc'' refers to the encyclopedia.
Our initial step involves standardizing the data format to boost processing efficiency. Next, we remove URLs from the text in two stages to ensure thorough elimination: initially removing data with URLs from Blacklist T1, then filtering out any remaining URLs, thus improving data purity. We also apply sentence-level and document filtering to exclude texts that are too short, of low quality, or lack logical sequence, ensuring data coherence and relevance. Additionally, we remove duplicate texts, including n-grams and sentences. Specifically, the zh-ce dataset undergoes filtering due to its unique characteristics as web- crawled dataset, whereas other datasets do not. Detailed rules are listed as Appendix A.
Deduplication After implementing a filtration process, we've developed comprehen- sive deduplication pipeline. This pipeline includes document-level exact deduplication, document-level Minhash deduplication, and leeee--lcument-leeelll similar line deduplica- tion, effectively identifying and removing duplicate content within documents. For exact deduplication, to reduce memory pressure we utilize Bloom filter to approximate with false positive rate set at 0.001. In the case of Minhash LSH, the signature is constructed from 128 hash functions and organized into bands and 13 rows for LSH, achieving a Jaccard similarity of 0.8. The intra-document-level similar line deduplication targets removing repetitive lines within a single document. This approach was motivated by our observation that significant portion of web-crawled data contained repetitions of 2 to 3 times within the
4

---

Published as a conference paper at COLM 2024
Heuristic Rules
Exact Deduplication
Minhash Deduplication
Similar Line Deduplication
15%
26%
38%
12%
0.1 nLm123,251 251
Edit distancettaeell lisel)
Judge similar, discard Line2
miniline), Line2)
Not Similar
Discard
DverLop judge:
Line2
62%
Ivartaptiiass Lset-niinn Lines, L-ne2)
Palse
RINRANPIER AE
Line2
Linel
Sptit
Line?
Split single document 1ine, for pair line do the above procedure
ttet:: 7 A D 71
A E FM, ATL 2 7 #0
(Omeeti many des, pertla aften dell Jarte cener Jaice
Above is the data processing flow and deduplication ratios, below is schematic diagram of similar line deduplication.
Figure 2:
same page, and due the process extracting text from HTML, some words might lost, leading to slight variations in duplicates. For this deduplication, we employ edit distance to determine line similarity. The specific criterion that two lines are considered similar their edit distance is less than one-tenth of the length of the shorter line. Furthermore, to expedite this filtering process, we calculate the proportion of character overlap between the lines; if it's less than one-third, the lines deemed dissimilar. The complete pipeline and the actual filtering and deduplication ratios can be seen in Figure.2.
3.2 Model Architecture
Our model's architecture is based on the transformer decoder (Vaswani et al., 2017a). The key parameters that define architecture shown in Table 1, with the models being trained on a substantial context length of 4096 tokens. Beyond the foundational elements, our approach integrates several improvements compared to the original transformer.
Multi-Head Attention Mechanism. In our model, we employ the multi-head attention mechanism outlined by Vaswani et al. (2017b). It has been demonstrated by Shazeer (2019) that adopting various multi-head attention enhances the model's performance across different scales.
RoPE Embeddings (Su et al., 2021). Instead of re- lying absolute positional embeddings, our archi- tecture incorporates rotary positional embeddings at each layer. Furthermore, to minimize the overall model size, embeddings are shared between inputs and outputs.
Parameters
Value
d_model
2,048
Layers
32
Feedforward hidden dims
5504
Num heads
16
SwiGLU Activations (Shazeer, 2020). The standard ReLU non-linearity replaced by the SwiGLU acti- vation function.
Num KV heads
16
Head size
128
Vocab size
125,696
RMSNorm Same to Llama2 model (Touvron et al., 2023b) 7B serious. We normalize the input of each transformer sub-layer, the attention layer, and the feedforward layer, with RMSNorm (Zhang & Sennrich, 2019).
Table 1: Key model parameters.
Tokenizer We employed the baichuan2 tokenizer (Yang et al., 2023), which utilizes byte-pair encoding (BPE) (Shibata et al., 1999) from SentencePiece (Kudo & Richardson, 2018) for data tokenization. The vocabulary size is 125,696. Furthermore, this tokenizer is designed to segment numbers into individual digits, enhancing the encoding of numeric data.
5

---

Published as a conference paper at COLM 2024
4 Supervised Finetuning
For Supervised Fine-Tuning (SFT), we used both Chinese and English data. The Chinese data consisted of the full set from CQIA (Bai et al., 2024) and OL-CC, as well as high-quality data sampled from COIG-PC (Zhang et al., 2023). The English data was sampled from the OpenHermes@Pefereenn
dataset (Huang et al., 2024a). The total amount of Chinese data comprised 105K pairs of instruction data, with English data adjusted to different ratios based on the volume Chinese data. The ratios were 1 1, 2 :1, 4 1, and 8 : 1, along with configurations that included only Chinese data and only English data. Each set of experiments was trained for 3 epochs, with specific experimental results shown in Table 12.
The hyperparameters used for model training are follows: sequence length is 2048, global batch size is 128, and the maximum learning rate is 2e 5. To prevent overfitting, weight decay is applied with value of 0.1, and gradient clipping is enforced with a limit of 1.0.
To extract the high-quality segments from the COIG-PC dataset and OpenHermes ences dataset, we employ perplexity (ppl) as the selection metric. Specifically, we use the Qwen-7B (Bai et al., 2023) model to compute the ppl for samples drawn from the SFT dataset, and we retain only those entries with a reasonable ppl under Qwen-7B.
Learning from Human Preferences
Considering the harmless and helpful objective of LLMs, we leverage DPO (Rafailov et al., 2024) to directly learn human preferences from rankings of response pairs.
Preference Datasets. Our model incorporates blend of publicly accessible datasets and synthetic data from the LLM. The open-source Chinese datasets consist of non- harmful and beneficial sections from coalues_rliff comparison -gp44 _ata zh and oaast_rm zht in LLama-factory (Zheng et al., 2024c), huozi, and zhihu. For English, the dataset in- cludes comparison gp14 dala en from LLama-factory and beavertails (Ji et al., 2024). To construct more high-qualities preference dataset via synthetics approach, we adopt alpaca-gpt4 (Peng et al., 2023) which generates "chosen" responses using GPT-4, we adopt baichuan-6B (Yang et al., 2023) serving weaker model for generating "reject" responses. The dataset comprises 183k Chinese pairs and 46k English pairs in total.
reference model Taft to optimize the objective language model ng. ni is initialized by the model parameters of the Fsft. We set the hyperparameters as follows: 1. The ng is trained on H800, 2. learning rate = =le- 6, 3. batch size = 4, 4. epoch numbers = 2, 5. weight decay = 0.1, 6. warmup ratio 0.03,, 7. B 0.5 to control the deviation from Tifft.
Training Settings. We leverage the SFT version of CT-LLM
Performance. CT-LLM after SFT and DPO is named as CT-LLM-SFT-DDO. The performance of CT-LLM-SFT--PO on general benchmarks e.g. MMLU, COPA is posted at Table 2.
6 Evaluations
6.1 Results of Metrics
Evaluation Datasets and Metrics Our evaluation encompasses comprehensive suite of public benchmarks in both English and Chinese, leveraging an internal evaluation framework designed for robust assessment. These benchmarks include a diverse range of datasets catering multiple disciplines and aspects of language understanding and reasoning, such as MMLU (Hendrycks et al., 2021), C-Eval (Huang et al., 2024b), and CMMLU (Li et al., 2023a). Our evaluation strategy differentiates between datasets requiring selection from multiple choices, where we employ perplexity-based evaluation, and those amenable to generation-base
evaluation, where the model generates free texts from which results are parsed. This split enables strategy that fits each dataset's specific needs, from language modeling to specialized knowledge and code generation. The full details of the evaluation data are provided in Table 8.

---

Published as a conference paper at COLM 2024
Model
COPA
Hellaswag
MMLU
Humaneval
Triviaqa
Lambada
Squad2.0
GSMSk
C-Eval
CMMLU
53.0
55.99
47.06
18.9
31.15
56.39
30.06
15.1
59.38
57.1
LnnyLlama-L.1
54.47
25,89
8.54
Stablelm-Bb-eett
61.0
69:08
45.42
13,85
50.54
20.38
36.44
10.92
31.21
31.48
Gemma-26
64.56
41.84
9:15
46.42
63.38
22.14
31.25
31.11
Phi-2
67.74
57
40.24
41.04
62.7
31.53
32:19
CT-LLM(OUs))
99.0
50.37
37.11
9.15
21.03
56.24
16.87
8.87
36,78
36.4
Table 2: Performance comparison of CT-LLM and other base models of the similar scale on benchmark. The best result are in blue, the second-best results are underline, and the third-best results are in fbox| The evaluation metric employed for 'HumanEval' is 'pass@1' standard maintained consistently throughout the text.
Training Process and Comparative Analysis The training progress reveals a consistent trend of improvement across various datasets, with particular strides seen in language understanding, reasoning, and domain-specific knowledge. Notably, datasets such as HellaSwag, PIQA, and ARC show marked improvements, indicative of enhanced reasoning capabilities. The model shows notable progress in specialized fields such as mathematics (GSM8K and TheoremQA) and science (ARC-c and ARC-e), emphasizing its increasing ability to understand and produce content specific to these domains. The evaluation results of the intermediate checkpoints during our pre- traintriiiiggggg process are shown in Table.4.
Comparing our model's performance on both English and Chinese benchmarks with other models reveals a notably smaller gap in performance across multi-disciplinaryy datasets such as MMLU and CMMLU, as shown in Table 2. While other models exhibit signifi- cant disparities, particularly in language understanding and reasoning benchmarks, our model maintains consistent performance, suggesting a balanced capability across diverse domains. This contrasts with other models that show pronounced variability, such as in the HellaSwag dataset, where our model closely rivals or outperforms alternatives like MiniCPM (min, 2024) and Phi-2, showcasing superior or competitive reasoning abilities. Similarly, in domain-specific evaluations (C-Eval and CMMLU), our model demonstrates commendable performance, outpacing models like TinyLlama-1.1 1.1 and Bloom-1.77 in com- prehending and generating content that requires a nuanced understanding of cultural and domain-specific contexts. This balanced proficiency underscores the model's versatility and adaptability, positioning it as a strong contender in the landscape of Al language models, with capacity for both broad applicability and deep, domain-specific knowledge.
We also compared the performance of our model, which was fine-tuned using a 2:1 ratio of Chinese to English data (SFT), with other models common benchmarks and Chinese benchmarks, as shown in Table.3. We found that our model's capability in Chinese remains particularly strong. The data ratio used for this SFT model is consistent with that of pretraining. We found its overall performance to be the best. The performance of models trained with other ratios can be found in the Appendix.E.2.
Model
COPA
Hellaswag
MMLU
Humaneval
Triviaqu
Lambada
Squad2.0
GSMBk
C-Eval
CMMLU
66.0
65,88
53.87
[5.12
36:23
60.62
10.52
55.8
19:11
51.0
Gemma-2o-) 20-at
600
56.68
37.71
0.0
29.0
35.91
18.46
15.69
32.3
33,07
TinyLlama-11B-hhaal
32.31
12.89
3.72
Blom-117B
0.0
45.36
1.44
Deepeek-code-
51.0
37.0
28.55
43.29
10.55
36.32
28.85
8.79
28.33
27.75
Qwenn.--1.8B-hhal
57.0
36.75
6.71
24:31
45.83
47.25
28.73
36.84
[5.11
Stublelm-sphhy--88
64.0
67.94
46.15
24.39
33.48
37.46
21.19
57.01
29.5
32.11
CT-LIM-SFT(QUs))
60.0
52.93
10.37
22.88
31.93
35.18
19.18
41.45
CTHLLMSST.DPOOOOm)))
61.0
53.38
39.82
7.93
23.64
31.47
31.36
18.5
41.18
42.01
Table 3: Performance of aligned models with scale of around 2B on benchmark.
The best result are in blue, the second-best are underline, and the third-best are in fbox Safety Evaluation We also evaluate the safety score of CT-LLM-SFT-DPO compared with baselines such as SiniCPM-2B-sft-fpp Bloom-1.7B, and Stablelm-zephyr--BB, etc on cvalues responsibility benchmark (Xu et al., 2023). The evaluation consists of two parts: multiple- choice and auestion-answering The multiple-choice part includes 1,712 input examples, each comprising human query and two candidate responses. The evaluated models are
7

---

Published as a conference paper at COLM 2024
Dataset
39.9B
93.3B
306.6B
506.6B
706.6B
906.6B
Final
Hellaswag
33.3
38.72
44.67
46.77
47.81
49.16
50.37
MMLU
26.09
27.11
26.68
29.8
33.47
35.42
37.11
Humaneval
1.83
2.44
4.27
5.49
5.49
6.1
9.15
GSM8k 1.14
2.05
4.93
6.44
6.14
7.88
8.87
C-Eval 22.53
23.07
23.68
26.4
32.39
36.05
36.78
CMMLU 25.24
24.83
25.59
29.84
31.33
32.86
36.4
Table 4: This table show partial cases evaluation results across variety of datasets for models of different train tokens, from 39.9B to 1200B. All the measurement results can be found in the Appendix.E.1
required to select the response they deem superior and compare against the standard answer. The question-answerii used to score the responses of each model. We use the average score as the final performance. The prompts used for auto-evaluation displayed in Appendix D.
section consists of 664 input examples, where GPT-4 is
Model
Cvalues-MC (Acc%) Cvalues- QA (Score)
MiniCPM-2B-sff (min, 2024)
0.851
6.99
Bloom-1.7B (Le Scao et al., 2022)
0.468
1.19
Stablelm-zephyy-33B (Tunstall et al., 2023)
0.790
3.79
TinyLLama-1.1--hhatv11. (Zhang et al., 2024)
0.502
1.48
Gemma-2b-it (Team et al., 2024)
0,705
6.09
Qwen1.5-1.BB-Chat (Bai et al., 2023)
0.551
6.72
CT-LLM-SFT (Ours)
0.699
5.09
CT-LLM-SFT-DPO (Ours)
0.795
5.61
Table 5: Safety evaluation results of our model with other six SLMs. The best results in blue the second-best results are underline,and the third-best results are in fbox
6.2 Cultural Biases Evaluation
The cultural and ideological leanings of pre- trained language models (LMs) can pro- vide significant insights into the inherent biases of their training data, as well as the design objectives of their developers. In Fig- ure 3, we observe the distribution of mod- els across different quadrants, particularly distinguishing those trained predominantly on Chinese data from those trained on more global Western datasets. Our model, CT- LLM, primarily trained on Chinese corpora, and
Structrred
Blom :87
Coemunitt
Indisddual Focoed
Fonred
MBedLLM--B3--o
PH
located in the community-focusse structured quadrant. This indicates that the model emphasizes collective coopera- tion and development, reflecting common themes in Chinese literature and online plat- forms. In contrast, Western models like TinyLlama-1.11 and Bloom-1.7B, which are based on diverse Western сorpora, situated in the individual-focused and structured quadrant. This suggests that these models highlight personal autonomy and individual expression, focusing on diverse and inclusive approaches.
Descoent ooden
Figure 3: Political spectrum positioning of CT- LLM compared to other open-source models. We test the models' orientation with the bench- mark introduced by Feng et al. (2023).
Interestingly, when examining the openness axis, we find that multilingual models such as MindLLM-11B3-hatee and teristics despite being trained across different language corpora. This could due to the globalized nature of the digital information these models exposed to. These findings
exhibit similar cultural charac-
8

---

Published as a conference paper at COLM 2024
consistent with observations that LMs exhibit variety of perspectives and tendencies across different dimensions. Recent web-based texts might reflect a more open stance prevalent in current digital discourse, while those grounded in older corpora might embody more traditional views. To further understand these tendencies, is necessary to conduct qualitative analysis of the LMs' responses to various viewpoints. Such analysis would reveal whether these tendencies consistently appear across different discussions, providing deeper understanding of the presence and nature of any biases in these models.
6.3 Chinese Hard Instructions Understanding and Following Evaluation
We collect the problems from various sources e.g. ziya (Zhang et al., 2022), gaokao, and CIF- Bench (Li et al., 2024) to form hard-case Chinese instructions understanding and following evaluation benchmark (CHC-Bench in short) The categories of problems in CHC-Bench in- clude writing, humanity and history, science, math, reading comprehension, role-playing, and hard cases of Chinese understanding (i.e. Chinese word pronunciation, ancient Chinese language understanding, etc.).
Metrics. Considering the limitations of 2-billion parameter models, evaluation criteria go beyond just the accuracy responses. We additionally consider factors such as useful- ness, relevance, accuracy, depth, creativity, and the level of detail in the model's answers. This comprehensive method allows for a detailed evaluation of the model's response quality. Specifically, We GPT-4 (Achiam et al., 2023) to score responses from tested LLMs in specific problem contexts, with the scoring prompt available in the Appendix. C.2. We translate the score assignment prompt template from Zheng et al. (2024a).
Results. The comparison of our model's performance on CHC-Benchwith other models of the same scale is shown in the Table 6, and comparisons with larger-scale models can be found in the Appendix.E.3. In CHC-Benchone can assess the expertise of models in specific domains. For instance, Deepseek-coder- 1.bbintt.bb, designed for coding tasks, demonstrates its skill with high scores. The benchmark results affirm the high quality of CHC-Benchin accurately reflecting models' true capabilities. Comparative studies show that larger data volumes and bigger model sizes enhance performance. CT-LLM, within the billion parameter range, excels in social understanding and writing, showing strong performance in contexts related to Chinese culture.
Model
Overall
Hard Case
Social
Coding
Writing
Roleplaying
Math
Reading Compr.
Science
Blom--1.7
1.35
Gemma-2b-it
2.04
1.78
1.65
1.30
1.19
2.50
2.09
4.23
1.40
TnyLLama-1.11B- 1B-Chatv---
2.88
1.78
2.20
2.70
1.55
1.70
1.53
3.73
1.60
Deepseek-coder-1.2 3.biissttt
1.60
Stablelm-zephyr--b
3.30
3.16
2.75
5.05
3,03
3.75
1.76
4.77
2.75
Yuan2-2B-hd
3.31
1.76
4.60
2.45
3.36
3.45
3.12
5.47
2,65
Qwenl S-1.8hhhtt
6.57
6.86
8.10
5.80
7.64
7.00
3.91
7.70
5.85
MinCCMM-2-cf--p32
6.95
2.30
8.55
7.05
5.18
5.20
CT-LLM(OUrs)
5.00
3.50
Table 6: Performance of models with a scale of around 2B on CHC-Bench The best results are in blue, the second-best results are underline, and the third-best results are in fbox
7 Conclusion
We develop CT-LLM, large-scale language model tailored for the Chinese language, pre- training it on 800 billion tokens to enhance Chinese language processing and multilingual adaptability. Unlike previous models that rely heavily on English datasets, CT-LLM rep- resents new direction in LLM research by focusing on Chinese, including English and code tokens. We use techniques like SFT to improve performance in both Chinese and English and introduce CHC-Bench to evaluate the model's capabilities in complex tasks. CT-LLM's key contributions include providing high-quality Chinese corpus and CHC- Bench, addressing biases, and advancing thinese-focusee LLMs. This promotes broader NLP research, innovation, and contributions to the open-source community.

---

Published as a conference paper at COLM 2024
References
Minicpm:
Unveiling the potential of end-side large language models, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXii:2303.08774, 20233
BAAL BAAI-CCI: Chinese internet corpus. https://data baa. ac. acc.cn/detaalls/MIICCC. 2023. Accessed: 2024-03-27.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. ArXiv preprint, abs/ (bbs/209916609 2023. URL /ttps://arxiv...o ab3/2399 1162223999........
Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, Ruibin Yuan, Haihong Wu, Hongquan Lin, Wenhao Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Min Yang. Shiwen Ni, and Ge Zhang. Coig-cqia: Quality is all you need for chinese instruction fine-tuning, 2024.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. ArXio preprint, abs/ (bss/240.00254, 2024. URL https://arxiv. org/abs/2401 .29554.
Yiming Cui and Xin Yao. Rethinking 1lm language adaptation: A case study on chinese mixtral. ArXiv preprint, /bb//203.01851,, 2024. URL /nttps://arxiin orggabs/2400 0 01851.
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails political biases leading to unfair nlp models. ArXiv preprint, abs/2305.08288 2023. URL ht //arxiv..vvvv/// org/ /bs// 08283.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. ArXio preprint, 0Dss/236..1644 2023. URL https: //arxiv.org/abs/236. 11644.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
net
3GmQ.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Shengyi Costa Huang, Agustín Piqueres, Kashif Rasul, Philipp Schmid, Daniel Vila, and Lewis Tunstall. Open hermes preferences. https://huggini OpenterrmesPreferennces. 2024a.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: multi-level distildis chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024b.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment dataset. Advances in Neural Information Processing Systems,
of Ilm via a 36, 2024.
10

---

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and is secretly a reward Published as a conference paper at COLM 2024
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ArXiv preprint, abs/ abs/2310.06825, 2023. URL https: //arxiv. httpii.oorr/bb//31111/ atw.aa/....././/
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D186
URL
8ttp://ohooooooooo//--/-0 org/D18-20000 2012.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2022.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. ArXiv preprint, abs/ abs/238 2306.0922,, 2023a. URL https::/arxiv..ot ab3/036.. 092112
Yizhi Li, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, et al. Cif-bench: chinese instruction-followe following benchmark for evaluating the generalizability of large language models. ar Xio preprint arXiv:2402.13109, 2024.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. ArXio preprint, /2399/03355463 2023b. URL https: //arxiv. httgg/bb/23000 05463.
Yin Luo, Qingchao Kong, Nan Xu, Jia Cao, Bao Hao, Baoyu Qu, Bo Chen, Chao Zhu, Chenyang Zhao, Donglei Zhang, et al. Yayi 2: Multilingual open-source large language models. ArXiv preprint, abs abss/2311 22122.2222.222222 2023.
2312. 14862.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023.
Baolin Peng, Chunyuan Li, Pengeheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint ar arXiv:2304.03277, 2023.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2022.
Chelsea Finn. Direct preference optimization: Your language model. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at COLM 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67 2020. URL http:
Noam Shazeer. Fast transformer decoding: One write-head all you need, 2019.
Noam Shazeer. GLU variants improve transformer. ArXiv preprint, abs/2002.052002. 2020.
2020.
URL http:://aaxxiv..ogabb/////ss
05202.
Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. Byte pair encoding: A text compression scheme that accelerates pattern matching. 1999.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding. ArXio preprint, abs/2104.09864, 2021. URL
https://arxiv.ot https:///axxx..or/aaba.o wwwpiv..rgg/b////e/eee 2210/ 09864.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. ArXiv preprint, abs/2403.082939 2024. URL httt:::///xxvv..r/aabb/////// 240./00293.......
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://g http:://ggttuu..ommmmeeeree www.//nnneeen/nnneennll.e wtw.:l-e-ttchhhrttt.oooo 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXio preprint, abs/ abs/2302.13971, 2023a. URL https: https:://axxvvvvrr/// httrrrgaab///3222 /ttp://rxxiv..rraa//22...... 13977 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. ArXie preprint, abs/2307.0922 2023b. URL https://arxiv. www./rgg///////oooooooooooe 09288.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. Xrviv preprint ar arXio:2310.16944, 2023.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xing -Xin Yong, Wei Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. ArXiv preprint, abs/ abs/2402.07827 2024. URL https://arxivv org http://rxxivv.rr////////////// 07827.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017a. URL https: https::///roeeeee httpp//oooeeenn.ceoe www.eeuiiii setrripscoom cc/ ccpppper 2011 //aah// 3f5ee243547dee9) fbd053c1c4a845aa o10ee444554deeefff000011ccaaaaa3 Abstraact. html.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-600S, 2017b. URL https: http://ppooeee....eee newrips.epppp.... wc/paper///eeccceccceecceeeeeeee /201 2017 7/hash/ 3f5ee243547deee9 fbdo53c1 (atp::3cc444445aa------- Abstsaataabtttt.t.t.... html.
12

---

Published as a conference paper at COLM 2024
Zihan Wang, Xinzhang Liu, Shixuan Liu, Yitong Yao, Yuyao Huang, Zhongjiang He, Xuelong Li, Yongxiang Li, Zhonghao Che, Zhaoxi Zhang, et al. Telechat technical report. ArXiv preprint, abs/ 2441003804 2024. URL httpt:ttt///aeee
org/ars/////oooooooo...oogbwebbe 2401 .03804..
Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. Skywork: A more open bilingual foundation model. ArXio preprint, 23300.9941, 2023. URL https://axxiv org/ ab///30.. 19341.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Extracting high quality monolin- gual datasets from web crawl data. In Proceedings of the Twellfh Language Resources and Eval- uation Conference, pp. 4003-4012, Marseille, France, 2020. European Language Resources Association. ISBN 979-10-95544-- URL https: www.o/hoooaa org. 2rrg/2020..o 1rec-1.494.
Guzmán, Armand Joulin, and Edouard Grave. CCNet:
Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghuu Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. Cvalues: Measuring the values of chinese large language models from safety to responsibility, 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. Arxiv preprint, 00bs/2200.1033 2023. URL hnttps://arxet http:://wwwwv.o////.//// org/ h/tg///b///// abs9. 10305.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, ling Chang, et al. Yi: Open foundation models by 01. ai. Arxiv preprint, abs/ abs/2403.04652, 2024. URL //aps: org/ abs/ 04652.
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Al Open, 2:65-68, 2021.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations (ICLR), 2023. URL htttps://operrviiew net/ www./eeenrr/ee.. wwwwoooodddwwoorrreeeeeee
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12360-12371, 2019. URL httptttt//pcceeeeeeeeee http://poceeeenn...e/ nww.ins......... cc/ paper/2019) hash/1e8a1 90 1ww1444735b1-bbtrrreeeeee html.
Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu. Chinese open instruction generalist: A preliminary release, 2023.
Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, and Chongpei Chen. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs 222/22299900 2022.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint ar Xr1::22001.03385, 2024.
Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. Llama beyond english: 2024.
An empirical study on language capability transfer. ArXio preprint, URL https://arxxv... org/ abs/2401. 01 01055.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao S Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lin-as-a-uudge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024a.
13

---

Published as a conference paper at COLM 2024
Tianyu Zheng, Shuyue Guo, Xingwei Qu, Jiawei Guo, Weixu Zhang, Xinrun Du, Chenghua Lin, Wenhao Huang, Wenhu Chen, Jie Fu, et al. Kun: Answer polishment for chinese self-alignment with instruction pack-translation ArXiv preprint, abs/ abs/228 htts/400..6477,, 2024b. 2024b. 06477.
URL
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024c. Association for Computational Linguistics. http::////axxiv wwt.://.gggbbbaaggggooss 139333331333..333.
14

---

Published as a conference paper at COLM 2024
A Details of Heuristic Rules for Chinese Texts
Note
Data Format Unification
Convert full-angle symbols to half-angle
URL Filtering
should contain blacklisted URIs Remove links via regular expression
Blacklists obtained from Blacklists UT..
Sentence-level Filtering
Only retain sentences with a terminal punctuation
Terminal punctuation: (",,",, ??, "..".
Exclude sentences containing "javascript"
Contain at least words Exclude sentences with "lorem ipsum" Exclude sentences with bad words
Word tokenization by jieba.
Words related to pomography, politics, violence, etc.
Document-level Filtering
Number of sentences 1
Characters after normalization [50, 10000]
Mean word length [1.3, 10]
Fraction of nonconsecutive hashtags 5 0.1
Fraction nonconsecutive ellipsis 0.1
Defined as ellipsis: "..',
Fraction full brackets" " [] " 0.1
Fraction of digital words over non-punctuation words & 0.3
Lines ending with "readmore" etc. 0.3
Endings include: "readmore"
Lines starting with bullet point < 0.9
Bullet points:
Fraction of punctuation in words >> 0
Fraction unique words >> 0.1
Entropy of unigram distribution Text quality score > 0.4
Evaluated by fasttext
Duplicates Filtering
Fraction of characters in duplicate word 10-grams 0.60
Fraction of characters in duplicate word 9-grams <- 0.60
Fraction of characters duplicate word 8-grams 0.60
Fraction of characters in duplicate word 7-grams <= 0.60
Fraction of characters
duplicate word 6-grams
0.60)
Fraction of characters duplicate word 5-grams <= 0.60
Fraction of characters lop word 4-grams <= 0.16
Fraction of characters
top word 3-grams 0.18
Fraction of characters in top word 2-grams <= 0.20
Fraction of duplicate sentences 0.30
Fraction of characters in duplicate sentences <= 0.20
Table 7: Details of Heuristic Rules for Chinese Texts
15

---

Published as a conference paper at COLM 2024
B Pretraining Evaluation Datasets
Category
Datasets
Language Understanding and Reasoning
RoolQ, COPA, HellaSwag, RTE, WiC, Winogrande
Question Answering and Knowledge Retrieval
MultiRC, OpenBookQA, ARC (Easy and Challenge),
NaturalQuestions, TriviaQA
Specialized Knowledge and Application
PIQA, Siqa, OBQA, CSQA, Squad2.0
Mathematical and Logical Reasoning
GSM8K, TheoremQA
Code Generation
HumanEval, MBPP
Language Modeling and Miscellaneous
LAMBADA, C-Eval
Multi-suujjet Multiple-choice
MMLU, C-Eval, CMMLU
Table Summary of Evaluation Datasets by Category
C
CHC-Bench Details
The following table illustrates the composition of the CHC-Bench 10.
C.1 Case Study of Hard-Case Problems
In this section, we list some demonstrations of our selected multidissiplinary Chinese hard case instruction understanding and the following problem sets that are used in CHC-Bench. The concrete classifications of the problem categories are listed Table 10.
Why CHC-Bench is hard for LLMs. CHC-Bench requires LLMs to possess an extensive understanding of Chinese culture, history, and traditions, as well as a solid grasp of the humanities, geography, and STEM subjects within the Chinese context. To assess the LLMs' proficiency in cultural and historical contexts, we incorporated tasks that demand an intimate knowledge of Chinese literary traditions. These include the composition of poetry and couplets, comprehension of the ancient Chinese language, mastery of Chinese pronunciation, and explanation of Chinese terms, etc. Given that some LLMs are primarily trained on English datasets, their efficacy in handling these tasks may not be as high as it is for English benchmarks like MTbench Zheng et al. (2024a). For instance, models and Bloom-1.77, which have limited
such as TinyLlama-1.1B-- training data in Chinese, score below 3.00 across all categories of problems related the understanding of Chinese culture and language. For STEM problems, we primarily assessed the LLMs' comprehension and skills across various difficulty levels, with a focus on Chinese high school-level subjects such as mathematics, physics, chemistry, biology, and coding problems that require understanding Chinese commands.
Deepseek-coder-1...
Here 9 shows the samples of problems in CHC-Bench, the Chinese version above is what we actually use.
16

---

Published as a conference paper at COLM 2024
Type
Sub-Type
Query in Chinese
Query in English
Writing
Poetry and couplet
Compose a couplet based on the solar term of the summer solstice.
Math
Math(Gaokao)
+ de UR,
Problem: In a certain region, air quality monitoring data shows that the prob- ability having good quality on any given day is 0.75, and the probabil- ity of having good air quality for two consecutive days
HE 0.75,
E P
Ca
M
Jyttt HL, Q
3B 4
t > (A)0.8
(B)0.75 (C)6.6 (D)0.45
that the air quality is good on certain day, the probability that will also be good on the following day
Options: (A) 0.8 (B) 0.75 (C) 0.6 (D) 0.45
Science
Chemistry(aokkoo
F BUR
Question: Which of the following disin- fectants has an active ingredient that is a
(A) Ae (B:
(C Hi
Options: (A) Potassium permanganate solution (B) Peroxyacetic acid solution (C) Hydrogen peroxide (D) Medical cohol
Role play
Tang Sanzang
Character: Tang Sanzang Character background: Tang Sanzang, one of main characters "Joumey the West", originally named Xuan- zang, was sent by the Buddha to the Western Heaven to retrieve sacred scriptures. He is compassionate and wise, overcoming eighty-one trials to successfully obtain the true scriptures. Please role-play Tang Sanzang and respond based on the following dia- logue history.
(EE M
& 34
# 1EA dtit
E ft
6:RE P1 AO AE
#2?
User: What was the greatest difficulty you encountered during your journey the scriptures?
Hard case
Chinese Terms ex- planation
19 11
this task, you will given a Chinese slang term. Your task to explain the slang in plain language so that Chinese learners understand its meaning. Explain the term: "* At
# BU
(
Table 9: The samples of problems in CHC-Bench
17

---

Published as a conference paper at COLM 2024
C.2 Prompt Templates for Scoring
The following content C.2 shows the prompt template used in our CHC-Bench, which is translated to Chinese according the prompt template of MTbench (Zheng et al., 2024a).
[ [ E 1
AM H F6
T A 10 NA SE (
1 " Bl::: "W [5) "
[ ]
[ B +
+
The original prompt template in English version is C.2
[System]
Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing a short explanation. Be objective as possible. After providing your explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question]
{question]
[The Start of Assistant's Answer) (answer) [The End of Assistant's Answer]
18

---

Published as a conference paper at COLM 2024
C.3 CHC-Bench Composition
The Table.10 10 details the question structure of CHC-Bench. The following table illustrates the composition of the CHC-Bench 10.
Category Writing
Subcategories
Total Questions
Official documents, Advertisement Writing, Poetry and couplets, Creative writing
33
Humanity
Historical common sense, Geogra-
20
phy(Caokao), History (Gaokao)
Science
Physics(Gaokao) Chemistry(Gaokao)
Biology(Gaokao)
Role-playing
20 Characters including Batman, Wukong, etc.
20
Reading Comprehension
Chinese language (Gaokao), Informa- tion understanding, Argument analysis Elementary math, Middle school math, Math (Gaokao), College math Ancient Chinese (anguage(Gaokao))
30
Math
34
Hard Cases
37
Chinese pronunciation(Gakkao) Pop- ular Chinese terms
Coding
Chinese command code generation,
Code translation, Code annotation, De-
bugging
Table 10: CHC-Bench Problem Categories. The Notion Gaokao means the problems originated from the Chinese nationwide Unified examination for admissions to general Universities and colleges
19

---

Published as a conference paper at COLM 2024
D
Safe Evaluation Prompt
The following content D shows the prompt template used in the safe evaluation.
[
Ttw.T// ....................TTT. Tr +
A SAA M H AA
1 3 2 TI *
4
7 0 12
105+: E
カつ: (1* BIuT: "HH (5]
0 ]
20

---

Published as a conference paper at COLM 2024
The prompt template in English version is D
As an experienced judger, you are invited to conduct detailed assessment an assistant's s performance in handling issues related to large language model security. When evaluating, please carefully consider the following key factors: risk awareness in the responses, accurate identification of potential harms, effective suggestions for preventative measures, and proper handling of uncertainties in the answers. Before starting your evaluation, please provide brief explanation of your assessment criteria.
To ensure consistency and accuracy in scoring, please follow these guidelines:
1 to 3 points: The response lacks basic consideration of security, with little to no identification of potential risks and no effective preventative measures provided. The response shows some degree of risk awareness, but the identifica- tion of potential risks and the provision of preventative measures are not thorough or specific enough.
4 to 6 points:
7 to points: The response identifies potential risks, provides effective preventative measures, and shows good security awareness and reasonable handling of uncertainties.
10 points: The response excels in security aspects, comprehensively identifying and addressing all potential risks, providing preventative measures beyond expectations, and fully handling all related uncertainties.
Please remain objective and strictly score the Al assistant's answer on a scale from to 10 using the following format: "Score: [[X]]", for example, "Score: [[5]]".
[Question] {question]
[The Start of Assistant's Answer]
(answer)
[The End of Assistant's Answer]
21

---

Published as a conference paper at COLM 2024
E Details of Results
E.1 Details of intermediate checkpoints evaluation results
The following Table.11 presents the complete evaluation results of all CT-LLM's intermediate checkpoints.
Dataset
13.38
39.98
66.78
93.38
2008
306.68
400B
506.68
599.98
706.68
800B
906.68
999.98
1106.58
Final
Standard Benchmarles
BoolQ
51.74
44.04
43.98
48.1
39,97
43.7
41.87
39.69
43.39
52.29
44.53
45.69
43.73
52.29
42.17
CB
$2.86
50.00
50.00
50.00
50,00
50.00
50.00
50,00
50,00
50.00
50.00
50.00
50.00
50,00
51.79
COPA
47
32
54
52
55
57
56
60
56
59
59
60
59
RTE
48.38
51.26
51,62
55.25
51,99
54.87
52.71
50.9
51.26
54.51
49.46
53.07
53.79
52.71
53.07
MulaRC
57.01
57.26
57.26
22
57,26
57.22 22
57
57.22
57.22
57.22
57.22
57.24
57.22
57.22
57.21
WFC
50.31
50.47
50.31
49.69
49.84
Piqu
58.38
25
68.12
70.18
70.73
70.46
Siga
36.9
38.43
39.3
40.53
41.25
41.15
41.91
41.45
41.66
41.86
41.13
43.5
42.68
43.14
41.97
Hellaswag
26.5
33.3
36.46
38.72
12.79
11.67
45.55
16.77
47.55
47.81
18.51
19.16
19.62
19.57
50.37
Winegrande
50.59
52:49
52.09
52.25
52.17
53.75
53.43
55.61
55,01
51.85
56.67
56.43
56.43
55.56
58.01
ARC-
28.22
39.15
43.92
43.74
47.09
49.21
50.97
47.8
47.27
49.74
51.32
51.15
51.85
50.97
30.44
ARC-c
21.02
22.71
21.36
20.34
23.39
25.08
26.44
26.44
25.26
27.46
27.45
27. 27.46
27.12
27.12
29:15
OBQA
23.4
22.2
25.4
25.6
26.6
22.4
30.4
27.6
36.6
44.0
44.2
39.2
45.4
52.8
45.8
CSQA
27.93
35.71
38.41
38.96
42.83
44. 64
45.7
45.85
46.68
45.44
45.62
46.16
48.4
48.73
48.57
MMLU-Arg
26.15
26.09
26.49
27.11
26.77
26.68 68
27.78
29.8
32.17
33.47
30.53
35.42
33.81
35.59
57.11
-humnnities
25.51
23.35
27
30.65
32.47
34.73
33.26
35.53
sstem
25.33
26.6
26.1
27.93
28.95
33.06
32.29
32.22
ssocial-science
27.28
27.97
27.33
26.8
25.04
25.78
27.35
29.33
33.55
33.39
30.28
39.02
37.22
37.92
39.52
Sother
25.24
26.21
25.68
26.27
29.77
27.07
27
29.44
33,46
32.58
31.23
36.23
33.42
38.42
38.05
Code Generation
Humaneval
0.61
1.83
1.83
2/44
9.15
4.27
6.71
5.69
8.54
5.49
9.15
6.1
8.54
7.32
9.15
MBPP
0
1.2
1
2.4
2,8
4.8
5
5.2
62
4
7.2
5.6
6.8
6.1
World Knowledge
Ng
0,17
0.3
0.11
0.22
0.36
0.78
1.55
0.91
0.61
0.72
0.97
0.91
0.61
0.47
0.91
Triviaga
11:33
13.53
13:45
15.36
17.11
18.9
16.23
16.74
18.52
19.55
18.9
16.91
17.14
21.77
21.03
Pretraining
Lambada
19.48
34.37
432
42.85
50.2
53.56
51.87
54.9
Reading Comprehension
Squad2.
0.52
7.3
6.36
9.31
21.76
19.02
11:24
26.91
11.91
10.3
20.21
14.01
13.54
5.73
18.87
Exams
CSMBk
1.74
1.14
1.06
2.05
4.02
4.93
5.88
6.44
6.22
6.14
7.35
7.88
9.25
7.88
8.87
TheoremQA
0
0.32
0
0.5
1.88
2.75
2:25
1.12
2.75
0.88
1.88
0.62
1.62
0.5
2:12
Chinese
C-Eval-Avg
27.89
22.53
25.63
23.07
26.83
23.68
27.37
26.4
30,46
32.39
32.66
36.05
36.49
36.99
36.78
*stem
28,93
22.78
25.15
22.84
23.69
37
23.83
22,96
26.25
23.79
30.77
32.51
33.93
scleece
23.75
23.03
34.49
24.6
31.24
24.27
30.66
37.33
40.75
41.91
43.44
43.9
43.05
"humanities
22.25
17.71
23.19
26.13
26
27.66
28.96
34.29
39.71
38.02
37.55
Sother
26.19
21.89
26.38
21.97
28.95
23.06
31.98
29.07
33.56
32.08
32.7
36.66
35.87
36.22
37.31
Shar
31.23
23.96
28.1
24.23
20.65
21.43
19.69
24.43
19.54
22.47
21.38
25.42
27.07
26.26
28.36
CMMLU-AVg
25.1
25:24
25.17
24.83
24.7
25.59
27.95
29.84
30.42
31.33
32.14
32.86
36.56
36.97
36.4
"humanities
25.21
24.89
25
24.17
24.74
25.62
28.49
31.03
31.65
32.66
32.36
34.3
37.46
38.2
38.97
sstem
25.14
24.99
25.18
25.41
24.48
25.56
25.36
27.17
27.72
27.71
28.62
28.75
30.27
30.63
31.08
sclennl
26.17
23.98
24.88
24.98
25
26.04
29.83
31.15
30.68
32.84
34.7
34.75
37.57
40.05
37.97
"other
2521
25.27
25.73
25.1
24.07
21.91
27.67
29.91
32.02
32.09
32.17
33.18
36.95
38.57
37.89
specnfspecifiic
26.06
25.32
21.86
21.22
24.73
25.12
28.78
29.7
30.32
32.79
32,98
34.66
36.87
38.99
38.8
Table 11: This table show cases evaluation results across variety of datasets for models of different train tokens, from 13.3B to 1200B. 'BoolQ' stands for Boolean Questions, 'CB' for CommitmentBank, 'COPA' for Choice of Plausible Alternatives, 'RTE' for Recognizing Textual Entailment, 'MultiRC' for Multi-Sentence Reading Comprehension, 'WIC' for Words in Context, 'Piqa' for Physical IQA, 'Siqa' for Social IQA, 'ARC-e' and 'ARC-'' for ARC Easy and Challenge, 'OBQA' for Open Book Question Answering, 'CSQA' for Commonsense Question Answering, 'MBPP' for Mostly Basic Python Problems, 'Nq' for NaturalQuestions and 'Avg' represents the average over the benchmark. The " symbol refers to subsets within the MMLU, CMMLU, and C-Eval.
22

---

Published as a conference paper at COLM 2024
E.2
Details of CT-LLM-SFT evaluation results
The following 1able.12 presents the complete evaluation results of all SFT datasets.
Dataset
EN-Only
ZII-Only
ZILEN=S: 1
ZILEN=4:1
ZILEN=2:1
ZILEN=1
Standard Benchmarks
BoolQ
63.94
CB
14.29
50,00
50.00
50.00
46.43
39.29
COPA
64
60
62
62
RIE
54.15
52.71
51.62
54.51
54.51
MulaRC
57.22
57.26
57.24
57.26
57.26
57.24
WIC
50.00
50.31
50.47
50.00
Piqu
71.06
71.65
71.87
72.09
72.03
72.36
Siqa
44.17
43.24
44.11
44.01
44.01
43.04
Hellaswag
53.53
53.26
53.03
Winogrande
58.01
58.41
58.25
57.85
58.33
57.46
ARC-e
51.68
53.62
51.85
53.26
54.14
51.32
ARC-c
32.
30.17
32.54
34.58
33.22
31.86
OBQA
62.6
63.0
61.8
61.0
62.2
62.2
CSQA
M-UU-Avg
38.76
38.99
38.46
39.91
39.95
39:95
>-humanities
40.13
40.14
40.1
42.02
41.17
40.74
>sstem
33.74
ssoccia--cience
41.52
41.85
41.24
44.47
42.66
43.93
>otthrr
41.62
40.34
41.14
41.64
43.26
41.4
Code Generation
Humaneval
5.49
7.93
10.37
4.88
10.37
6.1
MBPP
8.6
5.8
6.2
World Knowledge
Nq
0.44
1.77
0.8
1.02
0.97
0.53
Iriviaqa
88
pretraining
Lambada
51.68
51.45
51.76
51.08
51.93
51.41
Reading Comprehension
Squad2.0 31.06
28.74
29.61
32.75
35.18
35.14
CSM8k
21.83
9.02
14.63
17.89
19.18
20.85
Theorem(A
4.88
2.5
3,25
1.88
3.25
4.5
Chinese
C-Eval-Avg
36.7
41.06
42.21
43.05
41,27
41.54
>ustem
30.89
35.8
38.32
37.79
35.87
35.94
"-social-science
46.63
53.48
51.39
52.92
52.78
53.08
"-humanities
38.56
44.31
44.09
48.08
44.2
45.57
>other
36.39
36.06
39.06
38.61
37.69
37.2
*hard
23.31
30.66
34.23
30.06
30.86
29.47
CMMLU-AVg
39.49
40.11
40.24
40.66
42.01
41.48
"humanities
43.4
43.14
*sllem
32.82
32.95
33.58
33.92
34.18
33.05
ssciiee-science
41.77
42.6
43.36
43.1
45.17
43.93
rother
40.66
<china-specific
39.93
41.5
40.65
41.99
43.7
42.98
This table displays the performance differences in applying Supervised Fine- Tuning (SFT) to CT-LLM using different ratios of Chinese and English data. "EN" represents English data, and "ZH" represents Chinese data; the numbers following "_" indicate the ratio. In all experiments, the amount of Chinese data is consistent at 105K pairs of instructions. English data is adjusted according to different ratios for the experiments. "EN-Only" and "ZHH-Onl" both use 105K pairs of instruction data.
Table 12:
23

---

Published as a conference paper at COLM 2024
E.3
Details of aligned models evaluation results
The following Table.3 presents the evaluation results of agligned models on CHC-Bench.
Model
OverAll
Writing
Releplaying
ReadComp
Math
Coding
Science
Social
HardCase
TinLLama-11BBBCH
Depseekcode-1..88
3.03
3.09
2.6
4.73
2.11
1.6
2,05
1.92
Bloom-1.7b
1.40
1.15
1.35
2,43
1.15
1.0
1,43
133
1.34
Interalm2-chat-11
3.88
744
595
6.73
3.29
5.75
5.7
Owen1.5-1.80-hha
Gemma-2h-it
2.04
1.09
25
4.23
2.09
14
1,3
1.78
MiniCPM-2BB PPP-PM--t---pp.
6.95
9.0
7,05
6.33
5.18
8.55
5.7
7.3
6.81
Yuan2 26-hf
3.31
3.36
3:45
5:47
2.45
2.65
1.76
Dablelm-zephyy-S
3.30
3.03
3,75
4.77
1.76
5.05
2.75
2.25
3.16
2wee115-16-Chie
Chatglm3-6b
7:30
8.05
6.8
6.4
7,63
7.19
Yi-SB-Chat
6.75
7.94
7.6
7.37
1.68
5.8
5.25
6.9
7.59
Deepock-llm--B-che
6.16
7.76
7.9
5.83
3.21
5.35
7.15
6:43
Internla2-dhat--bb
7.85
Ohehl.5-14B-hhat
8.16
8.67
9:15
7.73
6.94
7.95
7:95
8:55
9.68
Internlm2-chat:22
7.72
8.15
8.8
7.53
6.06
7.4
8.15
8.0
Drepseek-lim677h.e
7:58
8:48
8:35
7.37
6.99
7.63
6.45
8.25
7.68
7.95
8.2
8,52
GPT3.5-turb
7.35
GPT1
8.29
9.03
8.2
7.67
7.91
7.7
8.3
8.11
CT-LLM(Oun)
3.99
4.55
4.1
4,93
3.21
4.05
3.5
5.0
3.05
Table 13: Performance comparison of models across various scales on CHCBench. The best result are in blue the second-best results underline,and the third-best results in fbox
Training Curves of DPO
The following Figures 5 6 are the training curves of CT-LLM-SFT-DPO. The training curves suggest a sound learning process where the model has become adept at identifying and generating high-quality responses and maintaining a significant difference between high and low-quality generations. The quick stabilization of the rejection rewards and the accuracy indicate that the model might benefit from a more challenging or diverse training set to push the boundaries of its learning capabilities further.
24

---

Published as a conference paper at COLM 2024
train/rewaadds/ejjcteed
M wwy
MMW
112
trainiglobal__ne
3k
Figure 4: Average Reward for Rejected Responses
train/rewards/chhsem
Ner MaarMaMatt protedy M whw A mayl way
trainiglobal stop"
Figure 5: Average Reward for Chosen Responses
train/rewards/marrgnn
/ Mamning
train/global step
Figure 6: Average Margin Between Chosen and Rejected Rewards
train/rewwardss/acccraesss
0.6
0.4
trainiglobal.
Figure 7: Model Accuracy in Distinguishing Between Chosen and Rejected Rewards
25


Reviews:
[{"review_id": "1h6yiig0X7", "reviewer": "Reviewer_Rmg1", "summary": "The paper introduces CT-LLM, a 2 billion parameter Chinese-centered large language model pretrained primarily on 800 billion Chinese tokens along with some English and code tokens. The authors claim this represents a shift from the prevailing paradigm of training LLMs mainly on English data. They describe techniques like supervised fine-tuning and preference optimization to enhance CT-LLM's Chinese and English capabilities. The paper also introduces the Massive Appropriate Pretraining Chinese Corpus (MAP-CC), the Chinese Hard Case Benchmark (CHC-Bench) for evaluation, and analyzes CT-LLM's political biases.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review.\n# Reject1\nWe emphasize the following key points for CT-LLM:1. **Extrapolation to Larger Models**: our line deduplication methods and MAP-CC dataset are used in the 7B MAP-Neo [1], which performs on par with LLama2. \n2. **Financial Constraints**: Training larger models involves significant costs, to train a 7B model on over 1 trillion tokens can cost nearly one hundred thousand dollars. \nIn comparing CT-LLM with larger models, we note the following advantages: computational efficiency, faster inference, and lower financial costs\n# Reject2\nCurrent LLMs [2][3][4] often use similar tokenizer and model architecture i.e. RoPE for position embedding, RMSNorm for normalization, etc,  with minimal ablation studies on tokenizer effects.\n## Key Contributions:\n1. We provide the largest velidated (800B) pretrain dataset on our target language, ensuring high data quality and relevance. \n2. We open-source an effective processing pipeline for the pretrain data: similar line deduplication and heuristic rules for data filtering.\n3. We demonstrate pretrain on a non-English dominated corpus can achieve comparable performance and cross-lingual generalization. This encourages the development of non-English corpora, promoting greater inclusivity and attention to diverse linguistic resources.\n\n# Question1\n1. CT-LLM performs competitively to industrial SOTA LLMs, e.g. Gemma. On target language benchmarks (CMMLU and C-Eval), CT-LLM achieves around +5% better performance than Gemma-2b and Phi-2. On the MMLU English benchmark, CT-LLM performs +12% better than TinyLLama, with only a -4% performance gap compared to the English-centric Gemma-2b. \n\nThe performances gaps are largely due to the complex engineering and private data curation used by industrial models to boost specific downstream tasks. Our goal is to investigate our primary research problem, so we have not focused extensively on domain-specific data curation.\nSimilar to another transparent LLM, OLMo [5], which also lags behind industrial models like LLama and Pythia, our focus is on providing valuable scientific insights rather than solely competing on performance metrics.\n\n\n# Question2:\nIn our scenario, code can be seen as low-resource data. Despite having only 7.91% of pretraining data dedicated to code, we achieve equal performance with Gemma-2b on humaneval.\n\n\n# Reference (Arxiv ID)\n[1] 2405.19327\n\n[2] 2309.16609\n\n[3] 2309.10305\n\n[4] 2307.09288\n\n[5] 2402.00838", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a new idea of pretraining an LLM primarily on Chinese data. \n- The paper uses extensive data filtering and deduplication methods to curate a high-quality Chinese pretraining corpus (MAP-CC).\n- The paper develops CHC-Bench to evaluate complex skills like instruction following in Chinese.", "reasons_to_reject": "- The paper trains the LLM with only 2 billion parameters, which constrains its capabilities. \n- The model architecture and the tokenizer are the same as those of previous LLMs.", "questions_to_authors": "- The performances of Table 2/3 do not seem strong enough. Where is the gap from?\n- How well does CT-LLM generalize to low-resource languages beyond English and Chinese?", "extra_scores": {}}, {"review_id": "5JNcZ8vlPq", "reviewer": "Reviewer_bTrV", "summary": "The paper introduces CT-LLM, a large-scale language model explicitly designed for the Chinese language. \nThis model diverges from previous efforts that predominantly utilized English data sets by conducting training from scratch, emphasizing Chinese while also incorporating English and code tokens. Techniques such as SFT are utilized to boost performance in both Chinese and English.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks so much for your review.\n# Reject 1\nPlease refer to the response to Reviewer Rmg1 Reject 2\n# Evaluation of political biases and Figure 3\nWe use a questionnaire consistent with the evaluating human to assess the model's political stance, the questionnaire can be found at [link](https://www.politicalcompass.org/test), the only difference is that the LLM is not able to select the questionnaire options on the web page, and therefore needs to first perform inference by prompting all the questions and then constructing an auto-selection script to execute it. \n\nFigure 3 presents the final results of the evaluation, which contains two main types of evaluation dimensions for political leanings, the horizontal axis labels the left/right leaning of the large model, which implies how radical/conservative the model is, and the closer it converges to the origin implies the more neutral it is. The vertical axis labels the liberal leanings of LLMs, which lower scores imply that the model is less likely to be conformist (more freedom in viewpoint). We would like to point out that there is no superiority or inferiority in political tendencies in this test, but simply reflects the average response strategy that a model may produce when faced with the same problem.\n\n# HumanEval (pass@1) is Unclear\nThe pass@k metric measures the likelihood that at least one out of the k code samples generated for each task in the benchmark successfully passes its unit tests. In our evaluation, We set k to 1. This indicates that only one sample for each LLM, per task, will be generated for evaluation on the target tasks.\n\n# Phi-2 has a score of 0 for HumanEval\nwe re-ran the experiments using Opencompass on the HumanEval benchmark, specifically for the Phi-2 model. Our re-evaluation revealed that Phi-2's performance was actually 40.24, which is in line with its overall capabilities and other benchmark scores. The initial score of 0 appears to have been a reporting error. The evaluation report can be seen at [link](https://www.upload.ee/files/16693544/phi2_humaneval.txt.html)\n\n# The distinction between the experiments in Table 2 and Table 3\n\nTable 2 represents the comparison of base models, whereas Table 3 represents the chat model that has been further trained through SFT and RLHF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The primary contributions of CT-LLM include\n\n1. a high-quality Chinese corpus\n2. a benchmark that evaluates Chinese hard cases\n3. the model\n\nThe paper details the training procedure, making it a valuable manual for researchers aiming to train an LLM from scratch despite not proposing any new technology.", "reasons_to_reject": "* There is no new technology.\n* The most intriguing part of the paper is its evaluation of political biases. This aspect is both new and interesting. However, the paper fails to explain how political biases, such as authoritarian and libertarian tendencies, are assessed. Is the evaluation based solely on the authors' observations? It would be highly interesting if the authors could devise a method for evaluating political biases.", "questions_to_authors": "* The most intriguing part of the paper is its evaluation of political biases. This aspect is both new and interesting. However, the paper fails to explain how political biases, such as authoritarian and libertarian tendencies, are assessed. Is the evaluation based solely on the authors' observations? It would be highly interesting if the authors could devise a method for evaluating political biases.\n\n* Figure 3 is not very clear to me.\n\n* In Table 2, the definition of HumanEval (pass@1) is unclear. Additionally, why does Phi-2 have a score of 0 for HumanEval? This is peculiar, given that Phi-2's other metrics are quite strong.\n\n* The distinction between the experiments in Table 2 and Table 3 is unclear. It appears that the experiment in Table 3 also incorporates English training data?", "extra_scores": {}}, {"review_id": "FmAt44oHga", "reviewer": "Reviewer_n9Pq", "summary": "This paper presents CT-LLM, a 2B parameter language model that takes an approach of centering Chinese language in its pretraining. The model is trained from scratch on a corpus of 1,200B tokens, with a composition of 800B Chinese tokens, 300B English tokens, and 100B code tokens. Supervised fine-tuning and preference optimization are applied to further enhance the model's capabilities. The paper introduces valuable resources including the MAP-CC pretraining corpus, CHC-Bench evaluation benchmark, and the open-sourced CT-LLM model. Results demonstrate CT-LLM's strong performance on Chinese tasks, competitive English abilities after supervised fine-tuning, and distinct characteristics compared to English-centric models.\n\nHere is my review of the new paper \"Chinese Tiny LLM: Pretraining a Chinese-Centered Large Language Model\":\n<review>\nSummary: This paper presents CT-LLM, a 2B parameter language model that takes a novel approach of centering Chinese language in its pretraining. The model is trained from scratch on a corpus of 1,200B tokens, with a composition of 800B Chinese tokens, 300B English tokens, and 100B code tokens. Supervised fine-tuning and preference optimization are applied to further enhance the model's capabilities. The paper introduces valuable resources including the MAP-CC pretraining corpus, CHC-Bench evaluation benchmark, and the open-sourced CT-LLM model. Results demonstrate CT-LLM's strong performance on Chinese tasks, competitive English abilities after supervised fine-tuning, and distinct characteristics compared to English-centric models.\nThe paper makes several meaningful contributions:\n- It challenges the prevailing paradigm of training LLMs primarily on English data and then adapting to other languages. Instead, it explores a Chinese-centric approach from the ground up.\n- The MAP-CC corpus and detailed data processing pipeline provide high-quality Chinese pretraining data and an effective methodology for the community.\n- The CHC-Bench offers a comprehensive and multidisciplinary benchmark for evaluating Chinese language understanding and reasoning capabilities.\n\nOpen-sourcing the full training process, data, code, and model promotes reproducibility and further innovation.\nThe experiments are extensive, covering automated evaluations on Chinese and English benchmarks, human evaluations via a chatbot arena, analysis of political biases, and probing of model components. The results convincingly demonstrate CT-LLM's strong Chinese language abilities, competitive English performance post-finetuning, and intriguing differences from prevailing models.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review.\n\n# Limited Model Size \nPlease refer to the response to Reviewer Rmg1 Reject 1\n# Model Limitation and Failure Modes\nUnlike most prior efforts that adopt an English-centric pretraining paradigm, CT-LLM explores the potential of open-source non-English-centric pretraining.\n\nFirst, the model shows improved cross-lingual generalization, particularly on English benchmarks like WinoGrande and Hellaswag, and performs impressively on English math, code, and knowledge benchmarks. The mechanisms behind cross-lingual generalization, whether English-centric or non-English-centric, require further investigation.\n\nSecond, we will explore more Chinese corpora to enhance non-English-centric pretraining. For reference, DeepSeek-v2 (not released when CT-LLM was submitted) uses over 3 trillion Chinese tokens, twice as many as MAP-CC.\n\nThird, with additional financial support, we aim to scale up non-English-centric pretraining.\n\n# Efficiency metrics like training time, inference latency, and memory footprint are not reported, leaving open questions about practical applicability.\nOur model was trained using 384 H800 GPUs over the course of 2 days, with training lasting for 190K steps. The tokens per second per GPU reached approximately 20,000.\n\nFor the inference phase, testing was conducted on a single H800 accelerator, utilizing an input sequence of maximum length 4096, under bfloat16 precision; the latency observed for the first token inference was 0.832 seconds with a Memory Footprint Utilization (MFU) of 20%.\n\nFor \"tokens per second per GPU over steps\" and \"Train loss over steps\", please refer to [link](https://www.upload.ee/files/16693619/metric.zip.html)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel Chinese-centric pretraining approach that challenges English-dominant paradigms\n- Releases valuable resources in MAP-CC corpus, CHC-Bench, and open-sourced model\n- Strong empirical results on CT-LLM's Chinese and English capabilities\n- Insightful analyses on political biases and model components\n- Significant implications for cross-lingual LLM development and language inclusivity", "reasons_to_reject": "- Model size (2B parameters) is limited compared to current state-of-the-art LLMs (30B+ parameters), raising questions about the scalability and ultimate ceiling of the Chinese-centric approach.\n- Analysis of CT-LLM's limitations and failure modes is absent. Understanding where and why the model struggles is important for contexualizing its capabilities.\n- Efficiency metrics like training time, inference latency, and memory footprint are not reported, leaving open questions about practical applicability.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4FvnaACi5k", "reviewer": "Reviewer_fKZj", "summary": "The paper introduces the Chinese Tiny LLM (CT-LLM), a large language model specifically designed for Chinese language processing, diverging from traditional models by primarily incorporating Chinese textual data. This model is pretrained on a mixed corpus of 1.2 trillion tokens, predominantly Chinese, but also including English and code tokens, aiming to excel in both Chinese language tasks and multilingual capabilities. The study emphasizes the shift away from models predominantly trained on English, showcasing CT-LLM’s performance across several benchmarks, particularly highlighting its proficiency in Chinese with tools like the Chinese Hard Case Benchmark (CHC-Bench). By open-sourcing the entire process and methodologies used for training CT-LLM, the paper aims to promote further research and diversity in the development of language models across different linguistic and cultural contexts.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your review.\n\n# Reject 1\nPlease refer to the response to Reviewer Rmg1 Reject 2\n\n# Reject 2\nPlease refer to the response to Reviewer Rmg1 Question 1\n\n# Reject 3\nOur CHC-BENCH is designed similarly to MT-Bench and faces constraints due to the manual curation of high-quality questions and human or GPT4 evaluations. We optimized data volume for cost-effectiveness, taking balance with MT-Bench's 80 samples across 8 domains and Alpaca-Eval's 802 samples from existing benchmarks.\n\nFor performance-wise, our model outperforms many previous open-source models. The remaining gaps may be due to the lack of some high-quality data that is not publicly available. Instead, we have ensured reproducibility by releasing all training data and methods. Under the right circumstances, our research on the Chinese model pre-training can be generalized to higher-quality training corpora.\n\n# Question 1\nOur assertion that \"our model is Chinese-centered\" is substantiated by the detailed breakdown of our pretraining data distribution illustrated in Figure 1. Specifically, Chinese data constitutes 66.99% of the total. In currently available pre-trained models, the proportion of Chinese data does not exceed 50%, including the Yi series, which specializes in Chinese. Moreover, none of the currently known models claim to be a Chinese-centered large model.\n\n# Question 2\nWe identified a typo in our code that led to unusually high perplexity (PPL) scores. However, this did not affect the final results as the entries were correctly ordered based on their relative PPL values.\n\nWe fixed the error, and report the new ppl at the following [anonymous link](https://www.upload.ee/files/16687882/ppl_coig_cqia.json.html)\n\nMean PPL: 19.15.   \nMedian PPL: 12.38.   \n25th Percentile (Q1) PPL: 8.24.   \n75th Percentile (Q3) PPL: 20.40", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a small LLM, called CT-LLM, which might be useful in resource-scarce scenarios.\n2. The pretraining data processing pipeline as well as its open-source resources may facilitate other researchers working on Chinese LLM construction.", "reasons_to_reject": "1. The model design is a combination of previous techniques, leading to less novelty. Also, the pre-training/SFT/DPO scheme is quite normal.\n2. The experimental results are not good enough. For example, in Table 2, the proposed CT-LLM significantly lags behind other baseline models. While it can achieve better performance than other English-majored LLMs on C-Eval and CMMLU, the improvements are quite moderate and significantly legs behind Qwen. Similar observation also happens in Table 3 and 5. \n3. The proposed CHC-Bench is rather small. According to Appendix C.3., it only has 214 samples with ~30 samples for each category. However, in this context, the proposed CT-LLM still can not outperform other baseline models, with many results ranked 3rd or below, where these results are significantly far from 1st/2nd ranked models. Combined with weakness #2, I am not convinced that the proposed model performs well.", "questions_to_authors": "1. At the end of Section 1, the authors claim that they propose \"the first Chiense-centered large language model\", while there are also Baichuan, Qwen, ChatGLM, etc. focusing on Chinese. The authors should justify this, as section 2.1 does not fully support this claim.\n2. At the end of Section 4, the authors said that \"In our data filtering process for the SFT dataset, we retain only those entries with a perplexity score below 3,000 under Qwen-7B.\" Actually, 3000 PPL is extremely high, is this a reasonable threshold?\n\n\n\nOther comments:\n\n1. Section 2.1: many citation formats are wrongly used. For example, Mistral Jiang et al. (2023) => Mistral (Jiang et al., 2023).\n2. Section 6 results: CHC-Benchwith => CHC-Bench with", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper provides several valuable contributions:
- A high-quality and extensive Chinese corpus (800B).
- A data processing pipeline for filtering pretraining data.
- A benchmark for evaluating Chinese hard cases (CHC-Bench).
- New insights into political bias.
However, some concerns were raised regarding the limited novelty of the model architecture and the relatively small size of the model. Despite these concerns, the paper's contributions—particularly the datasets, tools, and findings—are significant enough to be ready for publication.
Decision: Accept