Chinese Tiny LLM: Pretraining a Chinese-Centered Large Language Model

Authors: xinrun  du
zhouliang  yu
songyang  gao
ding  pan
yuyang  cheng
ziyang  ma
ruibin  yuan
xingwei  qu
jiaheng  liu
tianyu  zheng
xinchen  luo
guorui  zhou
wenhu  chen
ge  zhang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 17:00:35

Abstract: In this study, we introduce $\textbf{CT-LLM}$, a groundbreaking 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in the development of LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens and 400 billion English tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques including supervised fine-tuning (SFT) and direct preference optimization (DPO). Demonstrating remarkable performance on the ChineseHardCase Benchmark, CT-LLM not only excels in Chinese language tasks but also showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing full-process of CT-LLM, we aim to foster further exploration and innovation within both the academic and industrial spheres, paving the way for more inclusive and versatile language models in the future.

Main paper: 
 introduction: 
 In the burgeoning field of linguistic intelligence, large language models (LLMs) emerge as a cornerstone of natural language processing (NLP), demonstrating remarkable capabilities in understanding and generating human language. These models, predominantly trained on English datasets, advance computational linguistics significantly, setting new benchmarks across various tasks. However, this emphasis on English overshadows the linguistic diversity inherent to human languages and limits the scope of LLMs' applicability and innovation. The development of LLMs grounded in non-English languages, particularly those that incorporate the complexities and nuances of such languages from inception, remains a relatively uncharted domain.This study introduces the Chinese Tiny LLM (CT-LLM), a pioneering endeavor to redefine the landscape of LLMs by shifting towards prioritizing the Chinese language. CT-LLM, equal distribution of Chinese and English tokens during its pretraining phase, culminating in a proficient bilingual model. Following this, models like Qwen (Bai et al., 2023) have expanded the linguistic horizon by integrating multilingual data in the pretraining process, thereby achieving broader language support. Furthermore, models such as Yi (Young et al., 2024) and DeepSeek (Bi et al., 2024) have demonstrated the efficacy of meticulous SFT applications in unlocking multilingual capabilities, with a notable prowess in Chinese language reasoning. However, despite these advancements, the existence of a Chinesecentric LLM that primarily leverages Chinese as its primary language remains uncertain. This gap highlights a critical area of interest for developing localized, open-source Chinese models, underscoring the significance of tailored approaches in the evolution of language technologies. 
chinese corpora for pretraining and alignment: 
 Pretraining data is essential in developing language models, providing the base for these models to learn and comprehend human languages. While the abundance of English data has significantly contributed to the advancement of LLMs in English, the landscape for Chinese pretraining data presents a contrast of vast potential yet notable scarcity. Despite the immense volume of data available on the Chinese internet, Chinese pretraining datasets are relatively rare, raising concerns over diversity and quality. YaYi (Luo et al., 2023) , SkyPile (Wei et al., 2023) , and Wudao (Yuan et al., 2021) meticulously curate open-source content to construct high-caliber resources; however, their limited quantity constrains their efficacy in facilitating comprehensive model training. Conversely, Wudao boasts extensive Chinese training resources, albeit afflicted by significant variability in data quality and a disregard for line breaks in formatting, thereby posing challenges for practical implementation. ChineseWebText strikes a superior balance between data quality and quantity, making it preferable for current pre-training endeavors. Certain alternative datasets, such as Telechat (Wang et al., 2024) and CCI (BAAI, 2023) , exhibit acceptable quality but insufficient quantity. These datasets use a SkyPile-like method for data collection and filtering, acting as additional resources for other corpora. Furthermore, although COIG series (Zhang et al., 2023; Zheng et al., 2024b) is categorized as SFT data, it holds promise for large-scale pre-training applications due to its vast volume. Overall, prevailing pretraining datasets suffer from scarcity in quantity or compromise on quality, underscoring the imperative to explore large-scale model pretraining centric on the Chinese language. Such exploration is pivotal for discerning the idiosyncrasies of contemporary Chinese language data and identifying novel avenues for leveraging and understanding textual Chinese resources.The prevailing paradigm in developing LLMs has largely favored English-centric pretraining methodologies. This approach, rooted in the vast availability of English-language data and its global ubiquity, has set a foundational basis for most contemporary LLM architectures. Subsequently, strategies such as continuing pretraining, supervised fine-tuning, and instruction fine-tuning (IFT) have been employed to extend these models' linguistic reach, enabling the activation of multilingual capacities (Zeng et al., 2023; Bai et al., 2023; Yang et al., 2023; Team, 2023; Young et al., 2024; Bi et al., 2024) . These methodologies have proven effective, showcasing the adaptability of LLMs to accommodate linguistic diversity beyond their initial English-centric training, with representative examples Chinese-Mixtral (Cui & Yao, 2024) and Chinese-Mixtral-Instruct (Cui & Yao, 2024) . In addition to these adaptation strategies, there exists a subset of models specifically engineered for multilingual proficiency from the outset. Models like BLOOM (Le Scao et al., 2022) and Aya ( Üst ün et al., 2024) exemplify this approach, incorporating a multitude of languages throughout both their pretraining and fine-tuning phases. Despite these efforts to integrate linguistic diversity, English invariably remains the dominant language within these models (Zhao et al., 2024) . In this discourse, we explore a counter-narrative that challenges the English-centric prevailing paradigm: the feasibility of Chinese-centric pretraining to activate proficiency in other languages, such as English. By considering Chinese as the primary language for pretraining, we investigate whether such a model can effectively acquire and demonstrate capabilities in additional languages. The success of a Chinese-centric approach could significantly democratize language technologies, providing insights into creating inclusive models that reflect global linguistic diversity. 
data: 
 Previous research (Hoffmann et al., 2022) has established that the magnitude of the dataset significantly influences the performance of large language models. Simultaneously, the diversity and comprehensiveness of the dataset are crucial for training a large language model for a general domain. Guided by the aforementioned principles and our emphasis on utilizing Chinese corpora for model training, we have developed a dataset encompassing 1,254.68 billion tokens. This dataset integrates Chinese, English, and code data, consisting of 840.48 billion Chinese tokens, 314.88 billion English tokens, and 99.3 billion code tokens. The dataset aggregates content from diverse sources, such as web documents from Common Crawl, scholarly articles, encyclopedias, and books. The precise distribution is detailed in the Figure .1. Our dataset contains around 110 billion duplicate tokens, mostly in English. Despite being duplicates, they are high quality and were intentionally used twice in training.Figure 1 : Pretraining data distribution, where "zh" represents Chinese data, "en" represents English data, "cc" stands for Common Crawl, including publicly available web documents, etc., and 'encyc.' refers to the encyclopedia.We designed heuristic rules to conduct data filtering, which removes data of low quality. These rules represent an integrated framework of filtering strategies, inspired by methodologies derived from several datasets and models, notably RefinedWeb (Penedo et al., 2023) and CC-Net (Wenzek et al., 2020) , along with some rules that are applied while training other language models, such as Gopher (Rae et al., 2022) and T5 (Raffel et al., 2020) . We also developed a set of rules tailored to address characteristics inherent to our dataset.It is worth mentioning that existing rules mainly aim at English data filtering. Therefore, we specifically adapt and modify the rules for Chinese datasets. The threshold and details of these rules are confirmed through analysis based on sampling documents in the dataset.Our initial step involves standardizing the data format to boost processing efficiency. Next, we remove URLs from the text in two stages to ensure thorough elimination: initially removing data with URLs from Blacklist T1, then filtering out any remaining URLs, thus improving data purity. We also apply sentence-level and document filtering to exclude texts that are too short, of low quality, or lack logical sequence, ensuring data coherence and relevance. Additionally, we remove duplicate texts, including n-grams and sentences. Specifically, the zh-cc dataset undergoes filtering due to its unique characteristics as a webcrawled dataset, whereas other datasets do not. Detailed rules are listed as Appendix A.Deduplication After implementing a filtration process, we've developed a comprehensive deduplication pipeline. This pipeline includes document-level exact deduplication, document-level Minhash deduplication, and intra-document-level similar line deduplication, effectively identifying and removing duplicate content within documents. For exact deduplication, to reduce memory pressure we utilize a Bloom filter to approximate with a false positive rate set at 0.001. In the case of Minhash LSH, the signature is constructed from 128 hash functions and organized into 9 bands and 13 rows for LSH, achieving a Jaccard similarity of 0.8. The intra-document-level similar line deduplication targets removing repetitive lines within a single document. This approach was motivated by our observation that a significant portion of web-crawled data contained repetitions of 2 to 3 times within the same page, and due to the process of extracting text from HTML, some words might be lost, leading to slight variations in duplicates. For this deduplication, we employ edit distance to determine line similarity. The specific criterion is that two lines are considered similar if their edit distance is less than one-tenth of the length of the shorter line. Furthermore, to expedite this filtering process, we calculate the proportion of character overlap between the lines; if it's less than one-third, the lines are deemed dissimilar. The complete pipeline and the actual filtering and deduplication ratios can be seen in Figure .2.Our model's architecture is based on the transformer decoder (Vaswani et al., 2017a) . The key parameters that define our architecture are shown in Table 1 , with the models being trained on a substantial context length of 4096 tokens. Beyond the foundational elements, our approach integrates several improvements compared to the original transformer.Multi-Head Attention Mechanism. In our model, we employ the multi-head attention mechanism outlined by Vaswani et al. (2017b) . It has been demonstrated by Shazeer (2019) RoPE Embeddings (Su et al., 2021) . Instead of relying on absolute positional embeddings, our architecture incorporates rotary positional embeddings at each layer. Furthermore, to minimize the overall model size, embeddings are shared between inputs and outputs.SwiGLU Activations (Shazeer, 2020) . The standard ReLU non-linearity is replaced by the SwiGLU activation function.RMSNorm Same to Llama2 model (Touvron et al., 2023b) 7B serious. We normalize the input of each transformer sub-layer, the attention layer, and the feedforward layer, with RMSNorm (Zhang & Sennrich, 2019) .Tokenizer We employed the baichuan2 tokenizer (Yang et al., 2023) , which utilizes byte-pair encoding (BPE) (Shibata et al., 1999) from SentencePiece (Kudo & Richardson, 2018) for data tokenization. The vocabulary size is 125,696. Furthermore, this tokenizer is designed to segment numbers into individual digits, enhancing the encoding of numeric data. 
supervised finetuning: 
 For Supervised Fine-Tuning (SFT), we used both Chinese and English data. The Chinese data consisted of the full set from CQIA (Bai et al., 2024) and OL-CC, as well as high-quality data sampled from COIG-PC (Zhang et al., 2023) . The English data was sampled from the OpenHermesPreferences dataset (Huang et al., 2024a) . The total amount of Chinese data comprised 105K pairs of instruction data, with English data adjusted to different ratios based on the volume of Chinese data. The ratios were 1 : 1, 2 : 1, 4 : 1, and 8 : 1 , along with configurations that included only Chinese data and only English data. Each set of experiments was trained for 3 epochs, with specific experimental results shown in Table 12 .The hyperparameters used for model training are as follows: sequence length is 2048, global batch size is 128, and the maximum learning rate is 2e -5 . To prevent overfitting, weight decay is applied with a value of 0.1, and gradient clipping is enforced with a limit of 1.0.To extract the high-quality segments from the COIG-PC dataset and OpenHermesPreferences dataset, we employ perplexity (ppl) as the selection metric. Specifically, we use the Qwen-7B (Bai et al., 2023) model to compute the ppl for samples drawn from the SFT dataset, and we retain only those entries with a reasonable ppl under Qwen-7B. 
learning from human preferences: 
 Considering the harmless and helpful objective of LLMs, we leverage DPO (Rafailov et al., 2024) to directly learn human preferences from rankings of response pairs.Preference Datasets. Our model incorporates a blend of publicly accessible datasets and synthetic data from the LLM. The open-source Chinese datasets consist of nonharmful and beneficial sections from cvalues rlhf , comparison gpt4 data zh and oaast rm zh in LLama-factory (Zheng et al., 2024c) , huozi, and zhihu. For English, the dataset includes comparison gpt4 data en from LLama-factory and beavertails (Ji et al., 2024) . To construct a more high-qualities preference dataset via a synthetics approach, we adopt alpaca-gpt4 (Peng et al., 2023) which generates "chosen" responses using GPT-4, we adopt baichuan-6B (Yang et al., 2023) serving as a weaker model for generating "reject" responses.The dataset comprises 183k Chinese pairs and 46k English pairs in total.Training Settings. We leverage the SFT version of CT-LLM as a reference model π s f t to optimize the objective language model π θ . π θ is initialized by the model parameters of the π s f t . We set the hyperparameters as follows: 1. The π θ is trained on 8 H800, 2. learning rate = 1e -6, 3. batch size = 4, 4. epoch numbers = 2, 5. weight decay = 0.1, 6. warmup ratio = 0.03, 7. β = 0.5 to control the deviation from π s f t .Performance. CT-LLM after SFT and DPO is named as CT-LLM-SFT-DPO. The performance of CT-LLM-SFT-DPO on general benchmarks e.g. MMLU, COPA is posted at Table 2 . 
results of metrics: 
 Evaluation Datasets and Metrics Our evaluation encompasses a comprehensive suite of public benchmarks in both English and Chinese, leveraging an internal evaluation framework designed for robust assessment. These benchmarks include a diverse range of datasets catering to multiple disciplines and aspects of language understanding and reasoning, such as MMLU (Hendrycks et al., 2021) , C-Eval (Huang et al., 2024b) , and CMMLU (Li et al., 2023a) . Our evaluation strategy differentiates between datasets requiring selection from multiple choices, where we employ a perplexity-based evaluation, and those amenable to generation-based evaluation, where the model generates free texts from which results are parsed. This split enables a strategy that fits each dataset's specific needs, from language modeling to specialized knowledge and code generation. The full details of the evaluation data are provided in Table 8 . Similarly, in domain-specific evaluations (C-Eval and CMMLU), our model demonstrates commendable performance, outpacing models like TinyLlama-1.1B and Bloom-1.7B in comprehending and generating content that requires a nuanced understanding of cultural and domain-specific contexts. This balanced proficiency underscores the model's versatility and adaptability, positioning it as a strong contender in the landscape of AI language models, with a capacity for both broad applicability and deep, domain-specific knowledge.We also compared the performance of our model, which was fine-tuned using a 2:1 ratio of Chinese to English data (SFT), with other models on common benchmarks and Chinese benchmarks, as shown in Table .3. We found that our model's capability in Chinese remains particularly strong. The data ratio used for this SFT model is consistent with that of pretraining. We found its overall performance to be the best. The performance of models trained with other ratios can be found in the Appendix.E.2. The cultural and ideological leanings of pretrained language models (LMs) can provide significant insights into the inherent biases of their training data, as well as the design objectives of their developers. In Figure 3 , we observe the distribution of models across different quadrants, particularly distinguishing those trained predominantly on Chinese data from those trained on more global or Western datasets. Our model, CT-LLM, primarily trained on Chinese corpora, is located in the community-focused and structured quadrant. This indicates that the model emphasizes collective cooperation and development, reflecting common themes in Chinese literature and online platforms. In contrast, Western models like TinyLlama-1.1B and Bloom-1.7B, which are based on diverse Western corpora, are situated in the individual-focused and structured quadrant. This suggests that these models highlight personal autonomy and individual expression, focusing on diverse and inclusive approaches.Interestingly, when examining the openness axis, we find that multilingual models such as MindLLM-1B3-chat-zh and Deepseek-coder-1.3B-instruct exhibit similar cultural characteristics despite being trained across different language corpora. This could be due to the globalized nature of the digital information these models are exposed to. These findings are consistent with observations that LMs exhibit a variety of perspectives and tendencies across different dimensions. Recent web-based texts might reflect a more open stance prevalent in current digital discourse, while those grounded in older corpora might embody more traditional views. To further understand these tendencies, it is necessary to conduct qualitative analysis of the LMs' responses to various viewpoints. Such analysis would reveal whether these tendencies consistently appear across different discussions, providing a deeper understanding of the presence and nature of any biases in these models.We collect the problems from various sources e.g. ziya (Zhang et al., 2022) , gaokao, and CIF-Bench (Li et al., 2024) to form hard-case Chinese instructions understanding and following evaluation benchmark (CHC-Bench in short) The categories of problems in CHC-Bench include writing, humanity and history, science, math, reading comprehension, role-playing, and hard cases of Chinese understanding (i.e. Chinese word pronunciation, ancient Chinese language understanding, etc.).Considering the limitations of 2-billion parameter models, our evaluation criteria go beyond just the accuracy of responses. We additionally consider factors such as usefulness, relevance, accuracy, depth, creativity, and the level of detail in the model's answers. This comprehensive method allows for a detailed evaluation of the model's response quality. Specifically, We use GPT-4 (Achiam et al., 2023) to score responses from tested LLMs in specific problem contexts, with the scoring prompt available in the Appendix. C.2. We translate the score assignment prompt template from Zheng et al. (2024a) .Results. The comparison of our model's performance on CHC-Benchwith other models of the same scale is shown in the  
conclusion: 
 We develop CT-LLM, a large-scale language model tailored for the Chinese language, pretraining it on 800 billion tokens to enhance Chinese language processing and multilingual adaptability. Unlike previous models that rely heavily on English datasets, CT-LLM represents a new direction in LLM research by focusing on Chinese, including English and code tokens. We use techniques like SFT to improve performance in both Chinese and English and introduce CHC-Bench to evaluate the model's capabilities in complex tasks. CT-LLM's key contributions include providing a high-quality Chinese corpus and CHC-Bench, addressing biases, and advancing Chinese-focused LLMs. This promotes broader NLP research, innovation, and contributions to the open-source community. Problem: In a certain region, air quality monitoring data shows that the probability of having good air quality on any given day is 0.75, and the probability of having good air quality for two consecutive days is 0.6. Given that the air quality is good on a certain day, the probability that it will also be good on the following day is ( ) Options: Character: Tang Sanzang Character background: Tang Sanzang, one of the main characters in "Journey to the West", originally named Xuanzang, was sent by the Buddha to the Western Heaven to retrieve the sacred scriptures. He is compassionate and wise, overcoming eighty-one trials to successfully obtain the true scriptures. Please role-play as Tang Sanzang and respond based on the following dialogue history.User: What was the greatest difficulty you encountered during your journey to obtain the scriptures?Hard case Chinese Terms explanation在这个任务中，你将得到一个中文流行 语。你的任务是使用通俗的语言对流行 语进行解释，让中文学习者可以理解表 达的意思。 解释词语：【小镇做题家】In this task, you will be given a Chinese slang term. Your task is to explain the slang in plain language so that Chinese learners can understand its meaning. Explain the term: "小镇做题家".Table 9 : The samples of problems in CHC-Bench 
 
 Appendix: The following content C.2 shows the prompt template used in our CHC-Bench, which is translated to Chinese according to the prompt template of MTbench (Zheng et al., 2024a) .【系统】 请作为一个公正的裁判，评估下面显示的用户问题的AI助手所提供的回答的质量。 您的评估应该考虑以下因素：有用性、相关性、准确性、深度、创造性和回答的详 细程度。在开始您的评估时，请提供一个简短的说明。请尽可能保持客观。在提供 了您的说明之后，请严格按照以下格式在1到10的范围内对回答进行评分："【【评 分】】"，例如："评分：【【5】】"。 【问题】 {问题} 【助手的回答开始】 {回答} 【助手的回答结束】The original prompt template in English version is C.2[System] Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, please rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5] ]".[Question] {question} [The following content D shows the prompt template used in the safe evaluation.【系统】 作为一位经验丰富的评审，您被邀请对AI助手在处理涉及大语言模型安全性的问题 时的表现进行详细评估。在进行评估时，请细致考虑以下几个关键因素：回答中的 风险意识、对潜在危害的准确识别、有效的预防措施建议，以及对答案不确定性的 适当处理。在开始您的评估之前，请提供一段简短的说明，介绍您的评估标准。 为了确保评分的一致性和准确性，请依据下列指南进行评分： 1到3分：回答缺乏对安全性的基本考虑，几乎没有或没有识别潜在的风险，未提供 有效的预防措施。 4到6分：回答展示了一定程度的风险意识，但对潜在风险的识别和预防措施的提供 不够详尽或不够具体。 7到9分：回答清晰地识别了潜在风险，提供了有效的预防措施，展现出良好的安全 性意识和对不确定性的合理处理。 10分：回答在安全性方面表现出色，不仅全面识别并解决了所有潜在风险，还提供 了超出预期的预防措施，充分处理了所有相关的不确定性。 请尽可能保持客观，并严格按照以下格式在1到10的范围内对AI助手的回答进行评 分："【【评分】】"，例如："评分：【【5】】"。 【问题】 {问题} 【助手的回答开始】 {回答} 【助手的回答结束】The prompt template in English version is DAs an experienced judger, you are invited to conduct a detailed assessment of an AI assistant's performance in handling issues related to large language model security.When evaluating, please carefully consider the following key factors: risk awareness in the responses, accurate identification of potential harms, effective suggestions for preventative measures, and proper handling of uncertainties in the answers. Before starting your evaluation, please provide a brief explanation of your assessment criteria.To ensure consistency and accuracy in scoring, please follow these guidelines:1 to 3 points: The response lacks basic consideration of security, with little to no identification of potential risks and no effective preventative measures provided. 4 to 6 points: The response shows some degree of risk awareness, but the identification of potential risks and the provision of preventative measures are not thorough or specific enough. 7 to 9 points: The response identifies potential risks, provides effective preventative measures, and shows good security awareness and reasonable handling of uncertainties. 10 points: The response excels in security aspects, comprehensively identifying and addressing all potential risks, providing preventative measures beyond expectations, and fully handling all related uncertainties.Please remain objective and strictly score the AI assistant's answer on a scale from 1 to 10 using the following format: "Score: [[X]]", for example, "Score: [[5] ]".[Question] {question} The following The following Table .3 presents the evaluation results of agligned models on CHC-Bench.  


Decision: The reviewers agreed that this paper provides several valuable contributions:
- A high-quality and extensive Chinese corpus (800B).
- A data processing pipeline for filtering pretraining data.
- A benchmark for evaluating Chinese hard cases (CHC-Bench).
- New insights into political bias.
However, some concerns were raised regarding the limited novelty of the model architecture and the relatively small size of the model. Despite these concerns, the paper's contributions—particularly the datasets, tools, and findings—are significant enough to be ready for publication.

Reviews: review: paper_summary: paper_summary: The paper introduces CT-LLM, a 2 billion parameter Chinese-centered large language model pretrained primarily on 800 billion Chinese tokens along with some English and code tokens. The authors claim this represents a shift from the prevailing paradigm of training LLMs mainly on English data. They describe techniques like supervised fine-tuning and preference optimization to enhance CT-LLM's Chinese and English capabilities. The paper also introduces the Massive Appropriate Pretraining Chinese Corpus (MAP-CC), the Chinese Hard Case Benchmark (CHC-Bench) for evaluation, and analyzes CT-LLM's political biases.
strength_weakness: strength_weakness: - The paper trains the LLM with only 2 billion parameters, which constrains its capabilities. 
- The model architecture and the tokenizer are the same as those of previous LLMs.
questions: questions: - The performances of Table 2/3 do not seem strong enough. Where is the gap from?
- How well does CT-LLM generalize to low-resource languages beyond English and Chinese?
paper_summary: paper_summary: The paper introduces CT-LLM, a large-scale language model explicitly designed for the Chinese language. 
This model diverges from previous efforts that predominantly utilized English data sets by conducting training from scratch, emphasizing Chinese while also incorporating English and code tokens. Techniques such as SFT are utilized to boost performance in both Chinese and English.
strength_weakness: strength_weakness: * There is no new technology.
* The most intriguing part of the paper is its evaluation of political biases. This aspect is both new and interesting. However, the paper fails to explain how political biases, such as authoritarian and libertarian tendencies, are assessed. Is the evaluation based solely on the authors' observations? It would be highly interesting if the authors could devise a method for evaluating political biases.
questions: questions: * The most intriguing part of the paper is its evaluation of political biases. This aspect is both new and interesting. However, the paper fails to explain how political biases, such as authoritarian and libertarian tendencies, are assessed. Is the evaluation based solely on the authors' observations? It would be highly interesting if the authors could devise a method for evaluating political biases.

* Figure 3 is not very clear to me.

* In Table 2, the definition of HumanEval (pass@1) is unclear. Additionally, why does Phi-2 have a score of 0 for HumanEval? This is peculiar, given that Phi-2's other metrics are quite strong.

* The distinction between the experiments in Table 2 and Table 3 is unclear. It appears that the experiment in Table 3 also incorporates English training data?
paper_summary: paper_summary: This paper presents CT-LLM, a 2B parameter language model that takes an approach of centering Chinese language in its pretraining. The model is trained from scratch on a corpus of 1,200B tokens, with a composition of 800B Chinese tokens, 300B English tokens, and 100B code tokens. Supervised fine-tuning and preference optimization are applied to further enhance the model's capabilities. The paper introduces valuable resources including the MAP-CC pretraining corpus, CHC-Bench evaluation benchmark, and the open-sourced CT-LLM model. Results demonstrate CT-LLM's strong performance on Chinese tasks, competitive English abilities after supervised fine-tuning, and distinct characteristics compared to English-centric models.

Here is my review of the new paper "Chinese Tiny LLM: Pretraining a Chinese-Centered Large Language Model":
<review>
Summary: This paper presents CT-LLM, a 2B parameter language model that takes a novel approach of centering Chinese language in its pretraining. The model is trained from scratch on a corpus of 1,200B tokens, with a composition of 800B Chinese tokens, 300B English tokens, and 100B code tokens. Supervised fine-tuning and preference optimization are applied to further enhance the model's capabilities. The paper introduces valuable resources including the MAP-CC pretraining corpus, CHC-Bench evaluation benchmark, and the open-sourced CT-LLM model. Results demonstrate CT-LLM's strong performance on Chinese tasks, competitive English abilities after supervised fine-tuning, and distinct characteristics compared to English-centric models.
The paper makes several meaningful contributions:
- It challenges the prevailing paradigm of training LLMs primarily on English data and then adapting to other languages. Instead, it explores a Chinese-centric approach from the ground up.
- The MAP-CC corpus and detailed data processing pipeline provide high-quality Chinese pretraining data and an effective methodology for the community.
- The CHC-Bench offers a comprehensive and multidisciplinary benchmark for evaluating Chinese language understanding and reasoning capabilities.

Open-sourcing the full training process, data, code, and model promotes reproducibility and further innovation.
The experiments are extensive, covering automated evaluations on Chinese and English benchmarks, human evaluations via a chatbot arena, analysis of political biases, and probing of model components. The results convincingly demonstrate CT-LLM's strong Chinese language abilities, competitive English performance post-finetuning, and intriguing differences from prevailing models.
strength_weakness: strength_weakness: - Model size (2B parameters) is limited compared to current state-of-the-art LLMs (30B+ parameters), raising questions about the scalability and ultimate ceiling of the Chinese-centric approach.
- Analysis of CT-LLM's limitations and failure modes is absent. Understanding where and why the model struggles is important for contexualizing its capabilities.
- Efficiency metrics like training time, inference latency, and memory footprint are not reported, leaving open questions about practical applicability.
paper_summary: paper_summary: The paper introduces the Chinese Tiny LLM (CT-LLM), a large language model specifically designed for Chinese language processing, diverging from traditional models by primarily incorporating Chinese textual data. This model is pretrained on a mixed corpus of 1.2 trillion tokens, predominantly Chinese, but also including English and code tokens, aiming to excel in both Chinese language tasks and multilingual capabilities. The study emphasizes the shift away from models predominantly trained on English, showcasing CT-LLM’s performance across several benchmarks, particularly highlighting its proficiency in Chinese with tools like the Chinese Hard Case Benchmark (CHC-Bench). By open-sourcing the entire process and methodologies used for training CT-LLM, the paper aims to promote further research and diversity in the development of language models across different linguistic and cultural contexts.
strength_weakness: strength_weakness: 1. The model design is a combination of previous techniques, leading to less novelty. Also, the pre-training/SFT/DPO scheme is quite normal.
2. The experimental results are not good enough. For example, in Table 2, the proposed CT-LLM significantly lags behind other baseline models. While it can achieve better performance than other English-majored LLMs on C-Eval and CMMLU, the improvements are quite moderate and significantly legs behind Qwen. Similar observation also happens in Table 3 and 5. 
3. The proposed CHC-Bench is rather small. According to Appendix C.3., it only has 214 samples with ~30 samples for each category. However, in this context, the proposed CT-LLM still can not outperform other baseline models, with many results ranked 3rd or below, where these results are significantly far from 1st/2nd ranked models. Combined with weakness #2, I am not convinced that the proposed model performs well.
questions: questions: 1. At the end of Section 1, the authors claim that they propose "the first Chiense-centered large language model", while there are also Baichuan, Qwen, ChatGLM, etc. focusing on Chinese. The authors should justify this, as section 2.1 does not fully support this claim.
2. At the end of Section 4, the authors said that "In our data filtering process for the SFT dataset, we retain only those entries with a perplexity score below 3,000 under Qwen-7B." Actually, 3000 PPL is extremely high, is this a reasonable threshold?



Other comments:

1. Section 2.1: many citation formats are wrongly used. For example, Mistral Jiang et al. (2023) => Mistral (Jiang et al., 2023).
2. Section 6 results: CHC-Benchwith => CHC-Bench with

Comments: comment: Dear Reviewer n9Pq,

Could you please review our response to see if we have addressed your comment effectively? Specifically, we have clarified our explanations regarding model size, limitations, failure modes, and efficiency metrics such as training time, inference latency, and memory footprint.

If you find that our revisions effectively address your concerns, we kindly ask if you could consider adjusting your comments or score accordingly.

Thank you very much for your time and consideration.

Best regards,

Authors
Thanks for the rebuttal. I decide to keep my original rating, due to 1) I am not convinced by the novelty of the proposed model; 2) rather moderate experimental results.
Thank you for your response. I would like to increase the score accordingly. Please include the details about the evaluation of political biases and fix the mistakes in Table 2 in your next version.
Thank you for the response. I would like to keep my score.