MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Authors: Shengding Hu
Yuge Tu
Xu Han
Ganqu Cui
Chaoqun He
Weilin Zhao
Xiang Long
Zhi Zheng
Yewei Fang
Yuxiang Huang
Xinrong Zhang
Zhen Leng Thai
Chongyi Wang
Yuan Yao
Chenyang Zhao
Jie Zhou
Jie Cai
Zhongwu Zhai
Ning Ding
Chao Jia
Guoyang Zeng
dahai li
Zhiyuan Liu
Maosong Sun
Venue: COLM (2024)
Keywords: Scaling Law, Small Language Model, Training Dynamics, Training Strategy
TL;DR: Propose two small sized LMs MiniCPM (2.4B and 1.2B) with advanced training strategies, the performance on par with Mistral-7B and Llama7B/13B on general domain.

Abstract: The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly~\footnote{\url{https://github.com/OpenBMB/MiniCPM}}.

Published as a conference paper at COLM 2024
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
Shengding Hu', Yuge Tu², Xu Han 1.3x Chaoqun He', Ganqu Cui', Xiang Long?,
Zhi Zheng", Yewei Fang", Yuxiang Huang", Weilin Zhao', Xinrong Zhang',
Zheng Leng Thai', Kaíhuo Zhang Chongyi Wang', Yuan Yao', Chenyang Zhao', Jie Zhou², Jie Cai?, Zhongwu Zhai?, Ning Ding',
Chao Jia", Guoyang Zeng', Dahai Li?, Zhiyuan Liu' Maosong Sun Department of Computer Science and Technology, Institute for Artificial Intelligence, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China. Modelbest Inc.
SShanghai Artificial Intelligence Laboratory.
shengdinghu@gmail. com, (hanxu2022, liuzy,, sms @tssinguaa.edu.cn
Abstract
The burgeoning interest in developing Large Language Models (LLMs) with up trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as resource-efficient alternative. In this context, we introduce MiniCPM, specificclllllly the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Den Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute 1 optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPPO MiniCPM-MoF and MiniCPM-1128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly 1.
1 Introduction
Following the revelation of the scaling law (Kaplan et al., 2020), there has been a vigorous pursuit in the field of Large Language Models (LLMs) (Hoffmann al., 2022; Bai et al., 2023; Gemini et al., 2023; Chowdhery et al., 2023; Achiam et al., 2023), encompassing models with up an astonishing number of parameters in the trillions (Fedus et al., 2022). These models have emerged as a pivotal driving force in the evolution of artificial intelligence.
Nonetheless, the training of such large-scale models is both financially burdensome and operationally inefficient. On one hand, the empirical understanding of the mechanisms underpinning the training of LLMs remains elusive. Given the significant economic and
"Corresponding Authors. hhttpe://giccuub.co///ppeee cttp:///peepeaa////iii/C

---

Published as a conference paper at COLM 2024
environmental costs, experiments on LLMs are prohibitively expensive for most researchers and corporations. On the other hand, the deployment of these colossal models in everyday scenarios, such as on personal computers or smartphones, is either inefficient unfeasible. Both aspects underscore the imperative to refocus efforts on comprehensively exploring smaller, yet potent, language models (SLMs). These models on the one hand provide efficient solutions to practical deployment, on the other hand, if trained with scalable strategies, they can potentially guide the development of future larger models.
Recently, a resurgence of interest has been observed in the domain of SLMs, evidenced by the advent of series of innovative models such as the Phi series (Gunasekar al., 2023; Li et al., 2023b; Javaheripi & Bubeck, 2023), TinyLlama (Zhang et al., 2024a), MobileLLM (Liu et al., 2024), and Gemma (Banks & Warkentin, 2024), among others. While these models have significantly contributed to the expansion and diversification of the SLM landscape, there remain two pivotal areas where these models have yet to fully satisfy prevailing interests: (1) the development of comprehensive abilities akin to those exhibited by LLMs; and (2) the formulation of transparent and scalable training methodologies that could further propel the evolution of both SLMs and LLMs.
In this paper, we introduce MiniCPM, a series of SLMs, which primarily builds on two models, endowed with 2.4B and 1.2B non-embedding parameters respectively, and they rank preeminently in their respective 2B and 1B scale categories. MiniCPM also exhibits comparable capabilities to those of 7B~13B language models, such as Llama2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023), Gemma-7B (Banks & Warkentin, 2024), and Llama-13B (Touvron et al., 2023), etc. Notwithstanding their small model sizes, our training methodology is meticulously designed to facilitate seamless scaling of both model scale and data horizons. This is exemplified through our model wind tunnel experiments that encompass comprehensive hyper-paramete of a WSD for continuous training with an un un-rredefined pre-training token number and makes the reusing of model intermediate checkpoints highly feasible. A detailed analysis of the training dynamics of MiniCPM is presented, suggesting that the WSD scheduler demonstrates the intriguing loss landscape of model pre-training. With the WSD scheduler, we are now also capable of studying the data-model scaling law with linear effort on the model axis and negligible effort on the data axis, while the traditional ones need quadratic effort considering the scaling along both model and data axes. The result of the scaling law indicates much higher data size/ model size ratio compared with Chinchilla Optimal (Hoffmann et al., 2022).
optimization (Section 3), and the deployment learning rate scheduler (Section 4). The latter tailored
Moreover, we introduce the MiniCPM family, including MiniCPM-DPO, MiniCPM-128K, and MiniCPM-MoEE We conduct evaluations of the MiniCPM family against established benchmarks and illuminate their impressive capabilities as SLMs: (1) The foundation models surpass Mistral-7B, and LLama-13B. (2) The DPO model surpasses zephyr-7B (Tunstall et al., 2023) on MTBench (Zheng et al., 2024) (3) The 2.4B MiniCPM-128K model demonstrates performance either surpassing or matching that of models like Yarn-Miistral-7B-1288 (Peng et al., 2023) and C
(Du et al., 2021). (4) The MiniCPM-MoE, with 4B activated parameters, is on par with Llama2-34B (Touvron et al., 2023).
In summary, MiniCPM propounds new stage in the development of small language models, exemplifying the latent potential within SLMs and advocating for a more scientific and sustainable approach toward scaling up LLMs.
2 Related Work
Small Language Models. "Small Language Models" (SLMs) is an evolving concept that has undergone significant transformations over time. Presently, SLMs are generally construed as models that are smaller in scale compared to the well-known LLMs, typically not exceeding 7 billion parameters. These models are distinguished by their capacity for deployment on end-user devices, such personal computers and smartphones, even in the absence of a GPU. Notable examples within the current landscape of SLMs include the Phi series (Gunasekar et al., 2023; Li et al., 2023b; Javaheripi & Bubeck, 2023), TinyLlama (Zhang
2

---

Published as a conference paper at COLM 2024
et al., 2024a), MobileLLM (Liu et al., 2024), and Gemma (Banks & Warkentin, 2024), etc. variety of methodologies have been explored to augment the efficacy of SLMs. These include the incorporation of high-quality data (Gunasekar et al., 2023; Li et al., 2023b; Javaheripi & Bubeck, 2023), the application of structure pruning techniques (Xia et al., 2023), and the reconfiguration of model architectures (Liu et al., 2024), among others. MiniCPM enhances the capabilities of SLMs through a meticulous amalgamation of hyper-parameter optimization, strategic training methodologies, architectural design, and high-quality data.
Scalable Pre-training Strategies. Since the discovery of scaling law (Kaplan et al., 2020; Rae et al., 2021; Aghajanyan et al., 2023), scientifically and predictably (Achiam et al., 2023; Hu et al., 2023; Du et al., 2024) scaling up the LLMs has been pursued from diverse perspectives, especially for the pre-training stage. In terms of training stability, the Tensor Program series (Yang et al., 2022; 2023) is introduced to ensure optimal hyper-parameter consistency across varying model scales, technique employed in training CerebrasGPT (Dey et al., 2023). Furthermore, Wortsman et al. (2023) suggest leveraging smaller models to anticipate and mitigate instabilities in larger model training. From the training data standpoint, various data-centric strategies have been advocated (Xie et al., 2024; Shi et al., 2023; Ye et al., 2024). In the realm of training methodologies, prior research has delved into diverse learning rate schedulers (LRS) (Howard & Ruder, 2018; Raffel et al., 2020; Hundt et al., 2019), with the Cosine LRS (Loshchilov & Hutter, 2016) emerging as the predominant choice in LLMs. Kaplan et al. (2020) and Hoffmann et al. (2022) have meticulously examined the hyper-parameters of Cosine LRS, thereby laying a foundational groundwork for subsequent pre-training works. Of these, DeepSeek (Bi et al., 2024) bears the closest resemblance to our proposed WSD LRS. Concerning batch size scheduling, Smith et al. (2017) advocates for incrementing batch size as an alternative diminishing learning rate, a strategy recently adopted by Yi-9B (Young et al., 2024).
3 Model Wind Tunnel Experiments
Although we target at training SLMs that can be quickly deployed onto end devices, we envision that many aspects of model training are universal across scales. Extensive experi- ments should be conducted through an SLM to explore the limit of SLM before transferring the experience into LLMs. These experiments take the spirit of wind tunnel testing in developing an aircraft, thus we name it Model Wind Tunnel Experiments (MWTE). In this paper, the MWTE contains three parts: (1) Hyper-parameters; (2) Optimal Batch-size Scaling; and (3) Optimal Learning Rate Stability.
3.1 Scaling Hyper-parameters Invariant LM
Hlyper-parameeess have a significant impact on the performance of a model. However, adjusting hyper-parameters for each model in traditional training is not feasible for LLMs. Even for SLM like MiniCPM, extensive experiments on hyper-parameters search take a lot of resources. Tensor Program (Yang et al., 2022; 2023) proposes a framework to stabilize the hyper-parameters for models with different scales. The main part of the Tensor Program is the width scaling (Yang et al., 2022) and the depth scaling (Yang et al., 2023). The former technique supports CerebrasGPT (Dey al., 2023) to predict the loss of LLMs more accurately. In MiniCPM, we use both two scaling techniques. The specific scaling operations are listed in Table 3. We do not apply the attention softmax scaling techniques proposed in (Yang et al., 2022) (See Table 3 of their first version). Despite Yang et al. (2023) observing that depth scaling for a network with block depth larger than two is not satisfying, we find the resulting optimal learning rate is stable empirically. Details of the hyper-parameters and Tensor Program Operations are in Appendix A.1.
3.2 Optimal Batch Size
Batch size determines the balance between the convergence speed of the model and the consumption of computational resources. the batch size is too large, it will result in significant amount of data and computation costs. On the other hand, if the batch size
3

---

Published as a conference paper at COLM 2024
is too small, it will require a large number of training steps and may result in limited decrease the loss function. We follow Kaplan et al. (2020) to determine the batchsize from expected loss, with slight modification from their setting (see Appendix A.2). We conduct experiments on 0.009B, 0.03B, and 0.17B models, respectively, toward this goal. Each model size is trained on batch sizes with global learning rate of 0.01 and cosine learning rate scheduler. We observe the trend of the optimal batch size with loss on the C4 (Raffel et al., 2019) dataset (red line in the Figure 1).
12
bug (8- 21tlmmmmm211.1212121111
Slae
12
Batch
Batch
Figure 1: We demonstrate the loss curve of three size models trained using different batch sizes. Each vertical line formed by points with a gradient color represents training curve. Figure 2: The connected op- Lighter colors denote higher loss.
Loss
timal batch sizes.
As shown in Figure 1, we plot the batch size in the x-axis, and token consumption in the y-axis, the color of the points represents a loss. Thus vertical line formed by the color points denotes training curve. use parabolas to fit the equal-loss points and connect the minima of the parabolas with red lines. The lines demonstrate the optimal batch size shifts large as the loss decreases. We then connect the three lines (see Figure 2) and find that the lines connect each other well into a linear relationship in the log space, from which we
obtain the following relationship between batch size bs and C4 Loss L: bs x.21x10' We 7,224
note that it might seem strange that the batch size should be estimated from a rough loss prediction that we only have after training. We provide comment in Appendix A.2.
3.3 Optimal Learning Rate
Due to our use of Tensor Program (Yang et al., 2022; 2023), we anticipate that the learning rate, will not undergo significant changes during model scaling. To verify this, we conduct six sets of learning rate experiments at 0.04B, 0.1B, 0.3B, and 0.5B. In Figure 3, we find that although the model size has increased by ten times, the optimal base learning rate 2 does not show a noticeable shift and remains around 0.01. We further conduct a simple validation on scale of 2.1B and confirm that learning rate of 0.01 indeed achieves the lowest loss.
1B: Tats
1844
Figure 4: Cosine Learning Rate Scheduler with different periods. The Y-axis is the loss on the C4 corpus.
Figure 3: Loss vs Learning Rate. After ap- plying for the Tensor Program, the learn- ing rate shift becomes minimal.
2The actual learning of 2-D tensors will be scaled according to Tensor Program.
4

---

Published as a conference paper at COLM 2024
4 WSD Learning Rate Scheduler
4.1 Analysing Cosine LRS
The learning rate scheduler (LRS), which adjusts the learning rate used in different stages of training, is crucial for model performance. The current commonly used learning rate strategy the Cosine LRS (Kaplan et al., 2020; Hoffmann et al., 2022; Rae et al., 2021; Touvron et al., 2023; Bai et al. 2023; Almazrouei et al., 2023), which gradually decreases the learning rate following a cosine curve after reaches its maximum after the warmup stage.
A key hyper-parameter in the Cosine LRS is the step I at which Cosine decreases to the minimum for the first time. Typically, is set the total training step S for training with predefined training step. Generally, it is believed that the learning rate should be high to enable sufficient exploration. For example, Kaplan et al. (2020) demonstrate that the loss decreases when the summed learning rate over the entire training increases (see Figure 22 in their paper). This indicates setting < S is not optimal. On the other hand, Hoffmann et al. (2022) make a key observation that setting T > S results in dropped performance while setting S T results in improved training efficiency, confirming that the learning rate shouldn't be kept high throughout the training. To reproduce these observations, we conduct experiments on the 0.036B model. We try Cosine(T) and CosineLoop( (T) LRS, following the formula shown in Appendix B.1. The result can be seen in Figure 4. We can see that when the training step is S 20N, 40N, 60N, 80N, the lowest loss is always achieved by the Cosine(T) where T S. Both T < S and T S are not optimal.
We hypothesize that the Cosine LR performs exceptionally well when 1 S because of the following two reasons: (1) Cosine LRS with T S has a longer duration of high learning rate training compared to T < S and other LRS such as Linear LRS. This high learning rate might help the model find better global optimum. (2) Cosine LRS with T has more thorough learning rate decay phase compared to Cosine LRS with T > S and Constant LRS. This learning rate decay may involve unique training dynamics that enable the model to find a better local optimum.
4.2
WSD LRS
In light of the above perspective, we propose to explicitly divide the training stage into the high learning rate stage and learning rate decay stage. We name the Warmup-Stable- Decay (WSD) LRS. Especially, the WSD LRS contains three stages: the warmup stage (whose end step denoted by W), the stable training stage (whose end step is denoted by T), and the remaining decay stage. The function form of WSD is:
Wh, $ S<W W
WSD(T;s)
", W <s T Tss, T<s<s
(1)
where 0 f(s
decreasing function about s, 1/ the maximum learning rate. Typically, as long as the warmup stage is enough, it affects little performance, therefore, we omit W in the subsequent discussion. With an abuse notation, we will denote WSD with clear stop point
T)
4.3 Experiments
Next, we present several experimental findings of WSD LRS.
Loss Decreases Dramatically in Decay Stage. We try WSD LRS on 0,036B models. As shown in Figure 5, in the decay stage, as the learning rate begins to decrease, the loss experiences a significant rapid decline and quickly decreases to be equal to or lower than the Cosine LRS at step T = S. At the same time, can reuse the model before decay and continue training with the previous high learning rate. After more steps of training Š' we can also perform annealing to achieve the same loss as the Cosine LRS at Cosine(S") This
5

---

Published as a conference paper at COLM 2024
WOD S0
Oocar)000
(50
Figure 6: Continous training a 0.036B model match the performance of 0.17B model with an acceptable increase training com- pute.
Figure 5: Model training loss has sudden decrease in the decay stage of WSD LRS.
verifies our assumption that the training stage be explicitly split into the stable training and decay stages.
10% Steps are Enough. From the two-stage training perspective, shortening the decay stage will greatly benefit the fast test of different model checkpoints of stable training. Therefore, conduct experiments that start from the same stable training checkpoints and have different decay steps. Also shown in Figure 5, among all three stable training checkpoints in 40N, 60N, and S0N training data, having decay of 10% of the total tokens sufficient to achieve the best results, while a decay of 2.5% of total tokens falls short. Therefore, in the subsequent training experiments, we use decay about 10% to ensure full convergence.
Effective Data Scaling with WSD LRS. With WSD LRS, we can continuously train the LM to extreme convergence. To further demonstrate the potential training a fixed-sized model to convergence, we compare continuously training 0.036B LM with 0.17B model with 40N data. Figure 6, the green lines represent 0.036B models trained with different stable training tokens. Despite the last point of the 0.036B series being trained with many more tokens than Chinchilla Optimal (Hoffmann et al., 2022), still has space for performance improvement.
To find the limit of continuously training this fixed-sized LM, we estimate how the model's optimal performance changes with its computation during continuous training. By optimal performance, we mean the loss of training token is achieved by WSD(D,0.1D) With series of D, the losses will form the optimal loss envelope. Due to uncertainty about the function form of the loss envelope, we try two fitting formulas: (1) exponential: L(C)
ae ae-BC BC + Lo and (2) power-law: L(C)- = pC + Lp- The fitting results for both functions are in Appendix B.2. We find that the plwer-law form fits better (similar the Cosine LRS (Kaplan et al., 2020)). In Figure 6, the fitted curve is shown in green dotted lines. To intuitively estimate and comprehend the effect of continuous training such fixed-sized model, we also trained 0.17B model with WSD(40N, 4N), which is shown in pink in Figure 6. We can see that a 0.036B model can match the performance of 0.17B model with acceptable increase (~ 4 times) in training compute while saving lot of inference computation (Sardana & Frankle, 2023) (saving ~ times per inference call), indicating better inference-compuee inferene-comppte--ppinn setting (Sardana & Frankle, 2023).
Note
To deepen our understanding of WSD scheduler, we provide a detailed analysis of the model training dynamics in Appendix D. The WSD scheduler also helps us estimating the scaling law better and more efficiently, which can be found in Appendix E.

---

Published as a conference paper at COLM 2024
5 Two Stage Pre-training Strategy
Typically, the training of instruction following LLMs contains the pre-training stage and the supervised fine-tuning (SFT) stage (Zhang et al., 2023; Wei et al., 2021). In the pre- training stage, the data is composed of large-scale unlabeled data, while in the SFT stage, high-quality labeled data becomes the optimization target. In light of the pronounced loss decrease observed during the decay stage of the WSD LRS, we postulate that the integration of high-quality labeled data in this phase presents dual advantages:
Introducing this data during the annealing phase, in addition the SFT stage, fosters more comprehensive model learning. Specifically, it facilitates a more pronounced loss reduction in relation to the SFT data distribution, rather than the pre-training data distribution. This approach is more congruent with actual user scenarios.
contrast to uniform distribution of high-quality data throughout the entire pre-training process, this method enhances training by concentrating on data and sustaining continuous pre-training. we not predetermine a training step, we will repeat small dataset throughout an ongoing pre-training process, which could lead to negative effects.
Based on these two hypotheses, we propose the following training strategy: during the pre- training phase, only use large-scale coarse-quality pre-training data, which is abundant and can support continuous training when provided with more computational resources. During the annealing phase, we use diverse and high-quality knowledge and ability-oriented SFT data, mixed into the pre-training data.
To validate the advantages of our training strategy, conduct comparison experiments using (A) MiniCPM-2.4BB intermediate checkpoint the stable stage; and (B) MiniCPM- 1.2B's last checkpoints in the stable stage. Specifically, compare the following:
1. A-1: 2.4B model, decay using only pre-training data, followed by 4B token SFT.
2. A-2: 2.4B model, decay using the aforementioned high-quality data unlabeled data and SFT data mixed into pre-training data, also followed by 4B token SFT.
3. B-1: 1.2B model, decay using only pre-training data, followed by 6B token SFT.
4. B-2: 1.2B model, decay using only pre-training data, followed by 12B token SFT.
5. B-3: 1.2B model, annealing using the aforementioned high-quality data + SFT data mixed into pre-training data, also followed by 6B token SFT.
The results of the experiments are shown in Table 1. We can that, despite the A-2 and A-1 have undergone the same SFT distribution, adding SFT data to the decay stage pushes the boundary Comparison between B-2 and B-3 demonstrate that the deficiency of only SFT is not due to the insufficient training tokens in SFT stage.
C-Eval
CMMLU
MMLU
GSMSK
MATH
HumanEval
MBPP
A-1
40.0
41.5
44.6
27.7
5.1
27.7
24.4
A-2
52.6
51.1
50.9
42.3
5.4
30.4
30.3
B-1
40.9
41.5
47.9
34.2
7.9
43.9
30.5
B-2
41.2
42.0
47.9
34.4
7.3
43.9
29.8
B-3
49.1
46.8
49.6
31.8
10.5
44.5
32.8
Table 1: The ablation study of different training strategies.
The results indicate that the benefits of introducing high-quality data at the beginning of the decay stage much higher than simply adding it during the SFT phase. Therefore, recommend that specialization and enhancement of model capabilities should start from the decay phase.
7

---

Published as a conference paper at COLM 2024
6 Model
In this section, we begin to introduce the MiniCPM model that aggregates the aforemen- tioned observations and techniques. Due to page limitation, the model architecture details are in Appendix F.
Mixture Stago
Taty Misture Theay Stage
Ge
Cok Festrax
236
Open WA Xart
Aody
Saca Enchiig 90
48
ThraCha
Kooel
Ook-SPT
Figure Data mixture of different training stages. The stable stage is shown on the left and the decay stage is shown on the right.
6.1 Training Stages
The overall training of the MiniCPM base model includes three stages: stable training stage, decay stage, SFT stage (Zhang et al., 2023; Wei et al., 2021). Throughout the stages, use Adam Optimizer (Kingma & Ba, 2014).
Stable Training Stage. We utilize around 1T data (see Section 7 for data distribution), with the majority of the data sourced from open datasets. We use the optimal configuration discovered during the model wind tunnel experiments, WSD LRS, with batch size of 3.93 million and max learning rate of 0.01.
Decay Stage. We use a mixture of the pretraining data and high-quality SFT data. For the specific annealing form of the WSD scheduler, we employ exponential annealing, i.e.
f(s - 7)- 0.5(8-5)/T, in which T is set to be 5000 steps (20B tokens).
SFT Stage. We find it still necessary to conduct separate SFT phase. We utilize SFT data similar to the annealing phase excluding pre-training data and train with approximately billion tokens. The learning rate for SFT is aligned with the one at the end of annealing, and WSD Scheduler with exponential decay is also employed.
6.2 Training Data Distribution
We introduce our training data distribution in Figure 7. In the figure, CommonCrawl.co in Chinese Corpus derived from CommonCrawl raw corpus and goes through thorough cleaning. Dolma (Soldaini et al., 2024), C4 (Raffel et al., 2019), and Pile (Gao et al., 2020; Biderman al., 2022) are English corpora. They are deduplicated inner corpus and across corpus using MinHash algorithms (Broder, 1997). The Code Pre-train data contains the stack (Kocetkov et al., 2022) and StarCoder et (2023a), with inner deduplication and cross deduplication. In the decay stage, the data mixture contains more diverse data and proprietary data, including UltraChat (Ding et al., 2023), SlimOrca (Lian et al., 2023a;b), OssInstruct (Wei et al., 2023), Evollnstruct (Xu et al., 2023). The data with the suffix SFT is our proprietary data including LeetCode questions, Kindergarten through 12th grade (K12) textbooks and questions, etc.
8

---

Published as a conference paper at COLM 2024
3.0
2.9
2.
2.8
Loos on
2.7
Looe
2.7
2.6
2.0
2.5
2.4
2.00
4000
600
800
10000
1200
15.0
400
5001
800
1006
1200
Tolous (B)
Tolosus (B)
Figure 8: Loss curve on C4 dataset for MiniCPM-1.28 (Left) and MiniCPM-2.44 (Right). The orange segment at the tail the loss curve represents the remaining decay process, which is not utilized in the released version of MiniCPM.
Model
C-Eval
CMMLU
MMLU
HumanEval
MBPP
GSMSK
MATH
Llama2-7B
32.42
31.11
44.32
12.20
27.17
13.57
1.80
Qwen-7B
58.96
60.35
57.65
17.07
42.15
41.24
5.34
Deepseek-7B
42.82
44.45
47.82
20.12
41.45
15.85
1.53
Mistral-7B
46.12
42.96
62.69
27.44
45.20
33.13
5,00
Gemma-7B
42.57
44.20
60.83
38.41
50.12
47.31
6.18
Llama2-13B
37.32
37.06
54.71
17.07
32.55
21.15
2.25
MPT-30B
29.34
32.09
46.56
21.95
35.36
10.31
1.56
Falcon-40B
40.29
41.57
53.53
24.39
36.53
22.44
1.92
TinyLlama-1.1B
25.02
24.03
24.3
6.71
19.91
2.27
0.74
Qwen-1.8B
49.81
45.32
43.37
7.93
17.8
19.26
2.42
Qwen1.5-1.8B
55.00
50.85
43.81
5.49
24.82
26.16
3.25
Gemini Nano-3B
27.20
22.80
StableLM--Zehyyr-3B
30.34
30.89
45.90
35.37
31.85
52.54
12.12
Phi-2(2B)
23.37
24.18
52.66
47.56
55.04
57.16
3.50
Gemma-2B
29.26
28.56
38.49
24.39
29.74
16.83
3.34
MiniCPM-1122B
49.14
46.81
49.63
44.51
32.75
31.77
10.60
MiniCPM-2.4B
51.13
51.07
53.46
50.00
47.31
53.83
10.24
Model
BBH
ARC-e
ARC-c
HellaSwag
Avg
Avgen
Avgehn
Llama2-7B
33.23
75.25t
42.75
75.62*
35.40
36.21
31.77
Qwen-7B
37.75
83.42
64.76
75.32*
49.46
47.19
59.66
Deepseek-7B
33.38
74.58*
42.15*
75.45'
39.96
39.15
43.64
Mistral-7B
41.06
83.92
70.73
80.43*
48.97
49.96
44.54
Gemma-7B
39.19
89.35
76.79
79.47
52.22
54.18
43,39
Llama2-13B
37.92
78.87t
58.19
79.23*
41.48
42.44
37.19
MPT-30B
38.22
78.66*
46.08
79.72*
38.17
39.82
30.72
FaBcon-40B
36.24
81,94
57.68
83.26*
43.62
44.21
40.93
Llma--am
28.78
60.77*
28.15*
58.33*
25.36
25.55
24.53
Qwen-1.8B
29.07
63.97t
43.69
59.28t
34.72
31.87
47.57
Qwen1.5-1.88
28.82
64.86
45.56
59.39
37.09
33.57
52.93
Gemini Nano-3B
42.40
StableLM-Zephyr-33B
37.68
73.78
55.38
71.87
43.46
46.32
30.62
Phi-2(2B)
43.39
86.11
71.25
73.07
48.84
23.78
Gemma-2B
30.93
74.33
40.70
69.51
35.10
36.47
28.91
MiniCPM-1.2B 34.70
80.93
66.81
54.72
45.67
45.16
47.98
MiniCPM-2.4B 36.87
85.44
68.00
68.25
52.33
52.60
51.10
Table 2: Benchmark Score of MiniCPM-2.4B and MiniCPM-1.22 (both without RLHF). The two tables are continuous horizontally. Avg is over dataset in the table, Avgehn is the average of C-Eval and CMMLU while Avgen the average of remaining datasets, means the result is tested using PPL metrics (See Appendix G. Bold numbers represent the best score among the SLMs. Results of Gemini Nano-3B are borrowed from Gemini et al. (2023).

---

Published as a conference paper at COLM 2024
6.3 Training Loss
The overall training loss on the C4 dataset is shown in Figure 8. We can see that as expected in the preliminary experiments, the loss decreases sharply in the decay stage. Since we use the exponential decay, the loss still drops after the learning rate drops below 10% of the max learning rate. However, since we continue to SFT the model after the decay stage, we do not utilize the final checkpoints. The checkpoints we finetune from are shown in the last checkpoint of dark green segment. The first drop in MiniCPM-1122 is the result of enlarging batch size, which might have a similar effect as decreasing learning rate (Smith et al., 2017).
6.4 Evaluation
The overall evaluation utilizes our open-source tool UltraBval Ultralival is an open-source framework for assessing the capabilities of foundation models. It provides a lightweight and user-friendly evaluation system, supporting performance assessment for mainstream large models, and catering to the rapid evaluation needs of model training teams. The underlying inference and acceleration use the open-source framework vLLM (Kwon et al., 2023), and the dataset includes commonly used datasets: MMLU (Hendrycks et al., 2020) for English knowledge, CMMLU (Li et al., 2024) and C-Eval (Huang et al., 2024) for Chinese knowledge, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for coding, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for mathematics, and HellaSwag (Zellers et al., 2019), ARC-e (Clark et al., 2018), ARC-c (Clark et al., 2018) for commonsense reasoning, and BBH (Suzgun et al., 2022) for logic reasoning. More evaluation details can be found in Appendix G.
The overall evaluation results are in Table 7. Overall, on the mentioned datasets, have several observations.
(1) On average, MiniCPM-2244 ranks the highest among all the SLMs. (2) MiniCPM-2.4B performs similarly to Mistral-7B-v0.1 in English but significantly out- performs Mistral-7B-v0.1 in Chinese. (3) MiniCPM-224BB outperforms Llama2-13B except in MMLU, BBH, and HellaSwag, while MiniCPM-1.2B outperforms Llama2-7B except in HellaSwag. (4)Generally, BBH is harder for SLMs than LLMs compared to another knowledge-orienter dataset, demonstrating that reasoning ability might be more depen- dent on model size than knowledge. (5) Among SLMs, Phi-2 performance is on par with MiniCPM on academic-orienter datasets. This might be because their training data mostly involves textbook-style data that emphasize educational and academic scenarios. Since our pre-training data covers more distribution, we think MiniCPM is better at knowledge and ability coverage, which can be seen in Appendix K.
7 Conclusion
This paper introduces MiniCPM, comprising two SLMs with 2.4 B and 1.2 non-embedding parameters, respectively. These models demonstrate superior performance compared to their larger counterparis. Our training methodologies are scalable both in terms of model and data size, offering potential applicability in the development of LLMs. The introduction of our WSD scheduler is notable for promoting continuous training, exhibiting compelling training dynamics, and enabling efficient study of scaling law. We further introduce the MiniCPM family, including DPO, long context, and MoE versions. Future directions include in-depth analysis of the loss decrease the decay stage, and enhancing the capability of MiniCPM by scaling in both model size and data size.
Phtpp::/uutraevlllopmmbbbbb
10

---

Published as a conference paper at COLM 2024
Author Contributions
All authors contribute substantially to the MiniCPM project. Shengding Hu lead and participated all aspects the projects. This included the scaling experiments (conducted alongside Yuge Tu), babysitting the training of MiniCPM base models, and contributing to various other parts of the research. Shengding Hu wrote the paper. Chaoqun He was responsible for evaluating MiniCPM, while Ganqu Cui handled the RLHF training, Xiang Long, Zhi Zheng, Xinrong Zhang and Shengding Hu extended the context window to 128K. The MoE research was conducted by Yewei Fang and Zhi Zheng. Weilin Zhao and Kaihuo Zhang contributed the training and inference infrastructure. The open-sourcing of MiniCPM was prepared by Yuxiang Huang and Shengding Hu. Shengding Hu, along with Chenyang Zhao, also provided analysis on the WSD scheduler's 's training dynamics. Zheng Leng Thai developed the tokenizer. The development of MiniCPM-V was carried out by Chongyi Wang and Yuan Yao. The training corpus of MiniCPM was prepared by Jie Zhou, Jie Cai, Shengding Hu, Zhi Zheng, and Zhongwu Zhai. The paper was proofread by Xingrong Zhang and Chaoqun He. Insightful instructions on training MiniCPM were provided by Xu Han, Ning Ding, and Zhiyuan Liu. Finally, Zhiyuan Liu, Maosong Sun, Guoyang Zeng, Chao Jia, and Dahai Li offered essential resources for the training of MiniCPM.
Limitations
Although have proposed thorough study of the scaling law with SLMs, this paper does not extend to training an LLM to validate the scaling law. The application of WSD LRS on LLMs has not been fully explored to date. However, remain optimistic about its potential advantages.
Acknowledgement
This work is supported by the National Key R&D Program of China (No NN.22022700100001) and the National Natural Science Foundation of China (No. 62236004, 623B2065).
Scinncl
MiniCPM was initially published as technical blog on February 1st, 2024. Since then, we have received numerous insightful feedback from the community, significantly contributing to the development of this paper. We extend our gratitude to Chunting Zhou and Armen Aghajanyan for their valuable discussions. Special thanks go to Peiqin Sun and Yan Wang for their meticulous feedback on clarifying ambiguities in the blog. Additionally, the open-source community's efforts in integrating MiniCPM into inference frameworks like llama.cpp, etc.
appreciate
11

---

Published as a conference paper at COLM 2024
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint ar arXiv:2303.08774 2023.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine PMLR, 2023.
Learning, pp.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895 -900, Singapore, December 2023. Association for Computational Linguistics. doi: (0.18653/v1/ 2023. 1023.emnlp-malee maiin.298. URL Uttps http:::://caaaaa hcl.ooooooannn.ooo.oooooooo org/ emnlp-main. 298.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arvivv preprint (#Xie:2311.16867, 2023.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models, 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint ar arXiv:2309.16609, 2023.
Jeanine Banks and Tris Warkentin. Gemma: Introducing new state-of-the-art open models. https://blog hoollllggggggglo/ooooo httae///eeeooooggg///elleeeeeeee wewppeereeeeeeeaaoeee-eeeeeeeeee 2024. Accessed: date-of-access.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arv preprint ar arXiv:2401.022 002954, 2024.
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arv preprint arXiv:2201.0733 2022.
bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degrada- tion. https://www.rxt wwtp:ttwwwwdddoooo//oo/tttttt..t www.://oaaa/aLL//////////////
http:///mmmmttt/eeeeeeeeee /111t///////////// intkaware_scalln 2023. Accessed: [Insert Date of Access].
rope_allows lere_alloos__llmmmmddeee www.aaaaeeeeeeeeooooooeeeeeeeeee
Andrei Z. Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21-29. IEÉF, 1997.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. Xiv preprint ar Xiv:2111 .033744 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023.
12

---

Published as a conference paper at COLM 2024
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arv preprint ar arXii:1803.05455 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. Xiv preprint ar XXit:2110.14163 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high- quality feedback, 2023.
Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained the cerebras wafer-scale cluster. Xiv preprint arXiv:2304.03208, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint ar Xiv::2100. 10360, 2021.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arv preprint arXio:2403.15796 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post- training quantization for generative pre-trained transformers. CoRR, abs 22100.122333 2022. doi: 10.48550/ ARXIV:2210.17323 URL https://///do org/10 10. 38550/arXiv 2210. 17323.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arv preprint orXi:2201.00022, 2020.
Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: a family of highly capable multimodal models. ar preprint arXiv:2312.11855, 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. ar Xio preprint Xio:2206.11144, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Xrv preprint arXiv:2009.0330 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. Xiv preprint arXiv:2103.03874 2021.
Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query- key normalization for transformers. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4246-4253 Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/200 findings-emnlp.379. URL htttss:/acclatthlllo/ wwg/://0..020000 wttp://0..fiiinnneeeeeeoee wwndings--emnl..37...
13

---

Published as a conference paper at COLM 2024
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXio preprint ar xrXie:2203.15556, 2022.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. ar Xiv preprint AxXi::1801.06146 2018.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Unlock predictable scaling from emergent abilities, Xrviv preprint arXi::2310.0326,, 2023.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024.
Andrew Hundt, Varun Jain, and Gregory D Hager. differentiable architecture search. ar arv preprint ar arXi::1903009000 2019.
sharpdarts: Faster and more accurate
Mojan Javaheripi and Sébastien Bubeck.
Phi-2:
The surprising power of
small language models. attps://vnt wwwroooft com/ en -us uus.rueaarc/bblleeeeeeeeeeoeeeee models/ 2023. Accessed: date-of-
phi-2-the- hei-2-thh-surprisinne of access.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint ar arXiv:2310.06825, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural 2020.
language models. ar Xiv preprint ar
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar Xix::1412.69800 2014.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Fer- randis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: of permissively licensed source code. Preprint, 2022.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby: Sparse upcycling: Training mixture-of-experts from dense checkpoints. ar Xiv preprint ar arXiv:2212.050555, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2024.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arxi preprint ar Xte:2305.00661 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arxio preprint Xii::2309005433, 2023b.
14

---

Published as a conference paper at COLM 2024
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023a. /RR.httte://bttps////ugggg
co/Open CawaeOriaaaareeeeee
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, "Teknium", and Nathan Hoos. Slimorca dedup: A deduplicated subset of slimorca, 2023b. URL https: /tss:://uuuue https: htt.g/ggggggg....ceo co co/datasetss/peen.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mo- bilellm: Optimizing sub-billion parameter language models for on-device use cases. ar arviv preprint XrXiv22402.14905 2024.
Nla Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. Xiv preprint ar arXiv:1608.03995 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arviv preprint ar xxXt::7111.05001, 2017.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arviv preprint ar wwxx:22009000777 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arxiv preprint axXiv:2112.11446, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Adoances in Neural Information Processing Systems, 36, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Xiv e-prints, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. Xrv preprint ar Xi0:22401.00448, 2023.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/ 11611611166 URL htt.t://///// ReR..tttps::aclaathoooe /aapaaataaaannnoooc.///a arg/P16-110 1162.
Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah A Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. ar preprint arXiv:2310.10638, 2023.
Samuel Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. Xrv preprint ar Xix1:1711..0448, 2017.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Flazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Pe- ters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirz,, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. ar arviv preprint, 2024. URL https:: //trxivv.v.////////////////aaav/
15

---

Published as a conference paper at COLM 2024
Kaiqiang Song, Xiaoyang Wang, Sangwoo Cho, Xiaoman Pan, and Dong Yu. Zebra: Ex- tending context window with layerwise grouped local-global attention. arx preprint (rXiv:2312.08618, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.
LLMFarm team. LLMFarm, 2023a. UKt.attss//ggthhb....////mmo////
MLC team. MLC-LLM, 2023b. URL UKLittpss//gtthub../a//aaaa////p
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXio:2302.13971 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models zero-shot learners. ar Xiv preprint ar Xrix:22099.06522 2021.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code all you need. Xrv preprint arXiv:2312.02120 2023.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXix:2309.14 14322, 2023.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. ar Xiv preprint ar arXio:23100066 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Ábinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. ar Xiv preprint ar xrXiv:2309.16039 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXo:2230..12244, 2023.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer. ar preprint arXit:2203.03466 2022.
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. Xiv preprint arXiv:2310.02244 2023.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. Xiv preprint ar XrXiv:2403.16652, 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. ar Xiv preprint ar Xrxi::2403.0455 04652,, 2024.
16

---

Published as a conference paper at COLM 2024
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXio:1905.07830, 2019.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. Xiviv preprint ar Xiv:2401.002885, 2024a.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. Xrviv preprint ar Xiv: 10792, 2023.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. oobench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXio:2402.137118 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
17

---

Published as a conference paper at COLM 2024
A
Additional Results in Model Wind Tunnel Experiments
A.1 μPP hyper-parameter search
We conduct an extensive Bayesian search over set of predefined parametric spaces. For effi- ciency, we search for the N = 0.009B model. In our pilot experiments, we confirm that when hyper-parameter optimization is conducted using datasets scaled at magnitudes of 10N and 20N, there is a consistency observed in the efficacy of train the models with [D] 10N 0.09B tokens. Meanwhile, we also try QK-Norm (Henry et al., 2020) and independent weight decay (Loshchilov & Hutter, 2017) as well to stabilize the learning rate. The overall results are shown in Figure 9. After applying the QK-norm, we observe significant decrease in the learning rate sensitivity similar to Wortsman et al. (2023). However, as the MiniCPM project itself is an SLM, we do not require low learning rate sensitivity as long we find the best learning rate with TensorProgram (Yang et al., 2022; 2023). Therefore, we do not introduce QK-norm and independent weight decay in later experiments of MiniCPM. In Figure 9, we identify the best hyper-parameters for scale depth 1.4, scale emb = 12, init std 0.1, and Ir 0.01.
Therefore, we
0.00
Figure 9: Grid search over the uP parameterization spaces.
Name Embedding Output Scaling Residual Connection Scaling
Specific Operation
Multiply the output of the embedding by scale emb Scale the output tensor of a block before adding to each residual connection in each layer by scale_depth/ hadepthh/nuummle /numelayrss
Initialization of Tensors
Set the initialization standard deviation of each two- dimensional tensor parameter to init_std /dm/ / and set other parameters' initialization to 0.1 Adjust the learning rate of each two-dimensional tensor parameter to 1/ (dmz / times the learning rate of other parts (or the overall learning rate)
Learning Rate Scaling of Tensors
LM Head Scaling
Adjust the output logits to 1/(d, dbase) times the original value
Table 3: List of operations used when applying tensor program techniques.
18

---

Published as a conference paper at COLM 2024
A.2 Comment Optimal Batchsize
In Kaplan et al. (2020), OpenAl studies the relationship between the loss function and the number of tokens. In their experiments, they assume that consuming more steps is equivalent to consuming more time. Under this assumption, OpenAl defines critical batch size that achieves a certain loss without consuming too many steps or tokens. This rationale is valid if the experiments are provided with unlimited GPUs (at least within the scope of the experiments). Since GPUs are unlimited, enlarging batch size will not increase the single-step duration but will decrease the total number of steps. However, in our experiment, since we have a fixed resource (number of GPUs), we observe that doubling the batch size almost equals doubling the single-step time. Therefore, enlarging batch size to decrease total training steps has minimal effect on the total training time. In light of this observation, drop the goal of "not consuming too many steps" and turn towards minimizing the token quantity to achieve the lowest loss, instead.
The observation regarding the estimation of optimal batch size in relation to loss resem- bles the "Chicken-and-egs" paradox. Practically, there's often a preliminary estimate of the achievable loss for given model size, informed by prior knowledge of preliminary experiments. However, there is potential for the development of more refined estimation procedures in the future.
The optimal batch size and optimal learning rate are likely to be not independent. To overcome this correlation, we do a preliminary study on the learning rate first, then choose an optimal learning rate to do batch size experiment, and use batch size scaling to do the learning rate again. This is bit like the Coordinate Descent optimization method. However, more rigorous methods are welcomed in future work.
A.3 Model Architecture in Model Wind Tunnel Experiments
We list the model configuration used in the model wind tunnel experiments in Table The "shape" of the model, i.e., model width compared to model depth is kept similar possible to avoid any potential performance variation.
Name
N (B)
dm
dff
dp
11
L
9M
0.009
320
800
64
5
8
30M
0.036
512
1280
64
8
12
70M
0.066
640
1600
64
10
14
0.1B
0.109
768
1920
64
12
16
0.17B
0.166
896
2240
64
14
18
0.2B
0.241
1024
2560
64
16
20
0.5B
0.499
1344
3360
64
21
24
Table 4: Model configurations and training configurations of the models the scaling curve. N(B) represents the number of non-embedding parameters of the models, measured in billions.
B Additional Illustration on WSD LRS
B.1 Learning Rate Paradigm for Different LRSs
In this paper, we describe three kinds of LRSs, Cosine(T), CosineLoop(T), and WSD(T,D), Cosine and Cosine Loop take the form of the following:
An illustrative learning rate diagram for WSD and Cosine Scheduler is shown in Figure 10.
19

---

Published as a conference paper at COLM 2024
Cosine(Tis)
CosineLoop( (T;s)
{
W//, 8 W 0.91cos(TT)) + 0.1%, W<s<T 0.1%, 8 > T
w W {
0.91cos(TT)) + 0.1%, W <
WSSN IN;
WEERMNN AN)
0419
08:00
0009
Figure 10: Illustrative comparison be- Figure 11: We use two different function tween Cosine LRS and WSD LRS. The forms to fit the data scaling law achieved WSD LRS with different end steps share by WSD LRS and choose power law the same stable training stage.
the best fit.
B.2 Fitting the Data Scaling Law
In this section, we describe the fitted data scaling law for continue training with WSD LRS. Each point in Figure 11 is the end of the decay stage in WSD LRS with different end step. We try two function forms: exponential and polynomial. The fitted result shows that the polynomial scaling law is still the best for continue training.
B.3 Individual Figure for Model-Data Scaling Law
For each task and model, the scaling law L(N, DY's fitness with real loss values along the data axis is plotted in Figure 12.
B.4 Analysis of Llama2's Data-to-Model Ratio
As mentioned in Section E, analyze Llama2's Data-to-Model Ratio based their training loss curve. The extracted loss is plotted on the left of Figure 13. We convert the x-axis to computation Flops to compare the computed optimal regime on the right part of the Figure.
C MiniCPM's Vocabulary
Despite being small in parameter size, MiniCPM targets modeling diverse data distribution, excelling in English and Chinese. Therefore, our vocabulary is relatively large. For the 2.4B model, we use tokenizer consisting of 122,753 tokens (denoted by MiniCPMITokenizer 120K). This vocabulary is constructed from extensive and diverse language data, utilizing the sentencepiece library for Byte Pair Encoding (BPE) (Sennrich et al., 2016), and includes special symbols like traditional Chinese characters, rare characters, emojis, and special symbols such as Greek letters, Cyrillic letters, etc.
For the SLM, the embedding parameters will take up a lot of parameter space if the vocabu- lary is large. Therefore, for our 1.2B model, we use smaller vocab MiniCPMTTokenizer-70KK Compared to the MiniCPMMTokenizer- 120K tokenizer, have re-trained the tokenization
20

---

Published as a conference paper at COLM 2024
000015
1:11
01238
ILS3U5
real
3224
2.22
fitted
Cha
415-
3:3
039
3374
0.4
0000
18.2
0.50
02%
0.0
132
0.73
Clinse (Willlm))
319
6000
633
4124
300
0.50
088+
322
L10
0.505
332
341
0.00
2480
1.80
355
312
Clion
372
1.000
383
E305
15 Nimiom Thooe
05
24
1
Sumther Tobes
Nmmber Tibeu (0)
Nunsher Tchrax (I)
Number Tobrac
6.05-
0.7
000
0772
0.00
376
303
0m5
16
18
18
16
Figure 12: The fitted scaling law plotted along the data amount axis for each model and each task. The fitted result is satisfying except for the last checkpoints of the 0.11B and 0.25B model.
Figure 13: We extract the training loss data from Llama2 paper (left part) and estimate the Dapt in their paper using the right part. The straight lines plotted to compute optimal Nop estimate the optimal loss envelope assuming using WSD Scheduler.
21

---

Published as a conference paper at COLM 2024
26th Logerts Maaic552..
eLandd the Lave MicCFM4..30
0.7
0.5
000
0.5
Figure 14: Max Difference of Checkpoints.
on the same documents, while setting the max number of vocabs to 64,000. For the special characters, only add the traditional Chinese characters, emojis, and special symbols, but leave out the rare characters Chinese.
We conduct evaluations on 300,000 documents in Chinese, English, code, and academic papers that are not in the training set of the Tokenizer. The MiniCPM-120K tokenizer achieves the highest compression ratio (Bytes/Tokent
Baichuan2
ChatGLM2
Llama2
MiniCPM-120K
MiniCPM-70K
Vocab Size
125,696
64,794
32,000
122,753
73,440
Compression Rate (Bytes/
(lytes/Tokens) Tokens)
Chinese
3.64
3.54
1.87
3.73
3.56
English
4.12
4.02
3.78
4.14
4.02
Code
2.71
2.71
2.74
2.81
2.76
Paper
2.74
2.88
2.97
2.93
2.88
Average
3.30
3.29
2.84
3.40
3.31
Table 5: Compression ratio comparison.
D Analysis of the Decay Stage
In this section, we provide brief analysis of the loss drop in the decay stage, examining it through the prisms of checkpoint updates and gradient information. We calculate the maximum weight element update max;y (W/l (W((+1) w across all weight matrices the MiniCPM-2.4B (Introduced in Section F). As depicted in Figure 14, the updates exhibit robust correlation with the learning rate's magnitude. Notwithstanding the illustration of the two submodules ggate-proj and q-proj module of the 25th layer), this pattern is prevalent across every layer and submodule within the network. This observation may not be trivial: the model checkpoints experience significant updates preceding the learning rate's decay, yet the loss exhibits minimal reduction. Conversely, during the decay stage, despite less pronounced weight alterations, there is an accelerated decrease loss.
Further examination of the gradient data is undertaken by training 0.2B model, meticu- lously recording every step gradient information, and evaluating the differences between consecutive steps, thereby providing approximation of second-order gradient infor- mation. We treat the gradient at step as flattened vector ((!), and the parameter (also
flattened as a vector xf)) ) update between step and + is (t) = ((+1)) -x() The gradient norm take the 1.2 norm of the gradient [g( gradient inner product is gt (t),,
g (111) g
the cosine of the gradient's angle is given by as trajectory over high-dimension manifold, first order directional derivative along the
(())) OT Imaging the optimization process
22

---

Published as a conference paper at COLM 2024
g 1)+ wlt) v(
trajectory is computed as D1
and the second order directional derivative is
D2 [8 (1)))) g('))v()
D1, D2 enables an approximate estimation of the loss curvature on the
v
D2l The results of these statistics over time are shown in Figure 15. We
trajectory, K
(1+D))2
can see that the gradient norm diminishes during the decay phase, and upon commence- ment of this stage, the cosine between gradients predominantly assumes positive values, suggesting that in the decay phase, model parameters undergo consistent changes across steps. Concerning directional derivatives, it is remarkable that the first-order directional derivative diminishes exponentially with each step, aligning closely with the learning rate, while the second-order directional derivative exhibits a slight increase in magnitude. The curvature of the loss function also increases by magnitude, indicating the proximity to lo- cal optimum. These findings potentially offer deeper insight into the shape of optimization space, subject reserved för future exploration.
0.5
maciman gaalar
1.0
1.0
Contue vilee of i gallant' anal
0.4
0.5
0.5
0.3
0.2
-0.5
0.0
2300
6000
50000
100.00
0.0
30.0
4100
0020
S000
20000
40000
150M)
10000)
0101
Zyr onder divetioonal derieativee
10
magorian Tel onder dive vontool derication (og acale)
pammmmtee trayetery Duy wole))
0000
00000
310
0.012
1950
3000
Inemation
Docrtton
Figure 15: Gradient statistics over the training of 0.2B model using WSD LRS. The expo- nential decay stage begins at 8000 steps.
E
Measuring the Scaling Law with WSD LRS
Scaling laws serve as a fundamental guiding principle in the development of LLMs. Al- though these scaling laws exhibit variability in specific coefficients due to diverse configura- tions across model series, the compute optimal data-to-model ratio remains a meaningful metric across different scaling law functions, which "marginalizes" out the specific value of loss. Regarding this ratio, Kaplan et al. (2020) posit that a tenfold increase in model scale should equate to singlefold increase in data scale. Conversely, Hoffmann et al. (2022) argue for the same scaling rate between model size and data size. What's more, current models such LLama 2 (Touvron et al., 2023), train much more data than what Hoffmann et al. (2022) claims, still yielding considerable performance gain. Indicating higher data-to-model ratio.
This unaddressed uncertainty stems from the challenges inherent in training multiple models of varying sizes and data sizes in traditional scaling experiments. Previously, if the average cost of training one model size on one data size is C, then conducting the scaling experiments with model sizes and data sizes takes approximately O(m?)C.
In this section, we introduce the utilizaaion of the WSD scheduler as an effective approach to explore the scaling law with linear cost (O(mC)). Since the WSD scheduler has the advantage of arriving at the optimal loss of Cosine LRS after decaying from stable stage checkpoints of any step, without re-tra re-training the models from scratch to different amounts of tokens, thus making the scaling law measurement much more efficient along the data axis.
able to precisely measure the optimal scaling properties
23

---

Published as a conference paper at COLM 2024
We measure the scaling law along the data and model axes by training SLMs of 6 sizes ranging from 0.04B to 2B, each with decayed models starting from the checkpoint of 10N to 60N data during the stable training stage (N is the respective model size). The final loss is evaluated on five held-out evaluation datasets. To potentially compare the loss when the model uses different tokenizers, we take the average of loss by number of bytes instead of a number of tokens, following Achiam et al. (2023). The final loss of each pair of data size and model size is shown in the blue lines in Figure 12.
Then we fit the losses with model size N and data size D following Hoffmann et al. (2022) using scipy curvefit function:
L(N,D) = CNN R CDD +
The fitted curve along the data axis for each dataset and each checkpoint are shown in orange lines in Figure 12. Then we have the optimal model size Nopt, dataset size Dopt, given a fixed amount of compute C = 6ND (Rae al., 2021) as:
Nopt Dopt
K2 (6)
(3)
where K = ) and 11 f
aTP. The derivation of Nopt closely follows Hoffmann et al. (2022) by substituting D with 6N in Equation 2, and minimize L(N) given C. A similar way is adopted for Dopt. From Equation 3, when a = B, Nopt/ Dopt is a constant, supporting Hoffmann et al. (2022y's claim, and when < B, we should emphasize more on parameter scaling (Kaplan et al., 2020), and vise versa.
In our experiments, the fitted relationship between loss and N, D is shown in the contour plot of equal loss in Figure 17. The equation of fitted scaling law is shown in the first text box in each subplot. We can see that in all evaluation corpora, we have
a. More specifically, 0.29, & = 0.23, K² 0.01, 7] -0.10 (Note that N is under 10", D
on average, we have is under 109, and C is under 1018). Since a is slightly larger than B, this result shows that as the computation scale, we should slightly emphasize more on cata scaling than model scaling, which aligns with Hoffmann et al. (2022).
As for the concrete data-to-model ratio
we notice that there is huge gap in compute Nop!
Dept
optimal regime between ours and Hoffmann al. (2022) despite that the trend of Dept Nop with compute C is aligned between ours and theirs. Specifically, the data size should be 192 times larger than the model size on average, as opposed to 20 times in Hoffmann et al. (2022). We note that this aligns with the observation in Section 4.3 and Figure 6.
Nept Dap! we notice that their scaling
With respect to the large deviation from Chinchilla Optimal experiment was conducted in a not very recent configuration. To compare with more recent configuration such Llama2 (Touvron et al., 2023), we extract the training loss data from
Llama2 paper (left part) in Appendix Figure 13 and estimate the compute optimal Dagt in their paper using the right part of Figure 13. Since they use Cosine LRS, the loss is not optimal in the middle of the training, depicted by the concave curve during training in the right figure of Figure 13. We fill the concave part with a straight line to estimate the optimal loss envelope if they had used the WSD LRS. After that, the compute model size should roughly be the regime in which model's 's loss curve is about to intersect with larger model's loss curve. With this intuition, the 13B model is about to intersect with the 34B model at 10 EFlops (1018 Flops), and the 34B model is about to intersect with the 70B 10
Nopt
model 5 x 10S EFlops. Therefore, we estimate the Dopt be roughly 5x10 Now is 70 ~ 100. Therefore, under this approximate comparison, their data-model ratio is closer to ours. And our configuration can absorb more data into a smaller model compared to previous ones. However, we note that the above estimates are only a rough one.
6 which
6x34
24

---

Published as a conference paper at COLM 2024
Chappe
1.30
Eeglic 2M Siaeo
Chaase (i630))
1030
10
00.00
Chaamo Viei Qheped
$4.0
Regfii 200 (o0d)
Chaase (VVlaton)
174
Chiane Dies (hapen)
Conpres
Cucpple
Ouegrac
The result of scaling experiments with WSD Scheduler (above) and the fitted scaling curve (below). The x-axis is the computation Flops C = 6ND, each color of the line represents the same model with different computation Flops. We can see that smaller models are better than larger models when the Flops are small and worse when the Flops are large. Thus models different sizes will intersect with each other in the plot around the compute optimal regime.
Figure 16:
A larger data-to-model ratio means that we can absorb more data into smaller model than we previously thought, which is more efficient for inference and deployment. We hope WSD LRS will help more researchers explore L(N, D) with less effort and make the relationship clearer in LLMs.
Model Details
Vocabulary. We use two tokenizers of 122,753 vocabulary size for MiniCPM-2.4B and 73,440 vocabulary for MiniCPM-1.28. A small vocabulary for 1.2B favors efficiency without harming much performance. Details of the tokenizers are in Appendix C. Including the embedding parameters increases total parameters by 0.3B and 0.2B respectively.
Shared Input-output Layer. For SLM, the embedding takes up large parameter space. To make the model parameters smaller, we use the Embedding Sharing techniques for both MiniCPM-2.49 and MiniCPM-1.22.
25

---

Published as a conference paper at COLM 2024
Sajaak
Wikdewy
Cueppa (orn Fape)
14
6.50
000
602
6.15
Cougre tor
4-1.1
mm
Nossrilling Pananntes (1))
Senecsbeoiigg PARMAAAA (of)
Nasesaletiin Pacazatera (10))
Figure 17: The fit result of the scaling experiment with WSD Scheduler. The black dots horizontal line denote the decayed checkpoints in different compute within the same model size. The C in the legend is expressed Flops units for semantic completeness and does not use EFlops units. The C in Y-axis is in EFlops.
Deep-and-thin Network. We train MiniCPM-2.44B before training MiniCPM-1.2B. When training MiniCPM-2.4BB we adopt deeper and thinner architecture compared to Phi- (Javaheripi & Bubeck, 2023) (40 layers compared 32 layers). Recently, Liu et (2024) propose to train deep and thin networks for SLMs, which aligns with our perspective. Therefore, we further make the architecture deeper and thinner for MiniCPM-1.2B.
Group Query Attention. We train MiniCPM-2.4B without modification on the attention layer. Whereas we apply Group Query Attention (Ainslie et al., 2023) to MiniCPM-1.2BB inspired by Liu et al. (2024), to further reduce the parameters number.
Model
(B)
dm
dzf
d
f4g
1kke
Batch size (M)
Tokens (T)
MiniCPM-1.28
1,247,442,44 432
1,536
3,840
64
24
52
2M >> 4M
1.1T
MiniCPM-2.4B 2,442,057,984
2,304
5,760
64
36
36
40
4M
1.1T
Table 6: Model configurations for MiniCPM. N (B), dm dff, iff, da Mg. Bkur L, Batch size (M), Tokens (T) represents the number of non-embedding parameters of the model, model hidden dimension, feedforward layer bottleneck dimension, attention head dimension, number of queries, number key/valuee /values, number of layers, training batch size, total training tokens.
G Evaluation Details
UltraEval is an open-source framework for assessing the capabilities of foundation models. provides a lightweight and user-friendly evaluation system, supporting performance assessment for mainstream large models, and catering to the rapid evaluation needs of model training teams. The underlying inference and acceleration use the open-source framework vLLM (Kwon et al., 2023), and the dataset includes commonly used datasets: MMLU (Hendrycks et al., 2020) for English knowledge, CMMLU (Li et al., 2024) and C-Eval (Huang et al., 2024) for Chinese knowledge, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for coding, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for mathematics, and HellaSwag (Zellers et al., 2019), ARC-e (Clark et al., 2018),
26

---

Published as a conference paper at COLM 2024
-RC-c (Clark et al., 2018) for commonsense reasoning, and BBH (Suzgun et al., 2022) for logic reasoning.
Due to the difficulty of standardizing evaluations for large models and the lack of publicly available prompts and test codes for many models' evaluations, we try our best to adapt the evaluation methods to suit various model types. Specifically, input prompt during testing and adjust according to each model's 's appropriate input- output template. The evaluation scripts and prompts also open-source our repository, and we welcome developers to continually improve our evaluation methods.
start from standardized
When testing QA tasks (ARC-e, ARC-c, HellaSwag), two approaches typically employed. The first involves using Perplexity (PPL): we extend each option as the continuation of the question and use the PPL of the option as the selection criterion. The second is direct generation, where the model directly outputs answer options. We observe significant differences in results obtained using these two methods. MiniCPM performs perforrs similarly in direct generation and PPL tests, with better performance in direct generation. On the other hand, Mistral-7B-v0.1 performs better in PPL tests but exhibits poorer performance in direct generation. To address this phenomenon, when reporting the scores for each model, we adopt the score from the evaluation method that yields the highest score, ensuring fairness in comparison.
H Quantization
We conduct 4-bit quantization on our model. We do not quantize the parameters of em- bedding and layer normalization, since the performance of the model is sensitive to these parameters. Therefore, we only need to quantify each weight matrix. Denote the weight matrix as W Rdu We group every G consecutive parameter at the d dimension and form d G group. Then quantize each group of the parameters separately. For each group parameter w, we calculate the quantization scale and zero point as follows:
scale max(w) () miin w)
min (w)
23.
2eeo
24-1
scale
Group parameter are then quantized
t quant(w) round(
w
+ zero),
scale
where round operation round floating point to nearest integer. The dequantization opera- tion is approximately the reverse of the quantization method, which is
dequant(w)) = scale (=) zero).
dy
Finally, matrix W E Rouu Xiii is quantized to int4 w R dout xda, float scale E R dou 6 and
float zero E Iou m
To reduce the quantization loss, we adopt GPTQ (Frantar et al., 2022) to apply weight calibration. We sample calibration data X from similar distribution of SFT data. The quan- tization objective is minimize the disturbance of the quantization - dequant W) X)3? We follow GPTQ to quantize weight iteratively and update the remaining non-quantized weight by
87 Wg (eguant( (Ha').g)
H
49
where 9 is the quantization position in the current iteration while F denotes the remaining non-quantized weights. Hp is the hessian matrix of the objective function.
27

---

Published as a conference paper at COLM 2024
I MiniCPM Family
In this section, we introduce the other models that build on MiniCPM base models. Specif- ically, we trained the aligned model, long-context model, and MoE model for MiniCPM 2.4B.
L.1 MiniCPM-DPO
After SFT, we employ DPO (Rafailov et al., 2024) for human preference alignment of the model. During this stage, UltraFeedback (Cui et al., 2023) is utilized as the primary align- ment dataset, and a proprietary preference dataset is constructed enhance the model's code and mathematical capabilities. We conduct one epoch of DPO training with a learning rate of 1 X 10-5 and utilize Cosine LRS since we have pre-defined training step.
After applying DPO for preference alignment, the model's score on MTBench (Zheng al., 2024) increased from 6.89 after SFT to 7.25, surpassing even large models such as Llama2- 70B-Chat (see Figure 18). However, we also noticed that the performance on benchmarks slightly compromised, which is known as the alignment tax (Askell et al., 2021).
Model
C-Eval
CMMLU
MMLU
HumanEval
MBPP
GSMSK
MATH
ChatGLM2-6B
52.05
49.21
45.77
10.37
9.38
22.74
5.96
Mistral-7B-Instruut-r0..
38.06
36.96
53.56
29.27
39.34
28.73
3.48
Mistral-7B-Instruct-.0..
42.55
41.92
60.51
36.59
48.95
40.49
4.95
Qwen-7B-Chat
58.57
57.23
56.03
15.85
40.52
42.23
8.3
Yi-6B-Chat
70.88
71.11
62.95
14.02
28.34
36.54
3.88
Baichuan2-7B-CCn
53.28
53.50
53.00
21.34
32.32
25.25
6.32
Deepseek-7B-chat
46.95
49.72
51.67
40.85
48.48
48.52
4.26
Llama2-7B-Chat
34.54
32.64
47.64
14.02
27.40
21.15
2.08
MiniCPM-24B-DPO
48.64
48.37
53.05
51.22
48.01
53.37
9.86
Model
BBH
ARC-e
ARC-c
HellaSwag
Avg
Avgen
Avgehn
ChaiGLM2-6B
32.60
74.45
56.82
58.48*
37.98
35.17
50.63
Mistral-7B-instruu--01.11
39.52
81.61
63.99
73.47*
44.36
45.89
37.51
Mistrall7B-Inssruutt-0...
39.81
86.28
73.38
84.55*
50.91
52.83
42.24
Qwen-7B-Chat
37.34
64.44*
39.25*
74.52*
44.93
42.05
57.90
Yi-6B-Chat
37.43
84.89
70.39
74.60*
50.46
45.89
71.00
Baichuan2-7B-Chat
37.46
79.63
60.15
69.23*
44.68
42.74
53.39
Deepseek-7B-chat
35.70
76.85
63.05
76.68t
49.34
49.56
48.34
Llama2-7B-Chat
35.54
74.28
54.78
75.65*
38.16
39.17
33.59
MiniCPM-2.4B-DFO
36.22
85.02
68.17
65.67
51.60
52.29
48.51
Table 7: Benchmark scores for MiniCPM-2.4B-DPOO compared with larger chat models.
I.2 MiniCPM-128K
Tasks involving lengthy contexts depend on the implicit information within these contexts, circumventing the need for the extensive knowledge often absent in SLMs. In this section, we expand the context length MiniCPM-2.4B from 4,09 to 128,000 tokens, illustrating the capability of SLM to effectively process long contexts.
Initialization. For the initialization, we disable sharing embeddings between input and output, primarily to accommodate vocabulary parallelism essential for training with long context. The LM head initialized from the input embedding.
Training. Similar to MiniCPM, MiniCPM-224B-1888 utilizes the WSD as its learning rate scheduler and reuses the last checkpoint of the stable training stage of MiniCPM-2.4B. Concerning training data, categorize the dataset distribution detailed in Section 6.2 into "short data" and "long data". We classify books, wikis, and papers as "long data", and the other the "short data". The training comprises 44% long data and 56% short
28

---

Published as a conference paper at COLM 2024
Ta Srate"
MJ housh Pesecmme
MeaTT6-30
Misu Th Saasii
MPT--00--
Figure 18: MTBench score of MiniCPM-DPO-2.4B surpasses many models of larger size.
data for continued training. For the extension of long contexts, we apply Adjusted Base Frequency (ABF) (Xiong et al., 2023) in the 4K 32k range and employ NTK-Aware RoPE Scaling (bloc97, 2023) and curriculum learning from 32K to 128K. Both two stages involve future training. Furthermore,
indicated in Yi Tech Report (Young et al., 2024) and Zebra (Song et al., 2023), we use of synthetic long QA data that significantly enhances model performance in context-aware tasks.
Evaluation. We evaluate MiniCPM-2.444 ----PM----411188 128K in coBench (Zhang et al., 2024b), pioneering benchmark for long context evaluations. The tasks coBench (Zhang al., 2024b) extend beyond typical retrieval tasks and challenge the model with long context reasoning. We can see in Table 8, we achieve comparable results in Beatral-B--Instructe (ABF1000W) and outperform ChatGLM3-6BB128K despite being 2.5 times smaller.
Long Book
Long Book
Long
Long
Number String
KV Re- trieval
Book
Book
Model
Passkey
Choice
QA
QA
Sum
Eng
Chn
Eng
Eng
LWM-Text-128K
100
97.8
0.6
28.82
15.93
14.31
9,99
Yarm-Misttrl-77--128
92.71
56.61
0
27.95
15.49
9.55
9.06
Mistral-7B-Instrucc-
v0.2(ABF 1000w)
100
78.98
3.6
37.12
11.74
17.37
21.12
Yi-6B-200K
100
94.92
0
36.68
15.07
9.2
0.92
ChatGLM3-600118K
89.93
99.66
5.2
46.29
10.7
8.38
25.91
MiniCPM--24B-122EK
98.31
99.83
9
29.69
23,06
16.33
15.73
Avg w/o
Long Di- alogue
Math Cale
Math Find
Code Debug
Code Run
Model
Avg
Code &
QA Eng
Math
LWM--Text-128.
1.5
0
3.43
20.05
1
24.45
33.62
Yarn-Misttall-7b-128.
7.5
0
17.14
0.76
1.25
19.84
27.36
Mistral-7B-Instruet-
9.5
0
29.43
17.51
27.75
36.9
v0.2(ABF 1000w)
Yi-6B-200K
3.5
0
4.29
0.51
0.75
22.15
32.54
ChatGLM3-66-1228
6.5
0
8
5.33
25.58
36.57
MiniCPM-244B-122K
9.5
0
4.29
22.08
0
27.32
37.68
Table 8: MiniCPM-224B-128K result in coBench (Zhang et al., 2024b)
1.3 MiniCPM-MoE
We further extend the ability of MiniCPM using Mixture-of-Expert
29

---

Published as a conference paper at COLM 2024
Initialization. MiniCPM-MoE is initialized utilizing Sparse Upcycling (Komatsuzaki et al., 2022). The dense model checkpoint, derived from the stable phase of MiniCPM, undergoes transformation wherein each MLP layer is substituted by MoE layer. These new MoE layers are exact replicas of the original MLP layers from the dense checkpoint. The router parameters are randomly initialized following normal distribution with mean of 0 and variance of 0.01.
The number of total non-embedding parameters of MiniCPM-MoI is 13.6B. During training and inference, two out of eight experts are activated for each token, resulting in the number of activated parameters being approximately 4B. To prevent training from collapsing, an additional load balancing loss (Fedus et al., 2022) is applied to the final training objective. This auxiliary loss is multiplied by 0.01 which is large enough ensure balanced distribution of tokens assigned to different experts.
Routing Mechanism.
Training. Similar to MiniCPM, we employ WSD as our learning rate scheduler. Regarding the training data, we adhere strictly to the distribution specified in Section 6.2. The training batch size is maintained at 4M tokens during the stable training and decay stages and is reduced to 2M tokens during the SFT stage. The pre-training phase (including continue pre- train and decay stage) spans 130K steps, after which we notice diminishing improvement.
The benchmark results are detailed in Table 9.
Model
C-Eval
CMMLU
MMLU
HumanEval
MBPP
GSM8K
MATH
BBH
Llama2-34B
62.6
22.6
33.0
42.2
6.24
44.1
Deepseek-MoF (16B)
40.6
42.5
45.0
26.8
39.2
18.8
Mistral-7B
46.12
42.96
62.69
27.44
45.20
33.13
5.0
41.06
Gemma-7B
42.57
44.20
60.83
38.41
50.12
47.31
6.18
39.19
MiniCPPM-244B
51.13
51.07
53.46
50.00
47.31
53.83
10.24
36.87
MiniCPM-MoE (13.6B)
58.11
58.80
58.90
56.71
51.05
61.56
10.52
39.22
Table 9: Benchmark results of MiniCPM-MoE. means evaluation results on the full set of MBPP, instead of the hand-verified set (Austin et al., 2021). The evaluation results of Llama2-34B and Qwen1.5-78 are taken from their technical reports.
J Edge Device Benchmarking
After Int4 quantization in Appendix H, MiniCPM-2.4B'ss footprint reduced to 2GB, fa- cilitating deployment mobile edge devices. We adapted the model for Android and HarmonyOS using MLC-LLM (team, 2023b) and employed LLMFarr (team, 2023a) for adaptation to the iPhone system. This adaptation was tested various edge mobile devices.
It is important to emphasize that our efforts did not focus on optimization for mobile de- ployment, but rather on demonstrating the practicality of MiniCPM's inference capabilities on mobile platforms. We encourage further optimization and updates by the developer community to enhance the performance of large models like MiniCPM in mobile contexts.
The result is shown in Table 10, we can see that on the most advanced smartphone iPhone 15 Pro, the inference throughput as high as 18 tokens per second. In other devices, the inference throughput is also acceptable.
K Case Demonstration
A notable feature distinguishing the MiniCPM series from other robust SLMs like Phi (Li et al., 2023b) by Microsoft is that we train MiniCPM on general-purpose corpora, which guarantees the models' versatility and generalization across variety of tasks. In this section, we present several intriguing generation instances, typically achievable only by larger models, to demonstrate the capabilities of MiniCPM. Although extensive checks on whether specific cases exist in the training set, due to the diverse ways they can be expressed, we ensure that data similar to the test cases is intentionally added.
do not conduct
30

---

Published as a conference paper at COLM 2024
Phone Memory (GB)
Inference
SmartPhone
Operating System
Processor
Throughput
(token/s)
OPPO Find N3
Android 13
snapdragon Gen2
12
6.5
Samsung $23 Ultra
Android 14
snapdragon Gen2
12
6.4
snapdragon
Meizu M182Q
Android 11
888Plus
3.7
Xiaomi 12 Pro
Android
snapdragon Genl
8+3
3.7
Xiaomi Redmi K40
Android 11
snapdragon 870
3.5
Oneplus LE 2100
Android 13
snapdragon 870
12
3,5
Oneplus HD1900
Android 11
snapdragon 865
3.2
Oneplus HD1900
Android
snapdragon 855
3.0
Oneplus HD1905
Android
snapdragon 855
3.0
Oneplus HD1900
Android
snapdragon 855
3.0
Xiaomi M
Android
snapdragon 845
2.3
Huawei Nova 11SE
Harmony 4.0.0
snapdragon 778
12
1.9
Xiaomi 2IX
Android 9
snapdragon 835
1.3
iPhone Pro
iOS 17.2.1
A17 pro
18.0
iPhone
17.2.1
A16
15.0
iPhone Pro
iOS 16.5.1
A14
5.8
iPhone 12
iO5 17.2.1
A14
5.8
iPhone 11
iOS 16.6
A13
4.6
Table 10: Speed of MiniCPM-2.4B on different end devices.
Translation
Prompt:
Translate this sentence into French: "I am freshman Chinese, do you know how this sentence tth - (How simple crawler Python?)
MiniCPM-1.2B:
MiniCPM-24BBDPO.
suis un homme nouveau en chinois, vous êtes-vous courant de la manière dont cetle phrase est traduile: eréer un petit scraper weh simple Python
suis un nouvesu-né chinois, sais-tu comment on crée simple web crawler en Python ? Translation: "Je suis nouveau-nd en chinois, sais-tu comment on crée un simple web crawler en Python ?"
Figure 19: Examples on translation. We notice that MiniCPM-2.4B-DPP performs better in translating the technical part, but both models are unnatural in expressing "just starting to learn Chinese.". However, both models successfully understand the instruction with traps and generate French despite we do not include French corpus intentionally.
31

---

Published as a conference paper at COLM 2024
Math and Reasoning
Prompt: ST yo mit 430 H. aati 155-252---33
(If in the following equation, minus signs represent addition, and plus signs represent subtraction, give the result of the calculation: 15-25-12434-)
MiniCPM-11228
MiniCPM-224B-DPO:
10 TO/R 9, ME R 15+ + 12-34
30 5 1i 40 (
15-25-12-34
Mi 13 Ru 25
ME
PUbii 12-52
RE OF (5-3)
T ZA EUDDRI: 52
BI #ACNi : #1588188
(If the minus represents plus sign, then 25
(Li minus signs all represent addition, and plus signs actually represent subtraction, then equation can re-expressed as: 15-25+12-34.
rewrite equation
First, 13 + 40. Then, calculate
first numbers:
of the first three numbers:
calculate according to the order of addition and subtraction:
F'nally, calculate the difference between the last number and the sum of the first three numbers: 52 34 18.
(15+25)+12-34 40-12-34 52-34-18
So, the final result of the calculation is 18.)
Therefore, result the equation 18.)
Figure 20: Examples on math and reasoning. Both models are successful in symbolic substitution and math calculation.
Instruction Following
Prompt:
S EAG
(Writing an acrostic poem, with the hidden message "Good luck the Year of the Dragon.") ")
MiniCPM-1.2BB
MiniCPM-2.4B-DPO-
1880 3 "IAA..
#
10 22
(The spring iater in the dragon's pool rises, phoenix gate.
(Dragons soar for thousands of miles, their energy like a rainbow,
When will good fortune arrive.
As the new year begins, joyous occasions abound. Auspicious days worth enjoying logether, Good fortune and happiness bring rosy smiles.)
Peace prosperity hidden signs.
Figure 21: Examples instruction following. MiniCPM-2.4B-DDR successfully follows the instructions and generates a poem with the correct tonal pattern and the first character of each sentence. However, MiniCPM-1.2B only generates the correct tonal pattern.
32

---

Published as a conference paper at COLM 2024
Emoji
Prompt: W joon, - vralueeii I Mi MB 1
(Please give me JSON object where each key emoji and each value is the meaning of that emoji, providing
examples.)
MiniCPM-1.2B:
MiniCPM-2.4B-DPO-
#T BL 28 /h 1100 DE Zemoji. R
EL F
Temolk R
#1JSON:
#JJONNI
A
"
32 +
111 # (Ot course, this is JSON file containing around 10 emojiis their meanings:
MAAT f 19
"smiling",
"laughing",
I S 88 M RPP t u
du
"amazing",
"kissing",
JSON object containing emojis and
their corresponding meanings:
"mischief",
"disappointment",
means happy or satisfied, typically used press smiling and joy"
"thinkinng,
"It means anger dissatisfaction, typically used to intense emotions."
"money",
"help",
"It means laughing or happy, typically used to ex- press laughter and joy..
"rubbish"
means funny awkward, typically used to ex- press humorous situations or awlwwrddness."
Hope you found this JSON useful!)
"It means surprise or shock, typically used to ex- situation."
press a strong reaction
means disappointment discouragement, typ- ccaly used to express loss of interest or confidence in something someone.".
"It means heartache sadness, typically used express intense feelings pain caused by something or someone.".
"It means surprise or excitement, typically used to express the emotion discovery finding something someone.",
means cool stylish, typically confident, independent, and trendy attitude.". ",
express
"It means like or appreciate, typically used to ex- "
press affection admiration for something
hope this JSON object meets your neuds! If you have any questions, please feel free let me know.)
Figure 22: Examples emoji. Both models understand emojis and structured data well, and accurately control the number of items implicitly. The DPO model tends to generate more words.
33


Reviews:
[{"review_id": "OQEvqIRQgt", "reviewer": "Reviewer_n21G", "summary": "This work presents a suite of models called the MiniCPM models, which are language models of moderate size (~1.2B and ~2.4B parameters) that perform comparably to some of the strongest 7B models that are available (incl. Mistral-7B). Their main contribution is introducing a warmup-stable-decay (WSD) learning rate scheduler and analyze properties of this. They integrate WSD into their system and enjoy performance bumps. Finally they contribute an MoE model and long-context model based on the MiniCPM architecture.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful review and appreciation of MiniCPM. We are sorry about the confusion, and we will revise the paper carefully according to your suggestion.\n- More Careful Comparison Between WSD and Cosine\n    - We show that WSD can perform on par with cosine on loss on smaller models at any data scale, which already serves as an important reason for switching to the WSD scheduler. However, we can not immediately provide a very rigorous comparison when the training data is very large in the current version. We will leave it for future work.\n- Conflation of Contributions\n    - Thanks for pointing that out. Our paper mainly introduces mainly MiniCPM base models including their training strategy (WSD, hyper-parameter, two-stage training). These elements collectively contribute to the model's performance.\n    - We do think WSD is the most novel part. In the revision, we will make this core contribution clearer.\n    - For the other models, we mainly introduce them in the appendix (optional for reading). For these parts, it is more like a technical report that acknowledges all project members of our organization and also benefits relevant researchers.\n    - As for the vision part, we will remove the mention in the next revision. (During the submission of this work, we have not decided on whether to publish the vision paper).\n- Notation N's Meaning\n    - Yes, your understanding is correct. We will make it clearer in the next revision.\n- Writing Needs Improvement\n    - Thanks for pointing out the clear phrases. We have continued polishing the paper since our submission, and these improvements will be presented in the next revision.\n\n*Additionally, we list some of the recent implementations of our method in other powerful models as follows: (also see our comments to the last reviewer S8K9). We are impressed by the community's love of our method and the wide spread of it in such a short time.*\n```\n[1] Shen et al. JetMoE: Reaching Llama2 Performance with 0.1M Dollars https://arxiv.org/abs/2404.07413\n[2] Light-on.  Passing the Torch: Training a Mamba Model for Smooth Handover. https://www.lighton.ai/blog/lighton-s-blog-4/passing-the-torch-training-a-mamba-model-for-smooth-handover-54\n[3] Elie@Huggingface https://x.com/eliebakouch/status/1790772100585722088\n[4] Glorioso et al. Zamba: A Compact 7B SSM Hybrid Model. https://arxiv.org/pdf/2405.16712\n[5] MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series https://arxiv.org/pdf/2405.19327\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) WSD seems like quite a simple approach and can be integrated fairly easily into LM pretraining.\n\n2) Performance seems good in general, being on par with or close to models 3-4x the size.\n\n3) Analysis of the hyperparameters to get something that ends up working is good.\n\n4) Releasing a suite of models including an MoE, long context, and potentially a vision one (this was stated to maybe be a part of an upcoming paper, but also slated in here, which confused me).", "reasons_to_reject": "1) Although it seems like WSD works well on very small scales that you've tested, it would've been nicer to see how the exact same model with the exact same data order without WSD would do across the board. I think this could be computationally prohibitive, but that baseline would be nice to see.\n\n2) It seems like nearly all of the contribution here is the algorithm for WSD, but that's conflated with the fact that you're releasing a set of performant models, so its a little distracting and poorly organized. (minor point)\n\n3) There are some clarity issues that I expand a little bit more about in the questions to the authors. (minor point)", "questions_to_authors": "Clarity things\n\n1) N throughout is number of parameters of the model right? When you say S = 20N for a 1B model this would be 20B tokens of training, right? It would be great to make this more clear in the body of the text.\n\n2) The paper was hard to follow and there are a lot of awkward and/or redundant phrasing that made things unclear (e.g. in the batch size section 'the same number of GPUs is kept constant').", "extra_scores": {}}, {"review_id": "XWlLCp4xvN", "reviewer": "Reviewer_qVU8", "summary": "In this paper, the authors study the training dynamics of small language models (i.e., models ranging from 1.2-2.4 billion parameters).  Specifically, they focus on what they call “model wind tunnel” experiments. This involves training and studying model developing in 3 parts: 1) hyper-parameter selection (mostly focusing on network width and depth scaling parameters); 2) optimal batch sizing and; 3) learning rate scheduling. The most novel part of their work seems to be in the last part related to the development of what they call “warmup stable decay” (WSD), their learning rating scheduler that is conducive to continuous training and training without a fixed token length. \n\nThese experiments above lead to the following results (centering around experiments on C4): a new empirical bound on the optimal relationship between batch size and C4 loss that helps to minimize both token amount and loss (reported at the end of Section 3.2). Compelling empirical results on the efficacy of their WSD scheduling method (which breaks the scheduler into 3 stages: warmup, stable and decay stages, informed by systematic experiments they performed on cosine-warmup and its sensitivity to various step parameters); in particular, they report impressive losses with very small models on C4 (Figures 5 and 6). \n\nBased on these insights, they then train a family of small language models called miniCPM. Their training pipeline is comprehensive, ranging from full pre-training (at the scale of trillion of tokens using aggregations of multilingual publicly available pre-training corpora), to supervised instruction tuning and DPO-style alignment. They also tested across different architectures (e.g., MoE derivates and long-context versions of their models). The main novelty in this pre-training setting is the introduction of SFT data into the decay stage of their pre-training and scheduler (while I know this has become popular d recently, I worry that this makes their models have an unfair advantage over some of the other models they are comparing against; more about this below). \n\nThe results are impressive and, after looking through the details in the appendix, I was happy to see that not only were all the model comparisons produced by the authors themselves using a single tool, but that performance reported is based on each model’s best performance either through perplexity-based evaluation or direct generation. While I initially found Table 2 to be a bit confusing (highlighting would be very helpful here!), I was particularly struck by the relative average performance of 13-40B models against MiniCPM-2.4B.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the appreciation of our work. We apologize for any omissions in the initial submission and will address all the concerns in our revision to ensure clarity and accuracy.\n- Introduction of SFT Data into the Decay Stage\n  - Thank you for pointing this out. We believe that the comparison is not unfair due to the inclusion of SFT data during the decay stage. Current SOTA models often blur the boundaries between raw pretraining data and high-quality, human-curated data. As most models do not disclose their training data, it cannot be conclusively said that other models did not include such high-quality data. Given that most SOTA models release both base models and Chat models, we adhere to this convention. With the increasingly blurred boundaries between human-curated data and web-crawled data, we argue that directly comparing the SFT-ed models ensures a fairer comparison.\n  - We acknowledge that the comparison between Falcon-40B/MPT-30B and our model might be somewhat unfair concerning data differences. At the time of their training, it was less likely that they included high-quality data now being introduced. This comparison underscores that the advancements in pretraining techniques have significantly boosted model performance.\n- Model and License\n  - The model has been fully released, including intermediate checkpoints, under the Apache-2.0 license. However, due to anonymity constraints, I cannot directly point you to these resources.\n- Optimizer\n  - We use the Adam optimizer, and we will specify this in the revised manuscript.\n- SFT Data Mixture\n  - The data distribution for the decay stage, which includes the SFT data, is detailed in Appendix D.2, specifically in Figure 13. The SFT data consists of EvolInstruct, OssInstruct, SlimOrca, ShareGPT4, UltraChat, and a substantial portion of in-house SFT data covering logic, code, math, and knowledge.\n- Use \"Alignment\" Instead of \"RLHF\"\n  - You are absolutely correct. DPO is not RLHF, and we will make this correction in the revision.\n- Table 1 Unclear\n  - We will revise Table 1 to ensure it is clearer. Is there any particular point in Table 1 that you found especially unclear? Please let us know so we can specifically address it.\n\n*Additionally, we list some of the recent implementations of our method in other powerful models (See our comments to the last reviewer S8K9). We are impressed by the community's love of our training strategy and the wide spread of it in such a short time.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A very careful study on optimal model training strategies for small LLMs, which is an important area of research that, as the authors convincingly argue, touches on important issues related to practical deployment of LMs. \n\n- The paper is very easy to read and I found all the experiments to be well motivated. I believe that their experiments will motivate further research in this area. \n\n- A new suite of small LMs that I think will be used and embraced by the community (the question of whether these models will be released was not addressed; see questions below).", "reasons_to_reject": "- My main concern centers around the introduction of instruction data into the pre-training stage, and whether this makes the comparisons fair to many of the models in Table 2 (models that presumably didn't do this). While I don't think that this is grounds for a rejection, I would like to see the authors motivate and address this. If there were any ablations on this (maybe that I would missed), it would be helpful to report.  (Given that this was done in the most recent OLMO 1.7 model, this might be worthwhile to compare against)", "questions_to_authors": "- Will you models be released, and if so under what license? \n- I was surprised not to see (unless I missed it) details about the the particular optimizer you used. Can you clarify this? \n-  I was surprised to that your SFT stage includes training on 6 billion tokens. Did you report what sources you used? \n- you called the alignment stage RLHF, but as far as I can tell, you weren't using RL, because you were training with DPO, right? Please clarify.  \n\nother comments \n\n- The caption is Table 1 is very hard to read, please revise.", "extra_scores": {}}, {"review_id": "fCPbRFmwHo", "reviewer": "Reviewer_PFXg", "summary": "This paper investigates the capabilities of small language models (approximately 2 billion parameters), a critical area in current language model research. It introduces MiniCPM, a family of models with 1.2B and 2.4B parameters that exhibit competitive performance. This success is attributed to a thorough hyper-parameter scaling analysis on smaller-scale models and a novel training strategy employing a learning rate scheduler named Warmup-Stable-Decay. While the paper is generally well-written and provides detailed empirical results and discussions, some issues with notation clarity and figure presentation persist, and additional ablation studies are recommended to validate the effectiveness of the proposed methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sorry about the confusion, we will make them clear as much as possible in the revision. \n- Ambiguity.\n  -  Despite Equation 1 might help to understand WSD(80N, 8N), we will make further clarification as follows (and will be updated in the next revision):  WSD(T; S) means stable training on T tokens, and decay on S tokens, where T, S are usually represented relative to the model size N, i.e., KN tokens means the number of tokens is K times the number of parameters of the model.\n- Incorrect formula. Thanks for the careful review. We have revised the formula in our revision: $f(s-T) = 0.5^{(s-S)/T}$ \n- Unclear Figures\n  - We have revised the font size in our current revision.\n  - We will revise the color of the lines in our next revision.\n- Curve missing\n  - This curve is present in Figure 6. In Figure 6, we fit the optimal loss envelope for continuously training the 0.036B model. At $2.5\\times 10^{19}$ compute, the curve arrives at a 3.32 loss. The 0.17B model also arrives at a 3.32 loss with about $0.6\\times 10^{19}$ Flops. This information results in the conclusion you mentioned. We did not train the 0.036B models to $2.5\\times 10^{19}$ Flops in practice. \n- Ablation about the training stage\n  - We include the comparison that helps to show \"mixing SFT data at the beginning of annealing\" in Appendix C, Table 5. Where the models trained on SFT data but not at the beginning of annealing lag behind our proposed strategy significantly. We hope this addresses your concern.\n- Ablation of data mixture\n  - The success of MiniCPM undeniably involves the integral of SFT high-quality data. We have attempted annealing without any SFT data, and don't do SFT either, and the results, unsurprisingly, were unsatisfactory (e.g., MMLU~26). However, it is quite challenging to perform ablation studies on individual SFT datasets. We intend to leave the work on data mixture and data quality for future research.\n- optimal batch size and optimal learning rate\n  - They are likely to be not independent. To overcome this correlation, we do a preliminary study on the LR first, then choose an optimal learning rate to do a batch size experiment, and use batch size scaling to do the LR again. It is a bit like the Coordinate Descent optimization method. We will add this comment in the revision\n- code availability\n  - The training code is tightly integrated with in-house infra, which is not suitable for open source. However, the code of WSD and inference code are open-sourced publicly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Exploring the capabilities of small language models is significantly valuable for advancing research in both small and large language models.\n- The paper presents a comprehensive and effective training strategy for small language models, alongside clear decision-making insights derived from empirical results. These contributions are likely to inspire further research in the field.\n- The proposed model family shows impressive performance across a variety of benchmarks, even when compared to much larger models.", "reasons_to_reject": "- The paper contains ambiguities and possible errors in its notation, which impede understanding. For instance:\n  - This paper uses notations like \"WSD(80N,8N)'' multiple times. However, there is no definition about what the two items in parentheses represent.  $WSD(T,D)$ in Appendix B.1 is also not defined.\n  - In the \"Decay Stage\" section, the equation for exponential annealing $f(s-T)=\\eta \\times 0.5^{(s-S)/T}$ appears to be incorrect. The variable $\\eta$ seems out of place, and when $s=T$, the learning rate should equal $\\eta$.\n- Several figures are poorly presented:\n  - The labels in Figures 1 and 2 are too small to read.\n  - It is difficult to distinguish between the colors of different lines in Figures 5 and 6.\n  - Despite descriptions suggesting the presence of a curve showing the 0.036B model (with 4 times compute) matching the performance of the 0.17B model with WSD (40N, 4N), this curve is absent from Figure 6.\n- This paper highlights the benefits of introducing high-quality SFT data at the beginning of annealing. However, mixing SFT data with pretraining data after the pretraining could also improve the performance. That is to say, what may be important is the data mixing operation rather than the training stage. The ablation study did not shows whether mixing SFT data at the beginning of annealing is the best choice.\n- Most baselines in Table 2 do not use SFT, whereas MiniCPM is trained with both pretraining and SFT data. Additional discussion or experiments are needed to determine how much of MiniCPM’s success is due to the inclusion of SFT data.", "questions_to_authors": "- Are optimal batch size and optimal learning rate independent? Analyses in Section 3.2 and 3.3 seem to be done separately.\n- Will the training code, including the implementation of WSD, be made publicly available to facilitate reproducibility in research?", "extra_scores": {}}, {"review_id": "0YUgJHfKBU", "reviewer": "Reviewer_S8K9", "summary": "The manuscript presents a detailed study on the development and application of MiniCPM, a series of small language models (SLMs), which demonstrate comparable capabilities to much larger language models (LLMs) in the range of 7B-13B parameters. The authors have addressed a timely topic in the machine learning community, focusing on resource efficiency and practical utility of SLMs.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the appreciation of our work! We would like to clear up the confusion further you raised.\n\n- Emergence Pattern\n  - We assert that emergence may manifest similarly across both the model and data scales. More fundamentally, emergence patterns may be influenced by factors such as compute resources or pretraining loss/PPL [1]. This implies that a relatively SLM trained with extensive data might exhibit a comparable emergence pattern to that of an LLM trained with less data. Nevertheless, we acknowledge this as an insightful query: Is there any inherent difference in emergence patterns?  This question is worthy of future study. \n    - [1] Understanding Emergent Abilities of Language Models from the Loss Perspective \n- Insight transferability\n  - Regarding the \"insight gained from SLMs,\" our strategies are intentionally designed with scalability in mind, encompassing both model-wise scaling and data-wise scaling. Although our current resources restrict us from training a larger model, we firmly believe that the strategies presented in our paper can be effortlessly extended to train much larger models and applied to other models as well. In fact, we have observed several recent efforts that successfully implement our method in such a short time, underscoring its broader applicability. To name a few:\n\n```\n[1] Shen et al. JetMoE: Reaching Llama2 Performance with 0.1M Dollars https://arxiv.org/abs/2404.07413 \n[2] Light-on.  Passing the Torch: Training a Mamba Model for Smooth Handover. https://www.lighton.ai/blog/lighton-s-blog-4/passing-the-torch-training-a-mamba-model-for-smooth-handover-54\n[3] Elie@Huggingface https://x.com/eliebakouch/status/1790772100585722088\n[4] Glorioso et al. Zamba: A Compact 7B SSM Hybrid Model. https://arxiv.org/pdf/2405.16712\n[5] MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series https://arxiv.org/pdf/2405.19327\n```\n- Question about LLama3\n  - In a recent revision of this paper, we carefully measured the scaling law for our model. We found that current models can \"absorb\" much more data than the Chinchilla-Optimal amount. Specifically, the compute-to-data ratio for our model is now greater than 200. This finding is in line with the trend observed in models like Llama3 (adding more data is helpful).\n- Parameters count\n  - 2.7B parameters with embeddings.\n- Training resources\n  - For MiniCPM-2.4B itself, we utilize 160 A100 GPUs trained for 2 weeks (do not count in the preliminary studies).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper is well-motivated, addressing both the economic and practical challenges of deploying large models.\n2) The MiniCPM models are robustly presented, with detailed descriptions of the architecture, training methods, and innovative approaches like the WSD learning rate scheduler.\n3) The experimental section is comprehensive, demonstrating the models' performance across various benchmarks, which substantiates the authors' claims.", "reasons_to_reject": "1) Does the emergence pattern of Small Language Models (SLMs) mirror that of Large Language Models (LLMs)? If there are differences, can the insights gained from SLMs still be applicable for the development of LLMs?\n2) The recent advancements with models like Llama-3 suggest that smaller-scale models can achieve enhanced performance with increased training data. Does this paper offer any additional insights or conclusions in this regard?", "questions_to_authors": "1) What is the total model size when embedding parameters are included?\n2) The manuscript lacks a discussion on the utilization of hardware resources, such as GPUs. Could you provide details on this aspect?", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper the authors present a family of smaller LLMs in the 1.2-2.4B range, MiniCPM, that perform comparably to 7-11B parameter models. Their main contribution is the methodology for obtaining such performance using so many fewer parameters, which is achieved primarily though a new staged learning rate scheduler, warmup-stable-decay (WSD), designed for the continual training/adaptation setting, and a corresponding training recipe. The models have been released, including intermediate checkpoints, under the Apache-2.0 license. 

Reviewers agreed that the paper is reasonably well written and represents an interesting and impactful contribution, describing a new training recipe that allows for LLMs to perform competitively with respect to models that are 3-4x their size.
Decision: Accept