MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

Authors: shengding  hu
yuge  tu
xu  han
chaoqun  he
ganqu  cui
xiang  long
zhi  zheng
yewei  fang
yuxiang  huang
weilin  zhao
xinrong  zhang
zheng leng thai
kaihuo  zhang
chongyi  wang
yuan  yao
chenyang  zhao
jie  zhou
jie  cai
zhongwu  zhai
ning  ding
chao  jia
guoyang  zeng
dahai  li
zhiyuan  liu
maosong  sun
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 10:23:27

Abstract: The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly~\footnote{\url{https://github.com/OpenBMB/MiniCPM}}.

Main paper: 
 introduction: 
 Following the revelation of the scaling law (Kaplan et al., 2020) , there has been a vigorous pursuit in the field of Large Language Models (LLMs) (Hoffmann et al., 2022; Bai et al., 2023; Gemini et al., 2023; Chowdhery et al., 2023; Achiam et al., 2023) , encompassing models with up to an astonishing number of parameters in the trillions (Fedus et al., 2022) . These models have emerged as a pivotal driving force in the evolution of artificial intelligence.Nonetheless, the training of such large-scale models is both financially burdensome and operationally inefficient. On one hand, the empirical understanding of the mechanisms underpinning the training of LLMs remains elusive. Given the significant economic and environmental costs, experiments on LLMs are prohibitively expensive for most researchers and corporations. On the other hand, the deployment of these colossal models in everyday scenarios, such as on personal computers or smartphones, is either inefficient or unfeasible. Both aspects underscore the imperative to refocus efforts on comprehensively exploring smaller, yet potent, language models (SLMs). These models on the one hand provide efficient solutions to practical deployment, on the other hand, if trained with scalable strategies, they can potentially guide the development of future larger models.Recently, a resurgence of interest has been observed in the domain of SLMs, evidenced by the advent of a series of innovative models such as the Phi series (Gunasekar et al., 2023; Li et al., 2023b; Javaheripi & Bubeck, 2023) , TinyLlama (Zhang et al., 2024a) , MobileLLM (Liu et al., 2024) , and Gemma (Banks & Warkentin, 2024) , among others. While these models have significantly contributed to the expansion and diversification of the SLM landscape, there remain two pivotal areas where these models have yet to fully satisfy prevailing interests:(1) the development of comprehensive abilities akin to those exhibited by LLMs; and (2) the formulation of transparent and scalable training methodologies that could further propel the evolution of both SLMs and LLMs.In this paper, we introduce MiniCPM, a series of SLMs, which primarily builds on two models, endowed with 2.4B and 1.2B non-embedding parameters respectively, and they rank preeminently in their respective 2B and 1B scale categories. MiniCPM also exhibits comparable capabilities to those of 7B∼13B language models, such as Llama2-7B (Touvron et al., 2023) , Mistral-7B (Jiang et al., 2023) , Gemma-7B (Banks & Warkentin, 2024) , and Llama-13B (Touvron et al., 2023) , etc. Notwithstanding their small model sizes, our training methodology is meticulously designed to facilitate seamless scaling of both model scale and data horizons. This is exemplified through our model wind tunnel experiments that encompass comprehensive hyper-parameter optimization (Section 3), and the deployment of a WSD (Warmup-Stable-Decay) learning rate scheduler (Section 4). The latter is tailored for continuous training with an un-predefined pre-training token number and makes the reusing of model intermediate checkpoints highly feasible. A detailed analysis of the training dynamics of MiniCPM is presented, suggesting that the WSD scheduler demonstrates the intriguing loss landscape of model pre-training. With the WSD scheduler, we are now also capable of studying the data-model scaling law with linear effort on the model axis and a negligible effort on the data axis, while the traditional ones need quadratic effort considering the scaling along both model and data axes. The result of the scaling law indicates a much higher data size/model size ratio compared with Chinchilla Optimal (Hoffmann et al., 2022) . Moreover, we introduce the MiniCPM family, including MiniCPM-DPO, MiniCPM-128K, and MiniCPM-MoE. We conduct evaluations of the MiniCPM family against established benchmarks and illuminate their impressive capabilities as SLMs: (1) The foundation models surpass Mistral-7B, and LLama-13B. (2) The DPO model surpasses zephyr-7B (Tunstall et al., 2023) on MTBench (Zheng et al., 2024) (3) The 2.4B MiniCPM-128K model demonstrates performance either surpassing or matching that of models like Yarn-Mistral-7B-128K (Peng et al., 2023) and ChatGLM3-6B-128K (Du et al., 2021) . ( 4) The MiniCPM-MoE, with 4B activated parameters, is on par with Llama2-34B (Touvron et al., 2023) .In summary, MiniCPM propounds a new stage in the development of small language models, exemplifying the latent potential within SLMs and advocating for a more scientific and sustainable approach toward scaling up LLMs. 
related work: 
 Small Language Models. "Small Language Models" (SLMs) is an evolving concept that has undergone significant transformations over time. Presently, SLMs are generally construed as models that are smaller in scale compared to the well-known LLMs, typically not exceeding 7 billion parameters. These models are distinguished by their capacity for deployment on end-user devices, such as personal computers and smartphones, even in the absence of a GPU. Notable examples within the current landscape of SLMs include the Phi series (Gunasekar et al., 2023; Li et al., 2023b; Javaheripi & Bubeck, 2023) , TinyLlama (Zhang et al., 2024a) , MobileLLM (Liu et al., 2024), and Gemma (Banks & Warkentin, 2024) , etc. A variety of methodologies have been explored to augment the efficacy of SLMs. These include the incorporation of high-quality data (Gunasekar et al., 2023; Li et al., 2023b; Javaheripi & Bubeck, 2023) , the application of structure pruning techniques (Xia et al., 2023) , and the reconfiguration of model architectures (Liu et al., 2024) , among others. MiniCPM enhances the capabilities of SLMs through a meticulous amalgamation of hyper-parameter optimization, strategic training methodologies, architectural design, and high-quality data.Scalable Pre-training Strategies. Since the discovery of scaling law (Kaplan et al., 2020; Rae et al., 2021; Aghajanyan et al., 2023) , scientifically and predictably (Achiam et al., 2023; Hu et al., 2023; Du et al., 2024) scaling up the LLMs has been pursued from diverse perspectives, especially for the pre-training stage. In terms of training stability, the Tensor Program series (Yang et al., 2022; 2023) is introduced to ensure optimal hyper-parameter consistency across varying model scales, a technique employed in training CerebrasGPT (Dey et al., 2023) . Furthermore, Wortsman et al. (2023) suggest leveraging smaller models to anticipate and mitigate instabilities in larger model training. From the training data standpoint, various data-centric strategies have been advocated (Xie et al., 2024; Shi et al., 2023; Ye et al., 2024) . In the realm of training methodologies, prior research has delved into diverse learning rate schedulers (LRS) (Howard & Ruder, 2018; Raffel et al., 2020; Hundt et al., 2019) , with the Cosine LRS (Loshchilov & Hutter, 2016) emerging as the predominant choice in LLMs. Kaplan et al. (2020) and Hoffmann et al. (2022) have meticulously examined the hyper-parameters of Cosine LRS, thereby laying a foundational groundwork for subsequent pre-training works. Of these, DeepSeek (Bi et al., 2024) bears the closest resemblance to our proposed WSD LRS. Concerning batch size scheduling, Smith et al. (2017) advocates for incrementing batch size as an alternative to diminishing learning rate, a strategy recently adopted by Yi-9B (Young et al., 2024) . 
model wind tunnel experiments: 
 Although we target at training SLMs that can be quickly deployed onto end devices, we envision that many aspects of model training are universal across scales. Extensive experiments should be conducted through an SLM to explore the limit of SLM before transferring the experience into LLMs. These experiments take the spirit of wind tunnel testing in developing an aircraft, thus we name it Model Wind Tunnel Experiments (MWTE). In this paper, the MWTE contains three parts: (1) Hyper-parameters; (2) Optimal Batch-size Scaling; and (3) Optimal Learning Rate Stability.Hyper-parameters have a significant impact on the performance of a model. However, adjusting hyper-parameters for each model in traditional training is not feasible for LLMs. Even for SLM like MiniCPM, extensive experiments on hyper-parameters search take a lot of resources. Tensor Program (Yang et al., 2022; 2023) proposes a framework to stabilize the hyper-parameters for models with different scales. The main part of the Tensor Program is the width scaling (Yang et al., 2022) and the depth scaling (Yang et al., 2023) . The former technique supports CerebrasGPT (Dey et al., 2023) to predict the loss of LLMs more accurately. In MiniCPM, we use both two scaling techniques. The specific scaling operations are listed in Table 3 . We do not apply the attention softmax scaling techniques proposed in (Yang et al., 2022) (See Table 3 of their first version). Despite Yang et al. (2023) observing that depth scaling for a network with block depth larger than two is not satisfying, we find the resulting optimal learning rate is stable empirically. Details of the hyper-parameters and Tensor Program Operations are in Appendix A.1.Batch size determines the balance between the convergence speed of the model and the consumption of computational resources. If the batch size is too large, it will result in a significant amount of data and computation costs. On the other hand, if the batch size is too small, it will require a large number of training steps and may result in a limited decrease in the loss function. We follow Kaplan et al. (2020) to determine the batchsize from expected loss, with a slight modification from their setting (see Appendix A.2). We conduct experiments on 0.009B, 0.03B, and 0.17B models, respectively, toward this goal. Each model size is trained on 6 batch sizes with a global learning rate of 0.01 and cosine learning rate scheduler. We observe the trend of the optimal batch size with loss on the C4 (Raffel et al., 2019) dataset (red line in the Figure 1 ). As shown in Figure 1 , we plot the batch size in the x-axis, and token consumption in the y-axis, the color of the points represents a loss. Thus a vertical line formed by the color points denotes a training curve. we use parabolas to fit the equal-loss points and connect the minima of the parabolas with red lines. The lines demonstrate the optimal batch size shifts large as the loss decreases. We then connect the three lines (see Figure 2 ) and find that the lines connect each other well into a linear relationship in the log space, from which we obtain the following relationship between batch size bs and C4 Loss L: bs = 1.21×10 9 L 6.24 . We note that it might seem strange that the batch size should be estimated from a rough loss prediction that we can only have after training. We provide our comment in Appendix A.2.Due to our use of Tensor Program (Yang et al., 2022; 2023) , we anticipate that the learning rate, will not undergo significant changes during model scaling. To verify this, we conduct six sets of learning rate experiments at 0.04B, 0.1B, 0.3B, and 0.5B. In Figure 3 , we find that although the model size has increased by ten times, the optimal base learning rate2 does not show a noticeable shift and remains around 0.01. We further conduct a simple validation on a scale of 2.1B and confirm that a learning rate of 0.01 indeed achieves the lowest loss.  
analysing cosine lrs: 
 The learning rate scheduler (LRS), which adjusts the learning rate used in different stages of training, is crucial for model performance. The current commonly used learning rate strategy is the Cosine LRS (Kaplan et al., 2020; Hoffmann et al., 2022; Rae et al., 2021; Touvron et al., 2023; Bai et al., 2023; Almazrouei et al., 2023) , which gradually decreases the learning rate following a cosine curve after it reaches its maximum after the warmup stage.A key hyper-parameter in the Cosine LRS is the step T at which Cosine decreases to the minimum for the first time. EQUATIONwhere 0 < f (s -T) ≤ 1 is a decreasing function about s, η is the maximum learning rate. Typically, as long as the warmup stage is enough, it affects little performance, therefore, we omit W in the subsequent discussion. With an abuse of notation, we will denote WSD with a clear stop pointNext, we present several experimental findings of WSD LRS.Loss Decreases Dramatically in Decay Stage. We try WSD LRS on 0.036B models. As shown in Figure 5 , in the decay stage, as the learning rate begins to decrease, the loss experiences a significant rapid decline and quickly decreases to be equal to or lower than the Cosine LRS at step T = S. 6 , the green lines represent 0.036B models trained with different stable training tokens. Despite the last point of the 0.036B series being trained with many more tokens than Chinchilla Optimal (Hoffmann et al., 2022) , it still has space for performance improvement.To find the limit of continuously training this fixed-sized LM, we estimate how the model's optimal performance changes with its computation during continuous training. By optimal performance, we mean the loss of training token D is achieved by WSD(D, 0.1D). With a series of D, the losses will form the optimal loss envelope. Due to uncertainty about the function form of the loss envelope, we try two fitting formulas: (1) exponential:L(C) = αe -βC + L 0 and (2) power-law: L(C) = βC -α + L 0 .The fitting results for both functions are in Appendix B.2. We find that the power-law form fits better (similar to the Cosine LRS (Kaplan et al., 2020) ). In Figure 6 , the fitted curve is shown in green dotted lines. To intuitively estimate and comprehend the effect of continuous training such a fixed-sized model, we also trained a 0.17B model with WSD(40N, 4N), which is shown in pink in Figure 6 . We can see that a 0.036B model can match the performance of a 0.17B model with an acceptable increase (∼ 4 times) in training compute while saving a lot of inference computation (Sardana & Frankle, 2023) (saving ∼ 5 times per inference call), indicating a better inference-compute-optimal setting (Sardana & Frankle, 2023) .To deepen our understanding of WSD scheduler, we provide a detailed analysis of the model training dynamics in Appendix D. The WSD scheduler also helps us estimating the scaling law better and more efficiently, which can be found in Appendix E. 
two stage pre-training strategy: 
 Typically, the training of instruction following LLMs contains the pre-training stage and the supervised fine-tuning (SFT) stage (Zhang et al., 2023; Wei et al., 2021) . In the pretraining stage, the data is composed of large-scale unlabeled data, while in the SFT stage, high-quality labeled data becomes the optimization target. In light of the pronounced loss decrease observed during the decay stage of the WSD LRS, we postulate that the integration of high-quality labeled data in this phase presents dual advantages:• Introducing this data during the annealing phase, in addition to the SFT stage, fosters a more comprehensive model learning. Specifically, it facilitates a more pronounced loss reduction in relation to the SFT data distribution, rather than the pre-training data distribution. This approach is more congruent with actual user scenarios.• In contrast to a uniform distribution of high-quality data throughout the entire pre-training process, this method enhances training by concentrating on data and sustaining continuous pre-training. If we do not predetermine a training step, we will repeat a small dataset throughout an ongoing pre-training process, which could lead to negative effects.Based on these two hypotheses, we propose the following training strategy: during the pretraining phase, only use large-scale coarse-quality pre-training data, which is abundant and can support continuous training when provided with more computational resources. During the annealing phase, we use diverse and high-quality knowledge and ability-oriented SFT data, mixed into the pre-training data.To validate the advantages of our training strategy, we conduct comparison experiments using (A) MiniCPM-2.4B's intermediate checkpoint in the stable stage; and (B) MiniCPM-1.2B's last checkpoints in the stable stage. Specifically, we compare the following:1. A-1: 2.4B model, decay using only pre-training data, followed by 4B token SFT.2. A-2: 2.4B model, decay using the aforementioned high-quality data unlabeled data and SFT data mixed into pre-training data, also followed by 4B token SFT.3. B-1: 1.2B model, decay using only pre-training data, followed by 6B token SFT.4. B-2: 1.2B model, decay using only pre-training data, followed by 12B token SFT.5. B-3: 1.2B model, annealing using the aforementioned high-quality data + SFT data mixed into pre-training data, also followed by 6B token SFT.The results of the experiments are shown in Table 1 . We can see that, despite the A-2 and A-1 have undergone the same SFT distribution, adding SFT data to the decay stage pushes the boundary . Comparison between B-2 and B-3 demonstrate that the deficiency of only SFT is not due to the insufficient training tokens in SFT stage.A The results indicate that the benefits of introducing high-quality data at the beginning of the decay stage are much higher than simply adding it during the SFT phase. Therefore, we recommend that specialization and enhancement of model capabilities should start from the decay phase. 
model: 
 In this section, we begin to introduce the MiniCPM model that aggregates the aforementioned observations and techniques. Due to page limitation, the model architecture details are in Appendix F. The overall training of the MiniCPM base model includes three stages: stable training stage, decay stage, SFT stage (Zhang et al., 2023; Wei et al., 2021) . Throughout the stages, we use Adam Optimizer (Kingma & Ba, 2014) .We utilize around 1T data (see Section 7 for data distribution), with the majority of the data sourced from open datasets. We use the optimal configuration discovered during the model wind tunnel experiments, WSD LRS, with a batch size of 3.93 million and a max learning rate of 0.01. Decay Stage. We use a mixture of the pretraining data and high-quality SFT data. For the specific annealing form of the WSD scheduler, we employ exponential annealing, i.e. f (s -T) = 0.5 (s-S)/T , in which T is set to be 5000 steps (20B tokens).We find it still necessary to conduct a separate SFT phase. We utilize SFT data similar to the annealing phase excluding pre-training data and train with approximately 6 billion tokens. The learning rate for SFT is aligned with the one at the end of annealing, and a WSD Scheduler with exponential decay is also employed.We introduce our training data distribution in Figure 7 . In the figure, CommonCrawl Chn in a Chinese Corpus is derived from CommonCrawl raw corpus and goes through thorough cleaning. Dolma (Soldaini et al., 2024) , C4 (Raffel et al., 2019) , and Pile (Gao et al., 2020; Biderman et al., 2022) are English corpora. They are deduplicated inner corpus and across corpus using MinHash algorithms (Broder, 1997) . The Code Pre-train data contains the stack (Kocetkov et al., 2022) and StarCoder Li et al. (2023a) , with inner deduplication and cross deduplication. In the decay stage, the data mixture contains more diverse data and proprietary data, including UltraChat (Ding et al., 2023) , SlimOrca (Lian et al., 2023a; b) , OssInstruct (Wei et al., 2023) , EvolInstruct (Xu et al., 2023) . The data with the suffix SFT is our proprietary data including LeetCode questions, Kindergarten through 12th grade (K12) textbooks and questions, etc. The overall training loss on the C4 dataset is shown in Figure 8 . We can see that as expected in the preliminary experiments, the loss decreases sharply in the decay stage. Since we use the exponential decay, the loss still drops after the learning rate drops below 10% of the max learning rate. However, since we continue to SFT the model after the decay stage, we do not utilize the final checkpoints. The checkpoints we finetune from are shown in the last checkpoint of dark green segment. The first drop in MiniCPM-1.2B is the result of enlarging batch size, which might have a similar effect as decreasing learning rate (Smith et al., 2017) .The overall evaluation utilizes our open-source tool UltraEval3 . UltraEval is an open-source framework for assessing the capabilities of foundation models. It provides a lightweight and user-friendly evaluation system, supporting performance assessment for mainstream large models, and catering to the rapid evaluation needs of model training teams. The underlying inference and acceleration use the open-source framework vLLM (Kwon et al., 2023) , and the dataset includes commonly used datasets: MMLU (Hendrycks et al., 2020) for English knowledge, CMMLU (Li et al., 2024) and C-Eval (Huang et al., 2024) for Chinese knowledge, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for coding, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for mathematics, and HellaSwag (Zellers et al., 2019) , ARC-e (Clark et al., 2018) , ARC-c (Clark et al., 2018) for commonsense reasoning, and BBH (Suzgun et al., 2022) for logic reasoning. More evaluation details can be found in Appendix G.The overall evaluation results are in Table 7 . Overall, on the mentioned datasets, we have several observations. (1) On average, MiniCPM-2.4B ranks the highest among all the SLMs.(2) MiniCPM-2.4B performs similarly to Mistral-7B-v0.1 in English but significantly outperforms Mistral-7B-v0.1 in Chinese.(3) MiniCPM-2.4B outperforms Llama2-13B except in MMLU, BBH, and HellaSwag, while MiniCPM-1.2B outperforms Llama2-7B except in HellaSwag. (4)Generally, BBH is harder for SLMs than LLMs compared to another knowledge-oriented dataset, demonstrating that reasoning ability might be more dependent on model size than knowledge. (5) Among SLMs, Phi-2 performance is on par with MiniCPM on academic-oriented datasets. This might be because their training data mostly involves textbook-style data that emphasize educational and academic scenarios. Since our pre-training data covers more distribution, we think MiniCPM is better at knowledge and ability coverage, which can be seen in Appendix K. 
conclusion: 
 This paper introduces MiniCPM, comprising two SLMs with 2.4 B and 1.2 B non-embedding parameters, respectively. These models demonstrate superior performance compared to their larger counterparts.  
 
 Appendix: A.1 µP hyper-parameter searchWe conduct an extensive Bayesian search over a set of predefined parametric spaces. For efficiency, we search for the N = 0.009B model. In our pilot experiments, we confirm that when hyper-parameter optimization is conducted using datasets scaled at magnitudes of 10N and 20N, there is a consistency observed in the efficacy of hyper-parameters. Therefore, we train the models with |D| = 10N = 0.09B tokens. Meanwhile, we also try QK-Norm (Henry et al., 2020) and independent weight decay (Loshchilov & Hutter, 2017) as well to stabilize the learning rate. The overall results are shown in Figure 9 . After applying the QK-norm, we observe a significant decrease in the learning rate sensitivity similar to Wortsman et al. (2023) . However, as the MiniCPM project itself is an SLM, we do not require low learning rate sensitivity as long as we find the best learning rate with TensorProgram (Yang et al., 2022; 2023) . Therefore, we do not introduce QK-norm and independent weight decay in later experiments of MiniCPM. In Figure 9 , we identify the best hyper-parameters for scale depth = 1.4, scale emb = 12, init std = 0.1, and lr = 0.01. In Kaplan et al. (2020) , OpenAI studies the relationship between the loss function and the number of tokens. In their experiments, they assume that consuming more steps is equivalent to consuming more time. Under this assumption, OpenAI defines a critical batch size that achieves a certain loss without consuming too many steps or tokens. This rationale is valid if the experiments are provided with unlimited GPUs (at least within the scope of the experiments). Since GPUs are unlimited, enlarging batch size will not increase the single-step duration but will decrease the total number of steps. However, in our experiment, since we have a fixed resource (number of GPUs), we observe that doubling the batch size almost equals doubling the single-step time. Therefore, enlarging batch size to decrease total training steps has minimal effect on the total training time. In light of this observation, we drop the goal of "not consuming too many steps" and turn towards minimizing the token quantity to achieve the lowest loss, instead.The observation regarding the estimation of optimal batch size in relation to loss resembles the "Chicken-and-egg" paradox. Practically, there's often a preliminary estimate of the achievable loss for a given model size, informed by prior knowledge of preliminary experiments. However, there is potential for the development of more refined estimation procedures in the future.The optimal batch size and optimal learning rate are likely to be not independent. To overcome this correlation, we do a preliminary study on the learning rate first, then choose an optimal learning rate to do a batch size experiment, and use batch size scaling to do the learning rate again. This is a bit like the Coordinate Descent optimization method. However, more rigorous methods are welcomed in future work.We list the model configuration used in the model wind tunnel experiments in Table 4 .The "shape" of the model, i.e., model width compared to model depth is kept as similar as possible to avoid any potential performance variation. EQUATIONDespite being small in parameter size, MiniCPM targets modeling diverse data distribution, excelling in English and Chinese. Therefore, our vocabulary is relatively large. For the 2.4B model, we use a tokenizer consisting of 122,753 tokens (denoted by MiniCPMTokenizer-120K). This vocabulary is constructed from extensive and diverse language data, utilizing the sentencepiece library4 for Byte Pair Encoding (BPE) (Sennrich et al., 2016) , and includes special symbols like traditional Chinese characters, rare characters, emojis, and special symbols such as Greek letters, Cyrillic letters, etc.For the SLM, the embedding parameters will take up a lot of parameter space if the vocabulary is large. Therefore, for our 1.2B model, we use a smaller vocab MiniCPMTokenizer-70K.Compared to the MiniCPMTokenizer-120K tokenizer, we have re-trained the tokenization on the same documents, while setting the max number of vocabs to 64,000. For the special characters, we only add the traditional Chinese characters, emojis, and special symbols, but leave out the rare characters in Chinese.We conduct evaluations on 300,000 documents in Chinese, English, code, and academic papers that are not in the training set of the Tokenizer. The MiniCPM-120K tokenizer achieves the highest compression ratio (Bytes/Tokens). In this section, we provide a brief analysis of the loss drop in the decay stage, examining it through the prisms of checkpoint updates and gradient information. We calculate the maximum weight element update max ij (W(t+1) ij -W (t)ij ) across all weight matrices in the MiniCPM-2.4B (Introduced in Section F). As depicted in Figure 14 , the updates exhibit a robust correlation with the learning rate's magnitude. Notwithstanding the illustration of the two submodules (gate proj and q proj module of the 25th layer), this pattern is prevalent across every layer and submodule within the network. This observation may not be trivial: the model checkpoints experience significant updates preceding the learning rate's decay, yet the loss exhibits minimal reduction. Conversely, during the decay stage, despite less pronounced weight alterations, there is an accelerated decrease in loss.Further examination of the gradient data is undertaken by training a 0.2B model, meticulously recording every step gradient information, and evaluating the differences between consecutive steps, thereby providing an approximation of second-order gradient information. We treat the gradient at step t as a flattened vector g (t) , and the parameter (also flattened as a vector x (t) ) update between step t and t + 1 is v (t) = x (t+1)x (t) . The gradient norm take the L2 norm of the gradient ∥g (t) ∥, gradient inner product is g (t+1) • g (t) , the cosine of the gradient's angle is given byg (t+1) •g (t) ∥g (t+1) ∥∥g (t) ∥. Imaging the optimization process as a trajectory over a high-dimension manifold, first order directional derivative along the trajectory is computed asD 1 = g (t+1) •v (t) ∥v (t) ∥, and the second order directional derivative isD 2 = (g (t+1) -g (t) )•v (t) ∥v (t) ∥ 2. D 1 , D 2 enables an approximate estimation of the loss curvature on theEQUATION. The results of these statistics over time are shown in Figure 15 . We can see that the gradient norm diminishes during the decay phase, and upon commencement of this stage, the cosine between gradients predominantly assumes positive values, suggesting that in the decay phase, model parameters undergo consistent changes across steps. Concerning directional derivatives, it is remarkable that the first-order directional derivative diminishes exponentially with each step, aligning closely with the learning rate, while the second-order directional derivative exhibits a slight increase in magnitude. The curvature of the loss function also increases by a magnitude, indicating the proximity to a local optimum. These findings potentially offer a deeper insight into the shape of optimization space, a subject reserved for future exploration. Scaling laws serve as a fundamental guiding principle in the development of LLMs. Although these scaling laws exhibit variability in specific coefficients due to diverse configurations across model series, the compute optimal data-to-model ratio remains a meaningful metric across different scaling law functions, which "marginalizes" out the specific value of loss. Regarding this ratio, Kaplan et al. (2020) In this section, we introduce the utilization of the WSD scheduler as an effective approach to explore the scaling law with linear cost (O(mC)). Since the WSD scheduler has the advantage of arriving at the optimal loss of Cosine LRS after decaying from stable stage checkpoints of any step, we are now able to precisely measure the optimal scaling properties without re-training the models from scratch to different amounts of tokens, thus making the scaling law measurement much more efficient along the data axis.We measure the scaling law along the data and model axes by training SLMs of 6 sizes ranging from 0.04B to 2B, each with 6 decayed models starting from the checkpoint of 10N to 60N data during the stable training stage (N is the respective model size). The final loss is evaluated on five held-out evaluation datasets. To potentially compare the loss when the model uses different tokenizers, we take the average of loss by a number of bytes instead of a number of tokens, following Achiam et al. (2023) . The final loss of each pair of data size and model size is shown in the blue lines in Figure 12 .Then we fit the losses with model size N and data size D following Hoffmann et al. ( 2022) using scipy curvefit function:L(N, D) = C N N -α + C D D -β + L 0 (2)The fitted curve along the data axis for each dataset and each checkpoint are shown in orange lines in Figure 12 . Then we have the optimal model size N opt , dataset size D opt , given a fixed amount of compute C = 6ND (Rae et al., 2021) as: 6N in Equation 2, and minimize L(N) given C. A similar way is adopted for D opt . From Equation 3, when α = β, N opt /D opt is a constant, supporting Hoffmann et al. ( 2022)'s claim, and when α < β, we should emphasize more on parameter scaling (Kaplan et al., 2020) , and vise versa.EQUATIONIn our experiments, the fitted relationship between loss and N, D is shown in the contour plot of equal loss in Figure 17 . The equation of fitted scaling law is shown in the first text box in each subplot. We can see that in all evaluation corpora, we have β < α. More specifically, on average, we have α = 0.29, β = 0.23, K 2 = 0.01, η = -0.10 (Note that N is under 10 9 , D is under 10 9 , and C is under 10 18 ). Since α is slightly larger than β, this result shows that as the computation scale, we should slightly emphasize more on data scaling than model scaling, which aligns with Hoffmann et al. (2022) .As for the concrete data-to-model ratio D opt N opt , we notice that there is a huge gap in compute optimal regime between ours and Hoffmann et al. (2022) despite that the trend of D opt N opt ' with compute C is aligned between ours and theirs. Specifically, the data size should be 192 times larger than the model size on average, as opposed to 20 times in Hoffmann et al. (2022) . We note that this aligns with the observation in Section 4.3 and Figure 6 .With respect to the large deviation from Chinchilla Optimal N opt D opt , we notice that their scaling experiment was conducted in a not very recent configuration. To compare with more recent configuration such as Llama2 (Touvron et al., 2023) , we extract the training loss data from Llama2 paper (left part) in Appendix Figure 13 and estimate the compute optimal D opt N opt in their paper using the right part of Figure 13 . Since they use Cosine LRS, the loss is not optimal in the middle of the training, depicted by the concave curve during training in the right figure of Figure 13 . We fill the concave part with a straight line to estimate the optimal loss envelope if they had used the WSD LRS. After that, the compute model size should roughly be the regime in which a model's loss curve is about to intersect with a larger model's loss curve. With this intuition, the 13B model is about to intersect with the 34B model at 10 5 EFlops (10 18 Flops), and the 34B model is about to intersect with the 70B model at 5 × 10 5 EFlops. Therefore, we estimate the D opt N opt to be roughly 5×10 5 6×34 2 ∼ 10 5 6×13 2 , which is 70 ∼ 100. Therefore, under this approximate comparison, their data-model ratio is closer to ours. And our configuration can absorb more data into a smaller model compared to previous ones. However, we note that the above estimates are only a rough one. Chinese (Yayi Corpus)10 -1 10 0 10 1 10 2 10 3Compute5 × 10 -1 6 × 10 -1 7 × 10 -1 8 × 10 -1 9 × 10 -1Average 10 -2 10 -1 10 0 10 1 10 2 10 3 10 4 10 5 10 610 0 2 × 10 -1 3 × 10 -1 4 × 10 -1 6 × 10 -1Code 10 -2 10 -1 10 0 10 1 10 2 10 3 10 4 10 5 10 610 0 4 × 10 -1 6 × 10 -1English (Wikihow)10 -2 10 -1 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 0 6 × 10 -1Chinese (Wikihow)10 -2 10 -1 10 0 10 1 10 2 10 3 10 4 10 5 10 6Compute10 0 3 × 10 -1 4 × 10 -1 6 × 10 -1Ultratext 10 -2 10 -1 10 0 10 1 10 2 10 3 10 4 10 5 10 6Compute10 0 6 × 10 -1Chinese (Yayi Corpus) We can see that smaller models are better than larger models when the Flops are small and worse when the Flops are large. Thus models of different sizes will intersect with each other in the plot around the compute optimal regime.A larger data-to-model ratio means that we can absorb more data into a smaller model than we previously thought, which is more efficient for inference and deployment. We hope WSD LRS will help more researchers explore L(N, D) with less effort and make the relationship clearer in LLMs.Vocabulary. We use two tokenizers of 122,753 vocabulary size for MiniCPM-2.4B and 73,440 vocabulary for MiniCPM-1.2B. A small vocabulary for 1.2B favors efficiency without harming much performance. Details of the tokenizers are in Appendix C. Including the embedding parameters increases total parameters by 0.3B and 0.2B respectively.Shared Input-output Layer. For SLM, the embedding takes up a large parameter space. To make the model parameters smaller, we use the Embedding Sharing techniques for both MiniCPM-2.4B and MiniCPM-1.2B. 3.32×10 -2 N 0.37 + 2.17×10 -1 D 0.34 + 0.17 UltraEval is an open-source framework for assessing the capabilities of foundation models. It provides a lightweight and user-friendly evaluation system, supporting performance assessment for mainstream large models, and catering to the rapid evaluation needs of model training teams. The underlying inference and acceleration use the open-source framework vLLM (Kwon et al., 2023) , and the dataset includes commonly used datasets: MMLU (Hendrycks et al., 2020) for English knowledge, CMMLU (Li et al., 2024) and C-Eval (Huang et al., 2024) for Chinese knowledge, HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for coding, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for mathematics, and HellaSwag (Zellers et al., 2019) , ARC-e (Clark et al., 2018) , ARC-c (Clark et al., 2018) for commonsense reasoning, and BBH (Suzgun et al., 2022) for logic reasoning.Due to the difficulty of standardizing evaluations for large models and the lack of publicly available prompts and test codes for many models' evaluations, we try our best to adapt the evaluation methods to suit various model types. Specifically, we start from a standardized input prompt during testing and adjust it according to each model's appropriate inputoutput template. The evaluation scripts and prompts are also open-source in our repository, and we welcome developers to continually improve our evaluation methods.When testing QA tasks (ARC-e, ARC-c, HellaSwag), two approaches are typically employed. The first involves using Perplexity (PPL): we extend each option as the continuation of the question and use the PPL of the option as the selection criterion. The second is direct generation, where the model directly outputs answer options. We observe significant differences in results obtained using these two methods. MiniCPM performs similarly in direct generation and PPL tests, with better performance in direct generation. On the other hand, Mistral-7B-v0.1 performs better in PPL tests but exhibits poorer performance in direct generation. To address this phenomenon, when reporting the scores for each model, we adopt the score from the evaluation method that yields the highest score, ensuring fairness in comparison.We conduct 4-bit quantization on our model. We do not quantize the parameters of embedding and layer normalization, since the performance of the model is sensitive to these parameters. Therefore, we only need to quantify each weight matrix. Denote the weight matrix as W ∈ R d out ×d in . We group every G consecutive parameter at the d in dimension and form d in /G group. Then we quantize each group of the parameters separately. To reduce the quantization loss, we adopt GPTQ (Frantar et al., 2022) to apply weight calibration. We sample calibration data X from a similar distribution of SFT data. The quantization objective is to minimize the disturbance of the quantization ∥WXdequant( Ŵ)X∥ 2 2 . We follow GPTQ to quantize weight iteratively and update the remaining non-quantized weight byδ F = - w q -dequant(quant(w q )) [H -1 F ] qq • (H -1 F ) :,q ,where q is the quantization position in the current iteration while F denotes the remaining non-quantized weights. H F is the hessian matrix of the objective function. data for continued training. For the extension of long contexts, we apply Adjusted Base Frequency (ABF) (Xiong et al., 2023) in the 4K to 32k range and employ NTK-Aware RoPE Scaling (bloc97, 2023) and curriculum learning from 32K to 128K. Both two stages involve future training. Furthermore, as indicated in Yi Tech Report (Young et al., 2024) and Zebra (Song et al., 2023) , we use of synthetic long QA data that significantly enhances model performance in context-aware tasks.Evaluation. We evaluate MiniCPM-2.4B-128K in ∞Bench (Zhang et al., 2024b) , a pioneering benchmark for long context evaluations. The tasks in ∞Bench (Zhang et al., 2024b) extend beyond typical retrieval tasks and challenge the model with long context reasoning. We can see in Table 8 , we achieve comparable results in Mistral-7B-Instruct-v0.2 (ABF1000w) and outperform ChatGLM3-6B-128K despite being 2.5 times smaller. Je suis un homme nouveau en chinois, vous êtes-vous au courant de la manière dont cette phrase est traduite : Comment créer un petit scraper web simple à Python ?Je suis un nouveau-né en chinois, sais-tu comment on crée un simple web crawler en Python ? Translation: "Je suis un nouveau-né en chinois, sais-tu comment on crée un simple web crawler en Python ?"Figure 19 : Examples on translation. We notice that MiniCPM-2.4B-DPO performs better in translating the technical part, but both models are unnatural in expressing "just starting to learn Chinese.". However, both models successfully understand the instruction with traps and generate French despite we do not include French corpus intentionally.The actual learning rate of 2-D tensors will be scaled according to Tensor Program.https://ultraeval.openbmb.cn/homehttps://github.com/google/sentencepiece 


Decision: In this paper the authors present a family of smaller LLMs in the 1.2-2.4B range, MiniCPM, that perform comparably to 7-11B parameter models. Their main contribution is the methodology for obtaining such performance using so many fewer parameters, which is achieved primarily though a new staged learning rate scheduler, warmup-stable-decay (WSD), designed for the continual training/adaptation setting, and a corresponding training recipe. The models have been released, including intermediate checkpoints, under the Apache-2.0 license. 

Reviewers agreed that the paper is reasonably well written and represents an interesting and impactful contribution, describing a new training recipe that allows for LLMs to perform competitively with respect to models that are 3-4x their size.

Reviews: review: paper_summary: paper_summary: This work presents a suite of models called the MiniCPM models, which are language models of moderate size (~1.2B and ~2.4B parameters) that perform comparably to some of the strongest 7B models that are available (incl. Mistral-7B). Their main contribution is introducing a warmup-stable-decay (WSD) learning rate scheduler and analyze properties of this. They integrate WSD into their system and enjoy performance bumps. Finally they contribute an MoE model and long-context model based on the MiniCPM architecture.
strength_weakness: strength_weakness: 1) Although it seems like WSD works well on very small scales that you've tested, it would've been nicer to see how the exact same model with the exact same data order without WSD would do across the board. I think this could be computationally prohibitive, but that baseline would be nice to see.

2) It seems like nearly all of the contribution here is the algorithm for WSD, but that's conflated with the fact that you're releasing a set of performant models, so its a little distracting and poorly organized. (minor point)

3) There are some clarity issues that I expand a little bit more about in the questions to the authors. (minor point)
questions: questions: Clarity things

1) N throughout is number of parameters of the model right? When you say S = 20N for a 1B model this would be 20B tokens of training, right? It would be great to make this more clear in the body of the text.

2) The paper was hard to follow and there are a lot of awkward and/or redundant phrasing that made things unclear (e.g. in the batch size section 'the same number of GPUs is kept constant').
paper_summary: paper_summary: In this paper, the authors study the training dynamics of small language models (i.e., models ranging from 1.2-2.4 billion parameters).  Specifically, they focus on what they call “model wind tunnel” experiments. This involves training and studying model developing in 3 parts: 1) hyper-parameter selection (mostly focusing on network width and depth scaling parameters); 2) optimal batch sizing and; 3) learning rate scheduling. The most novel part of their work seems to be in the last part related to the development of what they call “warmup stable decay” (WSD), their learning rating scheduler that is conducive to continuous training and training without a fixed token length. 

These experiments above lead to the following results (centering around experiments on C4): a new empirical bound on the optimal relationship between batch size and C4 loss that helps to minimize both token amount and loss (reported at the end of Section 3.2). Compelling empirical results on the efficacy of their WSD scheduling method (which breaks the scheduler into 3 stages: warmup, stable and decay stages, informed by systematic experiments they performed on cosine-warmup and its sensitivity to various step parameters); in particular, they report impressive losses with very small models on C4 (Figures 5 and 6). 

Based on these insights, they then train a family of small language models called miniCPM. Their training pipeline is comprehensive, ranging from full pre-training (at the scale of trillion of tokens using aggregations of multilingual publicly available pre-training corpora), to supervised instruction tuning and DPO-style alignment. They also tested across different architectures (e.g., MoE derivates and long-context versions of their models). The main novelty in this pre-training setting is the introduction of SFT data into the decay stage of their pre-training and scheduler (while I know this has become popular d recently, I worry that this makes their models have an unfair advantage over some of the other models they are comparing against; more about this below). 

The results are impressive and, after looking through the details in the appendix, I was happy to see that not only were all the model comparisons produced by the authors themselves using a single tool, but that performance reported is based on each model’s best performance either through perplexity-based evaluation or direct generation. While I initially found Table 2 to be a bit confusing (highlighting would be very helpful here!), I was particularly struck by the relative average performance of 13-40B models against MiniCPM-2.4B.
strength_weakness: strength_weakness: - My main concern centers around the introduction of instruction data into the pre-training stage, and whether this makes the comparisons fair to many of the models in Table 2 (models that presumably didn't do this). While I don't think that this is grounds for a rejection, I would like to see the authors motivate and address this. If there were any ablations on this (maybe that I would missed), it would be helpful to report.  (Given that this was done in the most recent OLMO 1.7 model, this might be worthwhile to compare against)
questions: questions: - Will you models be released, and if so under what license? 
- I was surprised not to see (unless I missed it) details about the the particular optimizer you used. Can you clarify this? 
-  I was surprised to that your SFT stage includes training on 6 billion tokens. Did you report what sources you used? 
- you called the alignment stage RLHF, but as far as I can tell, you weren't using RL, because you were training with DPO, right? Please clarify.  

other comments 

- The caption is Table 1 is very hard to read, please revise.
paper_summary: paper_summary: This paper investigates the capabilities of small language models (approximately 2 billion parameters), a critical area in current language model research. It introduces MiniCPM, a family of models with 1.2B and 2.4B parameters that exhibit competitive performance. This success is attributed to a thorough hyper-parameter scaling analysis on smaller-scale models and a novel training strategy employing a learning rate scheduler named Warmup-Stable-Decay. While the paper is generally well-written and provides detailed empirical results and discussions, some issues with notation clarity and figure presentation persist, and additional ablation studies are recommended to validate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: - The paper contains ambiguities and possible errors in its notation, which impede understanding. For instance:
  - This paper uses notations like "WSD(80N,8N)'' multiple times. However, there is no definition about what the two items in parentheses represent.  $WSD(T,D)$ in Appendix B.1 is also not defined.
  - In the "Decay Stage" section, the equation for exponential annealing $f(s-T)=\eta \times 0.5^{(s-S)/T}$ appears to be incorrect. The variable $\eta$ seems out of place, and when $s=T$, the learning rate should equal $\eta$.
- Several figures are poorly presented:
  - The labels in Figures 1 and 2 are too small to read.
  - It is difficult to distinguish between the colors of different lines in Figures 5 and 6.
  - Despite descriptions suggesting the presence of a curve showing the 0.036B model (with 4 times compute) matching the performance of the 0.17B model with WSD (40N, 4N), this curve is absent from Figure 6.
- This paper highlights the benefits of introducing high-quality SFT data at the beginning of annealing. However, mixing SFT data with pretraining data after the pretraining could also improve the performance. That is to say, what may be important is the data mixing operation rather than the training stage. The ablation study did not shows whether mixing SFT data at the beginning of annealing is the best choice.
- Most baselines in Table 2 do not use SFT, whereas MiniCPM is trained with both pretraining and SFT data. Additional discussion or experiments are needed to determine how much of MiniCPM’s success is due to the inclusion of SFT data.
questions: questions: - Are optimal batch size and optimal learning rate independent? Analyses in Section 3.2 and 3.3 seem to be done separately.
- Will the training code, including the implementation of WSD, be made publicly available to facilitate reproducibility in research?
paper_summary: paper_summary: The manuscript presents a detailed study on the development and application of MiniCPM, a series of small language models (SLMs), which demonstrate comparable capabilities to much larger language models (LLMs) in the range of 7B-13B parameters. The authors have addressed a timely topic in the machine learning community, focusing on resource efficiency and practical utility of SLMs.
strength_weakness: strength_weakness: 1) Does the emergence pattern of Small Language Models (SLMs) mirror that of Large Language Models (LLMs)? If there are differences, can the insights gained from SLMs still be applicable for the development of LLMs?
2) The recent advancements with models like Llama-3 suggest that smaller-scale models can achieve enhanced performance with increased training data. Does this paper offer any additional insights or conclusions in this regard?
questions: questions: 1) What is the total model size when embedding parameters are included?
2) The manuscript lacks a discussion on the utilization of hardware resources, such as GPUs. Could you provide details on this aspect?

Comments: title: Please respond to rebuttal and other reviewers' comments
Response to the rebuttal
Reference updating
comment: Reviewer S8K9, could you please reply and/or update your review given the authors' response and discussion with the other reviewers? Thanks!
Thanks for your appreciation! We hope more people know MiniCPM series and use WSD in their practice.
Thanks for answering the questions and concerns I have. I think I under-valued the demonstrated utility of WSD, so after considering some of these things I'm inclined to raise my score to a 7. I'm updating the review to reflect this.
Thanks! We are updating the paper! You will see the changes in both future public versions and the camera-ready version if accepted.
I appreciate the author's response, which answers most of my concerns. I hope to see these changes in the updated version. I will increase my rating score.
Thank you for your feedback. I think you make fair points, especially for the first bullet point. I will keep my score as it is for now.
[3] was updated into a paper very recently:  Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations https://arxiv.org/pdf/2405.18392