Crystal: Illuminating LLM Abilities on Language and Code
Authors: Tianhua Tao
Junbo Li
Bowen Tan
Hongyi Wang
William Marshall
Bhargav M Kanakiya
Joel Hestness
Natalia Vassilieva
Zhiqiang Shen
Eric P. Xing
Zhengzhong Liu
Venue: COLM (2024)
Keywords: LM Pretraining, Language and Code, Transparency and Reproducibility, Multi-stage Pretraining, Open Science
TL;DR: The paper presents **Crystal**, a Large Language Model pretrained on 1.4T tokens that excels in both natural language and coding, featuring a multi-phase pretraining approach and openly sharing all resources to advance research.

Abstract: Large Language Models (LLMs) specializing in code generation (which are also often referred to as code LLMs), e.g., StarCoder and Code Llama, play increasingly critical roles in various software development scenarios. It is also crucial for code LLMs to possess both code generation and natural language abilities for many specific applications, such as code snippet retrieval using natural language or code explanations. The intricate interaction between acquiring language and coding skills complicates the development of strong code LLMs. Furthermore, there is a lack of thorough prior studies on the LLM pretraining strategy that mixes code and natural language. In this work, we propose a pretraining strategy to enhance the integration of natural language and coding capabilities within a single LLM. Specifically, it includes two phases of training with appropriately adjusted code/language ratio. The resulting model, CRYSTAL, demonstrates remarkable capabilities in both domains. Specifically, it has natural language and coding performance comparable to that of Llama 2 and Code Llama, respectively. CRYSTAL exhibits better data efficiency, using 1.4 trillion tokens compared to the more than 2 trillion tokens used by Llama 2 and Code Llama. We verify our pretraining strategy by analyzing the training process and observe consistent improvements in most benchmarks. We also adopted a typical application adaption phase with a code-centric data mixture, only to find out that it did not lead to enhanced performance or training efficiency, underlining the importance of a carefully designed data recipe. To foster research within the community, we commit to open-sourcing every detail of the pretraining, including our training datasets, code, loggings and 136 checkpoints throughout the training.

Published as a conference paper at COLM 2024
CRYSTAL: Illuminating LLM Abilities on Language and Code
Tianhua Tao Junbo Li*, Bowen Tan', Hongyi Wang?, William Marshall Bhargav M. Kanakiya², Joel Hestness Natalia Vassilieva Zhiqiang Shen*, Eric P. Xing & Zhengzhong Liu'
'Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates [junbo. 1i, zhiqiang. shen, gric.xing, hector. wiw.lluuuueeee Hiuu@mbzuai, ac ae
Univerrity of Illinois Arhana-Champpign Champaign, Illinois, United States (tianhua3 eeleelinilssleeee
CCarnegie Mellon University
Pittsburgh, Pennsylvania, United States (btan2, hongyiwa, epxing) gpxing)@andin @andrew. www..cc....mmm cdu.edu
$Cerebras Systems
Sunnyvale, California, United States william. fyfe. mmwrshall@gmii..cm
kanakiya, joel, natalia
Abstract
Large Language Models (LLMs) specializing in code generation (which are also often referred to code LLMs), egg., StarCoder and Code Llama, play increasingly critical roles various software development scenarios. It is also crucial for code LLMs to possess both code generation and natu- ral language abilities for many specific applications, such as code snippet retrieval using natural language teraction between acquiring language and coding skills complicates the development of strong code LLMs. Furthermore, there is lack of thorough prior studies on the LLM pretraining strategy that mixes code and natural language. In this work, propose pretraining strategy to enhance the integration of natural language and coding capabilities within a single LLM. Specifically, includes two phases of training with appropriately adjusted code/ /languagg ratios. The resulting model, CRYSTAI demonstrates re- markable capabilities in both domains. Specifically, has natural language and coding performance comparable to that of Llama 2 and Code Llama, respectively. CRYSTAL exhibits better data efficiency, using 1.4 trillion to- kens compared to the more than 2 trillion tokens used by Llama 2 and Code Llama. We verify our pretraining strategy by analyzing the training process and observe consistent improvements in most benchmarks. We also adopted typical application adaplation phase with a code-centric data mix- ture, only to find that it did not lead to enhanced performance or training efficiency, underlining the importance of a carefully designed data recipe. To foster research within the community, we commit open-sourcing every detail of the pretraining' including our training datasets², code3, loggings* and 136 checkpoints throughout the training.
code explanations. The intricate in-
Work done at Cerebras Systems.
WWbbaaee: httpps//www.lll http://wwwwm....../rrrrreteeeeee
Codde: http::///...tttoooooooo//o//rrra Wanddb: ottp
MModel weights: htpp:://wggiiggaae.////////////
and https: https://huugingfaccccom
LLM360/ MMM60/CrystalChat

---

Published as a conference paper at COLM 2024
Crystal Phase1
Crystal Phase2
Application Adaptation
siarcopeer
O
95%
</>
63%
python
5%
37%
345b tokens
927b tokens
110b tokens
Figure 1: The multi-phase training process for CRYSTAL
Introduction
Large Language Models (LLMs) for code gen- eration (i.e., code LLMs), such as Codex (Chen et al., 2021a), StarCoder (Li et al., 2023a), and Code Llama (Roziere et al., 2023), are advanc- ing rapidly due to their strong capability in generating code-related content (e.g,, functions), which helps improve the efficiency of software engineers and developers (Cognition Labs, 2024; Chen et al., 2021b; Li et al., 2023a; Roziere et al., 2023). These LLMs excel at generating functions and designing web page components based on engineers' instructions (e.g,, "Return True if all numbers in the list L are below threshold T.") (Calò
Coding Ability
50
Llama-2-78
Crystal-7B
Language Ability
40
CodeLlama-78
OLMo-7B
30
Starcoder- 15.58
25
40
55
70
Figure 2: CRYSTAL shows a good balance of language and coding abilities. The y-axis is the average over ARC-C, Hel- laSwag, MMLU, and GSM8K. The x-axis is the average of MBPP and HumanEval.
& De Russis, 2023). However, the abilities of code-oriented L.LSs are constrained in develop- ment contexts that necessitate interpreting high- level human instructions (egg,, through prompts or function descriptions) and producing com- prehensive, structured code accompanied by natural language documentation. Examples of such scenarios include solving GitHub issues (Jimenez et al., 2023), searching for code snippets based on natural language queries, generating entire Python libraries (which in- clude their complete code along with documentation and tutorials (Liu et al., 2023a; Luo et al., 2024)), or developing source code for websites, e.g., "Create a ticketing platform for travelers" (Calò & De Russis, 2023).
This underscores the ambition to create LLMs proficient both natural language process- ing and coding. Achieving this goal, however, is non-trivial. For instance, Code Llama, despite being continuously pretrained with code datasets on top of Llama2, suffers from catástrophic forgetting of natural language capabilities. In open-sourced LLMs, observe a prevalent issue: most models are tailored to specialize in either language or code, not both. For example, StarCoder is exclusively trained on code datasets accompanied by function documentation, thus limiting its exposure to varied natural language data. This trend indicates a notable gap in the design of most open-source LLMs, where there's lack of comprehensive curriculum that addresses both coding and natural language processing.
Therefore, are intrigued by the following research question: "Can an LLM efficiently obtain both language and coding abilities?" Existing studies have shown that the simultaneous acquisition of coding and language capabilities by LLMs is governed by complex dynamics: these skills may either conflict (Li et al., 2023a; Roziere et al., 2023) complement (Ma et al., 2024) each other, influenced by the data recipe and the model's s learning phase.
In this work, we propose a pretraining strategy designed specifically for code LLMs. Our strategy inspired by techniques such as multi-phase pretraining, curriculum learning (Ben- gio et al., 2009), continuous pretraining (Roziere et al., 2023), and multi-languuae training, and has two phases. We start the pretraining process with a data mixture of 95% natural lan- guage and 5% code. In the second phase, the data mixture is enriched to include 63% code data alongside 37% natural language. This two-phase design mimics the human learning process, where the acquisition of general language knowledge precedes the development
2

---

Published as a conference paper at COLM 2024
of coding skills, aiming to replicate this learning sequence. Pretraining using our strategy yields CRYSTAL, a code LLM that exhibits strong ability across both natural language (e.g., common sense reasoning) and code generation. Our strategy also demonstrates good data efficiency. That is, CRYSTAL, pretrained with 1.4 trillion tokens, performs comparably to Llama 2 and Code Llama, each pretrained with more than 2 trillion tokens.
Throughout the pretraining process, we continuously tracked the model's performance on downstream benchmarks, observing steady enhancements in both language and coding abilities across the two training phases. Despite a slight performance decline due to the distribution shift between Phase 1 and Phase 2, performance in Phase swiftly recovers and surpasses that of Phase 1. Additionally, implementing an experimental application adaptation phase, aimed at further enhancing coding abilities by incorporating an increased percentage of code data, could potentially boost performance. This phase is inspired by the Python-specializes
pretraining phase of Code Llama and StarCoder (Roziere al., 2023; Li et al., 2023a). Contrary to expectations, we observe mixed results from this phase, including a decline in language ability but marginal improvement in coding performance (see & 5.2), underscoring the necessity for carefully crafted data strategy.
Conducting thorough ablation studies for the entire pretraining process is computationally daunting. To mitigate these challenges, we embrace the principles of the LLM360 initia- tive (Liu et al., 2023b), ensuring full transparency in our pretraining process to support further scientific exploration and discoveries by the community. We release our training and fine-tuning datasets, source code for training, fine-tuning, and data preprocessing, and 152 intermediate model checkpoints. We also release a chat version, fine-tuned from CRYSTAL, namely CRYSTALCHAT, for user convenience.
2 Related Work
Open-source LLMs. The prevailing approach to developing modern LLMs involves two-step process: pretraining followed by fine-tuning. The extensive pretraining stage may involve using synthetic data, as demonstrated by the Phi series models (Gunasekar et al., 2023; Li et al., 2023b). However, the high-quality synthetic datasets used in the Phi models are not publicly available, whereas we aim make all our training details public and reproducible. Additionally, much of the pretraining is conducted on vast datasets comprising trillions of tokens that encapsulate nearly all available linguistic data. Notable projects in this domain include (Wang & Komatsuzaki, 2021; Andonian et al., 2023; Zhang et al., 2022; Scao et al., 2022; Biderman et al., 2023; Touvron et al., 2023a; Geng & Liu, 2023; Together Computer, 2023a; MosaicML NLP Team, 2023; Almazrouei et al., 2023; Touvron et al., 2023b; Bai et al., 2023; Jiang et al., 2023; Ol. 2023; Bi et al., 2024; Groeneveld et al., 2024). Of these, Pythia (Biderman et al., 2023), LLM360/ Amber (Liu et al., 2023b) and OLMo (Groeneveld et al., 2024) are particularly aligned with our work, sharing an emphasis on the complete reproducibility of LLMs. While Pythia stands out a pioneering effort, it does not embody the recent advancements observed in training with trillions of tokens and is not specifically tailored for code. Amber and OLMo, although newer, are also designed as general-purpose models. Our CRYSTAL utilizes advanced strategic data and training strategies to create a strong open-source model that excels in coding while also demonstrating strong overall capabilities.
Code LLMs. Applications at the core of fields such as software development engineer- ing (Fan et al., 2023) place a significant demand on language models equipped with special- ized code intelligence. Furthermore, models that are extensively trained on code datasets demonstrate enhanced reasoning capabilities and exhibit superíor performance in logical tasks, including mathematics (Ma et al., 2024; Fu et al., 2022). Motivated by the needs of both practical applications and research, code-oriented large models are increasingly gaining focus (Chen et al., 2021a; Li et al., 2022; Wang et al., 2023; Luo et al., 2023; Nijkamp et al., 2022; Li et al., 2023a; Roziere et al., 2023; Guo et al., 2024). The roles of code data differ across various works. StarCoder (Li et al., 2023a) exclusively trains code data, at the expense of general natural language understanding. DeepSeek Coder (Guo et al., 2024) incorporates more natural language into its pretraining, yet remains predominantly focused
3

---

Published as a conference paper at COLM 2024
code, with minimal enhancement in natural language benchmarks. Code Llama (Roziere et al., 2023) continues pretraining on code data atop Llama (Touvron et al., 2023a;b), which leads to forgetting issue that sacrifices general natural language understanding abilitiess WizardCoder (Luo et al., 2023) focuses on code instruction finetuning, where the ultimate performance significantly relies the foundational pretrained model. In contrast, CRYSTAL blends natural language and code data equally in pretraining, aiming to forge strong code model with enhanced natural language capabilities.
Multi Stage Training. Several LLM training efforts have also documented multi-stage approaches. For example, the XGen project (Nijkamp et al., 2023) includes relatively short second stage, comprising 4% of the total training tokens, which is similar to our adaptation stage. Our work further demonstrates successful strategy for mixing two domains during the major training stages. InternLM (InternLM Team, 2023) also reports using a multi-stage training method, though the details of these stages have not been disclosed.
Model Training
Drawing on the principles of coarse-to-fine methodologies for achieving domain adaptation without catastrophic forgetting, we design two phases in the pretraining process of Crystal. In the first phase, the model is expected to acquire broad spectrum of general language capabilities. In the second phase, we introduce coding ability into the model, ensuring that this augmentation does not compromise its existing natural language abilities. Table 1 summarizes the configurations of the phases.
Configuration. The architecture of CRYSTAL is adapted from prior work such as GPT-
2 (Radford et al., 2019), GPT-NeoX (Andonian et al., 2023), Llama (Touvron et al., 2023a) and BTLM (Dey al., 2023), featuring decoder-only models comprising 32 layers. The model is trained on non-GPU hardware architecture, using the Cerebras Condor Galaxy 1 (CG-1) (Cerebras, 2023). Taking advantage of the memory layout, the model can be trained efficiently using LayerNorm without RMSNorm. We incorporate a novel enhancement known as maximal update parameterization (μP), as described by Yang et al. (2022), de- ciding layer-wise learning rate, batch size, Adam coefficient, etc. We include all the final hyperparameters adjusted by pP in Table 1. The training time is 37 days on 16 CS-2 nodes.
Phase 1 for Fundamental Language Ability.
In the first phase, we focus on imbuing the model with foundational natural language derstanding, utilizing 345B tokens sampled from SlimPajama (Soboleva et al., 2023) dataset.
Training Loss
3
2
This dataset, primarily composed of natural language texts, includes modest portion (ap- proximately 5%) of coding data sourced from GitHub, subtly introducing the model to pro- gramming concepts. We expect this phase to es- tablish baseline comprehension of natural lan- guage, underpinning the model's subsequent specialization in code. The rationale behind starting with natural language is inspired in curriculum learning principles, positing that mastering the intricacies of natural language is prerequisite for tackling the structured complexity of programming languages.
50
100
Checkpoint
Figure 3: Pretraining loss curve. The gray dashed line divides Phase 1 and 2. We do not observe many major loss spikes; ob- served, we recovered by skipping specific data batches.
Phase 2 for Coding Ability. In the second phase, we expand the model's domain by integrating a 63% code data mixture, drawing from broad spectrum of programming lan- guages from the Stack (Li et al., 2023a) dataset (following the StarCoder mixture), resulting in a total of 927B tokens. This inclusion aims to transit the model from its natural language base towards more specialized understanding of code syntax and logic. In the meanwhile,
4

---

Published as a conference paper at COLM 2024
by keeping significant 37% of general language, we intend to prevent the catastrophic forgetting of general language ability.
Phase
Phase 2
Phase Adaptation
warm-up steps
86
86
276
total steps
79721
214387
27590
max LR
0.012
0.0087825
0.002
min LR
0.0086628
0.00013679
0.0002
optimizer
AdamW
betal
0.9
beta2
0.95
epsilon
le-9
weight decay
0.1
gradient clip
1.0
batch size
2112
sequence length
2048
trained tokens (current phase)
0.345T
0.927T
0.1T
trained tokens (accumulated) 0.345T
1.272T
1.372T
Table 1: Pretraining configuration. We choose the warm-up steps to be approximately 0.1% of the total steps in Phase 1. For Phase 2, we reuse the same numbers. In Adaptation Phase, set it to be 1% of the total steps.
Phase Adaptation. Following prior convention (Li et al., 2023a; Roziere et al., 2023), we conduct additional training to specialize the model on popular language tasks, using a small subset of Python and web-related data (HTML, CSS, and JavaScript) from the Stack dataset, totaling 100B tokens. A small portion (10B tokens) of SlimPajama is added to avoid catastrophic forgetting.
Finetuning. We perform tuning on top of Crystal using a collection of open-source datasets, tailored for chat applications, thereby enhancing usability for end users. We denote the resulting model as CRYSTALCHAT. Details can be found in Appendix C.
Figure 3 depicts the training loss curve throughout the training process, highlighting smooth and stable progression. Loss spikes are observed at the outset of Phase 1. It's important note that the loss scales vary across the two phases due the increasing proportion of code data. Code data tokens, which often represent shorter raw untokenized entities such as symbols, digits, and brackets, adhere to specific syntactical rules making them more predictable, resulting in a lower oss-pee-token.
4 Evaluation
We conduct an extensive evaluation CRYSTAL across multiple tasks, including language understanding, commonsense reasoning, code generation, and newly crafted benchmark for website generation. We compare CRYSTAL with models developed around the same time and trained with comparable number of FLOPs. However, some other open-weight models, such as Mistral (Jiang et al., 2023) and Mixtral (Jiang et al., 2024), do not disclose the size of their training data, making them unsuitable for direct comparison in studying the effect of data curiculumm For similar reasons, we also do not compare CRYSTAL with commercial endpoints.
4.1 Evaluating Natural Language Abilities
We evaluate Crystal's language ability on a key set of benchmarks maintained by EleutherA1 We show the benchmark results on a variety of aspects in natural language, including Reasoning: Hellaswag (Zellers al., 2019), ARC (Clark et al., 2018), Wino- grande (Sakaguchi et al., 2021), PIQA (Bisk et al., 2020); Question Answering: Open- BookQA (Mihaylov al., 2018), RACE (Lai al., 2017), BoolQ (Clark et al., 2019),
hhttps:///t
5

---

Published as a conference paper at COLM 2024
CRYSTAL Phase 2
Other Open Source Models
Phase
Adapt.
Llama 2
Code Llama
OLMo
StarCoder
Natural Language
Benchmarks
ARC-casy (0-shot)
64.73
20.75
67.34
74.50
62.29
68.51
50.17
ARC-challenge (0-shot)
37.54
42.58
38.91
46.16
35.24
40.27
27.73
ARC-challenge (25-shot)
42.83
47.44
47.01
53.33
42.75
45,93
32.16
Openbook QA (0-shot)
39.60
41.20
39.80
44.20
36,80
42.60
32.20
TruthfulQA (5-shot)
38.96
36.47
35.91
38.95
37.19
35.92
41.36
MMLU (0-shot)
28.05
42.46
42.33
41.71
34.76
28.19
27.55
MMLU (5-shot)
25.72
42
48.78
46.40
39,98
28.12
28.45
HellaSwag (0-shot)
69.65
72.89
70.35
75.92
75,56
46.65
HellaSwag (10-shot)
71.62
74.38
71.97
78.5
64.74
77.12
48.36
RACE (0-shot)
38.57
38.18
38.18
39.52
39.52
38.37
31.67
PIQA (0-shot)
75.84
78.07
76.77
78.78
72.58
79,92
65.61
COPA (0-shot)
86.00
83.00
80.00
87.00
80.00
88.00
67.00
BoolQ (0-shot)
66.64
74.43
72.36
78.07
74.65
72.66
57.16
Winogrande (0-shot)
63.14
67.01
65.51
69.38
65.51
67.24
55.10
Winogrande (5-shot)
64.80
68 82
67 67.40
73.64
65.75
68.90
56.04
GSMSK (5-shot)
2.12
12.36
10.39
14.71
11.15
9.02
Code Benchmarks
HumanEval (pass@1)
7.44
23.90
28.38
13.05
14.02
33.63
HvmanEval (pass@10)
14.64
45.12
52.76
22.61
58.36
24.56
59.38
MBPP (pass@1)
8.92
30.99
36.37
20.09
39.20
14.40
52.7
MBPP (pass@10)
17.24
58.62
56.37
34.69
64.00
26.42
65.44
Multipl-e Bash (pass@1)
0
10.76
6.96
2.53
10.13
1.26
10.12
Multipl-e C++ (pass@1)
6.83
24.22
23.60
6.83
26.08
13.04
29.81
Multipl-e C# (pass@1)
3.17
17.09
17.09
6.32
23.41
8.86
20.88
Multipl-e Java (pass@1)
3.17
22.79
27.21
11.39
33.54
13.29
29.74
Multipl-e [S (pass@1)
9,94
29.19
29.81
31,05
Multipl-e PHP (pass@1)
4.97
20.497
20.50
9.94
24.22
7.45
27.32
Multipl-e TS (pass@1)
10.06
25.15
30.18
13.21
32.70
12.57
33.96
Table Natural language and code generation evaluation results. The best scores for CRYSTAL are underlined. The best scores across all compared models are bold. All models in the table are around 7B in size, except that StarCoder is 15.5B model.
COPA (Reddy et al., 2019); General Knowledge: MMLU (Hendrycks et al., 2021b); Basic Arithmetic: GSMSK (Cobbe et al., 2021); Truthfulness: TruthfulQA (Lin al., 2021).
The results of these evaluations are detailed in Table 2. More results are available in our Weights Biases public dashboard online". Our results demonstrate that CRYSTAL achieves competitive performance across range of language tasks, with Phase 2 checkpoints outperforming other popular open-source LLMs, e.g., Llama 2, Code Llama, OLMo, and StarCoder-155. 15.5B several benchmarks et al., 2023b; Roziere et al., 2023; Groeneveld et al., 2024; Li et al., 2023a).
with fewer number of training tokens (Touvron
4.2 Evaluating Code Generation
For code generation, we evaluate the models on three benchmarks, i.e., HumanEval (Chen et al., 2021b), MBPP (Austin al., 2021) and Multipl-e (Cassano al., 2022), where Hu- manEval and MBPP are measuring functional correctness for synthesizing programs from docstrings, consists of more than 1,000 Python programming problems in total. Multiple-e is translation of HumanEval from Python to 18 programming languages. We omitted APPS (yendrycks et al., 2021a) and DS-1000 (Lai al., 2022) from our evaluation. The exclusion was due to the significant discrepancies in baseline model scores produced by our setup compared to those reported in their respective papers, primarily attributed to differences in prompts and evaluation configurations. The results can be found in Table 2.
WebMC: A New Benchmark Web Programs. In addition to the three benchmarks above, we introduce a new benchmark, WebMC, created for evaluating coding abilities related to website development. This dataset was generated using GPT-4 and manu- ally curated into 600 multiple-choice questions regarding website understanding, edit-

---

Published as a conference paper at COLM 2024
Base Models
Chat Models
Accuracy (o shot)
70
65
65
60
60
55
Phase
Adapt
LLaMA 2
CodeLLaMA
Phase 2+FT Adapt+FT
LLaMA 2 CodeLLaMA
Chat
Instruct
Figure 4: Evaluation results comparison across different models for zero-shot WebMC
ing and generation. WebMC has 3 types of tasks, including: (1) Generation: generat- ing a website according to specifications (e.g,, with a map widget), (2) Editing: chang- ing the font-family font-color, and (3) Understanding: Answer questions about ele- ment details, such as the content in the navigation bar. The dataset available at https://huggingf.e http://uuginnaaa...c.otottttssss co/dattaeets/LLMaG//M
We conduct an evaluation of CRYSTAL, CRYSTALCHAT against Llama 2, Code Llama, and their instruction-tuned variants. As shown in Figure 4, CRYSTAL demonstrates superior average performance in this comparison.
5 Analysis and Discussion
The evaluation results show that with an appropriate data curriculum, it's possible to train model excel in both language and code, sometimes even outperforming models with larger token budgets on certain benchmarks. In this section, we delve into an analysis of intriguing aspects observed during training.
GSM8K Example
Instruction: Question: James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run week?
Ground Truth: He sprints 3*3=9 times, he runs 9*60=540 meters #### 540
Phase 2 Response: 180
Phase 2+FT Response: James runs 3 sprints 60 meters each sprint = 180 meters per sprint. He runs this for 3 times a week, so he runs 180 meters per sprint times a week 540 meters week. The answer 540.
MBPP Example
Instruction: Write a function to remove first and last occurrence of given character from the string.
Phase Response:
def remove Occ (string, char):
# TODO: your code here
Phase 2+FT Response:
def remove Tamove_Occss,. c):
return replace(c, ", s.replace ", -1)
Table Examples of generation tasks enhanced by finetuning
5.1 Effect of Language and Code Mixing in Pretraining
In Fig. 5, we observe that during phase transition, there is notable drop in performance due to the abrupt change of domain on most benchmarks (ARC-C, Winogrande, and more in Appendix D.2). Within each phases, most benchmark scores for all intermediate checkpoints are rising smoothly, surpassing Phase performance the end of Phase 2. For instance, in Hellaswag 10-shot benchmark, decrease in score from approximately 71 to 67 after the initial 1k steps into Phase 2. It requires about 70% of Phase 2 to restore
Fxcept for MMLU, which fluctuates randomly during Phase 1.

---

Published as a conference paper at COLM 2024
Phase
Phase 2+FT
Adapt
Adapt+FT
Natural Language
Benchmarks
ARC-challenge (25-shot)
47.44
51.71
47.01
50.09
Openbook QA (0-shot)
41.20
42.00
39.80
38.8
TruthfulQA (5-shot)
36.47
47.29
35.91
45.13
MMLU (5-shot)
48.42
53.22
48.78
52.77
HellaSwag (10-shot)
74.38
76.12
71.97
72.76
Winogrande (5-shot)
68.82
70.64
67.40
68.19
GSM8K (5-shot)
12.36
28.05
10.39
27.98
Code Benchmarks
HumanEval (pass@1)
23,90
34.12
28.38
33.29
HumanFval (pass@10)
45.12
65.76
52.76
69.20
MBPP (pass@1)
30.99
39.40
36.38
40.20
MBPP (pass@10)
58.62
59.90
56.37
61.16
Multipl-e Bash (pass@1)
10.76
12.65
6.96
12.65
Multipl-e C++ (pass@1)
24.22
32.91
23.60
33.54
Multipl-e C# (pass@1)
17.09
24.05
17.09
23.41
Multipl-e Java (pass@1)
22.79
34.81
27.22
36.70
Multipl-e JS (pass@1)
29.19
31.67
29.81
42.23
Multipl-e TS (pass@1)
25.16
34.59
30.19
33.33
Table 4: Evaluation results for finetuned (FT) models based on Phase and Adaptation Phase (Adapt). More results on finetuned and instruction-following open source models can be found in Table 7.
performance levels, but after that the model can surpass the previous best value. The significant performance dips in these tasks the beginning of Phase 2 indicate that the abrupt shift in data distribution may be harmful. In reflection, more gradual transition to more code data may foster smoother adaptation.
For MMLU, the score is almost around a random baseline at 25% in Phase (with 5% code data). faster growing trend observed in Phase 2 (with 63% code data). The final MMLU score after two phases is even higher than strong baselines such as Llama 2. There seems to be an interesting interaction between language and code causing this phenomenon. One potential explanation can be data contamination. However, SlimPajama (a deduplicated version RedPajama) and StarCoder data (a dataset filtered from Github). We argue that the same similar datasets have been used by other open- source models of comparable scales(Tour) 2024), yet similar performance gain has not been observed in these models. This disparity in performance may suggest that our model's success cannot be fully attributable to data content. We hypothesize that can be attributed the unique combination of data selection, model architecture, and training curriculum. Although full pretraining ablation study is costly, our approach has been independently verified by the Snowflake Arctic team, which found that similar three-stage curriculum learning strategy can improve key evaluation metrics like MMLU (Snowflake AI Research, 2024). We also hope that our intermediate checkpoints will enable further study in this area.
pretraining datasets are
et al., 2023a; Li et al., 2023a; Groeneveld et al.,
We observe that supervised fine-tuning will boost the benchmark score significantly for generation tasks, such as MBPP and GSM8K. By inspecting the output (Table 3), we notice that the model tend to generate complete output, not mimicking the input question and follow coding instructions closely. Furthermore, the fine-tuned model will sometimes conduct chain-of-thought style generation, which is very helpful at solving arithmetic questions. Though we do not fully rule out the chances of data contamination.
Overall, the effect mixing surpass our expectation. The performance on language and code are both high and achieve good balance, depicted in Figure 2.

---

Published as a conference paper at COLM 2024
5.2
Adaptation Phase: Cost and Gain
MMLU 5 shot
HellaSwag 10 shot
80
55
Phase 1
Phase 2
HellaSwag Score
MMLU Score
50
Phase Adapt
70
45
Phase 2 + FT
40
Phase Adapt + FT
60
Phase
35
50
Phase
30
Phase Adapt
25
40
Phase
Phase Adapt + FT
20
100
200
300
100
200
300
ARC-C 25 shot
Winogrande shot
55
75
linogrande Score
50
70
Am V
ARC-C Score
45
65
40
35
Phase
60
Phase
30
Phase
55
Phase
Phase Adapt
Phase Adapt
25
Phase 2 + FT
50
Phase + FT
Phase Adapt FT
Phase Adapt
20
45
100
200
300
0
100
200
300
HumanEval pass @11s@10 10 (t=0.2)
GSMBK 5 shot
50
Phase 1
30
Phase
HumanEval Score
Phase
Phase
Phase Adapt
GSMBK Score
25
Phase Adapt
40
Phase 2 + FT
Phase 2 + FT
20
30
Phase Adapt FT
Phase Adapt + FT
15
20
10
10
5
0
0
100
200
300
0
100
200
300
Training Steps (thousands)
Training Steps (thousands)
Figure Benchmark scores for all intermediate checkpoints across phases. Contrary prior work, our Adaptation Phase does not improve the model. Instruction finetuning generally boost the model performance as expected.
Tasks that are more directly connected to the adaptation dataset such as HumanEval/MBPP (Python-related) and Multipl-e JS/TS (lvaacrrpp--elltted) show expected enhancements, For programming languages that are not reinforced, we observe mixed outcomes: some abilities improve (e.g., Java), while others suffer from certain declines (e.g., Bash and C++). However, for the majority of natural language tasks, the Adaptation Phase causes significant performance decrease". We hypothesize that the sudden decrease in the data ratio for language data may be responsible for this degradation
At a first glance, it seems that the Adaptation Phase brings expected specialization. However, we take into account of further finetuning, the adaptation gains are even out afterwards. Table 7 shows that, the model after finetuning Phase 2 (Phase 2+FT) and Phase Adaptation
Though the MMLU score remains unaffected.

---

Published as a conference paper at COLM 2024
(Phase Adapt+FT), ends up with similar performance. Contrary to the findings in prior work (Li et al., 2023a; Roziere et al., 2023), we observe that the marginal benefits brought by this phase is not worth the increased training cost, at least with this data curriculum.
Training Efficiency
Llama2-78, 7B,
Given that increased compute budget typically leads to better outcomes, we argue that LLMs should be evaluated based on compute efficiency in addi- tional final performance. Viewing the results of CRYSTAL through the lens of training efficiency can further validate the effectiveness of the training curricu- lum.
0.57
Stronger
Crystal-
Crystal-Phasez Phase2
Uama1-77
Average Performance
Performance
Crystel-AdapL
Falcon-78
MPT-78
Falcon-RuV77
0.53
CodeLlams
Crystal-Phasel
CodeLlama-PPther Python
Pythia-78
0.49
Efficient Computation
In Figure 6 we see the trade-off between total FLOPş10 spent on language data during pretraining and downstream performance. We that CRYSTAL phases and 2 models achieve stronger performance efficiently, even when com- pared to very strong baselines such as Llama 2.
0.45
BrarCoder-Base
Ours
Baselines
StarrCoden Python
22.0
22.4
22.8
Floating Point Operations (FLOPs)
Figure 6: The trade-off between the FLOPs (in log scale) required for language training and the average performance of ARC, MMLU, and Truth- fulQA.
Conclusion
In this work, we present a multi-phase LLM pretraining method designed encode natural language and coding abilities into single model. Through this approach, obtain model CRYSTAL, achieving natural language and coding performance on par with Llama and Code Llama, respectively. By tracking, observing and analyzing model performance throughout the pretraining, as well as a study of an additional adaptation phase, we obtain and present insights of the interplay of language and coding ability acquisition during the model training, highlighting the importance of data curriculum design.
Though we have included careful analysis on the training process, remains challenging to verify each design choice and explain every observed phenomenon during pretraining, largely due to constraints in computational resources. We will address the following limitation and release all our training artifacts, inviting the community to collaborate in overcoming these challenges:
Necessity of Smooth Transition Between Phases. Our examination in Section 5.1 reveals slight performance drop when transitioning from Phase Phase in certain benchmarks. This observation hints at the potential importance of ensuring smooth transition between training phases. Further investigation and validation of this hypothesis could further enhance training methodology.
Impact Code on Enhancing Reasoning Abilities. Notably, our second phase, which incorporates 67% code data, yielded unexpectedly high scores the MMLU, benchmark for natural language. This outcome suggests that structured code data may also boost language capabilities. A more definitive confirmation of this hypothesis could be obtained by re-running the second phase while omitting all code data, allowing for a direct assessment of its impact learning outcomes.
Acknowledgement
We thank the anonymous reviewers for their insightful comments and suggestions.
Fstimaated by FL.OPs per token times number of tokens of language data.
10

---

Published as a conference paper at COLM 2024
References
01.ai. 01-ai/yi: A series of large language models trained from scratch by developers @01-ai, 11 2023. URL https: thub. com/01-ai/////aa
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Êtienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guil- herme Penedo. The falcon series of open language models, 2023.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 9 2023. URL hwwph:ttps///// //ws:.///wwwweee thub. com/ eleutherai/gpt-neox.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen liang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arviv preprint arXiv:2108.0773 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
Yoshua Bérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek I1m: Scaling open-source language models with longtermism. arv preprint Xxxi:2401.022954 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430.
PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022.
Tommaso Calò and Luigi De Russis. Leveraging large language models for end-user website generation. In Infernational Symposium on End User Development, pp. 52-61. Springer, 2023,
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feld- man, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: scalable and extensible approach to benchmarking neural code generation, 2022.
Cerebras.
Introducing condor galaxy 1: puter for generative ai, 2023. wwwpeiiinn-oooooggglaayy
exaflops supercom- URL https://www.cerbbrr net.alog/////////////////// 4-eexxllpp----o www.-rermuuur-----o-rgrrrrratate
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained code. ar Xiv preprint ar arXim:2107.0337. 2021a.
11

---

Published as a conference paper at COLM 2024
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. ar Xiv preprint ar ar 00374,, 2021b.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/ questions. In NAACL,
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning 2018.
challenge. ar
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint ar Sife:2110.14160 2021.
Cognition Labs Introducing devin, the first ai software engineer. March 2024. URL
https://www wwwwonnnonnnniiooo wttcioottttooeeeee commintrodduing-deeinn
Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming, Chen, Robert Myers, Jacob Robert Steeves, Na- talia Vassilieva, Marvin Tom, and Joel Hestness. Btim-3b-8k: 7b parameter performance in a 3b parameter model, 2023.
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. Xrv preprint ar arXie:2310.0333. 2023.
Yao Fu, Hao Peng, and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's Notion, Dec 2022.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL llama.
https: //gis:://ttte thw...t.ttthtt../ggg com/ http//oenlmmreseereeeeeeee
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Ólmo: Acceler- ating the science of language models. arviv preprint ar xxXiv:2402.00888, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv wrggabs//3eee 11644.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint 4rXie:2401.14196 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. ArXiv, abs/2105.09gt 09999,, 2021a. URL https://api.
semanticscholar www.oolaarrrrgg/oooooooooooo wrgg/CrppusDD:....ooooooop (tt.orpuuII:22490111000 234790100.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
InternLM Team. Internlm: A multilingual language model with progressively enhanced 2023.
capabilities. https://gi thub. com wwwpeerwwwiIIIeeee--tte--e-
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xiv preprint ar arXit:2310006821 2023.
12

---

Published as a conference paper at COLM 2024
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv. org/ org/abs/2401. 044888
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan, Swe-bench: Can language models resolve real-world github issues? Xrv preprint ar arXiv:3310.06770, 2023.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint ar arXiv:1704.04633 2017.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Yih, Daniel Fried, Si yi Wang, and Tao Yu. Ds-1000: natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, 2022. URL https://api. seeanniccchollaa .acelaarrrrg//e httg/CorpuuID::53334533
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! Xrv preprint Xix::305.00666: 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 1.5 technical report, 2023b. URL https:
//arxiv. wwtrxii.orggabs//// http:/bbs/300...0 2ttp//229...0550.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-level code generation with alphacode. Science, 37866624)1009-11097 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091 2023a.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. Xiviv preprint ar Xiv:2312..06550, 2023b.
Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. Xiv preprint ar drXiv:2402.16667, 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint ar arXiv:2306.08566 2023.
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL httts:://opnneii /ttpss//////reeie..ommome net/forum?idt KIPJKST4gw.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023. URL www. www.mosaicml. ctw/cc/o/////oo hlomm/llg///// blog/mpt- 7b. Accessed: 2023-05-05.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. Xiv preprint ar XxXx::2203.13477 2022.
13

---

Published as a conference paper at COLM 2024
Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojciech Kryšcińsski, Lidiya Murakhovs'ker Prafulla Kumar Choubey, Alex Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingbo Zhou, Shafiq Joty, and Caiming Xiong. Xgen-7b technical report, 2023. URL https://arke org/abs/ 03450.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb catasstt for falcon llm: outperforming curated corpora with web data, and web data only. arviv preprint ar XrXii:2306.01110 2023.
Chen Qian, Jie Zhang, Wei Yao, Dongrui Liu, Zhenfei Yin, Yu Qiao, Yong Liu, and Jing Shao. Towards tracing trustworthiness dynamics: Revisiting pre-training period of large language models, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Lingwistics, 7:249-266, 2019. doi: 10.1162/tacl..00222 http:/allaa..00000 URL https: haap:ss//aalattholggg.. .rrg///oooooooooooooooo org. //rrg/Q19-116.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. ar arviv preprint arXi::2308. 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande; an adversarial winograd schema challenge at scale. Commuun ACM, 64(9):99-106, aug 2021. ISSN 0001-0782. doi: 10.1145/3474388 URL /ttttp://doiior/// 1145/3474381.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: 176b- parameter open-access multilingual language model. arXio preprint arXio:2211.05100, 2022.
Snowflake AI Research. Snowflake arctic: The best Ilm for enterprise ai ef- ficiently intelligent, truly open, 2024. URL https://www snowf lake. com/blog/ wta.guugeemmeelsssnoolllll--
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: https://www/ deduplicated version of RedPajama. 2023. URL https: httpt/////ugggiiaeeeeeeee/ttttte http://errrrrrssslllaaaaaaaaaaaa
627B token cleaned and
https://www.cerebra.s cerebras. net/blog/
June
Together Computer. Redpsjam---nnite-----ssss,, June 2023a. URL https:://hugiiiggaceee
gigetherrcmmputte//eddddajaaaa-- INCTTE 7B-Base.
Together Computer. Redpajama: an open dataset for training large language models, 10 2023b. URL httpp:://iithuu www.:/www..ooototttttttteoeeeeee www.e////tteeeoooooooooomeeeeeee computer /Redaajaaa-aaaa-aaaa
com/ tompuuer
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXio:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint (iXXi:23077.09888,, 2023b.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: 6 Billion Parameter Autoregressive Lan- guage Model.
thub http:tcco/kiiigoffo///////ttttte http://mmsh-tttaaaafmeee------ee May 2021.
14

---

Published as a conference paper at COLM 2024
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arxiv preprint ar XrXi:22355.0792, 2023.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXio:2203.03466, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Amnual Meeting of the Association for Computational Linguistics, 2019. URL https: wwwppeissssaaatcccccssccceec
wwrg/CCorusID:.. 159041722.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Xrv preprint ar Xft::2005.00066, 2022.
15

---

Published as a conference paper at COLM 2024
A Responsible Research
LLM360 is created with the mission to train and release open-source large language models to foster transparency, trust, and collaborative research. While large language models have demonstrated promise in advancing numerous domains throughout commercial and academic settings, the technology is still relatively poorly understood. Due the significant capital requirements to training and experimentation with LLMs, many learnings in the space happen behind closed doors. The lack of knowledge transfer will have negative effects for the ecosystem advances will be limited to small groups. To fully realize the potential large language models can deliver, we believe that the core tenets transparency, trust, and collaboration are paramount to the long term success of the field.
For each model released under LLM360, we will release the datasets, data preparation scripts, training code, numerous intermediate checkpoints, and complete analysis. We prioritize publicly available datasets such as The RedPajama (Together Computer, 2023b) and Refined Web (Penedo et al., 2023) and existing architectures and conventions such as LLaMA (Touvron et al., 2023a) to make our resource relevant and easy to access. By providing the listed artifacts, we hope to promote the reproducibility for all our work to encourage additional research.
Datasets are expensive to curate and are major competitive advantage for training per- formant models. By making all data available, our models are fully auditable. We provide clarity on all pretraining sources, the ethical manner in which data was sourced, and the actual data. Releasing checkpoints from the entire training process enables fine grained research into training dynamics (Qian et al., 2024) which would otherwise be restricted to those with the financial resources to pretrain models. We believe that the future should only constrained by our creativity, not man-made hurdles, and hope that access to our artifacts motivates others to pursue their own creative research unhindered.
Ethical Use. LLM360 models openly release our scores on safety evaluations such as Toxigen and TruthfulQA. These scores educate users to the potential risks that using our models may introduce when generating text. We gather our data from reputable sources and apply rigorous filtering to remove harmful data, but we cannot guarantee the outputs of our models will be completely safe. All users should conduct their own testing before adopting our models.
LLM360 models are also trained with coding abilities. When using code generated from large language models, users should always review the output before submitting it into their codebase. Generated code may introduce issues such insecure code which cannot be eliminated from the model. Users should perform their own safety testing and code reviews before deploying applications.
16

---

Published as a conference paper at COLM 2024
B Model Architecture
The Crystal language models employ a GPT-like architecture, featuring decoder-only mod- els comprising 32 layers. We incorporate a novel enhancement known as maximal update parameterization (muP), as described by Yang et al. (2022), enabling uniformity in hyper- citimiizationrrllater hyper-parameters, i.e,, learning rale, batch size, Adam coefficient, etc., and ing widths. This uniformity facilitates the optimization of hyper-parameters by tuning smaller, shallow model and directly transferring the optimized settings to the original wider model. Intuitively, muP achieves this by regularizing each linear layer relative to its width, rendering updates "independent" of width.
parameters including initialization-ei related hlpper-parameter across models of vary-
1. Input embeddings are scaled by mup_ www.ddinngsssca..e.e.o sccale.
2. Output logits scaled by mup_ happoutput_aaaae _alpha x mup wwidhhssclee
OKT
3. Attention weights scaling is refined to division by the hidden dimension size
OK
instead of its square root
We find this works better under the muP setting v
in early experiments.
4. Learning rates and weight decay are optimized for different parameter groups:
Embedding layer: LR=BASE_LR, WD=BASE.WD
Normalization layers: LR=BASE_LR, WD=0.
Other Parameters: LR=BASE_LR sup.width.scale WD=BASE.WD.
5. Initialization ranges are determined based on muP hyperparameters.
The muP hyperparameterss are set as follows:
mup initialization standard deviation: 0.073
mup.embeddingss.calles 14.6
mup output alpha: 2.22
mup.width.scale: 0.0625
mup base width: 256
For other architecture choices:
We use LayerNorm instead of RMSNorm.
Rotary position embeddings applied only the first 25% of hidden dimen- sions (Black et al., 2022), leaving the other 75% dimensions unchanged.
Training sequence length is 2048.
Embedding dimension 32032.
Full architecture details and comparisons with other models are available in table 5.
C Finetuning Details
C.1 Prompt Format
We introduced four special tokens to the tokenizer and model architecture to enhance instruction handling:

 Marks the beginning of system prompt.
< 
 Marks the end of system prompt.
<| iim_start/> Marks the start of an instruction message.
17

---

Published as a conference paper at COLM 2024
Crystal
Llama 2
Layers
32
Hidden Dimension
4096
4096
Embedding Dimension
32032
32000
Positional Embedding
Rotary
Rotary
Rotary Percentage
25%
100%
Layer Normalization
LayerNorm
RMSNorm
Num Heads
32
32
Activation
SwiGLU
SwiGLU
Sequence Length
2048
4096
Batch size
2112
1024
Bias
Linear & LayerNorm
None
muP
Yes
No
OK"
OKT
QK Dot Product Scaling
vd
Table 5: Architecture comparison.
<| iim_end|>>> Marks the end of an instruction message.
These tokens were integrated into the existing 32032-token vocabulary without necessitating expansion, leveraging reserved vocabulary slots. The conversation is wrapped by the tokens 
 and 
, framing the structure as follows:

 <|sys_stards system prompt <|sys_end]> <lim_starti> first user utterance 
 first model response <lim_start|> next user utterance <| im_end|> next model response 

C.2 Finetuning Datasets
Table summarizes the datasets we use for finetuning.
Subset
#Tokens
Avg. #Q
Avg. Q Len
Avg. #R
Avg. R Len
OASSTI-guanaco
4,464,640
1.36
38.28
1.36
271.69
SlimOrca
225,628, 160
1.00
259.16
1.00
151.12
ShareGPT
112,914,432
3.28
94.53
3.64
365.81
Evol-ShareGPT
85,954,560
1.00
145.99
1.00
425.17
ChatLogg
29,337,600
3.39
95.58
3.24
191.42
Aodaaa
2,623,488 488
1.00
32.46
1.00
67.68
Rosetta Code
7,987,200
1.00
450.09
1.00
533.52
Evol-CodeAAppca 1
73,803,776
1.00
210.33
1.00
437.92
Evol-CodeAlpaca
34,910,208
1.00
114.99
1.00
300.29
WebAlpaca
43,673,600
1.00
96.29
1.00
746.52
General Textbooks
85,590,016
Not instruction data
Programming Books
395,628,544
Not instruction data
Total
1,102,516,224
Table 6: Dataset Statistics. Q stands for Query. R Stands for reply. The summarizes the
average number and length of the queries and replies for the datasets. We also included textbook style datasets in the final finetuning dataset.
18

---

Published as a conference paper at COLM 2024
D Evaluation Details
D.1 Full Evaluation Details for Chat Models
We present the full evaluation results of our models and other open source models Table 7.
CRYSTAL
Other Open Source Models
Phase
Phase 2+FT
Adapt
Adapt=FT
Llama Chat
CodeLlama-Insiruct
OLMo-Instruct
Natural Language
ARC-easy (D-shot)
20.75
70.33
67.34
68.65
69.61
63.09
64.14
ARC-challenge (0-shol)
42.58
44.63
43.09
44.54
43.08
ARC-challenge (25-shot)
47.44
51.71
47.01
50.09
53.07
43.35
47.95
Openbook QA (0-shot)
42,00
45.20
TruthfulQA (5-shol)
36.47
47.29
35.91
45.13
45.30
39.23
45.52
MMLU (0-shot)
42.46
52.79
42.33
51.01
47.17
40.67
47.48
MMLU (5-shot)
48.42
53,22
48.78
52.77
48.42
42.75
48.57
HellaSwag (0-shot)
72.89
73.31
70.35
70.94
75.46
63.83
78.47
HellaSwag (10-shot)
RACE (U-shot)
38.18
41.15
38.18
41.44
43.63
40.00
40.57
PIQA (0-shot)
78.07
77.86
76.77
76.17
77
73.93
76.71
COPA (0-shot)
83.00
85.00
80.00
88.00
89.00
81.00
87.00
BoolQ (0-shot)
74.43
82.78
72.36
80.46
79.76
73.79
78.56
Winogrande (D-shot)
67.01
68.11
66.77
Winegrande (5-shot)
68.82
70.64
67.40
68.19
73.08
64.32
70.24
(5-shot)
Code Generation
HumanEval (pass(11
23.90
34.12
28.38
33.29
13.26
33.53
18.90
HumanEval (pass610)
45.12
65.76
24.89
59,6"
28.39
MBPP (passe1)
30.99
39.40
36.38
40.20
17.42
38.91
15.40
MBPT (pas=810)
58.62
59.90
56.37
61.16
32.12
66.71
27.32
Multiplee Bash (pass@1)
10.76
12.65
6.96
12.65
1.27
10.76
0.63
Multiplee C-+ (pass01)
32.91
13,66
Multipl-e C# (pass811)
17.09
24.05
17.09
23.41
9.49
24.05
7.59
Multipl-e Jawa (pass@1)
22.79
34.81
27.22
8.23
8.22
Multipl-e ]S (pas=@1)
29.19
31.67
29.81
42.23
14.91
31.05
1180
Multipl-e PHP (passi1)
20.50
33.54
20.50
31,05
10.56
27.33
9.32
Multipl-e (pass@1)
34.59
33.33
13.21
Table 7: Natural language and code generation evaluation results for chat models. The best scores for CRYSTAL underlined. Numbers with are adopted from original papers. Comparing the results for "Phase 2+FT" and "Adapt+FI", the benefits brought by the Adaptation Phase become insignificant after the finetuning,
D.2 Evaluation Results on Intermediate Checkpoints
We present evaluation results on all intermediate checkpoints.
19

---

Published as a conference paper at COLM 2024
Truthful-QA 0 shot
ARC-E O shot
50
Phase 1
Truthful-QA -all-QQA Score
Phase
70
wAY
45
Phase Adapt
ARC-E Score
Phase 2 + FT
Phase Adapt +
60
40
MM
Phase 1
50
Phase
35
Phase Adapt
Phase + FT
30
40
Phase Adapt FT
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
ARC-C 0 shot
ARC-C 25 shot
55
45
50
Vealle
ARC-C Score
ARC-C Score
40
45
40
35
Phase 1
35
Phase 1
Phase
Phase 2
30
30
Phase Adapt
Phase Adapt
25
Phase FT
25
Phase + FT
Phase Adapt FT
Phase Adapt FT
20
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
MMLU 0 shot
MMLU 5 shot
55
Phase
55
Phase 1
50
Phase
50
Phase
MMLU Score
Phase Adapt
MMLU Score
Phase Adapt
45
Phase + FT
45
Phase 2 + FT
40
Phase Adapt FT
40
Phase Adapt +
35
35
30
30
25
25
20
20
100
200
300
100
200
300
Training Steps (thousands)
Training Steps (thousands)
Figure 7: Evaluation results (part 1) on intermediate checkpoints.
20

---

Published as a conference paper at COLM 2024
GSM8K 5 shot
Openbook-QA 0 shot
30
Phase
Openbook- K--Q Score
45
Phase
GSM8K Score
25
Phase Adapt
Phase + FT
40
20
Phase Adapt FT
15
35
Phase 1
Phase
10
Phase Adapt
30
5
Phase + FT
Phase Adapt FT
0
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
HellaSwag shot
HellaSwag 10 shot
80
HellaSwag Score
70
HellaSwag Score
70
60
60
Phase 1
Phase 1
50
Phase
50
Phase 2
Phase Adapt
Phase Adapt
40
Phase 2 FT
40
Phase + FT
Phase Adapt FT
Phase Adapt FT
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
Winogrande 0 shot
Winogrande 5 shot
75
Winogrande Score
70
Winogrande Score
70
65
65
60
60
Phase 1
Phase 1
Phase 2
Phase 2
55
55
Phase Adapt
Phase Adapt
50
Phase FT
50
Phase + FT
Phase Adapt FT
Phase Adapt FT
45
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
Figure 8: Evaluation results (part 2) on intermediate checkpoints.
21

---

Published as a conference paper at COLM 2024
RACE 0 shot
COPA 0 shot
45
90
RACE Score
COPA COPA Score
85
40
Mufy
80
35
Phase 1
75
Phase 1
Phase
Phase
30
Phase Adapt
70
Phase Adapt
Phase FT
65
Phase + FT
Phase Adapt FT
Phase Adapt FT
25
60
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
BoolQ 0 shot
PIQA 0 shot
85
Phase
Phase
80
80
Ap Phase 1
BoolQ Score
Phase Adapt
PIQA Score
Bay
75
Phase + FT
75
Phase Adapt + FT
70
70
65
Phase 2
60
Phase Adapt
65
Phase 2 + FT
55
Phase Adapt FT
60
100
200
300
300 Training Steps (thousands)
100
200
Training Steps (thousands)
HumanEval pass@1 (t=0.2)
HumanEval pass@10 (t=0.2)
40
Phase
Phase 1
50
HumanEval Score
Phase
HumanEval Score
Phase
30
Phase Adapt
Phase Adapt
40
Phase + FT
Phase 2 + FT
Phase Adapt
Phase Adapt +
30
20
20
10
10
0
0
100
200
300
100
200
300
Training Steps (thousands)
Training Steps (thousands)
Figure 9: Evaluation results (part 3) on intermediate checkpoints.
22


Reviews:
[{"review_id": "fYd0PPvvmi", "reviewer": "Reviewer_oMAi", "summary": "The authors introduced a pretraining strategy that focuses on training LLMs with both text and code data. The resulting family of models, CRYSTAL, is stronger in both natural language and coding performance, compared to strong baselines like LLama2 and Code Llama. The authors are also committed to releasing open-sourced model checkpoints, increasing the replicability and transparency of LLM training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and helpful feedback! We respond to individual comments below.\n\n**Main Novelty Compared to CodeT5+**\\\nCodeT5+ starts with pre-trained LLMs, similar to CodeLlama. As we argued in the paper, such model development strategies can hurt the original language capabilities injected in the pre-trained LLMs, resulting in the final code LLM not achieving a perfect balance between language and code capabilities. This, in fact, is the key advantage of CrystalCoder.\n\n**Benchmark Crystal with more diverse coding tasks beyond HumanEval and MBPP**\\\nThanks a lot for the great suggestion! We are actively working on evaluating Crystal on APPS and DS-1000 as suggested. Due to the time constraint, we don’t have any number to share yet. But we will post the results as soon as they are ready.\n\n**Current strategy applied on top of other pretrained models** \\\nWhether our proposed strategy is compatible with pre-trained models remains an open question. Knowing the data composition of pre-trained LLMs is crucial to avoid issues such as introducing duplicated text to a model or using a wrong data weight. The lack of transparency from existing LLMs hinders this research. Additionally, conducting ablation studies becomes challenging without knowledge of their specific settings, as many open-weight LLMs only release their final model weights. Our fully open-sourced Crystal model provides a foundation for the community to conduct these experiments.\n\n**Question: ablation study with same number of tokens**\n- We agree with the reviewers that it would be great to include explicit ablation studies. However, pre-training is expensive, which is unfortunately hard to afford. We have started the ablation study as suggested by the reviewer but due to resource constraints, it is not finished.\n- Luckily, our approach has been verified by another team independently and has demonstrated strong performance. For example, Snowflake Arctic used a three-stage curriculum learning strategy inspired by our approach (starting with a focus on generic skills, then moving to more domain-specific knowledge and skills): https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/.\n- Olmo-7B and another of our 7B models, both trained under similar setups, show significantly lower MMLU compared to Crystal at a similar token scale. While not a direct comparison, these results further support the effectiveness of our strategy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Despite pretrained on a smaller set of tokens (1.4 trillion), CRYSTAL seems to possess strong/comparable performance against other models pretrained on a larger dataset (2 trillion tokens). \n- The evaluation is done comprehensively on many language tasks (reasoning, QA, general knowledge, etc.) and coding tasks (HumanEval, MBPP, WebMC). Interesting ablation results (e.g. data efficiency, training curves) are analyzed quite well, bringing useful insights to LLM training. \n- The authors’ commitment to open-source the models is commendable as it will help to push innovation and collaboration within the research community.", "reasons_to_reject": "- The use of a mixture of datasets of code and text over multiple stages of training is not so new. For instance, CodeT5+ (https://arxiv.org/abs/2305.07922) was also trained on both code and text data over two stages of pretraining. While the model was not evaluated on conventional NLP tasks, it performed quite well over multimodal test setups such as code-to-text and text-to-code benchmarks. Can the authors clarify what is the main novelty compared to CodeT5+? Have you tried to benchmark CRYSTAL in more diverse code-text tasks beyond HumanEval and MBPP? \n- Pretraining models from scratch is expensive. I wonder if the current pretraining strategy can be applied on top of any pre-trained models. If subsequently similar insights could be observed, I think the current method would be a lot more impactful.", "questions_to_authors": "- Typo (Introduction): “pertaining” \n- A useful ablation the authors can try is to use the data in phase 1 or phase 2 but train for the same number of training steps as phase 1 + phase 2 training combined. This would isolate the effect of training steps and demonstrate better how the data mixing contributes to the performance.", "extra_scores": {}}, {"review_id": "Av3R4oxc7m", "reviewer": "Reviewer_mxyS", "summary": "Authors propose a pre-training strategy that transitions from a phase of learning from natural language heavy to code heavy dataset. This has been claimed to achieve the most optimal balance between performance on natural language tasks and coding tasks for a smaller training budget than comparable work (in terms of model size) like Llama2 and CodeLlama.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your thoughtful reviews!\n\n**Motivation**\nWe’ve addressed the need for a single model in the introduction. While code LLMs can improve their natural language (NL) skills through finetuning, their NL proficiency may suffer in broader domains. Our goal is to develop a model that generally excels in both, proven by the performance of our models.\nMoreover, there is a hypothesis that code and NL can enhance each other. Though mentioned in prior works, it hasn’t been documented in pretraining literature. This research shows that the interplay between NL and code can indeed boost performance, even using less tokens.\nFinally, the ability to mix different types of data enables further scaling the total token size.\n\n**Data quality and Phi**\nThe focus of this work is multi-stage and the interplay between NL and code. While we acknowledge that data quality is essential, it is orthogonal to our research goal. Our training strategy can be effectively combined with higher-quality data.\n\nThe highlight of Phi-2 is its synthetic data, whereas we use publicly available corpora (SlimPajama, Starcoder) with validated quality. This saves time on massaging prompts, and compute on querying powerful LLMs. Moreover, the open source data makes our research more accessible and verifiable.\n\n**Questions**\n- RMSNorm is to resolve a computation bottleneck unique to GPU, which doesn’t exist on CS-2, allowing us to use Layernorm\n- 37 days on 16 CS-2 nodes\n- Loss spikes are commonly observed, while the explanation remain open. Spikes in Crystal pretraining is not frequent compared to other efforts (e.g. Llama). When a major loss spike is observed, it can be handled by skipping a specific data batch\n- RoPE applied to first 25% of embedding dimensions, inspired by GPT-NeoX\n- Based on our experiments, q*k dot product works better with muP\n- Prior work (e.g. CodeLlama) showed that a stage targeting a language (e.g., python) can boost performance. Our goal is to improve Python/Web programming. Data is sampled from Starcoder V1 \n- WebMC has 3 task types:\n  - generation: generate a websites with specifics (e.g., has a map)\n  - editing: change font-family/font-color\n  - understanding: ask about element details, such as what’s in the nav bar\n- Your hypothesis is inline with ours that the last stage’s data mix might be the culprit, which implies continuous training can be sensitive to the model’s data curriculum\n\nWe will add more details in the revision to address the questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work provide a closer look at the details of the pre-training stages of language models, which has been undisclosed in the case of most openly released models in recent times. Particularly, the stages of pre-training and their impact on downstream performance through empirical studies haven't been studied in this manner to the best of my knowledge. This can benefit the community's understanding in efficiently training models (with as small pre-training costs as possible) that are performance-wise optimal in both code and natural language domains. For instance, they highlight the variation in the loss scales across the two phases as code data is more predictable leading to a lower loss-per-token. The final model they obtain after the different training stages is indeed competitive in both domains when compared to specialised models like Llama 2 and CodeLlama that have been trained on a similar scale of data.", "reasons_to_reject": "Authors don't discuss the motivation of needing a single model to perform well at tasks from both natural language and code domain. One possible motivation could be that Code LLMs that can follow instructions are suited for a wide variety of agentic tasks as shown in Hong et al. However, one could argue if this capability can be achieved simply by instruction fine-tuning stage on top of a code pre-trained LM.\n\nWhile the impact of controlling the ratio of code to natural language data during pre-training has been studied, there is no discussion on the quality of the dataset used. Recent work on phi-2 (https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) has shown performance superior (MMLU, HumanEval, MBPP) to Crystal when trained on a similar size of data ~1.4T tokens, and using a smaller model in terms of parameter count (2.7B). Comparing this work to phi-3 or Llama 3 wouldn't be fair given the timeline of events.\n\nHong et al MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework", "questions_to_authors": "- The CG-1 architecture is not described in sufficient detail. Besides LayerNorm what architectural changes are required for training on this non-GPU hardware?\n- What is the training duration? (similar to how GPU hours are being reported in other studies)\n- In Fig 3, can the authors explain the occurrence of spikes in the training loss during stage 1?\n- Why is the qk dot product scaling different compared to the commonly used $\\sqrt{d}$ formula? (Table 4)\n- What does rotary percentage mean in Table 4?\n- What is the source of the 100B tokens in the phase adaptation stage? What is the motivation in including this stage?\n- Creating a benchmark like WebMC is an interesting contribution. Can the authors provide qualitative analysis to describe what the tasks here look like?\n- In 5.2 authors mention the adaptation phase to be hurting NL tasks. Could the reason be that the fraction of Slim Pajama is too little in this stage?\n\nSuggestions:\n- Pre-trained models like X-Gen, Phi series could be added to the related work section. XGen (by Nijkamp et al https://arxiv.org/abs/2309.03450) specifically discusses the approach of having different stages each with different sampling proportions. \n- Besides k-shot, authors could also try CoT prompting w/ the Crystal models to push performance on the NL and code benchmarks, particularly MATH and GSM8k.\n- A summary of model architecture details could be a part of the main paper (currently in Table 5 and Appendix B), similarly Table 7 should also be a part of the main paper. There is quite some whitespace (e.g. Fig 5 (could use shared legends), Fig 6, Section 6) that could be removed to achieve this compression.\n\nMinor typos/linguistic issues:\n- Abstract: Futhermore, there lack of --> there is a lack of?\n- Section 3 Phase 2: we intent to --> we intend to?\n- Section 5.2 Para 2: adaptation gains are even out", "extra_scores": {}}, {"review_id": "Tfe7R1XxZ4", "reviewer": "Reviewer_zzp6", "summary": "This paper investigates the integration of natural language and coding capabilities within code LLMs. The authors introduce a novel pretraining strategy that involves two training phases with a carefully adjusted mix of code and natural language data. Their proposed model, Crystal, demonstrates comparable performance in natural language processing and coding tasks to established models like Llama2 and Code Llama, but with greater data efficiency, using 1.4T tokens versus over 2T used by the other models. The effectiveness of the new pretraining approach is supported by detailed analysis of the training process and benchmarking, which show consistent improvements.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and helpful feedback! We respond to individual comments below.\n\n**Ablation Study**\\\nWe have started a 7B ablation but haven’t finished it due to resource constraints. We discussed the idea of smaller models but:\n- Scaling behaviors for multi-phase training is more complex.\n- MMLU is a key gain in our 7B setup. Yet our 1B models only have MMLU accuracy at random, the general trend is that knowledge of extensive tasks relies strongly on model sizes, hence ablation this way didn’t provide additional insights.\n\nWhile not a direct comparison, we collected some indirect supporting evidences:\n- Snowflake Arctic independently verified our approach, by using a 3-stage curriculum inspired by us (start with web text, then move to domain-specific (coding) ones): https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/.\n- Olmo-7B and another of our 7B models, both trained under similar setups, show significantly lower MMLU compared to Crystal at a similar token scale. \n\nResearch on pre-training is challenging, especially for detailed ablation studies. This is why we open-source the creation process of models, to empower the community to study this collectively.\n\n**Additional Comparison**\n- *GPT-3.5/4*: These models update constantly, and their model details and conditions (e.g., model size, instruction data used) are unknown. But we agree that adding them as references would provide a reference point.\n- *Mixtral 8x7B*: We compared Crystal with models of similar settings. Mixtral, a mixture-of-expert model, has a significantly larger effective parameter size, and an unknown token budget. We will include a discussion addressing the comparison with such models in our revised version.\n\n**Novelty of the method**\n- Pretraining research often focuses on scaling up straightforward ideas, we are happy to contribute to a simple yet effective method.\n- Due to resource limitations, we only conducted a two-phase experiment and found that a well-designed data curriculum benefits pretraining. We hope this can inspire future research into multi-phase or continuous pretraining.\n- Additionally, we released a full training trace, which, to the best of our knowledge, provides a unique resource for the community to build upon variations of our method.\n\n**Additional metrics**\\\nWe have taken your suggestion into account and have begun setting up evaluations for more coding tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation behind this paper is strong, highlighting a critical gap in open-source LLMs which typically excel either in coding or NLP tasks, but not both. This contrasts sharply with more versatile closed-source models like GPT-4.\n\n2. The paper provides detailed insights into model performance through the inclusion of evaluation results from different training checkpoints (Figure 5) and ablation studies (Table 3), clearly illustrating the impact of various training phases on Crystal’s effectiveness across tasks.\n\n3. Comparative analysis reveals that Crystal achieves strong performance on both coding and NL tasks, rivaling the capabilities of similarly sized models in two domain at the same time.\n\n4. The evaluation is comprehensive, particularly in the NLP domain, where Crystal is benchmarked against strong baselines such as LLaMa-2, Code LLaMa, and StarCoder across 11 distinct tasks.\n\n5. Crystal shows notable computational efficiency, achieving strong performance while requiring significantly fewer training tokens than comparable models.\n\nNote: While the paper mentions a commitment to releasing training code and all intermediate checkpoints, which could be a significant contribution, this is not yet verifiable in the review phase without supplementary material. Thus, it is not included as a strength here.", "reasons_to_reject": "1. It remains unclear whether the observed performance improvements are due to the two-phase training method or if similar outcomes could be achieved with a single-stage training using a combined dataset. The current ablation study does not address this question. Although retraining the full model is resource-intensive, conducting a smaller-scale experiment, perhaps with a 1B parameter model, could validate the effectiveness of the two-phase approach, which is central to this paper’s contributions.\n\n2. While the paper benchmarks Crystal against notable models, it omits comparisons with key baselines like GPT-3.5 and GPT-4, which are recognized for their proficiency in both coding and NL tasks. Additionally, including comparisons with models like Mixtral-8x7B, which also excel in both domains, would enhance the robustness of the evaluation.\n\n3. *[Minor]* The pretraining strategy proposed in this paper, which consists of two distinct phases using different mixtures of NL and code datasets, is too straightforward and lacks significant novelty.\n\n4. *[Minor]* The evaluation methodology appears somewhat biased towards NLP, as the evaluations on the coding aspects (3 tasks) are not as comprehensive compared with on the NLP side (11 tasks). Expanding the evaluation to include additional coding benchmarks such as APPS or DS-1000 could provide a more balanced view of the model's capabilities.", "questions_to_authors": "See weakness 1. Is it possible to see if performance improvements are due to the two-phase training method or if similar outcomes could be achieved with a single-stage training using a combined dataset?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a 2-phase pretraining process with the goal of training a single model that excels at both natural language and code. Basically, phase 1 contains mostly text with a small proportion of code, while phase 2 includes more code. They demonstrate that their Crystal model achieves competitive performance on both text and code tasks compared to other open-weight models of a similar size.

This work presents insights into the effect of different pretraining stages, and it would be a valuable contribution if the authors can fully open-source their model checkpoints and code. I recommend the authors to incorporate reviewers' feedback into their camera-ready version, especially with new experimental results they promised to add.
Decision: Accept