Crystal: Illuminating LLM Abilities on Language and Code

Authors: tianhua  tao
junbo  li
bowen  tan
hongyi  wang
william  marshall
bhargav m kanakiya
joel  hestness
natalia  vassilieva
zhiqiang  shen
eric p xing
zhengzhong  liu
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 21:59:38

Abstract: Large Language Models (LLMs) specializing in code generation (which are also often referred to as code LLMs), e.g., StarCoder and Code Llama, play increasingly critical roles in various software development scenarios. It is also crucial for code LLMs to possess both code generation and natural language abilities for many specific applications, such as code snippet retrieval using natural language or code explanations. The intricate interaction between acquiring language and coding skills complicates the development of strong code LLMs. Furthermore, there is a lack of thorough prior studies on the LLM pretraining strategy that mixes code and natural language. In this work, we propose a pretraining strategy to enhance the integration of natural language and coding capabilities within a single LLM. Specifically, it includes two phases of training with appropriately adjusted code/language ratio. The resulting model, CRYSTAL, demonstrates remarkable capabilities in both domains. Specifically, it has natural language and coding performance comparable to that of Llama 2 and Code Llama, respectively. CRYSTAL exhibits better data efficiency, using 1.4 trillion tokens compared to the more than 2 trillion tokens used by Llama 2 and Code Llama. We verify our pretraining strategy by analyzing the training process and observe consistent improvements in most benchmarks. We also adopted a typical application adaption phase with a code-centric data mixture, only to find out that it did not lead to enhanced performance or training efficiency, underlining the importance of a carefully designed data recipe. To foster research within the community, we commit to open-sourcing every detail of the pretraining, including our training datasets, code, loggings and 136 checkpoints throughout the training.

Main paper: 
 related work: 
 Open-source LLMs. The prevailing approach to developing modern LLMs involves a two-step process: pretraining followed by fine-tuning. The extensive pretraining stage may involve using synthetic data, as demonstrated by the Phi series models (Gunasekar et al., 2023; Li et al., 2023b) . However, the high-quality synthetic datasets used in the Phi models are not publicly available, whereas we aim to make all our training details public and reproducible. Additionally, much of the pretraining is conducted on vast datasets comprising trillions of tokens that encapsulate nearly all available linguistic data. Notable projects in this domain include (Wang & Komatsuzaki, 2021; Andonian et al., 2023; Zhang et al., 2022; Scao et al., 2022; Biderman et al., 2023; Touvron et al., 2023a; Geng & Liu, 2023; Together Computer, 2023a; MosaicML NLP Team, 2023; Almazrouei et al., 2023; Touvron et al., 2023b; Bai et al., 2023; Jiang et al., 2023; 01.ai, 2023; Bi et al., 2024; Groeneveld et al., 2024) . Of these, Pythia (Biderman et al., 2023) , LLM360/Amber (Liu et al., 2023b) and OLMo (Groeneveld et al., 2024) are particularly aligned with our work, sharing an emphasis on the complete reproducibility of LLMs. While Pythia stands out as a pioneering effort, it does not embody the recent advancements observed in training with trillions of tokens and is not specifically tailored for code. Amber and OLMo, although newer, are also designed as general-purpose models. Our CRYSTAL utilizes advanced strategic data and training strategies to create a strong open-source model that excels in coding while also demonstrating strong overall capabilities.Code LLMs. Applications at the core of fields such as software development engineering (Fan et al., 2023) place a significant demand on language models equipped with specialized code intelligence. Furthermore, models that are extensively trained on code datasets demonstrate enhanced reasoning capabilities and exhibit superior performance in logical tasks, including mathematics (Ma et al., 2024; Fu et al., 2022) . Motivated by the needs of both practical applications and research, code-oriented large models are increasingly gaining focus (Chen et al., 2021a; Li et al., 2022; Wang et al., 2023; Luo et al., 2023; Nijkamp et al., 2022; Li et al., 2023a; Roziere et al., 2023; Guo et al., 2024) . The roles of code data differ across various works. StarCoder (Li et al., 2023a) exclusively trains on code data, at the expense of general natural language understanding. DeepSeek Coder (Guo et al., 2024) incorporates more natural language into its pretraining, yet remains predominantly focused on code, with minimal enhancement in natural language benchmarks. Code Llama (Roziere et al., 2023) continues pretraining on code data atop Llama (Touvron et al., 2023a; b) , which leads to a forgetting issue that sacrifices general natural language understanding abilities. WizardCoder (Luo et al., 2023) focuses on code instruction finetuning, where the ultimate performance significantly relies on the foundational pretrained model. In contrast, CRYSTAL blends natural language and code data equally in pretraining, aiming to forge a strong code model with enhanced natural language capabilities.Several LLM training efforts have also documented multi-stage approaches. For example, the XGen project (Nijkamp et al., 2023) includes a relatively short second stage, comprising 4% of the total training tokens, which is similar to our adaptation stage. Our work further demonstrates a successful strategy for mixing two domains during the major training stages. InternLM (InternLM Team, 2023) also reports using a multi-stage training method, though the details of these stages have not been disclosed. 
model training: 
 Drawing on the principles of coarse-to-fine methodologies for achieving domain adaptation without catastrophic forgetting, we design two phases in the pretraining process of Crystal. In the first phase, the model is expected to acquire a broad spectrum of general language capabilities. In the second phase, we introduce coding ability into the model, ensuring that this augmentation does not compromise its existing natural language abilities. Table 1 summarizes the configurations of the phases.Configuration. The architecture of CRYSTAL is adapted from prior work such as GPT-2 (Radford et al., 2019) , GPT-NeoX (Andonian et al., 2023) , Llama (Touvron et al., 2023a) and BTLM (Dey et al., 2023) , featuring decoder-only models comprising 32 layers. The model is trained on a non-GPU hardware architecture, using the Cerebras Condor Galaxy 1 (CG-1) (Cerebras, 2023) . Taking advantage of the memory layout, the model can be trained efficiently using LayerNorm without RMSNorm. We incorporate a novel enhancement known as maximal update parameterization (µP), as described by Yang et al. (2022) , deciding layer-wise learning rate, batch size, Adam coefficient, etc. We include all the final hyperparameters adjusted by µP in Table 1 . The training time is 37 days on 16 CS-2 nodes. Phase 1 for Fundamental Language Ability. In the first phase, we focus on imbuing the model with foundational natural language understanding, utilizing 345B tokens sampled from SlimPajama (Soboleva et al., 2023) dataset. This dataset, primarily composed of natural language texts, includes a modest portion (approximately 5%) of coding data sourced from GitHub, subtly introducing the model to programming concepts. We expect this phase to establish a baseline comprehension of natural language, underpinning the model's subsequent specialization in code. The rationale behind starting with natural language is inspired in curriculum learning principles, positing that mastering the intricacies of natural language is a prerequisite for tackling the structured complexity of programming languages.Phase 2 for Coding Ability. In the second phase, we expand the model's domain by integrating a 63% code data mixture, drawing from a broad spectrum of programming languages from the Stack (Li et al., 2023a) dataset (following the StarCoder mixture), resulting in a total of 927B tokens. This inclusion aims to transit the model from its natural language base towards a more specialized understanding of code syntax and logic. In the meanwhile, by keeping a significant 37% of general language, we intend to prevent the catastrophic forgetting of general language ability. 1 : Pretraining configuration. We choose the warm-up steps to be approximately 0.1% of the total steps in Phase 1. For Phase 2, we reuse the same numbers. In Adaptation Phase, we set it to be 1% of the total steps. Phase Adaptation. Following prior convention (Li et al., 2023a; Roziere et al., 2023) , we conduct additional training to specialize the model on popular language tasks, using a small subset of Python and web-related data (HTML, CSS, and JavaScript) from the Stack dataset, totaling 100B tokens. A small portion (10B tokens) of SlimPajama is added to avoid catastrophic forgetting.Finetuning. We perform tuning on top of Crystal using a collection of open-source datasets, tailored for chat applications, thereby enhancing usability for end users. We denote the resulting model as CRYSTALCHAT. Details can be found in Appendix C.Figure 3 depicts the training loss curve throughout the training process, highlighting a smooth and stable progression. Loss spikes are observed at the outset of Phase 1. It's important to note that the loss scales vary across the two phases due to the increasing proportion of code data. Code data tokens, which often represent shorter raw untokenized entities such as symbols, digits, and brackets, adhere to specific syntactical rules making them more predictable, resulting in a lower loss-per-token. 
evaluation: 
 We conduct an extensive evaluation of CRYSTAL across multiple tasks, including language understanding, commonsense reasoning, code generation, and a newly crafted benchmark for website generation. We compare CRYSTAL with models developed around the same time and trained with a comparable number of FLOPs. However, some other open-weight models, such as Mistral (Jiang et al., 2023) and Mixtral (Jiang et al., 2024) , do not disclose the size of their training data, making them unsuitable for direct comparison in studying the effect of data curriculum. For similar reasons, we also do not compare CRYSTAL with commercial endpoints.We evaluate Crystal's language ability on a key set of benchmarks maintained by EleutherAI6 . We show the benchmark results on a variety of aspects in natural language, including Reasoning: Hellaswag (Zellers et al., 2019) , ARC (Clark et al., 2018) , Winogrande (Sakaguchi et al., 2021) , PIQA (Bisk et al., 2020) ; Question Answering: Open-BookQA (Mihaylov et al., 2018) , RACE (Lai et al., 2017) , BoolQ (Clark et al., 2019) , (Touvron et al., 2023b; Roziere et al., 2023; Groeneveld et al., 2024; Li et al., 2023a) .For code generation, we evaluate the models on three benchmarks, i.e., HumanEval (Chen et al., 2021b) , MBPP (Austin et al., 2021) and Multipl-e (Cassano et al., 2022) , where Hu-manEval and MBPP are measuring functional correctness for synthesizing programs from docstrings, consists of more than 1,000 Python programming problems in total. Multiple-e is a translation of HumanEval from Python to 18 programming languages. We omitted APPS (Hendrycks et al., 2021a) and DS-1000 (Lai et al., 2022) from our evaluation. The exclusion was due to the significant discrepancies in baseline model scores produced by our setup compared to those reported in their respective papers, primarily attributed to differences in prompts and evaluation configurations. The results can be found in Table 2 .WebMC: A New Benchmark on Web Programs. In addition to the three benchmarks above, we introduce a new benchmark, WebMC, created for evaluating coding abilities related to website development. This dataset was generated using GPT-4 and manually curated into 600 multiple-choice questions regarding website understanding, edit- We conduct an evaluation of CRYSTAL, CRYSTALCHAT against Llama 2, Code Llama, and their instruction-tuned variants. As shown in Figure 4 , CRYSTAL demonstrates superior average performance in this comparison. 
analysis and discussion: 
 The evaluation results show that with an appropriate data curriculum, it's possible to train a model to excel in both language and code, sometimes even outperforming models with larger token budgets on certain benchmarks. In this section, we delve into an analysis of intriguing aspects observed during training.Instruction: Question: James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week? Ground Truth: He sprints 3*3=9 times, so he runs 9*60=540 meters #### 540Phase 2 Response: 180Phase 2+FT Response: James runs 3 sprints * 60 meters each sprint = 180 meters per sprint. He runs this for 3 times a week, so he runs 180 meters per sprint * 3 times a week = 540 meters a week. The answer is 540.Instruction: Write a function to remove first and last occurrence of a given character from the string. 7 .performance levels, but after that the model can surpass the previous best value. The significant performance dips in these tasks at the beginning of Phase 2 indicate that the abrupt shift in data distribution may be harmful. In reflection, a more gradual transition to more code data may foster smoother adaptation.For MMLU, the score is almost around a random baseline at 25% in Phase 1 (with 5% code data). A faster growing trend is observed in Phase 2 (with 63% code data). The final MMLU score after two phases is even higher than strong baselines such as Llama 2. There seems to be an interesting interaction between language and code causing this phenomenon. One potential explanation can be data contamination. However, our pretraining datasets are SlimPajama (a deduplicated version of RedPajama) and StarCoder data (a dataset filtered from Github). We argue that the same or similar datasets have been used by other opensource models of comparable scales (Touvron et al., 2023a; Li et al., 2023a; Groeneveld et al., 2024) , yet similar performance gain has not been observed in these models. This disparity in performance may suggest that our model's success cannot be fully attributable to data content. We hypothesize that it can be attributed to the unique combination of data selection, model architecture, and training curriculum. Although a full pretraining ablation study is costly, our approach has been independently verified by the Snowflake Arctic team, which found that a similar three-stage curriculum learning strategy can improve key evaluation metrics like MMLU (Snowflake AI Research, 2024) . We also hope that our intermediate checkpoints will enable further study in this area.We observe that supervised fine-tuning will boost the benchmark score significantly for generation tasks, such as MBPP and GSM8K. By inspecting the output (Table 3 ), we notice that the model tend to generate complete output, not mimicking the input question and follow coding instructions closely. Furthermore, the fine-tuned model will sometimes conduct chain-of-thought style generation, which is very helpful at solving arithmetic questions. Though we do not fully rule out the chances of data contamination.Overall, the effect of mixing surpass our expectation. The performance on language and code are both high and achieve a good balance, as depicted in Figure 2 . Tasks that are more directly connected to the adaptation dataset (Python/JS/HTML/CSS), such as HumanEval/MBPP (Python-related) and Multipl-e JS/TS (JavaScript-related), show expected enhancements. For programming languages that are not reinforced, we observe mixed outcomes: some abilities improve (e.g., Java), while others suffer from certain declines (e.g., Bash and C++). However, for the majority of natural language tasks, the Adaptation Phase causes a significant performance decrease9 . We hypothesize that the sudden decrease in the data ratio for language data may be responsible for this degradation At a first glance, it seems that the Adaptation Phase brings expected specialization. However, if we take into account of further finetuning, the adaptation gains are even out afterwards. Table 7 shows that, the model after finetuning Phase 2 (Phase 2+FT) and Phase Adaptation (Phase Adapt+FT), ends up with similar performance. Contrary to the findings in prior work (Li et al., 2023a; Roziere et al., 2023) , we observe that the marginal benefits brought by this phase is not worth the increased training cost, at least with this data curriculum. Given that increased compute budget typically leads to better outcomes, we argue that LLMs should be evaluated based on compute efficiency in additional to final performance. Viewing the results of CRYSTAL through the lens of training efficiency can further validate the effectiveness of the training curriculum.Efficient Computation Stronger PerformanceIn Figure 6 we see the trade-off between total FLOPs10 spent on language data during pretraining and downstream performance. We see that CRYSTAL phases 1 and 2 models achieve stronger performance efficiently, even when compared to very strong baselines such as Llama 2. 
conclusion: 
 In this work, we present a multi-phase LLM pretraining method designed to encode natural language and coding abilities into a single model. Through this approach, we obtain a model CRYSTAL, achieving natural language and coding performance on par with Llama 2 and Code Llama, respectively. By tracking, observing and analyzing model performance throughout the pretraining, as well as a study of an additional adaptation phase, we obtain and present insights of the interplay of language and coding ability acquisition during the model training, highlighting the importance of data curriculum design.Though we have included careful analysis on the training process, it remains challenging to verify each design choice and explain every observed phenomenon during pretraining, largely due to constraints in computational resources. We will address the following limitation and release all our training artifacts, inviting the community to collaborate in overcoming these challenges:Necessity of a Smooth Transition Between Phases. Our examination in Section 5.1 reveals a slight performance drop when transitioning from Phase 1 to Phase 2 in certain benchmarks. This observation hints at the potential importance of ensuring a smooth transition between training phases. Further investigation and validation of this hypothesis could further enhance our training methodology.Impact of Code on Enhancing Reasoning Abilities. Notably, our second phase, which incorporates 67% code data, yielded unexpectedly high scores on the MMLU, a benchmark for natural language. This outcome suggests that structured code data may also boost language capabilities. A more definitive confirmation of this hypothesis could be obtained by re-running the second phase while omitting all code data, allowing for a direct assessment of its impact on learning outcomes. 
: 
 Figure 2 : CRYSTAL shows a good balance of language and coding abilities. The y-axis is the average over ARC-C, Hel-laSwag, MMLU, and GSM8K. The x-axis is the average of MBPP and HumanEval.Large Language Models (LLMs) for code generation (i.e., code LLMs), such as Codex (Chen et al., 2021a) , StarCoder (Li et al., 2023a) , and Code Llama (Roziere et al., 2023) , are advancing rapidly due to their strong capability in generating code-related content (e.g., functions), which helps improve the efficiency of software engineers and developers (Cognition Labs, 2024; Chen et al., 2021b; Li et al., 2023a; Roziere et al., 2023) . These LLMs excel at generating functions and designing web page components based on engineers' instructions (e.g., "Return True if all numbers in the list L are below threshold T.") (Cal ò & De Russis, 2023). However, the abilities of code-oriented LLMs are constrained in development contexts that necessitate interpreting highlevel human instructions (e.g., through prompts or function descriptions) and producing comprehensive, structured code accompanied by natural language documentation. Examples of such scenarios include solving GitHub issues (Jimenez et al., 2023) , searching for code snippets based on natural language queries, generating entire Python libraries (which include their complete code along with documentation and tutorials (Liu et al., 2023a; Luo et al., 2024) ), or developing source code for websites, e.g., "Create a ticketing platform for travelers" (Cal ò & De Russis, 2023) . This underscores the ambition to create LLMs proficient in both natural language processing and coding. Achieving this goal, however, is non-trivial. For instance, Code Llama, despite being continuously pretrained with code datasets on top of Llama2, suffers from catastrophic forgetting of natural language capabilities. In open-sourced LLMs, we observe a prevalent issue: most models are tailored to specialize in either language or code, not both. For example, StarCoder is exclusively trained on code datasets accompanied by function documentation, thus limiting its exposure to varied natural language data. This trend indicates a notable gap in the design of most open-source LLMs, where there's a lack of a comprehensive curriculum that addresses both coding and natural language processing. Therefore, we are intrigued by the following research question: "Can an LLM efficiently obtain both language and coding abilities?" Existing studies have shown that the simultaneous acquisition of coding and language capabilities by LLMs is governed by complex dynamics: these skills may either conflict (Li et al., 2023a; Roziere et al., 2023) or complement (Ma et al., 2024) each other, influenced by the data recipe and the model's learning phase.In this work, we propose a pretraining strategy designed specifically for code LLMs. Our strategy is inspired by techniques such as multi-phase pretraining, curriculum learning (Bengio et al., 2009) , continuous pretraining (Roziere et al., 2023) , and multi-language training, and has two phases. We start the pretraining process with a data mixture of 95% natural language and 5% code. In the second phase, the data mixture is enriched to include 63% code data alongside 37% natural language. This two-phase design mimics the human learning process, where the acquisition of general language knowledge precedes the development of coding skills, aiming to replicate this learning sequence. Pretraining using our strategy yields CRYSTAL, a code LLM that exhibits strong ability across both natural language (e.g., common sense reasoning) and code generation. Our strategy also demonstrates good data efficiency. That is, CRYSTAL, pretrained with 1.4 trillion tokens, performs comparably to Llama 2 and Code Llama, each pretrained with more than 2 trillion tokens.Throughout the pretraining process, we continuously tracked the model's performance on downstream benchmarks, observing steady enhancements in both language and coding abilities across the two training phases. Despite a slight performance decline due to the distribution shift between Phase 1 and Phase 2, performance in Phase 2 swiftly recovers and surpasses that of Phase 1. Additionally, implementing an experimental application adaptation phase, aimed at further enhancing coding abilities by incorporating an increased percentage of code data, could potentially boost performance. This phase is inspired by the Python-specialized pretraining phase of Code Llama and StarCoder (Roziere et al., 2023; Li et al., 2023a) . Contrary to expectations, we observe mixed results from this phase, including a decline in language ability but marginal improvement in coding performance (see § 5.2), underscoring the necessity for a carefully crafted data strategy.Conducting thorough ablation studies for the entire pretraining process is computationally daunting. To mitigate these challenges, we embrace the principles of the LLM360 initiative (Liu et al., 2023b) , ensuring full transparency in our pretraining process to support further scientific exploration and discoveries by the community. We release our training and fine-tuning datasets, source code for training, fine-tuning, and data preprocessing, and 152 intermediate model checkpoints. We also release a chat version, fine-tuned from CRYSTAL, namely CRYSTALCHAT, for user convenience. 
 
 Appendix: LLM360 is created with the mission to train and release open-source large language models to foster transparency, trust, and collaborative research. While large language models have demonstrated promise in advancing numerous domains throughout commercial and academic settings, the technology is still relatively poorly understood. Due to the significant capital requirements to training and experimentation with LLMs, many learnings in the space happen behind closed doors. The lack of knowledge transfer will have negative effects for the ecosystem as advances will be limited to small groups. To fully realize the potential large language models can deliver, we believe that the core tenets of transparency, trust, and collaboration are paramount to the long term success of the field.For each model released under LLM360, we will release the datasets, data preparation scripts, training code, numerous intermediate checkpoints, and complete analysis. We prioritize publicly available datasets such as The RedPajama (Together Computer, 2023b) and Refined Web (Penedo et al., 2023) and existing architectures and conventions such as LLaMA (Touvron et al., 2023a) to make our resource relevant and easy to access. By providing the listed artifacts, we hope to promote the reproducibility for all our work to encourage additional research.Datasets are expensive to curate and are a major competitive advantage for training performant models. By making all data available, our models are fully auditable. We provide clarity on all pretraining sources, the ethical manner in which data was sourced, and the actual data. Releasing checkpoints from the entire training process enables fine grained research into training dynamics (Qian et al., 2024) which would otherwise be restricted to those with the financial resources to pretrain models. We believe that the future should only be constrained by our creativity, not man-made hurdles, and hope that access to our artifacts motivates others to pursue their own creative research unhindered.Ethical Use. LLM360 models openly release our scores on safety evaluations such as Toxigen and TruthfulQA. These scores educate users to the potential risks that using our models may introduce when generating text. We gather our data from reputable sources and apply rigorous filtering to remove harmful data, but we cannot guarantee the outputs of our models will be completely safe. All users should conduct their own testing before adopting our models.LLM360 models are also trained with coding abilities. When using code generated from large language models, users should always review the output before submitting it into their codebase. Generated code may introduce issues such as insecure code which cannot be eliminated from the model. Users should perform their own safety testing and code reviews before deploying applications. Table 5 : Architecture comparison.• 
 -Marks the end of an instruction message.These tokens were integrated into the existing 32032-token vocabulary without necessitating an expansion, leveraging reserved vocabulary slots. The conversation is wrapped by the tokens 
 and 
, framing the structure as follows: 
We present the full evaluation results of our models and other open source models in Table 7 . Table 7 : Natural language and code generation evaluation results for chat models. The best scores for CRYSTAL are underlined. Numbers with † are adopted from original papers.Comparing the results for "Phase 2+FT" and "Adapt+FT", the benefits brought by the Adaptation Phase become insignificant after the finetuning, .We present evaluation results on all intermediate checkpoints. https://github.com/EleutherAI/lm-evaluation-harnesshttps://wandb.ai/llm360/CrystalCoder?nw=hdze3lfpuerExcept for MMLU, which fluctuates randomly during Phase 1.Though the MMLU score remains unaffected.Estimated by FLOPs per token times number of tokens of language data. 


Decision: This paper presents a 2-phase pretraining process with the goal of training a single model that excels at both natural language and code. Basically, phase 1 contains mostly text with a small proportion of code, while phase 2 includes more code. They demonstrate that their Crystal model achieves competitive performance on both text and code tasks compared to other open-weight models of a similar size.

This work presents insights into the effect of different pretraining stages, and it would be a valuable contribution if the authors can fully open-source their model checkpoints and code. I recommend the authors to incorporate reviewers' feedback into their camera-ready version, especially with new experimental results they promised to add.

Reviews: review: paper_summary: paper_summary: The authors introduced a pretraining strategy that focuses on training LLMs with both text and code data. The resulting family of models, CRYSTAL, is stronger in both natural language and coding performance, compared to strong baselines like LLama2 and Code Llama. The authors are also committed to releasing open-sourced model checkpoints, increasing the replicability and transparency of LLM training.
strength_weakness: strength_weakness: - The use of a mixture of datasets of code and text over multiple stages of training is not so new. For instance, CodeT5+ (https://arxiv.org/abs/2305.07922) was also trained on both code and text data over two stages of pretraining. While the model was not evaluated on conventional NLP tasks, it performed quite well over multimodal test setups such as code-to-text and text-to-code benchmarks. Can the authors clarify what is the main novelty compared to CodeT5+? Have you tried to benchmark CRYSTAL in more diverse code-text tasks beyond HumanEval and MBPP? 
- Pretraining models from scratch is expensive. I wonder if the current pretraining strategy can be applied on top of any pre-trained models. If subsequently similar insights could be observed, I think the current method would be a lot more impactful.
questions: questions: - Typo (Introduction): “pertaining” 
- A useful ablation the authors can try is to use the data in phase 1 or phase 2 but train for the same number of training steps as phase 1 + phase 2 training combined. This would isolate the effect of training steps and demonstrate better how the data mixing contributes to the performance.
paper_summary: paper_summary: Authors propose a pre-training strategy that transitions from a phase of learning from natural language heavy to code heavy dataset. This has been claimed to achieve the most optimal balance between performance on natural language tasks and coding tasks for a smaller training budget than comparable work (in terms of model size) like Llama2 and CodeLlama.
strength_weakness: strength_weakness: Authors don't discuss the motivation of needing a single model to perform well at tasks from both natural language and code domain. One possible motivation could be that Code LLMs that can follow instructions are suited for a wide variety of agentic tasks as shown in Hong et al. However, one could argue if this capability can be achieved simply by instruction fine-tuning stage on top of a code pre-trained LM.

While the impact of controlling the ratio of code to natural language data during pre-training has been studied, there is no discussion on the quality of the dataset used. Recent work on phi-2 (https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) has shown performance superior (MMLU, HumanEval, MBPP) to Crystal when trained on a similar size of data ~1.4T tokens, and using a smaller model in terms of parameter count (2.7B). Comparing this work to phi-3 or Llama 3 wouldn't be fair given the timeline of events.

Hong et al MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
questions: questions: - The CG-1 architecture is not described in sufficient detail. Besides LayerNorm what architectural changes are required for training on this non-GPU hardware?
- What is the training duration? (similar to how GPU hours are being reported in other studies)
- In Fig 3, can the authors explain the occurrence of spikes in the training loss during stage 1?
- Why is the qk dot product scaling different compared to the commonly used $\sqrt{d}$ formula? (Table 4)
- What does rotary percentage mean in Table 4?
- What is the source of the 100B tokens in the phase adaptation stage? What is the motivation in including this stage?
- Creating a benchmark like WebMC is an interesting contribution. Can the authors provide qualitative analysis to describe what the tasks here look like?
- In 5.2 authors mention the adaptation phase to be hurting NL tasks. Could the reason be that the fraction of Slim Pajama is too little in this stage?

Suggestions:
- Pre-trained models like X-Gen, Phi series could be added to the related work section. XGen (by Nijkamp et al https://arxiv.org/abs/2309.03450) specifically discusses the approach of having different stages each with different sampling proportions. 
- Besides k-shot, authors could also try CoT prompting w/ the Crystal models to push performance on the NL and code benchmarks, particularly MATH and GSM8k.
- A summary of model architecture details could be a part of the main paper (currently in Table 5 and Appendix B), similarly Table 7 should also be a part of the main paper. There is quite some whitespace (e.g. Fig 5 (could use shared legends), Fig 6, Section 6) that could be removed to achieve this compression.

Minor typos/linguistic issues:
- Abstract: Futhermore, there lack of --> there is a lack of?
- Section 3 Phase 2: we intent to --> we intend to?
- Section 5.2 Para 2: adaptation gains are even out
paper_summary: paper_summary: This paper investigates the integration of natural language and coding capabilities within code LLMs. The authors introduce a novel pretraining strategy that involves two training phases with a carefully adjusted mix of code and natural language data. Their proposed model, Crystal, demonstrates comparable performance in natural language processing and coding tasks to established models like Llama2 and Code Llama, but with greater data efficiency, using 1.4T tokens versus over 2T used by the other models. The effectiveness of the new pretraining approach is supported by detailed analysis of the training process and benchmarking, which show consistent improvements.
strength_weakness: strength_weakness: 1. It remains unclear whether the observed performance improvements are due to the two-phase training method or if similar outcomes could be achieved with a single-stage training using a combined dataset. The current ablation study does not address this question. Although retraining the full model is resource-intensive, conducting a smaller-scale experiment, perhaps with a 1B parameter model, could validate the effectiveness of the two-phase approach, which is central to this paper’s contributions.

2. While the paper benchmarks Crystal against notable models, it omits comparisons with key baselines like GPT-3.5 and GPT-4, which are recognized for their proficiency in both coding and NL tasks. Additionally, including comparisons with models like Mixtral-8x7B, which also excel in both domains, would enhance the robustness of the evaluation.

3. *[Minor]* The pretraining strategy proposed in this paper, which consists of two distinct phases using different mixtures of NL and code datasets, is too straightforward and lacks significant novelty.

4. *[Minor]* The evaluation methodology appears somewhat biased towards NLP, as the evaluations on the coding aspects (3 tasks) are not as comprehensive compared with on the NLP side (11 tasks). Expanding the evaluation to include additional coding benchmarks such as APPS or DS-1000 could provide a more balanced view of the model's capabilities.
questions: questions: See weakness 1. Is it possible to see if performance improvements are due to the two-phase training method or if similar outcomes could be achieved with a single-stage training using a combined dataset?

Comments: title: Response to rebuttal
Thank you for your response
comment: I thank the authors for the clarifications provided, and am revising my score accordingly.
Thanks to the authors for replying to my questions. I will keep my original review score. I look forward to the additional results and the full release of the models, both would significantly help the research community with useful insights.
Thank you for your response and clarifications. I decide to keep my original review score.