LLM360: Towards Fully Transparent Open-Source LLMs

Authors: zhengzhong  liu
aurick  qiao
willie  neiswanger
hongyi  wang
bowen  tan
tianhua  tao
junbo  li
yuqi  wang
suqi  sun
omkar  pangarkar
richard  fan
yi  gu
victor  miller
yonghao  zhuang
guowei  he
haonan  li
fajri  koto
liping  tang
nikhil  ranjan
zhiqiang  shen
roberto  iriondo
cun  mu
zhiting  hu
mark  schulze
preslav  nakov
timothy  baldwin
eric p xing
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-15 17:42:21

Abstract: The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present **LLM360**, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and Crystal, including their training code, data, intermediate checkpoints, and analyses. We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.

Main paper: 
 introduction: 
 The landscape of Large Language Models (LLMs) has experienced a remarkable transformation in the past one year, witnessing an unprecedented surge in both the popularity and capabilities of these models. At the forefront of this evolution are proprietary LLMs such as GPT-4 (OpenAI, 2023) and Claude (Claude, 2023) , which have captured the attention of the AI community due to their power and versatility. At the same time, the recent emergence of openly accessible yet highly capable LLMs such as LLaMA (Touvron et al., 2023a; b) , Falcon (Penedo et al., 2023), and Mistral (Jiang et al., 2023) have allowed researchers and practitioners at large to easily obtain, customize, and deploy LLMs in more diverse environments and for more diverse use cases.Despite the growing influence of open-source LLMs, a notable trend has been to restrict visibility and access to their training, fine-tuning, and evaluation processes, including crucial components such as their training code and data. This practice limits the ability of the broader AI research community to study, replicate, and innovate upon advanced LLMs. A more transparent approach to sharing not just the final model but also training details and artifacts is crucial for fostering a more inclusive and collaborative research environment. Motivated by the above, we note the following specific challenges in LLM research today. Data Provenance. Understanding the origins and characteristics of training data is crucial for assessing the reliability and biases inherent in LLMs. A lack of transparency about data sources and composition hinders the ability to identify and mitigate biases which can be perpetuated in model outputs. Simultaneously, data leakage-where training datasets overlap with benchmark datasets-can lead to misleading performance metrics that obscure a model's general effectiveness (studied in Wei et al. (2023) ; Zhou et al. (2023) ). These issues highlight the need for clear documentation of data origins and usage in LLM development.Even with full disclosure of data sources, the lack of access to complete training code, configuration details, and specific datasets can make it challenging to reproduce the results reported in studies. For example, although training data mixtures are disclosed by LLaMA (Touvron et al., 2023a) , the data processing and training code are not released. Yet, LLMs trained using an open reproduction of LLaMA's data (e.g., RedPajama (Together Computer, 2023b;a)) do not fully reproduce its benchmark evaluations (Geng & Liu, 2023) , indicating that additional data processing or training procedures may be necessary.Open Collaboration. The practice of only releasing final model weights not only leads to redundant efforts but also poses unique challenges in conducting certain research. For instance, research into the emergent abilities of LLMs (Biderman et al., 2023a; Wei et al., 2022) or the investigation of how different training data affects model behavior (Yu et al., 2023; Xie et al., 2023) becomes more challenging without access to intermediate training checkpoints. Researchers are often forced to either work with the final model, which offers limited insights into its developmental nuances, or start from scratch, leading to unnecessary duplication of work and expenditure of compute.LLM3601 aims to address the issues above through a comprehensive open-source LLM effort. Models in LLM360 are published with all training and model details (e.g., hyperparameters, schedules, architecture, and designs), all intermediate model checkpoints saved during training, and full disclosure of the exact pre-training data used.Our contributions are:• We outline the LLM360 approach, focusing on its design principles and the rationale for fully open-sourcing LLMs. We detail the core open-sourced artifacts, including datasets, code and configurations, model checkpoints, and metrics. We provide a target for transparency that all present and future LLM360 models strive to meet.• We pretrain two new LLMs from scratch and release them under the LLM360 framework. The first LLMs in the LLM360 series: AMBER is a 7B English LLM pretrained on 1.3T tokens. CRYSTAL is a 7B English and code LLM pretrained on 1.4T tokens. We discuss the development details, preliminary evaluations, observations, and lessons we learned from AMBER and CRYSTAL. The rest of this paper is organized as follows. In §2, we discuss related work and the predecessors that inspired LLM360. In §3, we provide a description of the LLM360 framework and the release artifacts that fall into its purview. In §4, we discuss the AMBER and CRYSTAL models, and preliminary analyses of both. §7 concludes. 
related work: 
 The closest previous work to LLM360 is Pythia, which ensures full reproducibility (Biderman et al., 2023b) . The Pythia project provided 154 checkpoints for model sizes from 70M to 12B with the same data order to better support research on interpretability, learning dynamics, ethics and transparency. While Pythia is a pioneering work, it no longer reflects many recent LLM practices, such as training with more tokens (Touvron et al., 2023a) than the compute-optimal size (Hoffmann et al., 2022) , or multi-phased, curriculum training on different data mixes (Gemini-Team et al., 2023; InternLM Team, 2023) . On the other hand, LLM360 takes an approach prioritizing transparency and reproducibility under which up-to-date models can continue to be released, and our new 7B models surpasses the 12B Pythia model in public benchmarks (Beeching et al., 2023) . Overall, Pythia set an early precedent for transparency and reproducibility of LLMs that we aim to perpetuate and expand in LLM360 to modern LLM pretraining regimes. In general, open-source LLMs span a wide spectrum of transparency and reproducibility when it comes to their release artifacts. Many recent LLMs only release their final model architecture and weights, keeping their data sources and most training details undisclosed (Touvron et al., 2023b; Bai et al., 2023; Jiang et al., 2023; 01.ai, 2023) . Some are trained on publicly available datasets (Wang & Komatsuzaki, 2021; Black et al., 2022; Scao et al., 2022; Biderman et al., 2023b; Geng & Liu, 2023; Together Computer, 2023a; Shen et al., 2023) , whereas others disclosed their data mixtures but do not make training-ready data available to the public (Zhang et al., 2022; Touvron et al., 2023a; MosaicML NLP Team, 2023; Almazrouei et al., 2023) . Several LLMs of note have been released with substantially more transparent details and artifacts. The release of GPT-J (Wang & Komatsuzaki, 2021) and GPT-NeoX (Andonian et al., 2023) (Groeneveld et al., 2024; Zhang et al., 2024; Üst ün et al., 2024) similar to the LLM360 setting.✓ ✓ ✓ * ✓ ✓ ✓ 1.3 CRYSTAL Dec'23 ✓ ✓ ✓ ✓ ✓ ✓ ✓ 1.4 OLMo 7B † (Groeneveld et al., 2024) Feb'24 ✓ ✓ ✓ ✓ ✓ ✓ ✓ 2.46Overall, we observe a trend that (before late 2023), more recent and capable LLMs are becoming more closed in their release artifacts. In contrast, the goal of LLM360 is to release modern and high-quality models while maintaining a high degree of release transparency. 
the llm360 initiative: 
 In this section we present LLM360, an initiative to open source LLMs that promotes transparency, reproducibility, data/model provenance, and collaborative research. LLM360 provides guidance and recommendations for release artifacts that are collected during LLM pre-training and subsequently made publicly available to the community.As part of the launch of LLM360, we also release two new pre-trained LLMs, which we hope will foster immediate interest and collaboration in the open-source research community. First, AMBER, an English language LLM with 6.7B parameters trained on 1.25 trillion tokens. Second, CRYSTAL, an English and code LLM, also with 6.7B parameters, trained on 1.4 trillion tokens. Details on AMBER and CRYSTAL are reported in §4. Training Dataset and Data Processing Code. The pretraining dataset is a main ingredient of an LLM and it is important for users and adopters to have visibility into this data to assess potential behavior issues and biases. For example, recent concerns about benchmark data leakage into LLM pretraining is much easier to study when pretraining datasets are available (Zhou et al., 2023; Wei et al., 2023) .Furthermore, visible data improves the extensibility of LLMs in further training. For example, recent work suggests that training on repeated data disproportionately degrades final model performance (Hernandez et al., 2022) . Given the breadth of data in modern pretraining, visibility into the original pretraining data is essential for avoiding repeated data in downstream fine-tuning or continual pretraining on specialized domains.LLM360 advocates for the public release of the pre-training data.When applicable, details about data filtering, processing, and training order should be released as well. Doing so equips the community with better tools to assess the capabilities and risks of LLMs and to reproduce and build upon existing LLMs for future use cases.Training Code, Hyperparameters, and Configurations. These code and settings have a significant impact on the performance and quality of LLM training, and are not always publicly disclosed. For example, we observed that a carefully balanced hybrid data-model-pipeline (3D) parallelism (Narayanan et al., 2021) can outperform the standard FSDP in PyTorch by up to 15% on our Nvidia A100 clusters. Another example is that it is essential to keep the inverse frequency matrix in RoPE positional embedding in FP32 (Su et al., 2021) , which aligns with the observation in Qwen (Bai et al., 2023) .In LLM360, we open-source all of our pre-training frameworks, hyperparameters, and configurations, including the full training source code, training parameters such as learning rates and batch sizes, and system configurations such as parallelism dimensions. LLMs undergo training over weeks to months, and the trends and evolution patterns over this training period can offer valuable information. However, access to detailed logs and intermediate metrics for LLMs is typically limited. These statistics often contain key insights that cannot be directly derived, and even a simple analysis on the metrics, such as computing metric variances or norms, can reveal significant findings. For instance, the team behind GLM proposed an effective gradient shrinking algorithm for handling loss spikes and NaN losses by analyzing gradient norm behaviors (Zeng et al., 2023) .Our aim with LLM360 is to alleviate this problem by completely open sourcing the logs and metrics we collect. This includes system statistics (e.g., GPU workload), training logs (e.g., loss, gradient norm), and evaluation metrics (e.g., perplexity, downstream tasks). Access to these logs may facilitate a deeper understanding of the whole training process, including how LLMs evolve during various training scenarios. We provide easy access to the figures by sharing directly on the LLM360 Weights & Biases page2 . A few example metrics include downstream evaluation results, training loss, gradient norm, etc.In §5.1, we introduce how one can make use of the metrics, and illustrate an experiment tracking the memorization behavior of a model throughout training. The metrics are released in coordination with the data chunks and checkpoints for researchers to easily find their correspondence. Furthermore, we provide open access to the analysis and evaluation code used to foster reproducibility. 
amber and crystal pre-trained llms: 
 In this section, we introduce two model families: AMBER, the first model in the LLM360 family, along with finetuned models AMBERCHAT and AMBERSAFE; and CRYSTAL, a codefocused model, along with the finetuned model CRYSTALCHAT.Below we review the details of our pre-training dataset, including data preprocessing, format, data mixing ratios, architectural details, and specific pre-training hyperparameters. We also publish all details in the LLM360 code base.We summarize all details in Table 2 . AMBER uses a similar model architecture as LLaMA 7B, including model dimensions, use of RMSNorm (Zhang & Sennrich, 2019b) , and rotary positional embeddings (RoPE) at each layer of the network (Su et al., 2021) . CRYSTAL further incorporates maximal update parameterization (µP) (Yang et al., 2022) The instruction following ability (Ouyang et al., 2022) of language models has significantly simplify the adoption of LLMs in pratical applications. Similar methods are adopted by the open source community (Taori et al., 2023; Chiang et al., 2023) . We released both supervised instruction finetuning and preference tuning versions of the models.AMBERCHAT and AMBERSAFE. We train AMBERCHAT, a supervised finetuning model, on enhanced ShareGPT data (Xu et al., 2023) . We use FastChat (Zheng et al., 2023) to finetune the model for 3 epochs, where learning rate is 2×10 -5 , gradient accumulation steps is 16, and warmup ratio is 0.04. We further create a preference tuned version AMBERSAFE, using Direct Parameter Optimization (DPO) (Rafailov et al., 2023) , using the SafeRLHF dataset (Ji et al., 2023) . Here we set β to 0.1, gradient accumulation steps to 4, and the learning rate to 5×10 -7 .CRYSTALCHAT. The selection of finetuning data for CRYSTALCHAT features both language and coding proficiency. We use a 1.1 billion token dataset collection (see Appendix G.2 for details). CRYSTALCHAT is proficient in programming and excels in web programming.The goal of AMBERCHAT and CRYSTALCHAT are to produce instruction-following models which are commonly used for research purposes. An additional goal of AMBERSAFE is to carry out a stage of safety-alignment in addition to instructing-tuning, so that the effects of each stage can be separately studied, whereas most open-weight models today only release the final result after both stages. 
amber and crystal evaluation: 
 Benchmark Results. We conduct a wide range of evaluation on our models, and release the evaluation results on checkpoints throughout the training for observation across the pretraining process. All metrics are publicly available5 . Figure 4 : Results for AMBER on the Open LLM leaderboard metrics. Figure 4 shows four benchmark metrics for AMBER: ARC (Clark et al., 2018) , Hel-laSwag (Zellers et al., 2019) , MMLU (Hendrycks et al., 2021) , and TruthfulQA (Lin et al., 2022) , evaluated with the Open LLM Leaderboard6 settings. We can see that the HellaSwag and ARC evaluation scores monotonically increase during pre-training, while the Truth-fulQA score decrease as the training proceeds. Interestingly, the MMLU score decreases and fluctuates in the initial stage of pretraining and then starts to increase at around 70% progress.Figure 5 shows the metrics for CRYSTAL, with two additional metrics evaluating code generation: MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021) . For most of the language metrics, we oberve a drop during transitions between each stage, since there is a drastic shift of domain composition (Table 2 ). Interestingly, the performance on MMLU consistently improves over phases 1 and 2, with scores starting to rise above a random guess quickly in stage 2, in contrast to the trend observed in AMBER. The performance of AMBER model lags slightly behind some, especially newer 7B models.While our results are close to models released around the same time. We hypothesize this is due to the dataset choices, and the use of an incorrect precision when storing early checkpoints (Appendix E). Our next training effort CRYSTAL, with a more careful data strategy (e.g., curriculum, deduplication), produced significantly better results on benchmarks. To evaluate our tuned models, we evaluated AMBER, AMBERCHAT, and AMBERSAFE with MT-Bench (Zheng et al., 2023) . The models scored 2.49, 4.95, and 4.97, respectively (Table 9 ).Prior work such as Pythia (Biderman et al., 2023b) has shown insight results by analyzing intermediate model checkpoints during training. Our hope is that LLM360 can also provide the community with useful resources for both reference and research purposes. To this end, we release the initial version of the ANALYSIS360 project, an organized repository to analyze various aspects of the model behavior, including model characteristics and downstream evaluation results.As an example of the analysis that can be performed using the full set of model checkpoints, we conduct an initial study on memorization in LLMs. Recent work (Carlini et al., 2021; 2022) shows that LLMs may memorize a significant part of their training data, which can be extracted with appropriate prompting. Such memorization not only raises privacy concerns in leaking private training data, but also downgrades the performance of LLMs if the training data contains unintended duplicates or peculiarities. To study this behavior, we carry out memorization experiments using the set of checkpoints and data from AMBER.We adopt the memorization score introduced in (Biderman et al., 2023a) and also test 32extractable (Carlini et al., 2021) sequences (see § D for full details and definitions). We sample 1000 sequences from each of the 360 data chunks, and use the first 64 tokens of each sequence to conduct experiments. Results are shown in Figure 6 . On the left we show the distribution of memorization scores over 10 selected checkpoints. On the right, we group the data chunks based on their next selected checkpoint and plot the memorization score for each data chunk for each checkpoint. As demonstrated, the LLM360 released artifacts enable reproducible research and provide means for the community to study LLM technology. 4 : Evaluation comparisons among a few notable code and language base models. The last column is the average of the language task average and the code task average. CRYSTAL strikes a good balance between both language and code tasks. readers to understand the details of pre-training (e.g., loss curve behaviors, how evaluation metrics emerge) without needing to do so themselves. We also provide some of our learnings and potential use cases showing how one can use LLM360 for their own projects.• In AMBER, NaN losses were periodically observed, which may have been caused by certain random states, training precision, or data quality issues. Our solutions included switching to a different random seed or skipping those data chunks. We noticed some "misbehaved" data chunks can cause NaN loss regardless of when they appear in training-for example, even after we moved them to the end of the training.• We observed that a hybrid and carefully tuned parallelism strategy-combining data, tensor-model, and pipeline (also referred to as 3D) parallelism strategies (Narayanan et al., 2021 )-achieves better system throughput than FSDP (Zhao et al., 2023) , especially in distributed clusters with limited inter-node bandwidth.• Data quality, mixing, and sequencing are crucial aspects of LLM pre-training. For AMBER, we adhered as closely as possible to the settings in LLaMA. However, AMBER's performance still lags behind LLaMA's-a key omission in LLaMA's technical report is a detailed description of its data preparation. On the other hand, our CRYSTAL dataset and which mixes English and code in three stages, achieves competitive performance with LLaMA on both the Open LLM Leaderboard and Code Evaluation benchmarks.• One can conduct experimental studies at any stage of model training. As previously mentioned, the optimal data mixing ratio remains a significant open problem in LLM pretraining. However, it is nearly impossible to verify a specific mixing ratio by conducting full LLM pre-training. A more feasible approach is to adjust the data mixing ratios on the fly, i.e., starting from an LLM360 checkpoint, and either increasing or decreasing a specific data ratio from a particular category, e.g., increasing the data weight in Wikipedia.• Studies of trustworthiness in LLMs can benefit substantially from access to intermediate checkpoints, plus controllability of trustworthiness (Qian et al., 2024) .• For building domain-specific LLMs (e.g., medical, finance, law, etc.), one may not necessarily want to start from the final LLM checkpoint (which is more akin to fine-tuning). Instead, one can always pick one of the LLM360 checkpoints (e.g., from 50% of the pre-training stage) and resume the pre-training to obtain a domain-specific LLM.• Many algorithmic approximation frameworks for efficient training require partially trained model weights (Wang et al., 2021; 2023) . LLM360 can provide suitable model initializations for these methods. Issues Encountered During Pre-training. We believe that it is important to be fully transparent and discuss issues encountered during pre-training, which can be a major help to the community. We discuss several major issues encountered during the pre-training process of AMBER in Appendix § E. These issues could potentially impact our final model performance, and we aim to address most of these issues in subsequent LLM pre-training efforts. Carbon Footprint. For AMBER, the estimated GPU-only power consumption is 63.9 MWh, representing a carbon footprint of 24.6 tCO2eq. We do not have the specifications to estimate the full system consumption. Estimating chip-only consumption for Cerebras CS-2 systems is difficult, but the estimated total carbon emission for CRYSTAL is at a similar scale of AMBER. LLM360 and Responsible Usage. Given the wide-ranging applicability and high performance of LLMs, it is essential to carefully manage the potential impact and risks associated with them. We believe the transparent nature of LLM360 can help make the potential risks known to all stakeholders. As one example, one form of risk associated with LLMs includes biases (Weidinger et al., 2022) , such as social stereotypes, discrimination and exclusion, that may result from under-representation of certain languages or domains. By inspecting the exact training data and bias analysis (e.g., BOLD (Dhamala et al., 2021) ) in ANALYSIS360 (also see Appendix A), stakeholders can have a thorough review of these risks before deploying the models. LLM360 can also help with risk mitigation. The project shares reproducible traces and exact data during LLM training, providing a reusable environment for researchers to conduct experiments to design better guardrails to contain potential risks. 
conclusion and future work: 
 In this paper, we introduced LLM360, an initiative for comprehensive and fully-transparent open-source large language models. As an initial effort, we have released two pre-trained 7B LLMs: AMBER (a general-purpose English language LLM) and CRYSTAL (an LLM pre-trained with a focus on code generation), along with the fine-tuned models AMBERCHAT, AMBER-SAFE, and CRYSTALCHAT. In terms of artifacts, we released pre-training code, configurations, hyperparameters, intermediate model checkpoints, optimizer states, as well as the data sequence and data processing code. Our vision is to significantly advance and promote transparency within the open-source LLM pre-training community.For future work, we are conducting a more detailed analysis of AMBER and CRYSTAL's base models as well as their fine-tuned models. Detailed results will be released and discussed in their respective technical reports. Our team is also pre-training a much larger LLM, which will be fully released as soon as the pre-training is complete. Additionally, we will explore the optimal ratios for mixing different subsets in the pre-training datasets.LLM360 models are also trained with coding abilities. When using code generated from large language models, users should always review the output before submitting it into their codebase. Generated code may introduce issues such as insecure code which cannot be eliminated from the model. Users should perform their own safety testing and code reviews before deploying applications. AMBER. We choose AMBER pre-training hyperparameters by following LLaMA as close as possible (Touvron et al., 2023a) . We use RMSNorm (Zhang & Sennrich, 2019a ) with ϵ = 1e -6 . We use the AdamW optimizer with β 1 = 0.9 and β 2 = 0.95, the initial learning rate is set to η = 3e -4 , following a cosine learning rate schedule that decreases to a final rate of η = 3e -5 . We apply a weight decay of 0.1 and use gradient clipping at 1.0. The model is warmed up over 2,000 steps. Differing from the LLaMA setup, based on our hardware setting with 224 GPUs, we use a pre-training batch size of 2,240 (224 × 10) instead of 2,048. We uses FlashAttention (Dao et al., 2022) for fast attention computation. 
 
 Appendix: The architecture of CRYSTAL is adapted from prior work such as GPT-2 (Radford et al., 2019) , GPT-NeoX (Andonian et al., 2023) , LLaMA, and BTLM (Dey et al., 2023) , featuring decoder-only models comprising 32 layers.The model is trained on a non-GPU hardware architecture with Cerebras Condor Galaxy 1 (CG-1), hence some of the model architetures differ from the common choices. Due to the memory layout of the hardware, the model can be trained efficiently without using RMSNorm.We incorporate a novel enhancement known as maximal update parameterization (µP), as described by Yang et al. (2022) , enabling uniformity in hyperparameters including optimizationrelated hyper-parameters, i.e., learning rate, batch size, Adam coefficient, etc., and initializationrelated hyper-parameters across models of varying widths.The µP hyperparamter scalling are allpied to the following parameters.1. Input embeddings are scaled by mup embeddings scale.2. Output logits are scaled by mup output alpha × mup width scale.3. Attention weights scaling is refined to division by the hidden dimension size AMBER is trained on a GPU cluster of 56 DGX A100 nodes, each equipped with 4× 80GB A100 GPUs. Each GPU is connected with 4 links NVLink. Cross node connection setting is 2 port 200 Gb/sec (4× HDR) InfiniBand. The throughput we achieved with our distributed training framework is around 582.4k tokens per second. Our pretraining framework is lit-llama7 developed based on PyTorch Lightning. We used mixed-precision during pre-training with BF16 for activations and gradients and FP32 for model weights (Micikevicius et al., 2017) .Here we provide additional details about our analysis on memorization over the sequence of LLM360 model checkpoints. As mentioned above, to show an example of the analysis that can be performed over the set of model checkpoints, we conduct an initial study on memorization in LLMs. Recent work (Carlini et al., 2021; 2022) shows that LLMs may memorize a significant part of their training data, which can be extracted with appropriate prompting. Such memorization not only raises privacy concerns in leaking private training data, but also downgrades the performance of LLMs if the training data contains unintended duplicates or peculiarities. As we release all checkpoints and data, we can conduct a comprehensive analysis of memorization across the whole stage of training.We adopt the memorization score introduced in (Biderman et al., 2023a) , indicating the accuracy of tokens in the continuation of length l with a prompt of length k,score(k,l) = 1 l l ∑ i 1[S k+i = G k+i ],where S 0:k+l is the sequence from training data, while G k:k+l is the generated sequence with prompt S 0:k . A memorized or k-extractable (Carlini et al., 2021) sequence has a memorization score of 1. Following (Biderman et al., 2023a; b) , we conduct our experiments with k = l = 32. We sampled 1000 sequence from each of the 360 data chunks, and use the first 64 tokens of each sequence to conduct the following experiments.We show the distribution of memorization scores for 10 selected checkpoints in Figure 6 (left), and additionally annotate the percentage of score = 1. For every checkpoint, we only include the data chunks it has already been trained on. From the result, we learn that 1) More than 1% of the sequences are 32-extractible from AMBER; 2) AMBER can memorize more sequences as training progresses; 3) The spike at score = 1 indicates that AMBER can memorize a much larger number of tokens than our preset threshold 32 (consistent with prior work (Carlini et al., 2022; Biderman et al., 2023a) ).We group the data chunks according to the selected checkpoints, and plot the memorization score on each data chunk group for each checkpoint in Figure 6 (right). We find that 1) AMBER checkpoints memorize the latest seen data much more than previous data; 2) For each data chunk, the memorization score drops a bit with additional training, but keeps increasing afterwards.In addition, we show the correlation between sequences in terms of memorization score or k-extractable metric in Figure 7 . We witness a strong correlation between the checkpoints. In this section, we discuss several major issues encountered during the pre-training process of AMBER. These issues could potentially impact our final model performance. We have addressed most of these issues in subsequent LLM pre-training efforts.NaN loss on a few data chunks. During the pre-training procedure, we encountered NaN loss in four out of 360 data chunks. Whenever we faced this issue, we tentatively skipped the entire data chunk. Initially our plan was to train on these four data chunks in later stage of the training, however, we found that these data chunks tend to cause NaN loss regardless of the position of training. We end up finishing our training by taking the first four chunks from the training sequence to complete our learning rate schedule.Missing optimizer states. In our pre-training framework, we did not manage to save the optimizer states; we only saved model checkpoints for each data chunk. This oversight might be the cause of the NaN loss issue observed in the four data chunks, as mentioned earlier.Each time we resumed pre-training from a previous model checkpoint, the optimizer state in the AdamW optimizer was re-initialized. This re-initialization could potentially affect model training stability.Discrepancies on the precision of checkpoints. In the initial phase of pre-training, our codebase had an issue where model checkpoints were saved with BF16 precision, despite our mixed precision training process maintaining model weights at FP32. This issue was later identified and rectified by our team, ensuring that all subsequent model checkpoints were saved with FP32 precision. We anticipate that the initial BF16 model checkpoints may have contributed to some degree of accuracy drop in the model.We also provide additional evaluations of our models against other open-weight and opensource language models. In Table 8 , we evaluate all models using the exact OpenLLM leaderboard settings.In comparison, AMBER performs similar to open source models released around the same time (Falcon, MPT, Incite) with a slight advantage on MMLU but weaker in other metrics like ARC. CRYSTAL, our newer model, is comparable with other models released around the same time (Llama2, OpenLlama), where it shows a strong MMLU score of 48.78, as compared to Llama2's 43.80. As LLMs are advancing quickly, newer models in the list, such as Gemma, Qwen1.5 and Olmo 1.7 are generally better.To study the effect of instruction tuning and safety alignment, we compute the MTbench (Zheng et al., 2023) score of AMBER, AMBERSAFE and AMBERCHAT, as long as a few reference models.To evaluate our models on further metrics, such as coding, we have conducted additional evaluations with newer version of LM-Harness and other benchmark suite (the exact evaluation settings are available together with our released evaluation code). We show evaluation results for these metrics in Table 10 . Mistral-7b-sharegpt-90k 6.787975Table 9 : MT-Bench Scores for Amber model family and reference models, averaged over 3 runs. We compared it with Mistral 7B trained on ShareGPT-90K, to get a rough perspective.We introduced four special tokens to the tokenizer. These tokens were integrated into the existing 32032-token vocabulary without an expansion, leveraging reserved vocabulary slots. The conversation is wrapped by the tokens 
 and 
, framing the structure as follows:
 
 system prompt 
 
 first user utterance <| → im_end|> first model response 
 next user utterance 
 → next model response 
 The name LLM360 signifies open-sourcing LLMs from all angles, and that 360 data points (i.e., checkpoints, data chunks, evaluation results) are released for our first model.https://wandb.ai/llm360https://github.com/openlm-research/open llama#dataset-and-traininghttps://www.cerebras.net/condor-galaxy-1https://wandb.ai/llm360https://huggingface.co/spaces/HuggingFaceH4/open llm leaderboardDiscussion and Take-home MessagesPre-training LLMs is a computationally daunting task that many academic labs or small organizations cannot afford. We hope LLM360 provides comprehensive knowledge, allowinghttps://github.com/Lightning-AI/lit-llama 


Decision: The authors introduce LLM360, an effort towards building LLMs entirely in the open. In addition to releasing the code, data, and checkpoints from the training of two 7B models, Amber and Cystal, the authors introduce the LLM360 initiative, which is an ongoing LLM-in-the-open project.

The main weaknesses the reviewers raised include:

- There are already similar open LLM projects, e.g., Pythia, that the authors added a comparison two in rebuttal. Notably, two other more-open-efforts are discussed: OLMo (early 2024) and MAP-Neo (released after COLM deadline).
- A few missing comparisons, e.g., chat benchmarks, carbon footprint, etc. (which the authors added in rebuttal)

Overall, I concur with the reviewer consensus that this work is a strong contribution: I see no reason why another promising open LLM effort should not be presented at COLM, esp. given that this work is mostly-concurrent with OLMo/MAP-Neo (and outperforms the prior similarly open model pythia, which is acknowledged/cited/discussed; the authors also promised quantitative comparisons in the camera ready and this seems feasible.)

[ethics comments from PCs] The authors discuss about bias being a dimension of Analysis360, but do not evaluate on any datasets. Moreover, there is no discussion of Personally Identifiable Information (PII) in training datasets and ways to avoid them while releasing these datasets. It would be useful to either provide quantitive numbers on bias or provide an ethics statement that discusses both bias and PII protection. **Ethics statement is mandatory for the revision of the paper.**

Reviews: review: paper_summary: paper_summary: This paper discusses the importance of open-source large language models and makes a step in this direction by proposing LLM360, an initiative which encourages the community to make publicly available training data, code, model checkpoints and intermediate results. The authors make a step in this direction by releasing two 7B LLMs pretrained from scratch for the English language, Amber (1.3T tokens) and Crystal (1.4T tokens), altogether with their pre-training data, code, intermediate checkpoints and analyses. In addition, the authors also release corresponding fine-tuned models AmberChat, AmberSafe, CrystalChat.
strength_weakness: strength_weakness: The paper does not discuss the goal of fine-tuning the models. It is unclear from the paper what AmberChat, AmberSafe, CrystalChat have been finetuned to do and for which tasks they can be readily used.

The authors do not provide an explanation why on some metrics the Amber model does well, while on some other metrics (ARC) the model is behind the curve. 

This statement should be clarified: “We also find that our finetuned AMBER models are relatively strong, even compared with other similar models.”

The paper frequently mentions the Pythia (Biderman et al., 2023b) model, but there is no direct comparison between the models of this paper with Pythia.

No discussion related to data quality used for pre-training and safety aspects of the model. The paper would greatly benefit from more detailed discussion of data selection and analysis of model biases, controllability and trustworthiness. 

“For future work, we are conducting a more detailed analysis of AMBER and CRYSTAL’s base models as well as their fine-tuned models. Detailed results will be released and discussed in their respective technical reports.” - In my view, this detailed model analysis should not be left to future work, but included in the current paper.
questions: questions: What guardrails are in place to contain the potential risks associated with the models you plan to release?
paper_summary: paper_summary: The LLM360 initiative addresses a critical need in the AI research community for complete transparency in the training of Large Language Models (LLMs). The quality of this work is evident in the comprehensive approach to open-sourcing, which includes training code, data, model checkpoints, and intermediate results.
strength_weakness: strength_weakness: 1. The paper lacks extensive comparative analysis with existing LLMs, which could better demonstrate the efficacy and efficiency of the LLM360 models.
2. The long-term sustainability of continually open-sourcing such extensive resources is not discussed, which could be a potential hurdle in widespread adoption.
questions: questions: 1. How do you plan to sustain the LLM360 initiative in terms of funding and resource allocation?
2. Can you elaborate on any legal or ethical considerations in open-sourcing such extensive datasets and model details?
paper_summary: paper_summary: This work present a fully open-source LLMs initiative LLM360, where not just model weights, but all the training data, training code, model checkpoints and intermediate results are made available to the community. This initiative is significant for the community as it addresses several important challenges LLM research is dealing with - data provenance, reproducibility and open collaboration. 

In the paper, the authors outline the LLM360 approach, as well as pretrain two new LLMs (Amber and Crystal) from scratch and release them under the LLM360 framework, together with all the associated core open-sourced artifacts. The paper is very well written and clearly structured, the rationale behind open-sourcing the core artifacts is properly explained, the process of pre-training newly presented LLMs and their evaluation are both discussed in sufficient details. The take-home messages and potential use cases of LLM360 make this work useful also for the future researchers and practitioners. The authors are transparent about the issues encountered during training and make suggestions about the responsible usage of their models.
strength_weakness: strength_weakness: I do not see significant reasons to reject this paper. In order to improve it for the camera-ready version, I would suggest adding a couple of recent models to the related work discussion, specifically Aya (Üstün, Ahmet, et al. Aya model: An instruction finetuned open-access multilingual language model", 2024) and StarCoder 2 (Lozhkov, Anton, et al. "StarCoder 2 and The Stack v2: The Next Generation", 2024 ). 

It would also be beneficial to read a bit about how the authors see the future of LLM360 in terms of the open collaboration with the community, e.g. do they expect external contributions to LLM360 in the future?
questions: questions: Correct the typo under the Issues Encountered During Pre-training section on page 9: "...which can be a major help ***the*** the community".
paper_summary: paper_summary: This paper presents a fully open-sourced language modeling project, whose aim is to release all code, training data, intermediate checkpoints and results, and also serves as a technical report for two 7B parameter language models, Amber (trained on 1.3T tokens of English text) and Crystal (trained on 1.5T tokens of English text and code). Along with these pretrained models, three finetuned variants of these models, AmberChat and CrystalChat (SFT) and AmberSafe (DPO) were described and evaluated. Amber and AmberChat were evaluated on ARC, Hellaswag, MMLU, and TruthfulQA, and Crystal was evaluated additionally on HumanEval (from the Codex paper) and MBPP.
strength_weakness: strength_weakness: The paper could use more detailed evaluations and further discussion of existing results. CrystalChat does not have any evaluations in the paper currently.

1. Chat-focused evaluation (e.g. AlpacaEval) and safety evaluations (e.g. Toxigen) of AmberChat, CrystalChat and AmberSafe would helpful to see.
2. Further discussion of the performance on Amber (the base model) on the OpenLLM leaderboard benchmarks would be informative. What would make the model's performance more comparable to better 7B models? Is it the training data mix or the size? I understand that concrete observations cannot be made here because all the training details of the other models are not known, but a discussion on how the model could be improved would be helpful.
3. The draft currently does not have any details of the carbon footprint of training the models.

Comments: comment: Thank you for the response. Considering the author's response, I have increased my score.
I thank the authors for the response. I am still confident with my score and would be glad to see this paper published.
We understand your concern of comparing the model with existing LLMs and agree that a comprehensive report will be helpful. Note that in our paper we compare models with similar conditions (e.g. dataset, release dates), and report the numbers following the exact settings of public OpenLLM leaderboard as possible. We have more evaluation results to share here:

**Metrics using Open LLM leaderboard setting**

We have evaluated both models using the exact OpenLLM leaderboard settings as below, the numbers of several other models are referenced from the leaderboard.

| Model  | ARC-C | HellaSwag | MMLU | TruthfulQA| 
| -------- | ------- | ------- | ------- | ------- |
| Amber  | 41.89   | 71.63   | 30.76   | 34.00   |  
| Crystal |  47.01   | 71.97   | 48.78   | 35.91   |
| Llama1-7B | 50.94 | 77.80 | 35.67 | 34.34 |
| Llama2-7B | 53.07 | 77.74 | 43.80 | 38.98 | 
| Mistral-7B | 59.98 | 83.31 | 64.16 | 42.15 | 
| Gemma-7B | 61.09 | 82.20 | 64.56 | 44.79 | 
| Qwen1.5-7B | 54.18 | 78.51 | 61.97 | 51.08 |
| Codellama-7B | 39.93 | 60.80 | 31.12 | 37.82 | 
| OpenLlama-v2-7B | 43.69 | 72.20 | 41.29 | 35.54 | 
| Olmo-7B | 45.65 | 77.31 | 28.13 | 35.93 |
| Olmo-1.7-7B  | 49.4 | 78.68 | 53.52 | 35.89 |
| Falcon-7B | 47.87 | 78.13 | 27.79 | 34.36 |
| MPT-7B | 47.70 | 77.57 | 30.80 | 33.44 |
| RedPajama-Incite-7B | 46.25 | 71.63 | 27.68 | 33.03|

In comparison, Amber performs similar to open source models released around the same time (Falcon, MPT, Incite) with a slight advantage on MMLU but weaker in other metrics like ARC.

CrystalCoder, our newer model, is comparable with other models released around the same time (llama2, OpenLlama), where it shows a strong MMLU score of 48.78, as compared to Llama2’s 43.80. 

This field is advancing quickly, and newer models in the list, such as Gemma, Qwen1.5 and Olmo 1.7 are generally better.


**Other Metrics**

To evaluate our models on metrics on additional benchmarks, such as coding, we have conducted additional evaluations with newer version of LM-Harness and other benchmark suite (the exact evaluation settings will be available together with the evaluation code)

| Model  | Openbook QA  | RACE | BoolQ | PIQA | HumanEval p@1 | MBPP p@1  | Winogrande | GSM8K |
| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
| Amber | 40 | 37.70 | 68.90 | 79.43 | - | - | 64.25 | - | 
| Crystal | 41.20 | 38.18 | 74.43 | 78.07 | 23.90 | 30.99 | 67.01 | 12.36 | 
| Llama2-7B | 44.20 | 39.52 | 78.07 | 78.78 | 13.05 | 20.09 | 69.38 | 14.71 | 
| Llama1-7B | 44.40 | 40.28 | 75.01 | 78.94 | 10.61 | 17.04 | 70.24 | 8.87 | 
| CodeLlama-7B | 36.80 | 39.52 | 74.65 | 72.58 |  30.06 |  39.20 | 65.51 | 11.15 | 
| Mistral-7B | 44.20 | 40.86 | 83.73 | 82.15 | 29.11 | 38.78 | 74.19 | 37.68 |
| Falcon-7B | 43.80 | 37.42 | 73.70 | 80.57 | 9.42 | 13.38 | 67.24 | 4.62 |
| MPT-7B | 42 | 38.66 | 74.00 | 80.30 | 16.52 | 22.49 | 68.51 | - | 
| Olmo-7B | 42.60 | 38.37 | 72.66 | 79.92 | 14.02 | 14.40 | 68.90 | 4.09 |
| Pythia-6.9B | 25.5 | - | 62.10 | 75.20 | 7.68 | 6.00 | - | - |
| Starcoder-13B | 32.80 | 32.15 | 63.91 | 65.72 | 30.70 | 37.56 | 54.30 | 9.02 | 
| Phi1.5 | 48.20 | 37.51 | 74.74 | 75.95 | 35.36 | 35.19 | 72.92 | 31.23 |



**Safety related metrics**

For instruct-tuned/chat models, the finetuned data can affect the model performance on tasks significantly. Here we provide a comparison on safety/biase of AmberChat, AmberSafe and a few reference models. 

| Model | Toxigen | BOLD Avg. | BOLD Race std. |
| -------- | ------- |  ------- | ------- | 
| AmberSafe | 2.0 | 0.51 | 0.092 | 
| AmberChat | 10.77 | 0.64 | 0.069 |
| Llama2-7B-Base | 21.28 | 0.304 | - |
| Llama2-7B-Chat | 0.01 | 0.482 | - | 
| Code Llama 7B | 22.64 | 0.230  | - | 
| Code Llama 7B Instruct | 0.04 | 0.503 | 0.042 |
| Falcon-7B-Base | 14.53 | 0.283 |  - | 
| Falcon-7B-Instruct | 5.78 | 0.332 | 0.035 |
| MPT-7B-Base | 22.32 | 0.32 | - |
| MPT-7B-instruct | 16.33 | 0.302 | - |

We found that safe tuning with DPO makes Amber less toxic, but not necessarily less biased. Specifically, the BOLD score (sentiment towards certain social groups) shows that AmberChat shows a better sentiment score over different social groups, and has a smaller standard deviation across groups (fair). 

Here, we only show the standard deviation across races, we have conducted more analysis and could include them in the revision. Compared with other models, AmberSafe shows good scores against a few open source models released around the same time. However, we can see that the Llama models, after tuning with their internal safety data, show a much safer behavior.  

Due to resource constraints, we prioritized comparing models with comparable settings and models released around the same time of our models, and avoided comparing with moving targets such as commercial APIs. Please let us know if there are specific results you would like to see.
I appreciate the author's response. I still have concerns regarding comparative analysis with existing LLMs. It is helpful if the authors could report the results in the response.
I thank the authors for their informative response, and am glad to see the additional evaluations and carbon footprint details. Including these details in the paper would make it significantly stronger. I have increased my score accordingly.