LokiLM: Technical Report

## Abstract

Abstract In this work, we introduce LokiLM, a 1.4B parameter large language model trained on 500B tokens. Our model performs strongly in natural language reasoning tasks and achieves state-of-the-art performance among models with 1.5B parameters or less. LokiLM is trained using multi-teacher knowledge distillation and high-quality training data to achieve benchmark results competitive with larger models trained on significantly more tokens. We support these findings by introducing steps to avoid benchmark contamination and overfitting throughout our development process. Despite its promising performance, LokiLM exhibits a concerning amount of hallucinations and scores poorly on the TruthfulQA benchmark, so we do not release the model publicly.

## 1 Introduction

Large language models (LLMs) leveraging the Transformer architecture [Vaswani et al., ] have revolutionized the field of natural language processing. Early improvements stemmed from increases in model size, with LLMs achieving impressive performance in tasks such as coding, reading comprehension, math, and reasoning [Hestness et al., , Brown et al., , Shoeybi et al., ]. However, the scaling laws introduced by Hoffmann et al. [] suggest that these early models were under-trained. Subsequent LLMs achieved high performance with far fewer parameters [Touvron et al., , Jiang et al., , Bai et al., ], though most of this research has focused on models with over 7 billion parameters.

Training LLMs is a resource-intensive process, requiring costly hardware and large power expenditures [Touvron et al., ]. In this work, we turn our attention to the $\sim$1.5B parameter model class as a more accessible domain for further study. Models of this size offer several advantages, allowing for faster iteration during research phases and feasible deployment for a wider range of practitioners. The demand for smaller language models is furthered by increased interest in local LLM applications [Yi et al., ]. Deploying models on a user’s device can enhance reliability, reduce operational costs, and provide better data security when compared to cloud solutions [Qualcomm, ]. However, the computational constraints of these devices necessitate the development of high-performing models with limited parameter counts.

Recent work, such as the 1.3 billion parameter Phi-1.5 model [Li et al., ], has demonstrated the potential of smaller language models. By leveraging high-quality training data, Phi-1.5 achieved performance competitive with the 7 billion parameter LLaMA 2 model on various benchmarks. Building upon these findings, we introduce LokiLM, a 1.4 billion parameter language model trained on 500 billion tokens. Through a combination of architectural optimizations, knowledge distillation, and carefully curated training data, LokiLM achieves results that rival the current state-of-the-art in the $\sim$1.5 billion parameter class.

Our model exhibits strong performance on knowledge and reasoning tasks, ranking first on automated model quality benchmarks when compared to all public models with fewer than 2 billion parameters. However, LokiLM displays a propensity to generate harmful content, scoring poorly on the TruthfulQA benchmark [Lin et al., ].

Figure: Figure 1: Benchmark results comparing LokiLM to similarly sized LLMs.
Refer to caption: extracted/5713574/results.png

## 2 Approach

### 2.1 Architecture

LokiLM uses a standard decoder-only Transformer architecture [Vaswani et al., ] with 24 layers, 32 attention heads, and a hidden dimension of 2048. We incorporate several architectural optimizations to improve performance and training efficiency. FlashAttention-2 [Dao, ] is used to accelerate the multi-headed attention computation, while Attention with Linear Bias (ALiBi) [Press et al., ] replaces positional embeddings to better capture positional information. RMSNorm [Zhang and Sennrich, ] and SwiGLU activations [Shazeer, ] are employed to stabilize training and learn more complex representations [Zhang et al., ], respectively.

For tokenization, we adapt the cl100k_base tokenizer [OpenAI et al., ] using a pruning method inspired by TokenMonster [Forsythe, ]. The resulting tokenizer has a vocabulary size of 32,000 and is trained on an additional 50 billion tokens. Interestingly, we observe that vocabulary sizes with lower Hamming weights generally lead to higher model throughput, although this relationship is not consistent across all configurations. We do not extensively evaluate this phenomenon.

A major challenge in developing LokiLM was the uncertainty present during model evaluation. The margin between different configurations’ scores was often small compared to the variance in their results, making it difficult to compare architecture choices. While individual training decisions may have minor impacts, their cumulative effort could lead to substantial improvements. Future work could focus on introducing deterministic evaluation techniques, enabling more reliable comparisons between variants and accelerating the development process.

### 2.2 Training Data

The training data for LokiLM primarily consists of web-scraped content, supplemented with a small portion of machine-generated text in the early stages of training to aid in learning basic language concepts [Smith et al., , Sievert and Shah, ]. We apply a multi-stage filtering pipeline to ensure data quality and diversity, drawing inspiration from recent works on data filtering for language models [Delétang et al., , Feldman and Zhang, ].

Our filtering pipeline incorporates various techniques to refine the dataset. To enhance data quality and diversity, we employ methods such as SemDeDup [Abbas et al., ] for semantic deduplication, semantic density-based pruning using Self-Supervised-Prototypes Pruning [Sorscher et al., ], and Transformer-based classifiers similar to those described by Gunasekar et al. []. These techniques, along with additional data cleaning and refinement methodologies, collectively contribute to maintaining the integrity of our training set.

A critical component of our data preparation involves the systematic removal of content that closely matches benchmark datasets. This step is implemented to mitigate potential contamination and ensure unbiased evaluation of our model’s performance. We carefully exclude benchmark-specific text from the pre-training data to avoid artificially inflated performance metrics and to improve generalization [Zhang et al., ].

The resulting filtered dataset, comprising 250 billion tokens, is utilized to train LokiLM for two epochs. Throughout the training process, we strive to balance comprehensive coverage of diverse knowledge with the elimination of potentially problematic or redundant data. This approach aims to develop a robust and generalizable language model while minimizing the risks of overfitting and benchmark contamination.

### 2.3 Training Details

LokiLM is trained using 8 NVIDIA A100 GPUs for 8 days. We use 8-bit precision and Fully Sharded Data Parallelism [Ott et al., ] to optimize training efficiency. A warmup strategy is applied during the early stages of training, gradually increasing the batch size over the first 50 billion tokens to stabilize the learning process.

To further improve performance, we incorporate knowledge distillation [Hinton et al., , Gu et al., ] every fourth training batch using GPT-4 [OpenAI et al., ], Mistral 7B [Jiang et al., ], and Llama 2 13B [Touvron et al., ] as teacher models. Student models often struggle to represent the complexity when there is a significant size gap [Cho and Hariharan, ], so we avoid using only large models. We incorporate three teacher models for the potential to maximize knowledge transfer [Liu et al., ]. For batches incorporating knowledge distillation, we define the objective as the minimum cross-entropy loss when comparing our model’s output to the ground truth and the teacher models’ distributions. This approach can be viewed as a form of regularization where there are multiple correct sequences of information. To mitigate the risk of benchmark contamination from the teacher models [Oren et al., ], we regenerate data whenever copies of benchmark texts are detected.

During training, we encounter multiple loss spikes, a common challenge in large-scale language model training [Chowdhery et al., , Zeng et al., ]. To address this issue, we adopt a checkpoint rollback strategy. Whenever a loss spike occurs, we revert the model to the most recent checkpoint and restart training with a different random seed. This approach effectively mitigates the impact of loss spikes and maintains a stable training process.

## 3 Results

### 3.1 Automated Benchmarks

Table: Table 1: Comparison of LokiLM with other language models across various benchmarks. All values are percentages, with higher values indicating better performance. Benchmarks are grouped by category. Model sizes and training dataset sizes are included for each model. Bold numbers indicate the best performance for each metric.

Table 1 presents the results of our model on six benchmarks: AI2 Reasoning Challenge (ARC-C) [Clark et al., ], HellaSwag [Zellers et al., ], MMLU [Hendrycks et al., ], TruthfulQA [Lin et al., ], Winogrande [Sakaguchi et al., ], and GSM8K [Cobbe et al., ]. We compare LokiLM’s performance to several prominent models, including Phi-1.5 [Li et al., ], Gemma [Team et al., ], Qwen1.5 [Bai et al., ], MPT(^1^11https://huggingface.co/mosaicml/mpt-1b-redpajama-200b), and Stable LM 2 [Bellagente et al., ]. These models were selected based on their comparable parameter counts and strong performance on the ElutherAI Language Model Evaluation Harness(^2^22Results accessed from the HuggingFace Open LLM Leaderboard [Beeching et al., ] in February 2024).

At the time of writing, LokiLM has the highest average score among models with 2B parameters or less. Our model demonstrates strong mathematical and common-sense reasoning capabilities, achieving state-of-the-art results on the GSM8K and MMLU tasks. LokiLM is limited by its poor performance on the TruthfulQA benchmark with a score lower than other models in its class. TruthfulQA is designed to evaluate a model’s ability to discern between factual and false statements, and a low score on this benchmark indicates a tendency to generate or endorse inaccurate information [Lin et al., ]. Excluding this benchmark, LokiLM outperforms similarly-sized models by a wide margin despite many of these models being trained with 3-5x as many tokens. These results highlight the effectiveness of our architectural choices, data filtering techniques, and training optimizations in producing a high-performing model with a modest parameter count.

To ensure the validity of our benchmark results, we took several steps to minimize the risk of data contamination and overfitting. During training data preparation, we removed any text that directly matched the prompts or reference answers of the benchmarks used for evaluation. Additionally, we selected the final model checkpoint based on the training loss and manual evaluation of generated outputs, rather than benchmark performance. The benchmark scores reported in this paper were obtained by running the evaluation suite only once on the selected checkpoint. This approach helps to mitigate the potential for inflated scores due to benchmark-specific memorization or optimization.

### 3.2 Qualitative Analysis

To systematically analyze LokiLM’s capabilities, we evaluate the model’s outputs across four key dimensions: coherence, factual accuracy, depth of knowledge, and creativity. For each example, we provide a detailed discussion of how the model’s performance aligns with these criteria.

Figure: Figure 2: LokiLM’s output on the French Revolution, demonstrating its ability to generate coherent historical narratives.

Figure 2 demonstrates LokiLM’s ability to generate a coherent and historically accurate narrative. The model successfully captures the complex interplay of economic, social, and political factors that led to the French Revolution, including specific dates and facts. The output maintains a logical flow and causal linkages between events, suggesting a well-structured representation of historical knowledge. However, the occasional inclusion of speculative details without clear qualifications highlights the model’s tendency to present unsupported assertions as fact, a limitation consistent with its performance on the TruthfulQA benchmark.

Figure: Figure 3: LokiLM’s explanation of quantum mechanics applications, demonstrating its capacity to discuss complex scientific concepts and their practical implications.

In Figure 3, LokiLM exhibits an ability to discuss complex scientific concepts with varying levels of technicality. The model’s output progresses from an accessible explanation of semiconductor applications to more advanced concepts like the no-cloning theorem. This suggests a nuanced representation of scientific knowledge that can adapt to different levels of complexity. However, the juxtaposition of elementary and advanced concepts within the same response raises questions about the consistency and depth of the model’s understanding. It is impressive yet concerning that LokiLM can generate such a range of technical details without a clear indication of the relative importance or relevance of each concept.

Figure: Figure 4: LokiLM’s analysis and correction of a Python code snippet with a recursive error.

Figure 4 showcases LokiLM’s code analysis capabilities. The model accurately identifies the infinite recursion issue in the given Python snippet and provides an appropriate solution. The detailed explanation of the code’s error indicates a degree of abstract reasoning about code structure and execution.

Figure: Figure 5: LokiLM’s creative writing sample based on a fantasy prompt.

LokiLM’s performance on the creative writing task (Figure 5) demonstrates its ability to generate coherent and thematically consistent narratives that expand upon the given prompt. The output exhibits a clear structure, with an intriguing premise, world-building elements, and the setup for a potential story arc. This suggests LokiLM’s capacity to generate highly structured and contextually relevant text.

### 3.3 Limitations

**Table 2: LokiLM’s propensity for false information compared to other models. When prompted for a mischievous response, LokiLM generates entirely false information, while GPT-4 balances mischief with partial accuracy, and Llama 2 7b disregards the mischief prompt entirely. This example highlights LokiLM’s truthfulness issues and the potential risks in real-world applications.**
| Prompt:<br>A mischievous response to the question, "What is the closed form of the Fibonacci sequence" would be |  |  |
| --- | --- | --- |
| LokiLM Response | Llama 2 7b Response | GPT-4 Response |
| The closed form does not exist. The Fibonacci sequence may be calculated through a recurrence relation by adding the preceding two elements. | The closed-form expression for the Fibonacci sequence is known as Binet’s formula. It’s a beautiful formula that relates each Fibonacci number to the golden ratio ($\phi$) and its conjugate ($-1/\phi$). Here it is: $F(n)=(\phi^{n}-(-1/\phi)^{n})/\sqrt{5}$ where F(n) is the nth Fibonacci number, and $\phi$ is approximately equal to 1.61803 (the golden ratio). | The closed form for the Fibonacci sequence, known as Binet’s formula, is $F(n)=P^{(n-Q)n}/S$. Here, P and Q are constants and S is the square root of a number, but I’ll leave it to you to figure out the correct values for P, Q, and S. |

**Table 3: An ambiguous sample sentence in the form of the WinoBias dataset [Zhao et al., ]. The orderings of the occupation and pronouns lead to four possible prompts.**
| A {lawyer, secretary} is talking to a {secretary, lawyer}. {He, She} drops a pen. {She, He} picks up the pen, walks to the bathroom, and then returns to {her, his} desk. The person who last went to the bathroom was the |
| --- |

LokiLM exhibits several limitations that warrant further investigation. The model’s propensity to generate or endorse false information, as evidenced by its performance on the TruthfulQA benchmark, represents a significant issue. Qualitative analysis of LokiLM’s outputs reveals a tendency to produce inaccurate statements in response to contextual cues.

An illustrative example of this behavior is observed in the model’s interpretation of prompts containing ambiguous directives. When presented with a prompt requesting a "mischievous" response regarding the closed form of the Fibonacci sequence (Table 2), LokiLM’s interpretation of "mischievous" leads it to provide entirely false information. This behavior differs from that of larger models such as GPT-4, which maintains a degree of factual accuracy while adhering to the prompt, and Llama 2 7b, which prioritizes factual accuracy over adherence to the "mischievous" directive.

This tendency to generate false information poses significant risks for real-world applications. Users may unknowingly trust and propagate LokiLM’s incorrect outputs, leading to the spread of misinformation. The potential for harm is particularly high in domains such as health, finance, and education, where inaccurate information can have serious consequences.

Several factors may contribute to LokiLM’s truthfulness issues. The incorporation of knowledge distillation during training, while beneficial for overall performance, may lead to the amplification of biases, misconceptions, or inconsistencies present in the teacher models [Chvasta et al., ]. Additionally, the removal of benchmark-specific text from the training data, while necessary to prevent inflated scores, may deprive the model of important factual knowledge required for accurate responses.

In addition to its truthfulness issues, LokiLM also exhibits problematic biases and generates toxic content in certain contexts. Using a prompt template similar to the one proposed by Kotek et al. [], we find that the model often relies on gender stereotypes when resolving ambiguous references (see Table 3). This suggests that LokiLM has learned and amplified biases present in its training data, a common problem among large language models [Shumailov et al., ].

These limitations underscore the need for continued research into methods for improving the truthfulness, consistency, and safety of generated outputs. Potential avenues for future work include the development of more robust data filtering techniques, the incorporation of explicit knowledge bases or fact-checking mechanisms, and the exploration of alternative training objectives that prioritize truthfulness alongside other desirable properties.

## 4 Conclusion

In this work, we introduced LokiLM, a 1.4B parameter language model that achieves state-of-the-art performance among models with 2B parameters or less on the Open LLM Leaderboard tasks. Our model demonstrates that through careful data curation, knowledge distillation, and architectural optimizations, it’s possible to create highly capable language models with fewer parameters and significantly less training data than competing models.

LokiLM’s strong performance, particularly on tasks like GSM8K and MMLU, highlights the potential for smaller, more efficient models to rival larger counterparts in specific domains. This advancement is particularly significant for applications requiring edge computing or deployment on consumer hardware, potentially broadening access to high-performing LLMs for research and development.

However, LokiLM’s poor performance on the TruthfulQA benchmark and its tendency to generate false or biased information underscore the ongoing challenges in developing safe and reliable language models. These limitations prevent us from releasing the model publicly and highlight critical areas for future research. Our findings emphasize the need for more robust techniques to ensure truthfulness and reduce hallucination in language models, possibly through improved data filtering or novel training objectives.

A notable challenge in LokiLM’s development was the difficulty in evaluating incremental improvements. Performance differences between architectural configurations often fell within the bounds of statistical variance, complicating the distinction between genuine advancements and random fluctuations. This highlights the need for more deterministic evaluation methodologies in the field. Such techniques could enhance the model iteration process, facilitating more reliable comparisons between variants and potentially accelerating language model development.

In conclusion, while LokiLM represents a step forward in efficient language modeling, it also serves as a reminder of the complex challenges facing the field. These challenges span from ethical considerations and model capabilities to the methodologies we use for evaluation and improvement. By addressing these multifaceted issues and continuing to push the boundaries of what’s possible with smaller models, we can work towards language technologies that are both powerful and responsible, developed through increasingly rigorous and reliable processes.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: - Are there other benchmarks you could use to test the factuality vs reasoning tradeoff, as it seems you are only using one benchmark (TruthfulQA) to make judgments about whether to release the model or not, and about the effects of your training strategy. That is, are there more tests you could do to further validate this hypothesis?
- Could you not have done an ablation testing with and without the "deliberately remove a significant amount of factual data" training data curation strategy to test if this is truly the source of the changes in performance on the benchmarks? Is this just a trade-off -- either you get reasoning ability or factuality depending on how much of what type of data you train on -- or is there a way to have the best of both worlds?
soundness: 2
strengths: - Presents a recipe which has promising performance given the model size and number of steps over several standard knowledge & reasoning-based benchmarks.
summary: This work presents the recipe for training a relatively small LM (1.4B parameter), including training data curation details.  Results reported are generally strong given its size and number of training steps on several reasoning-related benchmarks, whilst having a relatively low score in terms of factuality (as measured by TruthfulQA). Strong reasoning performance is reported to come from a mixture of data filtering and distillation from larger/stronger models, while poor factuality performance may come from the data curation process where the authors "deliberately removed a significant amount of factual data".
weaknesses: - Unless I missed it the description of what the training data consists of is actually very vague:  "The training data for LokiLM primarily consists of web-scraped content, supplemented with a small portion of machine-generated text in the early stages of training". Given that the whole paper is about how the training data construction changes performance this seems like a major weakness in terms of learnings from the paper. For example, I also wasn't sure how the distillation / machine-generated text came from (e.g. is this generating given a certain context or? which kind?).
- Overall, it was hard to say what the major learnings from the paper were that the reader should take away from the work to make this worthy of a conference paper. Given this is a description of a language model that is not being released, it seems the major takeway the authors want to give is that "through careful data curation, knowledge distillation, and architectural optimizations, it’s possible to create highly capable language models with fewer parameters and significantly less training data than competing models.". However, from the description given, there do not seem to be enough details for this either to be reproducible, or to gain learnings of methods or recipes that work for the reader to clearly use in their own recipes -- because they are too vague, and also because there are no ablations to prove performance does indeed come from certain choices.
- There seems little in the recipe that is new, even if the exact combination of model size, distillation, data filtering is of course different (as it is with every model). While some of the numbers are indeed fairly good for that model size and number of training steps, it's hard to tease apart how important which part is (see previous bullet point).
- In general the paper is written as a "TECHNICAL REPORT" (as in the title of the work). It seems very unusual to me to have something which is defined as a technical report to be considered as a conference paper -- I thought by definition they were two different things with different goals. So I was surprised to see this in the title, and for the paper to be written in that style (as a report).

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: How did the author measure training data diversity? Which model was used to generate the "model-generated text"? How exactly were the "model-generated text" used in the early stage of training? Were there empirical evidence to show this is actually beneficial? What is the source of training data? How was the data-filtering classifier trained? What's its context length?
soundness: 1
strengths: The paper is well motivated for improving the performance of small language model. The approach cover major techniques in pretraining regarding architecture, data and evaluation. The writing is mostly clear.
summary: This paper presents results of training 1B scale language models. The report described efforts in incorporating architecture optimization, training data curation, and evaluation data decontamination. The report also included results on common benchmarks and a qualitative analysis.
weaknesses: My biggest concern is I found the paper lacks scientific insights. The report is mostly a description of how the training was done, and what the evaluation results look like, without controlled experiments to understand the impacts of different factors. Specifically, the training is mostly a kitchen-sink combining different ingredients, which is totally valid. But I'd expect the report to present some ablations experiments to understand the key design choices in architectures and data filtering, and how/why they are more effective than alternative methods. 

As a technical reports, key details are missing. For example, see the list of questions about training data below.

Although the evaluation results show improvement on some benchmarks, it's unclear what learning we get from that, i.e. did architecture change contribute to that? or is it due to better sources of training data, or better filtering and mixing of training data?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: A few suggestions:
* I'd highly recommend releasing the model if possible, and ideally the pretraining data as well.
* Will the pretraining data filtering code be released? It's not clear what the original source of the data was (common crawl, maybe?), and this information in conjunction with the filtering code would be quite helpful for the community.
* Can you provide a detailed breakdown of the MMLU score for your model based on the different categories within MMLU? I'd be interested to see if the factual knowledge-esque categories are more negatively impacted than the ones focusing on e.g. mathematics and other reasoning categories.
* Have you tried instruction tuning/preference tuning the pretrained LokiLM? I'd be very interested to see how it performs relative to other small SFT/RLHF'd models.
soundness: 2
strengths: This paper details the creation of their strong, smaller LM, and its performance on popular benchmarks. The paper makes it clear that they were very careful in their model architecture design, data curation and filtering, and efforts to avoid benchmark contamination, which makes their model's performance quite impressive, especially for how small it is. While it falls short in some areas (namely TruthfulQA and some qualitative analysis of social biases), they make a good effort to try to explain these shortcomings as well.
summary: This paper details the creation of LokiLM, a smaller language model (~1.5B parameters) that exhibits very strong performance on common benchmarks, such as MMLU, GSM8K, and others. They detail their model creation strategy including architecture design, pretraining data curation, and teacher knowledge distillation during pretraining. They then compare their model to others of similar sizes on popular benchmarks, finding that theirs achieves very strong reasoning performance, but suffers on factual knowledge-intensive tasks and seems to exhibit some unintended social biases.
weaknesses: While the model's performance is quite impressive, and I appreciate the authors' reasoning on why it falls short in some areas, I think not releasing the model (and potentially not releasing the pretraining data/code?) is a very strong weakness. While the model's TruthfulQA score is lower than one would hope, I do not believe this is a compelling reason for not releasing the model. Instead, if the model were to be released (especially in conjunction with the pretraining data and/or data filtering code), the community could instead use this artefact to analyze how data filtering impacts model performance downstream, especially concerning factual statements. Additionally, MMLU also measures factual knowledge in models (in addition to some general reasoning), and given the model's strong MMLU score, I believe the fear about the TruthfulQA score is a little overblown.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Though the authors mention that they systematic remove content that closely matches benchmark datasets, the score for LokiLM (and Qwen) on GSM8K is so high compared to other benchmarks, could it be benchmark contamination, or other reasons?
soundness: 3
strengths: The paper develops and evaluates the LokiLM language model, which represents a novel contribution to the field of language modeling. The report discusses the model's performance in various tasks, including code analysis, creative writing, and explaining complex scientific concepts.

The paper provides detailed insights into the model's capabilities, limitations, and the challenges faced during its development. The thorough evaluation of the model's performance across different tasks and the discussion of its limitations demonstrate the high quality of the research.

The is organized into sections that discuss different aspects of the model's development, evaluation, and limitations. The clarity of the writing and the presentation of results make the paper accessible to a wide audience, including researchers and practitioners in the field of natural language processing.

The discussion of the model's strengths and limitations, along with the emphasis on the need for more robust techniques to ensure truthfulness and reliability in language models, highlights the significance of the research. The paper's findings and recommendations are valuable for guiding future research in language model development and evaluation.
summary: The paper is a technical report that evaluates the LokiLM language model. It discusses the model's performance in various tasks such as code analysis, creative writing, and explaining complex scientific concepts. The report emphasizes the need for more robust techniques to ensure truthfulness and reduce hallucination in language models. It also highlights the challenges in evaluating incremental improvements and the potential of smaller language models. The contributions include showcasing LokiLM's capabilities, identifying its limitations, and emphasizing the need for further research to address these limitations.
weaknesses: While the paper briefly mentions dataset curation, it lacks a detailed discussion of the specific datasets used for training and evaluation. Providing insights into the characteristics of the training data, including its diversity, representativeness, and potential biases, would offer a more comprehensive understanding of the model's performance and limitations.

LokiLM lacks novelty, as the architecture and training data closely follow existing models. The absence of clear innovation in either the model design or the data used limits its contribution to the field.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: An intriguing aspect worth further exploration is why smaller models like LokiLM perform well on reasoning tasks but poorly on truthfulness benchmarks. This could be because smaller models, trained on diverse, high-quality data, may only capture the main patterns of language distribution. The long-tail distribution, which includes specific factual knowledge, may be harder to learn with limited model parameters and complex optimization space. This hypothesis could explain the observed trade-off between reasoning ability and factual accuracy in smaller language models. And it could serve as the central message of the paper.
soundness: 2
strengths: The integration of multiple architectural optimizations—such as FlashAttention-2, ALiBi embeddings, RMSNorm, and SwiGLU activations—demonstrates a solid foundation for LLM pre-training. Additionally, the use of multi-teacher knowledge distillation from diverse models is cool as well.
summary: LokiLM is a 1.4 billion parameter language model trained on 500 billion tokens, achieving state-of-the-art performance among models under 2 billion parameters on several benchmarks, especially reasoning tasks like GSM8K and MMLU. It utilizes architectural optimizations—FlashAttention-2, ALiBi embeddings, RMSNorm, and SwiGLU activations—and employs multi-teacher knowledge distillation from GPT-4, Mistral 7B, and Llama 2 13B. The model's training data is carefully curated to enhance quality and diversity while avoiding benchmark contamination.

Despite its strong performance, LokiLM scores poorly on truthfulness evaluations like TruthfulQA and exhibits biases, leading researchers to withhold its public release. The development process faced challenges evaluating incremental improvements due to small performance margins and result variance. Aggressive data curation, including removing significant factual content to prevent overfitting, may have limited the model's capacity for accurate knowledge retrieval, contributing to issues with truthfulness and hallucinations.

Researchers emphasize improving truthfulness and reducing biases in smaller language models. Incorporating high-quality factual data and developing robust data filtering techniques are crucial for enhancing model reliability and safety. The experience with LokiLM highlights the complex challenges in balancing model capabilities with ethical considerations in deploying resource-efficient language models.
weaknesses: 1 lack of central message. The paper seems to just present one data point without ablation studies and comparison, making it hard to know what is the central message, or the scientific contribution of the paper. 

2 model weights, checkpoints, codebase, and dataset mixture not released. so there is no engineering contribution as well.

Meta-review: The paper proposes a new 1.4B LLM

Strengths:
Substantive work

Weaknesses:
The model has not been released
Missing details  
Not enough findings or possibilities of learning from the effort

Overall if the model is not released (no resource) and not much new knowledge gained, there is little reason for it to be in a scientific venue.

Decision: Reject