Surprising Effectiveness of pretraining Ternary  Language Model at Scale
Authors: Ayush Kaushal
Tejas Vaidhya
Arnab Kumar Mondal
Tejas Pandey
Aaryan Bhagat
Irina Rish
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Models
low-bit language models
quantization-aware training
pretraining of large language models
and scaling laws
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces Spectra LLM suite, showcasing that Ternary Language Models (TriLMs) outperform traditional and quantized models in efficiency and scaling, especially beyond one billion parameters.

Abstract: Rapid advancements in GPU computational power has outpaced memory capacity and bandwidth growth, creating bottlenecks in Large Language Model (LLM) inference. Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but it suffers from significant performance degradation below 4-bit precision. This paper addresses these challenges by investigating the pretraining of low-bitwidth models specifically Ternary Language Models (TriLMs) as an alternative to traditional floating-point models (FloatLMs) and their post-training quantized versions (QuantLMs). We present Spectra LLM suite, the first open suite of LLMs spanning multiple bit-widths, including FloatLMs, QuantLMs, and TriLMs, ranging from 99M to 3.9B parameters trained on 300B tokens. Our comprehensive evaluation demonstrates that TriLMs offer superior scaling behavior in terms of model size (in bits). Surprisingly, at scales exceeding one billion parameters, TriLMs consistently outperform their QuantLM and FloatLM counterparts for a given bit size across various benchmarks. Notably, the 3.9B parameter TriLM matches the performance of the FloatLM 3.9B across all benchmarks, despite having fewer bits than FloatLM 830M. Overall, this research provides valuable insights into the feasibility and scalability of low-bitwidth language models, paving the way for the development of more efficient LLMs.

Published as a conference paper at ICLR 2025
SURPRISING EFFECTIVENESS OF PRETRAINING TERNARY LANGUAGE MODELS AT SCALE
Ayush Kaushal' 2 Tejas Vaidhya 14 Arnab Kumar Mondal 4, Tejas Pandey' 3, Aaryan Bhagat 5 Trina Rish 124 Nolano AI PUniversity of Montreal 'IIT Kharagpur *Mila Quebec Al Institute 'UC Riverside
ABSTRACT
Rapid advancements in GPU computational power has outpaced memory capacity and bandwidth growth, creating bottlenecks in Large Language Model (LLM) inference. Post-training quantization is the leading method for addressing memory- related bottlenecks in LLM inference, but it suffers from significant performance degradation below 4-bit precision. This paper addresses these challenges by investi- gating the pretraining of bow-bitwidth models specifically Temary Language Mod- els (Tr) as an alternative to traditional floating- peiiin models (FloatI.Ms) and their post-training quantized versions (QuantI We present Spectra I.IM suite, the first open suite of LLMs spanning multiple bit-widths, including FloatL. Mss, QuantLMs, and TriLMs, ranging from 99M to 3.9B parameters trained on 300B tokens. Our comprehensive evaluation demonstrates that TriLMs offer superior scaling behavior in terms of model size (in bits). Surprisingly, scales exceed- ing one billion parameters, TriLMs consistently outperform their QuantLM and FloatLM counterparts for given bit size across various benchmarks. Notably, the 3.9B parameter TriLM matches the performance of the FloatL.M 3.9B across all benchmarks, despite having fewer bits than FloatLM 830M. Overall, this re- search provides valuable insights into the feasibility and scalability of low-bitwidth language models, paving the way for the development of more efficient LLMs.
INTRODUCTION
The computational power of GPUs, measured in FLOPs, is increasing exponentially, doubling approximately every 1.26 years. In contrast, memory capacity and bandwidth are growing at slower pace, doubling every 2 and 2.9 years, respectively (Gholami et al. (2024). This disparity highlights that computing capabilities are outpacing advances in memory technology. In Large Language Models (L.LMs) inference, the primary bottlenecks are caused by model size (bits), which affects memory usage (memory capacity) and data transfer to processors (memory bandwidth). These issues are becoming more critical than the growing number of model parameters which affect the computational limits (FLOPs). For instance, wfate-of-the-an LLMs such as 340B Nemotron 4 (Nvidia et al. 2024) have sizes (in bits) exceeding the memory capacity of data center GPUs, such as 8xH100s. Token generation speed, latency, is now limited by memory bandwidth (Kim al. 2024). Addressing these bottlenecks requires more expensive training, exceeding Chinchilla's 's compute-optimal regime (Hoffmann et al.. 2022). with less than 10B parameter models being trained up to 15 trillion
tokens (Touvron et al. (202bb AI@ Meta 2024 Team et al.) 2024). Another widely used technique is post-training quantization during deployment (huu et al. 2023): however, we demonstrate in Section
5 that this approach is sub-optimal.
In post-training quantization, LLMs initially trained using 16-bit floating point precision (referred to as FloatLMs) undergo parameter quantization, and the resulting models are termed QuantLMs. These models use optimized kernels for deployment, offering speedups nearly proportional to the compression factor (Frantar& & Alistarh] 2024). However, quantizing to very low bitwidths creates significant mismatch between the representations of pretrained FloatLM and the deployable QuantLM, resulting in undesired behavior and quality degradation (Li ct al. 2024 |Huang et al. 2024). Some oate--o-tthe-nn methods (Frantar et al. 2022 Egiazarian et al. 2024) mitigate this issue by using
"qqual contribution. listed in alphabetical order.
Coorespondence: tejas ai

---

Published as a conference paper at ICLR 2025
Commonsense & Reasoning Across Size
Cammonsense Reasoning Across Parameters
Avg. Score Across 6 Berchmarks
Avg. Score Across Berchmarks
FleeLMM
Flaetl M
CuanntL 3-50
Tr..
TriMM
10
20
30
40
60
60
190M
580M
1.58
3.88
Size in Bits
10
Parameters
(a) C&R: acc vs size
(b) C&R: acc vs parameters
LAMBADA Accuray Across Size (Bits)
LAMBADA Accuracy Across Parameters
FloatLM
QuentL
GuantL M 3-B8
TriLM
Accuracy (kg scall)
Accuraa)
40
30
FleatLM
GuantLM 4-Bit
QuantLM
TriLM
190M
560M
1.58
3.98
(10")
Parameters
(c) LAMBADA: acc vs size
(d) LAMBADA: acc vs parameters
Figure 1: Common Sense and Reasoning (C&R) & LAMBADA Accuracy for temary TriLM, FP16 FloatLM and quantized QuantL models across different model sizes, bits and number of parameters. C&R scores are averaged across benchmarks. 3B+ scales, TriLMs demonstrate better performance their size than QuantLM and competitive performance to FloatLM of the same parameters. See Tables and 10 for details.
calibration and re-training data from target domains; however, this approach increases sensitivity to the calibration data. For instance, seemingly simple choices, like whether to length-normalize the calibration data, can significantly impact QuantLM's performance (Malinovskii et al. 2024). Other works have observed that QuantLM at 4 bits (4-bit QuantLMs) have about 65% lower knowledge capacity per parameter compared to trained and aligned FloatLMs (Allen-Zhu & Li 2024).
An alternate approach to reducing model bitsize while maintaining parameter count involves training neural networks with low effective bitwidths (Zhou al. 2018). This approach offers compression benefits beyond post-training quantization without its associated drawbacks. While low-bitwidth approaches typically employ binary (1-bit) or ternary quantization (1.58-bit), binary quantization generally underperforms compared to regular FP16 models (Liu et al., (202a)) (see Appendix B In contrast, ternary modeling can match performance while sigmificantly reducing memory requirements (as we demonstrate in section 55.. Nevertheless, the relative performance of pretrained biw-bitwidth language models compared QuantLMs across similar sizes (in bits) and similar parameter counts remains unclear. This is crucial unanswered question, given the high inference costs during the deployment of very large-scale L.MMs. Moreover, the training dynamics and scaling law of these low-bitwidth models also poorly understood, partly due the lack of publicly available systematic suites and associated comparative studies.
Motivated by these challenges, we make the following contributions in this paper:
Feasibility and Scalability of Training Ternary Language Models (TriLMs) We discuss the deployment advantages (in section 2.1 and theoretical feasibility (in section 2.2 of training low- bitwidth models scale. We then introduce ternary language models (TriLMs) and systematically study their scaling laws compared to FloatLMs. Our analysis reveals that TriLMs offer better scaling behavior in terms of model size, measured in bits (refer Section 4.3) Moreover, the number
2

---

Published as a conference paper at ICLR 2025
of parameters increases, the difference in validation loss between TriLMs and FloatLMs becomes insignificant, indicating TriLM'ss competitive performance at scale.
Spectra LLM suite. We present Spectra, the first open suite of LLMs spanning many bit-widths. This suite includes Mss, the corresponding Quantl Ms at 3, 4, 6, and 8 bits, and ternary L.MSs (TriLMs). The latter uses ternary weights (-1. 0, +1 J. The suite features 9 models ranging from 99M to 3.9B parameters, all trained on the same 300B token dataset, totalling 54 models. We believe that the Spectra LLM suite makes valuable contribution to the LLM research community by facilitating comparative studies, exploring the scalability and efficiency of ternary modeling, and improving interpretability from neuronal to connection levels.
Evaluation and comparative analysis of TriLMs, FloatLMs, and QuantLMs at different scales. We evaluate MriLMs, FloatLMs,, and QuantLMs across multiple benchmarks, spanning commonsense, reasoning, knowledge capacity and toxicity. At the billion parameter scale, TriLMs consistently out- perform their Quantl .M and Floatl .M counterparts of the same bit-size across all the aforementioned benchmarks (see Figure T.. Surprisingly, TriLM 3.9B matches the performance of FloatLM 3.9B across all benchmarks, despite getting a higher perplexity and being 5.9 times smaller in bitsize.
However, also note that certain challenges remain in TriLMs. For instance, TriLM 3.9B exhibits the same level of toxicity and stereotyping as FloatLM 3.9B. significantly higher than a similarly sized Fl 830M when measured in bits. While TriLM 3.9B and FloatLM 3.9B show similar validation perplexity on some datasets, such as Penn Tree Bank and Lambada. a gap persists at this scale on web corpora, both in-domain (i.e., on test subset of SlimPajama, the same domain used to train the models) and out-of- (e.g.. on Dolma, C4 and RefinedWeb datasets). We provide detailed perplexity results in the appendix D.4
2 MOTIVATION FOR LOW-BITWIDT MODELS
2.1 MEMORY BOTTLENECKS AND LANGUAGE MODEL DEPLOYMENT
Memory Capacity of GPGPUs and Model Size (in Bits): Figure 24a in Appendix G reveal that memory capacity has consistently lagged behind computational power across various accelerators, including recent hardware like Blackwell (Nvidia Team 2024), MI325X (AMD Team 2024). and Gaudi3 (Intel Gaudi Team| 2024). This gap is further exacerbated by advanced computational techniques like Ampere sparse FP8. As shown in Figure 25a of Appendix G the model sizes (in GB) for TriLM, QuantI. 4-Bit, and FloatLM scale differently with parameter counts. For simplicity, the figure excludes overhead from KV Cache, activations, and compilation during model deployment. FloatL.M reaches the memory capacity of a single H100 at 34B parameters, with larger models exceeding the capacity of multiple GPUs In contrast, QuantL 4-Bit supports up to 300B parameters on a single MI300X. TriLMs, with over 300B parameters and appropriate packing, can fit on a single H100, making them not only efficient for GPU deployment but also ideal for edge devices.
Memory bandwidth of GPGPUs and model inference speedup: Figure 24b in Appendix G
demonstrate the trends of Memory Bandwidth over FLOPs for accelerators over the years, highlighting the slower growth of memory bandwidth compared to computation. This trend, indicating a downward slope, aligns with Kim et al. (2024)'s establishment of the memory wall in autoregressive LLM computation. They found that the speed of token generation is bottlenecked by the rate at which data fed from memory to processors, rather than the processing speed of the hardware. Consequently, the autoregressive decoding of LLM inference can theoretically achieve speedup proportional to its compression factor. Figure 25b Appendix G illustrates the maximum possible speedup for QuantLM 4-Bit and TriLM compared to FP16 at different parameter counts. Even at 7 billion parameters, TriLMs can be more than times faster at autoregressive decoding than FloatLM and times faster than QuantLM 4-bit. While QuantLM 4-Bit plateaus at maximum possible speedup factor of 4x TriLMs plateau much higher at 10x for FloatLM.
3

---

Published as a conference paper at ICLR 2025
2.2 Low BITS CAN CAPTURE WEIGHT VARIANCE EFFECTIVELY AT SCALE
In this section, we use information theory to support our hypothesis: as the number of parameters increases, training language models with low-bifwidth can effectively capture the necessary weight variance withoul significant information loss. Assuming a fixed training dataset, we base this hypothesis on analyzing weight distributions in FloatLMs ranging from 99M 3.9B parameters.
Shannon Entropy
Differential Entropy
-325
4.5
RozLLM
-3.0
3.5
RosLM 190M
RozLM
-350
2.5
RozLM 830M
RoaLM
2.0
-360
100
402
500
Number of Bins
Model Parameters (Millions)
Figure 2: Shannon entropy (in bits) discretized weight distribution with increasing number of bins.
Figure Differential entropy of Gaussian fits on weight distributions across different scales.
Assuming that the weights of a trained model follow a Gaussian distribution (see Appendix E}. we fit a Gaussian to these weights to understand their statistical behavior across model scales. The differential entropy is calculated using the expression: H(W = logz (2merdy where ow is the standard deviation of the weights (Papoulis & Pillai 2002). As shown in Figure differential entropy decreases with an increase in the number of parameters. This decrease indicates that the weights become more concentrated around the mean as the model size increases, suggesting higher predictability and less variability (MacKay 2003). This reduced variability is due to overparameterization, which leads to redundancy in the weights Zhang et al. (017): Neyshabur al. (2018).
Additionally, we also use Shannon entropy, calculated by discretizing the weight distribution into (N) bins and computing HShannon - Pi log2 Pi, where p; is the probability of the weights falling into the i-th bin. The probability is given by the normalized frequency of weights within each bin. Discretization allows to apply discrete entropy measures to continuous data, with the number of bins determining the representation granularity. Shannon entropy measures the average minimum number of bits needed to encode the weight values based on their distribution, providing a quantifiable measure of their "information content" (Shannon) 1948). Lower Shannon entropy indicates more predictable, less diverse distribution that be effectively encoded using fewer bits. Examining Shannon entropy across scales and bin sizes, we validate our finding that larger models require fewer bits for effective representation. As plotted in Figure 2 there general trend of decreasing Shannon entropy with increasing parameter count for given number of bins.
These analyses support the potential of low- biw-bitwitth models to match full fill-precision performance. especially as model sizes grow. In Section 4.3 we further substantiate this by analyzing the scaling behavior of our low-bitwidth TriLMS compared to FloatLMSs
2.3 SELECTING THE APPROPRIATE LOw-BITWIDTH MODEL
Selecting the appropriate quantization level is crucial for balancing computational efficiency and performance bitwiit models. Binary models quantize weights to {-1, 1}, simplifying multiplications to efficient XOR operations (Courbariaux et al. (0T66), making them suitable for resource-constrained environments (Hubara al.) 2017). Ternary models introduce a zero state {-1,0, 0, +-1),, replacing multiplications with additions and subtractions (Li et al. 2016), and improve efficiency by skipping calculations when weights are zero (Zhu et al. 2017). Conversely, models quantized to 2, 3, or 4 bits require more complex operations, including full multiplications, which are computationally more demanding than those required for binary and ternrry models (Zhou et al.) 2016). Therefore, this work only considers binary and ternary language models for further analysis.
The reduction in computational complexity in binary models often comes at the cost of significant performance degradation (Rastegari et al. (016)). In contrast, ternary quantization adds state,
4

---

Published as a conference paper at ICLR 2025
better approximating weight distributions without significantly increasing computational demands (Li et al. 2016), making them a sweet spot for balancing performance with efficiency gains (Zhu et al. 2017). Our observations align with these, as the scaling trends for given data size for our TriLMs consistently outperform those of MiLMs (Binary LLMS) across all parameter counts and bit sizes considered in this work, shown in Appendix [ and Figure [2] As result, this work primarily focuses on the TriLM model, which we describe in the next section.
3 TRILM: TERNARY LANGUAGE MODEL
Davice
Forward Equation
Thoslodinng
FlaatLM
Device 2
TrILM
(Wel
Mabic Suboactin
17 round mia [nax
1)
1).1)
Matiicc
Prouar
Y - XiP*
Device N-1
2as 2oy [Wyl
Idtrrediaa Reproentatica (aztvaatoon)
Wiy signt Wy
aWf XXP""
Device
Figure 4: The computational flow of the forward, backward, and inference processes in TriL.'ss linear layer with N-Way model parallelism is shown the left. Additionally, we provide the equations (on left) for the forward pass during the training of our Float M, TriL.,, and BiI. (for details, see
TriLM is LLaMa-style Touvron et al., 2023a) autoregressive transformers Vaswani et al. 2017)
model with RMSNorm (Zhang & Semnrich, 2019), SwiGLU Gated MLP (Shazeer 2020). Rotary Position Embedding (RoPE) (Su et al.] (022T)
Attention and no bias terms. In TriLMs.
we represent the weights of all the linear layers in one of three possible ternary states {-1, 0, }}. along with an additional floating-poii training, we maintain the latent (or master) weights in floating point precision, allowing for the accumulation of small updates over iterations that eventually contribute to a transition in the estimated ternary state of a parameter. As shown in Figure 4 during the forward pass, we ternarize the floating point latent weights on-the-fly. fly. This process involves calculating the scale value to the absolute mean of the latent weights. After scaling, we estimate the ternary state of each parameter by rounding off to the nearest ternary state. In the backward pass, we use a straight-through estimator to compute gradients of the floating point latent weights (Bengio et al.) 2013). Since we only need the scale values and the ternarized states during inference, we achieve significant reduction in both model size and inference time at larger scales compared FloatLMs. We provide formal description of these forward pass, backward pass, and inference time equations in the Appendix ($A.I). We represent the embedding and language model head in half- floating point across all our experiments.
number called 'scale value' shared across the matrix. During
Since, training of TriLMs requires on-the-fly fly computation of scale values, synchronizing for a single scalar across devices in model parallel training (Shoeybi et al.] 2019) can cause significant communication overhead. To address this, we allow each device to independently compute these scale values over its own matrix shard. This approach introduces additional artifacts, where the number of scalar values for each matrix is the same the degree of model parallelism used during training (see appendix A.6).. However, the impact on modelsize is negligible, for matrices with millions of parameters, we only add 6 scalar values each.
Concurrent research on low-bit LLMs, like BitNet 1.58 (Ma et al., 2024), also demonstrates the feasibility of training LLMs with ternary weights. Our experiments demonstrate that TriLM's architecture not only outperforms BitNet b1.58 but is also simpler and more stable. Moreover, both the larger BitNet 1. 1.3B model presented in their paper and our replication of the BitNet 1.1B model
5

---

Published as a conference paper at ICLR 2025
underperform compared to our TriLM 1.1B commonsense and reasoning benchmarks. We provide further details on these comparisons in Appendix A.7
4 SPECTRA SUITE: SPANNING PARAMETERS AND BITWIDTHS
The Spectra suite includes comprehensive families of large language models designed to span different parameter counts and bit-widths. This suite includes three main model families: TriLMs, FloatLMss and QuantLMs (3, 4, 6, and 8 bits). Drawing inspiration from established model suites such those by BBdderman et al. (2023). ), Liu et al. (202cc,, and Groeneveld et al. (2024). Spectra aims to facilitate scientific research on low-bitwidth LLMs, interpretability, and safety.
4.1 OVERVIEW OF SPECTRA SUITE
The spectra suite of LLMs is distinguished by several key attributes listed below.
1. Scale. The open suite spans broad spectrum of scales across parameter count (99M to 3.9B), sizes (9 * 10s to 6.4 * 10 bits) and bitwidths (1.58 bits to 16 bits).
2. Uniform Training, All the TriLMs and FloatLMs are trained on identical data sequences, specifically 300B subset of Slim Pajama (Soboleva et al., 2023) dataset (see Appendix ensuring training consistency. Data ordering and batch sizes are also kept consistent within each model family. All QuantL undergo quantization using the same calibration data on identical data sequences, maintaining uniformity in model quantization procedures.
A.2
3. Public Accessibility, Training data well as intermediate training checkpoints of TriLMs and FloatLms are publicly available for future research.
4. Consistent Model Parameter Mapping. All the model families maintain consistent one-to-one mapping in terms of parameter count for studying scaling phenomena.
Figure II demonstrates the Spectra LM suite spanning across two dimensions size (bits) and parameters. For each parameter count, we have 6 models across different bitwidths. Due to the availability of FloatLM, Spectra can easily be extended with new QuantL.s by using different Post Training Quantization methods. We provide details on the dataset and tokenizer, pretraining setup, hyperparameters, and optimization schedule across the families of models Appendix [A] Moreover, we describe our FloatLMs and their quantized versions in the next section.
4.2 FLOATLMS AND QUANTLMS
Family of FloatLMs. We utilize LLaMa-style (Touvron et al. 2023a) architecture akin to TriLM. In MloatLMs, we represent the parameters the weight matrices of linear layers as floating-poin numbers (FP16/BF16). The optimization schedule for FloatLM follows cosine decay scheduling with weight decay and includes a learning rate warmup. This is consistent with the practices established in models such Pythia, OLMo, LLM360. For more details, refer the Appendix (A.4).
Family of Mus Recently, data-aware quantization techniques like GPTQ (Frantar et al. 2022) have emerged as efficient solutions for losslesss weight quantization down to 4-bit precision DDttteers & Zettlemoyer 2023). In this work, we implement GPTQ post-training quantization FloatLM, creating the QuantLM family of models across 4, 6, and bits. We quantize all transformer layer weights. For 3-bit and 4-bit quantization, we employ group size of 128, which results in effective bit rates of 3.25 and 4.25 bits per parameter, respectively. We refine our approach by incorporating best practices from recent research by Malinovski et al. (2024), particularly in terms of calibration data and scaling it to a million tokens for improved reconstruction. To ensure fair comparison with TriLM, we maintain certain components in their original precision. Specifically, we do not quantize the embedding, language model head, or activations. Additionally, we employ symmetric quantization (without zero offset) for its simplicity, fast inference kemnlss support (Frantar & Alistarh, 2024), and comparable performance to asymmetric quantization, which uses separate zero offsets and scale values for each group. This approach also ensures consistency and facilitates fair comparison with TriLMs. Notably, our Spectra suite is designed with flexibility in mind, allowing for easy extension to other quantization methods as needed.

---

Published as a conference paper at ICLR 2025
4.3 TRAINING DYNAMICS
M: Tokees Thained va Traring loss
THA oullh lobers Tramed
2309
2.
1330
1.40
24
Pisits 2.54
22
F
2008
2003
Tokens Train Trained
Trained
(a) Training loss over time for TriLMs.
(b) Training loss of TriLM vs FloatL..
Figure 5: Training Cross Entropy Loss across steps the TriLM family of models. At the halfway point (150B tokens) when we lower the peak learning rate, we observe a sudden drop in training loss. In the two- third way, removing weight decay lcads to faster convergence.
Optimizing low bitwidth neural networks, such as in Quantization Aware Training (Liu et al. 2023b Yuan et al. 2024 Bethge al. 2018 Le & Li 2023), requires specific considerations like higher initial learning rates and reduced weight decay. Our optimization schedule for TriLM incorporates two key interventions within standard linear decay learning rate schedule with warmup and weight decay (L2 regularization). First, we reduce the peak learning rate at approximately the halfway point of training. Second, remove the weight decay regularization about two-thirds into the training, as ternarization provides sufficient regularization (Courbariaux et al. 2016b). Figure 9 Appendix (A.5 [A.5) shows an ablation study for 1.IB parameter model trained on 100B tokens, demonstrating that both interventions are necessary to achieve the lowest validation perplexity.
Figure 5a illustrates the training loss curves for all the TriLMs trained and Figure 5b shows the relative training loss of a TriLM in comparison to two smaller FloatLMs. The loss curves demonstrate continuous and consistent improvement in TriLMs with an increase in parameter count. During training, we make several key observations. First, minor spikes and drops in training loss occurred consistently across different TriLM scales at the same token counts, as all models were trained on identical data with the same ordering. Notably, the two largest models-TTrilee 3.9B-cach experienced a large loss spike in the first half of training. Second, adjusting the learning rate at the midpoint led to a sharp decline in training loss over few hundred million tokens, though its impact varied by model size: for the larger models (2.4B and 3.9B). the rate of loss reduction returned the prior pace after the initial sharp drop, while for smaller models (I.IB and 1.5B), the loss reduction plateaued, and models below IB parameters saw an increase in training loss. Lastly, the removal of weight decay at the two-thirds mark accelerated convergence for all models, with the effect being most pronounced in the largest TriLM models.
2.4B and TriLM
4.4 SCALING LAWS
In this section, we derive the scaling law for TriLMs and compare it to that of FloatLMs. All models, including our largest 3.9B parameter model, are trained on 300B tokens, placing them in the overtrained regime as determined by Chinchilla's 's Hooffmann al. (2022) compute-optimal token calculation. Figures and [6) illustraeee the final validation loss across different model size in terms of bits and number of parameters (N) respectively. In terms effective model size in bits (6a). which is crucial during inference, TriLMs exhibit significantly better scaling than FloatLMs. Notably, TriLM 3.9B validation error matches with FloatLMs 1. IB, which is nearly 1.7 times larger in terms of effective bit size. To investigate the scaling behaviour in terms of N with fixed data, we fit the validation loss to a power-law with offse Hoffmann et al. (2022) (see Figure 6b and Appendix
Anye Na
where ( [ARaMM = 185,
@TiLM 0.26,
eTiLM 1.76
Cyype(N]
+ Cype:
(1)
A = 159,
O'loonLM = 0.26,
eFlcaLL = 1.67
derived using fixed data regime of 300B tokens

---

Published as a conference paper at ICLR 2025
We employ the Levenberg- Levenberg-Marquard algorithm (Levenberg (944); Marquardi 1963)
for efficient non-linear least squares fitting. Both FloatLM and TrLM share the scaling exponent a = 0.26, indicating similar scaling behavior with the number of parameters N. However, the offset terms <TiLM = 1.76 and M = 1.67 reveal that their validation losses converge as N increases. Although TriLM starts with higher coefficient A 185, suggesting greater initial validation loss than FloatLM (A = 159), this difference becomes insignificant with larger N, aligning their performance asymptotic scales shown in Figure 6b and
Scaling with Bitsize
Scaling Law Fits
THIM
xa 1.66
1.2
My x.x
Validation Locs
3.5
Validation
2.8
3.-
2.8
2.2
1Ga Model Size params) LLg Scale
Model Size
1013
(a) Scaling laws - perplexity across size (bits).
(b) Scaling laws - perplexity across parameters.
Figure 6: Final validation loss across sizes (in bits) and parameters. TriLMs with increasing size offer better performance than MloatLMs of the same number of bits; and the gap in validation perplexity closes at scale.
Despite the observed differences in validation loss at the scale of models considered in this work, we demonstrate in Section 5 that at 3.9B parameters, TriLM offers competitive downstream performance compared to FloatLM of the same parameter count across a variety of benchmarks in commonsense reasoning and knowledge-based tasks. Moreover, as discussed in Appendix $D.4 both models show similar perplexity on clean datasets such Penn Tree Bank and OpenAl's Lambda. However, the gap in perplexity is observed in overlapping web-based datasets like Dolma and RefinedWeb.
RESULTS
We evaluate the families of LLMs on three aspects commonsense & reasoning tasks, knowledge- based tasks, and toxicity, all of which are crucial measures of their downstream performance. Readers may refer to the appendix for more details regarding the benchmarks ($D).
Commonsense and Reasoning. We evaluate Spectra Suite models using eight distinct common- sense and reasoning benchmarks consisting of tasks from logical and reasoning questions to grounded and physical commonsense tasks: Arc Easy, Arc Challenge (Clark et al.] 2018). BoolQ (Clark et al. 2019), HellaSWAG (Zellers et al., 2019). WinoGrande (Sakaguchi et al. 2021). PIQA (Bisk et al. 2019), LAMBADA (Paperno et al. 2216). LogiQA (Liu et al.) 2021). all under zero-shot settings. Figures Ta and Th display the average performance of the L.MMs on the first six benchmarks across size in bits and number of params. Figures Ic and present the performance for the LAMBADA dataset. The results show that TriLMs consistently demonstrate superior performance for their size across all benchmarks the 2.4B and 3.9B parameter scales. At the largest scale of 3.9B, TriLM surpasses FloatLM on LAMBADA and achieves competitive average scores across six benchmarks. Additionally. TriLMs at the largest scales consistently outperform 4-bit QuantLMs of equivalent parameter count. However, across the considered scales, all LLMs show poor performance on LogiQA, making it difficult to identify a clear performance trend. For detailed benchmarking across all datasets -se Tables 7 8 and 10
Knowledge. Several downstream practical uses of LLMs require them to have knowledge about common subjects like science or politics. To evaluate the performance of LLMs these subjects, we use SciQ (Welbl et al. 2017). TriviaQA (Joshi et al. 2017) and MMLU (Hendrycks et al. 2021) benchmarks in zero-shot settings. Figures 7a and 7b show the accuracy of the Spectra suite LLMs on
SciQ across size in bits and parameter counts. Figures 21c and 21d depict the accuracy for TriviaQA, while and db the same for MMLU. Across both benchmarks, at large 2.4B+ scales, TriL offer
8

---

Published as a conference paper at ICLR 2025
Sc and) Aaresa
Seil Accursy (horms und) Aeros Panmmrraa
Trhiude Exect ManchAArros Siee (8
TIKAOA Execct Match Paramalen
20
Gussriat 4o
Oueerti 3
2
PoutUi Queet
Hustae
Ouerdh 1-5
12
89
30
(a) Size SciQ
(b) Params SciQ
(c) Size TrivisQA
(d) vs. Parums TriviaQA
Figure 7: Performance of temaay TriL.,, Float.M and quantized Quantl M (3-bit 4-bit) models on SciQ and TriviaQA tasks across Size (Bits) and Parameters. Refer Tables T0 and T3 for details.
the best performance at a given size (bits). Surprisingly, despite having fewer bits, the knowledge capacity of TriLM does not have any significant degradation as observed in the case of Lua
(Allen-Zhu Zhu & Li 2024). This indicates that biwtiit LLMss like TriLMs have similar knowledge capacity to Float Ms, indicating that knowledge capacity is parameterized via the presence and nature of connection (+1 or -1), rather than its strength. Tables TO and 15 expand on these results.
MMLUU
MM Acroon Phraemeders
MM Steer Paramelers
Plottd
20
P.
Coint L.V 3:1
art 3.50
Trl
TV
20
Plat.M
24
Ract TYEM
Quar 6.2 4-a
THLM
1904
550M
1.00
330
10
1504
55300
1.50
132
(a) Vs. Size (Biis) Avg
(b) Vs. Pammeeter- Aug
(c) Vs. Size- STEM
(d) Vs. Purameters- STEM
Figure 8: MMLU Accuracy for ternary TriLM, FloatLM and quantized QuantLM (3-bit & 4-bit) models across Size and Parameters. Please refer to Table for details.
Toxicity. We evaluate the Spectra suite across various safety and toxicity benchmarks of TruthfulQA (Lin et al. 2021), Big Bench BBQ Lite (Parrish et al. 2022) and CrowsPairs (Nangia et al., 2020). These scores are listed in the Appendix in Table 15 We observe that none of the LLMs, even those with up to 3.9 billion parameters and trained on 300 billion tokens, perform significantly better than random guessing the TruthfulQA benchmark. For the other two datasets, there is noticeable correlation between the occurrence of toxicity and stereotypes and the I.LMs' performance on various tasks. In particular, TriLMs with fewer than one billion parameters exhibit less stereotyping than FloatLMs with a similar parameter count, However, this difference diminishes as the scale increases, with TriLM 2.4B and TriLM 3.9B exhibiting biases comparable to those of FloatLM 2.4B and FloatLM 3.9B, respectively, on these benchmarks. This suggests that, although TriLMs initially show reduced bias compared to similarly sized FloatLMs, their performance aligns with FloatL.ss of equivalent parameter counts at larger scales. This also highlights that TriLMs exhibit considerably more stereotyping than FloatLMs of comparable size (measured in bits), yet perform comparably to Ms with similar parameter counts.
RELATED WORK
Quantization of Large Language Models after Training. Post-training quantization (PTQ) algorithms convert pretrained high-precision model (FP32 / FP16 BF16) into lower precision format without requiring the original training process (Cai et al. 2020 Hubara et al.) 2020 Choukroun et al., 2019). These methods be either data- inteainndeeendent Additionally, PTQ for LLMS presents unique challenges due to numerical outliers in both weights and activations (Bondarenko al. 2021). GPTQ (Frantar et al. 2022 is pate-of-the-an one-shot
need a small ca calibration dataset.

---

Published as a conference paper at ICLR 2025
weight quantization method aimed at finding matrix of quantized weights (say W) that minimizes the squared error relative to the full precision layer output. By leveraging second-order information, GPTQ derives a closed-form solution to this optimization problem, making it scalable to large LLMs. Other methods (Dettmers et al. 2023 Lin et al. 2024; Lee et al. 2024) emphasize the importance of outlier weights that correspond high-magnitude activations. Some recent methods also quantized activation along with the weights (Xiao et al., 2224;) Yao et al. [022] 2023). [Ahmadian et al. (2023)
demonstrate that large activation outliers can be effectively mitigated scale by making appropriate optimization decisions during the pretraining phase.
Training Language Models At Lower Precision. Several prominent language models such as GPT (Brown et al. 2020a), NeoX (Black et al. 2022), Llama and Pythia families have been traditionally trained using mixed precision (FP32/FPI6 or FP32/BF16) (Micikevicius et al. 2018) or half-precision (FP16/BF16) (Kalamkar et al. 2019). Recently, Tao et al. (2022) introduced QuantGPT, model that incorporates contrastive and logit distillation from a full- full-precision teacher to a quantized student model during pretraining. Further developments, such BitNet (Wang et al.] 2023) and BitNet b1.58 (Ma et al.] 2024), have specifically focused on quantization-aware training for extremely low-bitwidth networks in transformer-based models. In their work, models are trained at low "effective" precision of binary and ternary respectively- where the latent (or master) weights during training are maintained in higher precision like FP16. The model weights are binarized or ternarized on the fly during the forward pass and gradients are backpropagated for the latent weights using the straight-through estimator (Courbariaux et al. 2016b)). Prior works emphasize the importance of maintaining latent (or master) weights at high precision to allow accumulation of small updates during training for example, Peng et al. (2023) observed a significant performance drop on the language model when the latent (or master) model weights were switch from 16-bits (FP16/BF16) 66) to 8-bits (FP8) during training. Concurrent architectural improvements such as Flash Attention (Dao et al. 2022; Dao 2023), the mixture of experts (Zoph et al.) [2022). XLSTM et al. (2024) and state space models
(Gu & Dao, 2024; Dao & 2024 Gu et al. 2022) complement these advancements in lower
precision modeling.
7 CONCLUSION AND FUTURE WORK
In this work, we address memory limitations in large language model (LLM) deployment by exploring both post-training quantization and direct low-bitwidth training. We introduce the Spectra LLM suite, featuring 54 models ranging from 99 million to 3.9 billion parameters, trained on 300 billion tokens. This suite includes 16 LLMs (FloatLMs), quantized QuantLMs (3-8 bits), and our proposed ternary LLMs (TriLMs). Our findings reveal that TriLMs scale better than their half-precision Float counterparts in terms of effective model bit size, and they can achieve comparable validation loss when scaled to a large number of parameters. Additionally, our results demonstrate that TriLMs surpass other models in bit-size efficiency and achieve performance comparable to FloatLMs at 3 billion+ parameters across multiple benchmarks.
Future work should address the remaining challenges of toxicity, stereotyping, and performance gaps on web corpora associated with bow-bitwidth models. Investigating scaling laws across diverse data regimes and bit sizes in low- bit-bitiitth models trained with quantization- aatiiiiiatioaawareeeee training can provide deeper insights into their behavior and limitations. Additionally, combining these models with state- space architectures like Mamba Gu & Dao (2024) could further enhance efficiency and performance without sacrificing accuracy. These research directions hold promise in advancing efficient language modeling further.
ACKNOWLEDGEMENT
We acknowledge the support from the Mozilla Responsible AI Grant, the Canada CIFAR AI Chair Program and the Canada Excellence Research Chairs Program. This research was enabled by the computational resources provided by the Summit supercomputer, awarded through the Frontier DD allocation and INCITE 2023 program for the project "Scalable Foundation Models for Transferable Generalist AI" and SummitPlus allocation in 2024. These resources were supplied by the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, with support from the Office of Science of the U.S. Department of Energy.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale, 2023. URL
attps::/arxiis org/ abs 19365.19266
@I@ Meta. Llama model card. /tt.t//sss////ttbbb.o////eeeeeee /aammm///aaa//////mm
#lob/main/MOn/ CAR___AAR... md
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling
laws, 2024. httpssss:: wttp://.rrrrab//22..0000///
AMD Team. Amd instinct mi210 accelerator
https:
Files/ Baceupaaatoss//ancccct////////22 httm:000mmmmm1mmmmmmmmmmmm..mm 2022a. Accessed: July 3, 2024.
AMD Team. Amd instinct mi250 and mi250x acceleretees....ooooooooo http://raoooootttttttts/s////a amd. com/ system/
2022b. Accessed: July
3, 2024.
AMD Team. Amd instinct mi300a accelerator.
[ccolaeattoo//aaassccccc/////1/0 html 2023a. Accessed: July 3, 2024.
AMD Team. Amd instinct mi300x accelerator. https: /wwp://wwwww. amd. www/ wtt.d//ooo//////uuueeeeeeeeeeeee
html 2023b. Accessed: July 3, 2024.
AMD
Team.
Amd
instinct
mi325x
accelerator.
https:
ir. amd com.coo////eee/e/ news http://wwweeeeets/reeeeeeee htt.://ess...eeeeeee/eeeaaaaaa/a
amd acce
2024.
Ac-
cessed: July 3, 2024.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Halla- han, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 9 2023. URL https:: //www www.gggtthuuu.u.... thub. com/ eutheeaa//ptttttteeee nexx
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXhv:2405.04517 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation, 2013.
Joseph Bethge, Marvin Bornstein, Adrian Loy, Haojin Yang, and Christoph Mcinel. Training competitive binary neural networks from scratch. ArXiv, abs/18120 1812.0196,, 2018. URL https
//api. semant i www.iiccchhol...ooccoooeeec htt.://ma...cccc......ooooorrrpo
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https::///// www. wandb Coftwareevvaaee available from wandb.com.
com/ Software
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. AAAI Conference on Artificial Intelligence, 2019.
URL https://ppp.pp. semant iommaatiissen cascool ar org/ htgp/CorpusTD 208 90939
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022. URL https: arriv. oog.orggaa.... abs//bb//20 06745
11

---

Published as a conference paper at ICLR 2025
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization, 2021. URL httpp::/arxiv.oor wwwpiii.rr//ab...
2109. 12948
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Werss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020a. URL https: http://abb2000.... l/aaxivoorg///oo..vaaaaoaa abs/ 2005. 14165
Tom B. Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq; A novel zero shot quantization framework, 2020. URL https::/arxxv.crrtt//
2001
00281
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantiizaiinn of neural networks for efficient inference, 2019. URL http:::://axxxv..rr//oo
06822
Christopher Clark, Kenton Lee, Ming-Woi Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologiex, Volume 1 (Long and Short Pupers), pp. 2924-29366 Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 101.8655/V//N19-130 URL
1100
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URLhttps: //api semant wtw.smannnncccco.......cccc www.llar.rrrgggooe wttp://oooooooooooooooooa 3922881
Matthieu Courbariaux, Yoshua Bengio, and Sean-Pierre David. Binaryconnect. Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 18:3123-3131, 2016a.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations, 2016b. httppstttt////xiii abs/
1511.00363
Tri Dao. Flashattention Faster attention with better parallelism and work partitioning, 2023, URL htttpss//arxx..oorr/////////3
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL URL.ttttps///arvvv...c////s//// 2ttp:xiv.oro//bb/2055.2222 21060
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL httpp::/aaxxv..oo//////
2205. 14135
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws, 2023.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spgr: A sparse-quantiinn repre- sentation for near-lossless Ilm weight compression, 2023. URL https: arxix. /rtp.v.oggaaab///
2306 0.3078
12

---

Published as a conference paper at ICLR 2025
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization, 2024. URL
https://arxiv/ http:/ivwwrg//..ooooo
org. wttp/////////// 2222400666111 06118
Elias Frantar and Dan Alistarh. Marlin: a fast 4-bit inference kernel for medium batchsizes.
https
//github. com -OT--DSSSSDDSDD marlin 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:22 17323, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoft, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023. /RLLittps:///eeodd http://o....rreecorrr///33333550
Amir Gholami, Zhewei Yao, Schoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. Ai and memory wall, 2024. URL httpppss///www.....///////222//2
Google TPU Team. Google cloud tpu v3. htte:s//lbood..ooool...////////s 2018. Accessed: July 2024.
Google TPU Team. Google cloud tpu v4. httpp://sbboddggoggle..e////////
2021. Accessed: July 2024.
Google TPU Team. Google cloud tpu v5c. httppp//aaaaaa...oo..../c///////
2023a. Accessed: July 3, 2024.
Google TPU Team. Google cloud tpu v5p. httpss///ooos.ppppeeecc////ooo/0 2023b. Accessed: July 3, 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Sol- daini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024. URL http:: http/aaxivivvo org. abs
00838
Albert Gu and Tri Dao. Mamba: L. time sequence modeling with selective state spaces, 2024. URL
2 00752
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference Learning Representations (ICLR), 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimat large language models, 2022, URL https: / arxiv. org/abs///. 220.3. 15556
13

---

Published as a conference paper at ICLR 2025
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang. Chongyi Wang. Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL https:: wwwpariww..rr.oo.oomm abs/
1904.06399
Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lx, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study, 2024. URL.ttps:://aze
org. abs 2404 14047
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yaniv, and Yoshua Bengio. Binarized neural networks. Neural Computation, 291):322-334, 2017.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming, 2020. URL https
7/
Intel Gaudi Team. Intel gaudi 2 and gaudi 3 ai accelerators, https::/cdrdw/:: wwtpettwwwpubbl...om.. intel com/ 818888174///ua/ii - 3--i acce aa-aalaccceeeaatooo lerator- wwwp:eewwwttetpppppppppe paper. 2024. Accessed: July
3, 2024.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: large scale distantly su- pervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611. Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 1111665111/1111111
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang. Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. study of 6 for deep learning training, 2019. URL https: /wtp///aavvvvvv. org. abs 1901.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https:: htttps://axii..ooo org. www.oog////aa/.o..oooo/.......a/ /2002/0011..o 08361
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Phuoc-Hooee Charles Le and Xinlin Li. Binaryvit: Pushing binary vision transformers towards convolutional models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pp. 4664-4673, June 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models, 2024. URL https:://axiis www.rxiv..oggaaaa.
2306.04.. 02272
Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2:164-168, 1944.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv: 1605.0460 2016.
Jason Li. Mechanistic interpretability of binary and ternary transformers, 2024. URL
https
1 /arxiv. hww.vorggabs//24000 17703
Shiyao Li, Xuefei Ning, Luning Wang. Tengxuan Liu, Xiangsheng Shi, Shengen Yan. Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models, 2024, URL
fttps://aarxiv
http:///wwivv.oo//a///////// wtt.:/o////aeooo.o 18158
14

---

Published as a conference paper at ICLR 2025
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration, 2024. URLhttps: http:////wwwww..///////.. 00977
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, ICAI'20, 2021. ISBN 9780999241165.
Zechun Liu, Barlas Oguz, Aasish Pappu, Yangyang Shi, and Raghuraman Krishnamoorthi, Binary and 01841
ternary natural language generation, 2023a. https://: http:////axxxiooo/aaaeees/
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mchdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. /7888, 2023b.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. Llm360: Towards fully transparent open-source llms, 2023c.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits, 2024.
David J. C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003, ISBN 978-0521642989.
Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. Pv-tuning: Beyond throught estimation for extreme 1lm compression, 2024. httltttss///xxi..ooto wttp://gga........ooo
abs 2405. 14852
Donald Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics, 11(2):431444 1963.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. 2018. URL httl:ttts///axiv.ooott wttp://///aaaoaooo/o././
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space, 2013. URL URL /ttp:/ttt/////tteel org/ abs//b///133 attgg/ab//331..377 3781
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pais S-iiis::--:---- A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online, November 2020. Association for Computational Linguistics.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks, In Proceedings of the International Conference on Learning Representations (ICLR), 2018
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brun- dyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derezynski, Yi Dong. Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabalesh- warkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski,
15

---

Published as a conference paper at ICLR 2025
Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping. Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https: https://arxiv. arxiv. org/ abs /20e/24666.. 10704
Nvidia Team.
Nvidia
tesla
v100
gpu
accelerator.
https: :/
tappf//
pdf 2018.
Accessed: July
3, 2024.
Nvidia Team.
Nvidia a100
tensor
core
gpu.
https://ww..nvii
Com/ http:/ooonettd///rrtneeeee http://///...aaaooooo.oooo/o/o//
nvidia a100- datasheet us mwidia- 10 14 pef. 2020.
Accessed:
July 3, 2024.
Nvidia Team.
Nvidia h100 tensor core gpu.
https://eesourees.nt www.:uurrc.e..iiiii......
en in-us-tenie oatp://ee......oodddddd..eoeerrp July 3, 2024.
2022. Accessed:
Nvidia Team. Nvidia h200 Me core
2023.
Accessed: July 2024.
Nvidia Team. Nvidia blackwell architecture.
http:://resources.vvidii..
en us-uusulln www.:-wee-aaccchettetteeeeeeeeee
2024. Accessed: July 3, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Pupers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 18,18//6vlll 1/133//1/P111111 144. http:tppsss/allllaooooloor.ryyo/
Athanasios Papoulis and S. Unnikrishna Pillai. Probability, Random Variables, and Stochastic Processes. McGraw-Hii 4 edition, 2002. ISBN 778800771226111
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086-2105, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 101118665/1/2222 htt.:///////2.....nnnnna......l. 165. URL https: http:////caaatthooccooooo
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang. Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng. Fp8-Im: Training fp8 large language models, 2023. URL https: pttttts://arxiv..oo www.:iiwiwoorgg/eeeeeeeeeeee www./o/r//bbbb/eee
18313
Ofrr Press and Lior Wolf. Using the output embedding to improve language models, 2017. URL hittps://arxiln wrg/abs//600 hccpsss//arav.orrrra////003 05859
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020. ISBN 9781728199986.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &
16

---

Published as a conference paper at ICLR 2025
Data Mining, KDD '20. pp. 3506-3506, New York, NY, USA. 2020. Association for Com- puting Machinery. ISBN 9781450379984. doi: 10.144533944866346770.. URL https //doi.org/10. 1et..11114/318888 011453339866.4467033 3406703
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525-542, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an ad- versarial winograd schema challenge scale. Commun. ACM, 64(9):99 106, aug 2021. ISSN 0001-0782. doi: 10.1 10.1144/377438. URL
Claude E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-423, 623-656, 1948.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Ommiquant: Ommidirectionally calibrated guantization for large language models, 2024. httph::
2308 13137
Noam Shazeer. Glu variants improve transformer, 2020.
Mohammad Shoeybi. Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-bilioo parameter language models using model parallelism. 2019.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hest- ness, and Nolan Dey. Slimpajama: A 627b token cleaned and deduplicated
version of redpajama, 2023. URL https: http::/wwwwwweebbeaaaeeeeeeeee www.bras. het.brlggnee//oogg///oo s Limpa j jama- a- 62166276tooo token c cleaned- and seccuuliiatted-versiii
renf-reepaaao jama
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 482 4821-4836, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/.28 331.
org
/20////22222222 acl- 331
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Chone
Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskly., Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang. Oscar Wahltinez, Paige Bailcy, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comaneseu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec. Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. 2024. URL https://arx: www.//ag/////.eoo/ 08295
George-Christian
17

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana: Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lec, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jían Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL /ttps:://prroceedngsss..u www..pp..cc/pppppe.......... ELle//Ffe225557aee1110033Sss
files//aperr20117
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models, 2023.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. ArXiv, abs/1707.06209, 2017. URL. htttps://p/ semanticscholalne wraschoolar..r//Coouussss
1553193
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL httpt::t/////tt www. ac www.wwwbb.... wtt..//.......oooogo/////oo 2020, emnlp demos 6
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024. URL https
//arxiv.com org abs / 2211. 10438
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable prstinrai quantization for large-scale transformers, 2022.
URL htlphhtts///xxii..ooom wwtp:/iwww.rrroooooo.o.oooooooot http://bbb//200. 01861
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization in llms from comprehensive study to low rank compensation, 2023.
URL URRpsssss://axxi orgg abs/23033 08302]
Zhihang Yuan, Yuzhang Shang, and Zhen Dong. PB-LLM: Partially binarized large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
https: http:://pppeeeieiee net http:/eneeeeettteeeeee
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P19 1472. URL.ttts::
antho ogy. wwwphhhoogy....gg//oo.oo
18

---

Published as a conference paper at ICLR 2025
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR), 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 2016.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients, 2018. URL https?
//axxiv.org/abs// 1606.66660 06160
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained terarry quantization. arXiv preprint 161.v 2017.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey on model compression for large language models, 2023.
http://wwxxw....a/aa/a// 077633
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL
https: /ttps://arrtie wttparwwwwwwo./////////000///
owt. /20p 08906
19

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Architecture and Pre Training Details
20
A.1
Forward Pass, Backward Pass and Inference Equations
21
A.2
Data and Tokenizer
21
Pre Training Setup
22
A.4
Hyperparameters
................................
22
A.5
Optimization Schedule
22
A.6
Known Implementation Artifacts
23
A77
Diflerences from BitNet Architecture
23
A.8
Spectra Suite LLMss: Parameter Scaling and Bitwidth Variations
25
A.9
Details Selection Hyperparameters)
25
A.I0
Perplexity for Increasing Training Tokens TnLM Models
26
B
Binary vs Ternary Large Language Models
27
B.1 BiLM: Binary Large Language Model
27
B.2 Scaling Binary vs Ternary Model
27
B.3 Results
27
C Scaling Law
29
Benchmark Details
31
Commonsense and Reasoning
31
D.2
Knowledge
................................
31
D.3 Toxicity
34
D.4
Perplexity on other datasets
34
D.5
OmniQuant: 3-Bit Post- Trainin Quantized Models
36
E
Weight Distribution of Linear Layers
F
Learning Dynamics Analysis of Peak LR drop and decay stage
39
G
Memory Bottlenecks and Low-Bitwidth Language Modelling
41
Overview of Recent Datacenter GPUs and Accelerators.
41
Memory Trends Speedup Opportunities in Low-Bitwidth Language Modeling
41
H Ablations
43
Mustrative examples of TriLM 3,9''s completion capabilitics
46
Broader Impact
48
A ARCHITECTURE AND PRETRAINING DETAILS
This section provides comprehensive overview of the architectural design and pretraining for TriLM (Ternary Language Model) and FloatLM (Floating Point Language Model). We outline the forward and backward pass equations specific to their linear layers, highlighting the contrast between the FP16 matrices FloatLM and the termary matrices with scalar scaling in TriLM. Additionally, covers dataset selection, tokenizer usage, and preprocessing methods employed for training data preparation. These discussions provide information on pretraining setups, implementation nuances, and key hyperparameters critical the models' development.
20

---

Published as a conference paper at ICLR 2025
A.1
FORWARD PASS, BACKWARD PASS AND INFERENCE EQUATIONS
Table 1 shows the equations across TriLM vs FloatLM for forward pass, backward pass and inference.
Type
Forward Pass
Backward Pass
Inference
o1 ay
FloaLLM
Y XW2
==XWT
at
0L
BY
7 7
W
TriLM
= round
min
max
Wiy
0L AL
BY
Compute W. and y once and cache
W W Y XW
Wij
Wes
AL 7L
X
aw BY
Y = XW"
1 Ei= Ej= [Will
a um
Compute W and a once and cache
Wij sign( Wi: nm - Ej-1 Ws2)
8L aL
BiLM
ax BY
W = a
Wi = aW;
aL OL
X
aW BY
XIP
y = XWT
Table Equations in Linear Layer of TriLMs and FloatLMs.
Reason for restricting the quantization approach to linear weights in TriLMs. In developing extremely large language models like TriLMs. a key architectural strategy is to quantize only the linear layer weights while keeping the embedding layers and language model head higher precision. This is driven by the need to reduce model size while maintaining performance. Linear layers (dense layers) hold the bulk of the parameters in transformer models (Vaswani al.) (017). Quantizing these weights to ternary states significantly reduces the model size, facilitating deployment on memory-ccnsttrinee hardware. However, the embedding layers and language model head remain in higher precision (e.g., half- precision floating point) to preserve critical functions in language understanding and generation. Embedding layers encode important semantic and syntactic information, and quantizing them would degrade performance (Mikolov al., 2013). Similarly, the language model head, which maps internal representations to the vocabulary space, requires high precision to maintain prediction quality (Press & Wolf 2017)
A.2 DATA AND TOKENIZER
Let input be X E Raxn for a linear layer with FP16 weight matrix W E Roxx and Y Roxm be the output. The same matrix W is also used denote latent weights in TriLMs during training.
Dataset Selection:
For ternarized layers in TriLMs, we also have scalar scale 7 R, matrix with ternarized states W E (-1,0, 1axm and ternarized matrix W E Raxm- We set = le - 5.
Due to lack of availability of Pile 300B (Gao et al. 2020) used in Pythia, opted to use 300B token sample of deduplicated Slim Pajama datase We sample from each subset with the probability proportional to its size.
Training Data Preparation:
Main experiments (Spectra suite): We used the full 300B token sample.
Ablation studies: Training runs with 100B tokens, we sample from these 300B tokens with equal probability weight to each data-point.
Fine-Wel Edu experiments: We tokenized one-third of a 350B token sample, from which we then sampled 100B tokens for our experiments.
We also make this subset public
21

---

Published as a conference paper at ICLR 2025
QuantLM: For the creation of QuantLM, utilized subset the Slimpajama-62777 dataset, consisting of 512 samples with sequence length of 2048. These samples were normalized for length. Our approach closely follows the methodology outlined in (Malinovskii et al. 2024).
Dataset
Size (Tokens)
Arxiv
13B
Book
13B
C4
80B
Tokenizer and Optimization Techniques: We use the GPT-NeoX 20B tokenizer following Pythia. For speed- ing up training, we round embedding rounding of to the nearest multiple of 128 times the model parallel size.
Common Crawl
156B
GitHub
16B
Stack Exchange
10B
Wikipedia
12B
A.3 PRETRAINING SETUP
Total
300B
We scale using 2D parallelism with Megatron-styl shard- ing (Shoeybi
Table 2: 300B Subset of Slim Pajama
2019) and use ZeRO stage Deepspeed
(Rasley et al.) (020) for ZeRO (Rajbhandari et al. 2020). Our implementation was based on GPT NeoX Codebase (Andonian et 2023). We use AdamW (Kingma & Ba 2017) for optimization. We train on nodes with IBM Power9 PC CPUs and 6x 16GB V100. Due to the lack of BFloat 6 support in V100, we train both TriLM and FloatLM in FP16 using Mixed Precision Training and Dynamic Loss Scaling. Please refer to $A.6 for more implementation specific details. We extensively use Huggingface (Wolf et al., 2020) and Wandb (Biewald 2020) for handling the checkpoints and experiment tracking.
A.4 HYPERPARAMETERS
Table shows the hyperparameters for TriLM and FloatLM's transformer architecture and their learning rate. We set Adam 3 are (0.9, 0.95) for both families of models and all the reported runs are trained to 2048 sequence length. FloatLM and TriLM are respectively trained with batch sizes of 2M and 1M tokens respectively.
Params
Hidden
GLU
Heads
Layers
MP
FloatLM LR
TriLM LR
99.74M (99M)
512
1280
8
4.0 10
2. 1.5 10
190.0M (190M)
768
2048
12
4. 10
2.1+10-3 1.5x 10-3
392.4M (390M)
1024
2560
16
24
3.0 10
1.8 1.2 10-3
569.2M 2M (560M)
1280
3072
20
24
2.8+
16+10
1.1x
834.0M (830M)
1536
4096
24
24
2.5+10-1
15+10-3 - 110x
1,149B (I.IB)
1792
5120
28
24
2
2.2 10-4
13+ 3 10
1.515B
2048
24
2.0 10
1.2+10 1.0
2.461B (2.4B)
2304
7680
36
30
2.0 10
1.2
8.0 10
3,989B (3.9B)
3072
9216
24
30
1.5 10
1.2 10
8.0
Table 3: Hyperparameters across model sizes for TriLM and FloatLM.
Params
99M
190M
390M
560M
830M
1.1B
1.5B
2.4B
3.9B
FlsaLM
1.60
3.05
6.28
9.11
13.34
18.39
24.23
39.38
63.83
QuaniLM
QuantLM 6-Bit
1.11
1.92
3.38
4.70
6.55
8.70
11.15
17.09
27.03
QuaniLM 4
1.03
1.72
2.88
3.93
5.36
7.00
8.86
13.18
20.59
QuantLM 3-Bit
0.98
2.59
6.03
7.55
10.95
TrLM
0.90
1.42
2.11
2.76
3.55
4.42
5.36
7.23
10.76
Table 4: Sizes in bits (* ($109) for Spectra suite of LLMs across varying parameter counts.
A.5 OPTIMIZATION SCHEDULE
In this section, we ablate the two interventions a vanilla linear decay learning rate scheduling with warmup and weight decay (I.2 Regularization). (1) Peak I.R roughly the halfway point, reduce
22

---

Published as a conference paper at ICLR 2025
the peak learning rate. (2) L2 Reg. at roughly two-thirds of the training, we remove the weight decay regularization as ternarization provides sufficient regularization (Courbariaux et al. (0166)) Figure 1 demonstrates the ablation run performed for 1.IB parameter model on 100B tokens with both, only one and neither of these interventions.
Among these four runs, notice the lowest final training loss when both, the L2 Regularization and Peak LR in- tervened, closely followed only L2 Reg- ularization 1/2 and 2/3) being inter- vened and then only Peak LR being in- tervened. Dropping the peak LR the halfway point leads to quick sharp drop in training loss. Similar phenom- ena have also been observed in sched- ules with small episodes of fast learning rate decaying like MiniCPM (Hu et al., 2024). On the other hand, removing L2 regularization, weight decay, leads to accelerated convergence, which can even mostly have the same effect as low- ering peak LR leading to quick drop in loss. These relative training loss ob- servation at 100B tokens also go hand in hand with relative downstream per- formance across commonsense and rea- soning tasks, which are listed in Table [4]and [3] Thus, we fix the TriLM opti- mization schedule where we drop in the peak learning rate at the halfway mark and the weight decay removed the two-thirds mark.
TriLM Training Loss Across Optimization Schedules
2.75
Hyperparameker Intervened
Training Cross Entropy Loss
Peak LR L2 Reg.
Only L2 Rog.
2.70
Only Posk Baseline
2.60
2.55
2.50
25B
50B
75B
100B
Tokens Trained
Figure 9: Training loss for 1.IB parameter TriLM, across different optimization schedules. We intervene for combinations of two hyperparameters peak learning rate and L2 regularization. Intervention for both hyperparameters given best training loss.
A.6 KNOWN IMPLEMENTATION ARTIFACTS
Similar to BitNet (Wang et al. 2023), our models exhibit artifacts resulting from model parallelism. key issue arises when computing the scale, 7, across the entire weight matrix, which is sharded across multiple devices. This process introduces significant communication overhead due the all-reduce operations. In our implementation, we address this by computing the scales over the portion of the weight matrix local to each device. Consequently, during inference with TriLM models, scales are computed independently within each model parallel group. Importantly, this modification has negligible impact on the bits per parameter, amounting less than 10-5 even at the highest model parallelism level of 6 for our largest model.
Given that we train in FP16, some artifacts are expected result of this training method. However, we do not anticipate significant performance differences when comparing mixed precision training with BF16 or even FP32. This expectation is based on the observation that the lowest loss scales recorded during our runs were consistently at or above the recommended value of 128 (Micikevicius) ct al. 2018 (refer to Table
A.7 DIFFERENCES FROM BITNET ARCHITECTURE
TriLM differs from BitNet b1.58 in several ways for better performance as well as for fairer com- parison with FloatLMs. Adopting the GPT-3's Pre-Normalization approach as outlined by (Brown) ct al. 2020b), normalization is applied prior to each linear layer. This method has proven essential for maintaining stable training under FP 6 precision (Wang et al. 2023). Consequently, normalization occurs twice within each transformer layer: once at the input representations to the attention sub-layer and again at the input representations to the Gated MLP sub-layer. This approach contrasts with BitNet, where activation or intermediate representations are normalized, scaled, and quantized to bits before each linear layer, which occurs between 4 to times per transformer layer depending on
23

---

Published as a conference paper at ICLR 2025
Model
Min. Loss-Scale
# Skipped Batches
# Skipped Tokens
FloatLM 99M
256.0
181
0.37B
TriLM 99M
1024.0
303
0.33B
FloaLLM 190M
512.0
0.35B
THLM 190M
512.0
0.33B
FloatLM 390M
1024.0
170
0.35B
TrLM 390M
512.0
312
0.34B
FloaLLM 560M
256.0
164
0.33B
TriLM 560M
512.0
294
0.32B
FloatLM 830M
2048.0
0.36B
THLM 830M
128.0
0.33B
Floatl M 1.1B
2048.0
158
0.32B
TrLMM 1.1B
512.0
306
0.33B
FloatLM 1.5B
256.0
170
0.35B
TrLM 1.5B
512.0
318
0.34B
FloatLM 2.4B
1024.0
0.34B
TriLM 2.4B
256.0
0.32B
Floatl .M 3.9B
256.0
164
0,34B
TriLM 3.9B
128.0
309
0.33B
Table Final loss-scale and number batches skipped across TriLM FloatLM training runs We are able to maintain above the recommended loss scales of 128 for mixed precision training (Micikevicius et al. 2018).
Relative Performance Across Architectures
Avg. Score Across Benchmarks
Se.
52
RILA 1.18
RealLM 1.10
Batker 1. 18 (Ours)
Bisee 52 TOSM
BRoeet by 1.39
Figure 10: Performance across various architectures TriLM 1.IB, FloatLM L.IB, BitNet b1.58 I.IB (our replication) along with reported scores of BitNet b1.58 at 700M and 1,3B params. Scores are averaged across common sense and reasoning benchmarks, mentioned in Table 14 and 13
the specific implementation. Furthermore, TriLM employs RMSNorm with scale parameter the parameterless R.M.Norm.
Figure 10 shows the commonsense and reasoning performance of TriLM I.IB, FloatLM 1.1B and our replication of BitNet b1.58's architecture at L.IB scale, along with the reported performance for BitNet 1.55 700M and 1.3B. All these models have been trained for 100B tokens. Our BitNet
24

---

Published as a conference paper at ICLR 2025
replication achieves performance between the 700M and 1.3B models. However, all the BitNet models, including the larger 1.3B parameter model perform worse than TriLM 1.1B. It should be noted that at this 1.IB scale, TriLMs do not achieve parity with FloatLMs of the same parameter count. Table 14 and 13 ists the detailed performance of these models across common sense henchmarks.
A.8 SPECTRA SUITE OF LLMS: PARAMETER SCALING AND BITWIDTH VARIATIONS
Figure IT iillstraats the Spectra LM suite spanning two key dimensions: bitwidth and the number of parameters. The suite includes the TriLM, FloatLM, and QuantLM model families (in 3, 4, 6, and bits). The suite features different parameter scales ranging from 99M to 3,9B parameters. In total, the suite comprises 54 models.
S Spectra Suite Spans Across Parameters & Bits
FloatLM
Model Size in Bits (Log Scale)
50
QuantL
Quenii 6-Bt
CuenILM 4-Bt
20
QuaniL
TriLM
10
2
190M
560M
1.5B
3.9B
Parameters (Log Scale)
Figure 11: The Spectra Suite spans across two dimensions: parameters and bitwidth. Each point corresponds to a language model in the suite.
A.9 DETAILS ON THE SELECTION OF HYPERPARAMETERS
Model architecture hyperparameters. To ensure stable training and mitigate risks of loss spikes or slow convergence, we carefully selected key architectural hyperparameters such as layer normalization techniques and other training parameters. To optimize hardware efficiency, we padded the vocabulary size and rounded the hidden dimensions to the nearest power of two, facilitating alignment with accelerator constraints. Given computational limitations. our scaling and coefficient analyses were conducted with nine data points within the boillion-paramett range, each trained on datasets containing 300 billion tokens. These constraints directly influenced the design choices for the number of layers, hidden dimensions, embedding size, feedforward network dimensions, and attention heads across model variants.
Training hyperparameters. We adopted a rigorous evaluation-driven approach to refine training hyperparameters, encompassing model initialization, optimizers, and learning rate schedules. This methodology mirrors the training-time evaluations utilized in other model suites such as those by Biderman et al. (202)) and and Groeneveld et al. (2024). To systematically explore the hyperparameter space, we employed grid search over configurations, including learning rate, weight decay, batch size, and optimizer variants. Evaluations are conducted at different stages of training. from 4 billion up to 20 billion tokens, providing continuous and carly-stage feedback model performance. The key metrics evaluated include validation loss, commonsense reasoning, knowledge based task performance, and toxicity assessments, as outlined in the paper.
25

---

Published as a conference paper at ICLR 2025
A.10 PERPLEXITY FOR INCREASING TRAINING TOKENS IN TRILM MODELS
Impact of Training Tokens on Perplexity in TriLM Models. The performance of language models generally improves with increased number of training tokens, demonstrated in prior work (Hoffmann et al. 2022: Kaplan et al.] 2020). This trend is clearly reflected in the validation perplexity of our TriLM models across varying training token counts. The results, summarized in Table 6 highlight the critical role of scaling up training data to reduce perplexity. Notably, the larger consistetyy achieves lower perplexity all token scales. This reinforces the scaling law relationship among larger model sizes, more extensive training data, and reduced validation perplexity, mirroring the scaling laws observed for FloatLMs.
Training Tokens
Perplexity of TriLM 99M
Perplexity of TriLM 560M
50B
27.11
15.48
150B
26.30
14.50
300B
25.70
14.01
Table 6: Perplexity comparison for TriLM models with 99M and 560M parameters across different training token counts.
The reduction in validation perplexity, even after training on over 3000 times the number tokens relative to number of parameters, is particularly evident smallest model (99M) of our suite. As shown in Table the 99M model continues to demonstrate improvement with up to 300B tokens, suggesting that TriLM remains effective even at higher training ratios. Extrapolating from this trend, our largest model (3,9B) could theoretically train on up to 11.8 trillion tokens without reaching convergence. This scale comparable to the magnitude of training tokens used for LLaMA 3, state-of-the-ard open-source language model AT Meta (2024). However, due to resource constraints, training at such a scale remains direction for future work.
Impact of Token-to-Parran
26

---

Published as a conference paper at ICLR 2025
B BINARY VS TERNARY LARGE LANGUAGE MODELS
In this section, we will comprehensively compare Binary Large Language Models (BiLMs) with Ternryy Large Language Models (TriLMs). We will start by describing BiLMs, followed by studying scaling laws and presenting results on various benchmarks, as well as comparisons with TriLMs across parameter count and model size (in bits).
B.1 BILM: BINARY LARGE LANGUAGE MODEL
In Binary Large Language Models (BiLMs), the weights of the linear layers represented by binary values -1 or 1, with an accompanying floating point scaling factor, similar to the method employed in TriLMs. Comprehensive formal descriptions of the forward pass, backward pass, and inference time calculations are provided in Appendix A We have trained three BiLM models distinct sizes: 99M, 560M, and 1.1B parameters. These models were trained the same dataset and in the same sequence as the MriLMs, adhering to the identical optimization schedule detailed in Appendix A.5
Cross Entropy Model Parameters
Cross Entropy Model Size (Bits)
Cross Entropy (Log Per perplexity)
Temary LLMS
Binary LLMg
Binary
Temaay
Entropy
0.22
Model Parameters
Model Size (Bits)
Vs. Params
(b) Vs. Size (in Bits)
Figure 12: Final Validation loss across size (measure in bits) and parameters.
B.2 SCALING BINARY VS TERNARY MODEL
Figures 12a and 22 show the final validation loss across model sizes (in bits) and parameter counts, respectively for the three distinct sizes previously mentioned. At the Billiont model scale, Ternary Models appear to be preferable in terms of both the number of parameters and model size (in bits). However, the gap seems be narrowing, which suggests that BiLMs have the potential be competitive with TriLMs at higher parameter counts (100B+). Therefore, we decide to only scale TriLMss further to study the scaling laws FloatL.Ms vs TriLMs up 3.9B parameters.
B.3 RESULTS
We conducted comprehensive benchmark analysis of Binary Large Language Models (BILMs) across three key dimensions: commonsense and reasoning tasks, knowledge-based tasks, and toxicity evaluation, detailed in Tables 7 TTO and 15
27

---

Published as a conference paper at ICLR 2025
Accuracy vs Model Size
Accuracy vs Model Parameters
AV3 Score Acrocs Benchmans
Benchmanks
40
40
Size Bitx (10F)
Modal Parameters (Millions)
(b)
Figure 13: Performance Lemary TriLMs and BiLMs models on commonsense Reasoning and MMLUs lasks across Size (Bits) and Parameters. Refer to Tables 7 and for details.
LAMBADA vs Size in Bits
LAMBADA vs Model Parameters
Acourery (M
BILM THM
BiLM THLM
Size (Bits 10"
Model Parameters (Billions)
(a) vs. Size LAMBADA
(b) vs. Parmss in LAMBADA
Figure 14: Performance of ternary TriLMs and BiLMs models on LAMBADA tasks across Size (Bits) and Parameters. Refer to Tables 7 and for details.
Parameters
BILM
THLLM
THLM
Siow (B.bx 10-9)
Model (Billame))
Sie
vs. Size MMLU
vs. Purams MMLU
Figure 15: Performance of temaay TriLMs and Ms models on MMLU across Size (Bits) and Parameters. Refer to Tables 15 for details.
28

---

Published as a conference paper at ICLR 2025
C SCALING LAW
In this section, provide additional insights into the scaling fits discussed in Section 43 In addition to fitting a power law with an offset, we also explore a standard power law following Kaplan et al. (2020). Our findings suggest that the standard power law fits slightly less precise than those incorporating an offset term. However, both models indicate decreasing difference in validation loss the number of parameters (N) increases.
Scaling Law Fits (Linear Scale)
TriLM PowerLaw with of -2 1.76
FloatLM PowerLaw with y 1.67
TriLM: TriLM-Powwet PowerLaw:y 18.13 x -9.03
4.0
FloatLM PowerLaw:y
0.055
Validation Loss
3.5
2.5
2.0
Model Size
1e9
Scaling Law Fits (Log-Log Scale)
TriLM: TriLM:Poweet PowerLLw with offset: 185 x- 1.76
Validation Loss (log scale)
10a
FloatLM PowerLaw with offsettt==ytt: 92 1.67
M.MM::oowrr.wwy 18.13 x x -0.00
:PowerLaw:
3 10P
2x 10a
107
10a
109
1040
10a
10a2
10aH
Model Size (log scale)
Figure 16: Comparison Power Law Power Law-with-offset Fits TriLM and FloatLM.
As shown in Figure using the scaling equations for TriLMs and FloatLMs, derive the relationship between parameter count and the percentage difference in validation loss relative to FloatL We observe that at 330B and 15.6B parameters, the validation losses for TriLMs
29

---

Published as a conference paper at ICLR 2025
within 6% and 7% of FloatLMs' validation losses, respectively. This indicates that TriLMs are likely to closely match the performance FloatLMs at larger scales.
Parameter Count vs. Validation Loss Difference (% of FloatLM's Loss)
1022
Number of params (log scale)
N(k) = expl
0.26 26.1 1.59K
log
0.01671k 0.0835
1020
10
1026
1024
10
329.92 B
15.65 B
10
2.14.B
0.44
0.11 B
10
10
Validation Loss difference in % of FloatLM's loss
Figure 17: Comparison of Power Law and Power Law-with-offset Fits for TriLM and FloatLM.
30

---

Published as a conference paper at ICLR 2025
D BENCHMARK DETAILS
We benchmark TriLM, FloatLM and QuantLM across Knowledge, Commonsense, Reasoning and Toxicity benchmarks. We average our scores across 3 different 'seeds' by preparing three different QuantLM models quantized using different calibration sets. We also add Pythia (deduplicated with consistent 2M batch size across families) suite of models (70M to 2.8B params) and BitNet b, 158 performance scores from their paper for comparison. We use the LM Evaluation Harness (Gao et al. 2023 to benchmark.
D.1
COMMONSENSE AND REASONING
We report commonsense and reasoning benchmark scores across benchmarks previously considered by BitNet 158 in Table 8 and rest in Table I0 Each is considered in zero-shot setting, Following the details of each the benchmarks considered:
ARC Challenge and Easy: (Clark et al. 2018) ARC dataset comprises 7787 multiple- choice science questions divided into two sets: Challenge and Easy. We calculate accuracy and normalised accuracy across both of these sets.
BoolQ: (Clark et al. 2019) BoolQ is reading comprehension dataset consisting of naturally occurring yes/no questions. We calculate the accuracy of this task.
HellaSwag: (Zellers al. 2019) HellaSWAG is a dataset of multiple choice questions for
testing grounded commonsense. The incorrect options are generated through Adversarial Filtering (AF) to fool machines but not humans. We calculate the accuracy and normalized accuracy on this task.
(Sakaguchi et al.) 2021) WinoGrande is collection of 44k problems for testing commonsense reasoning formulated as a fill-in-a-blank task with binary options. We report the accuracy on this task.
WinoGrande:
PIQA:
Bisk et al. 2019) Physical Interaction Question Answering (PIQA) is a physical commonsense reasoning benchmark dataset to test the physical knowledge of language models. We calculate accuracy and normalized accuracy on this task.
(Paperno et al., 2016) LAMBADA is dataset to evaluate text understanding by next-word prediction. It collection of narrative passages BooksCorpus To succeed on LAMBADA, models must integrate broader discourse information, not solely rely on local context. We calculate the perplexity and the accuracy of the model on this task.
LAMBADA OpenAl:
LogiQA: (Liu et al. 2021) LogiQA is a dataset for testing human logical reasoning.
contains questions spanning multiple types of deductive reasoning. We calculate the accuracy and normalized accuracy on this task.
D.2 KNOWLEDGE
We report performance on SciQ, TriviaQA in Tables 10 15 and 16 Each is considered in a zero-shot setting. Following are the details of each of the benchmarks considered:
The knowledge-based evaluation included the following tasks:
SciQ: (Welbl al. 2017) The SciQ dataset contains multiple-choice questions with 4 answer options from crowd-sourced science exams. The questions range from Physics, Chemistry and Biology and several other fields. We calculate the accuracy and length normalized accuracy on this task.
TriviaQA: (Joshi al. 2017 TriviaQA is reading comprehension dataset containing question-answer-evideeee triples. We calculate the exact match accuracy on this task.
MMLU (Hendrycks et al.) 2021): The benchmark aims to assess the knowledge gained during pretraining by evaluating models solely in zero-shot and few-shot scenarios. It spans 57 subjects, including STEM fields, humanities, social sciences, and more.
31

---

Published as a conference paper at ICLR 2025
Models
Challenge
Are Easy
BoolQ
Acc Nore.
Acc
Acc Nomn.
Acc
Ace
Pythis
22.01
22.11 1.2
2484 0.9
2484 0.9
38.51
FloiLM 99M
2388 1.2
1991 12
39.1 1.0
43.11 1.0
58.21 0.9
Quant
23.8
45.3H
58.54
QuanL
23.2
3884
M
37.11:
Quant
23.24
3434
36.1
48.4+
TeLM
24.14
Mo6-t:
3984
61.34
Binary 99M
20.84
1833
35.81-0
40.1
61.04 08
Pythia 100M
23.84
23.1 12
26.34- 09
26.64
38.3 0.9
M 190M
24.11
20.5 12
4300 1.0
48.41
59.11::0 0.9
QuanLLM 190M S-Bi
24.4 1.3
2133 122
43.0 1.0
48.1 1.0
59.3+ 0.9
Quant M 190M 6-Bi
23.8
2002 1.2
42.0 1.0
48.0 1.0
99.1 0.9
QuamLM 190M 4 Bi
25.24 1.3
1999 12
26.1 09
26.8 0.9
40.9+ 0.9
QuanLM 190M B-Bit
22.5 1.2
1942 122
37.1 1.0
39.7.1 1.0
$6.11
TrLM 100M
23.0 1.2
19.5 12
30.6 1.0
43.9 1.0
4688 0.9
Flouil 390M
24.7 1.3
21.34 12
463 1.0
$1.0 1.0
$4.:2
Quant M 390M 8-Rir
24 1.3
21.22 12
46 1.0
$1.0 1.0
$4.::00
QuamLM 290M 6-Biz
24.8 1.5
21.54 12
4634 1.0
5184 1.0
55.3 0.
QuanLM 390M 4-Bi
25.11
2133 12
45.1 1.0
19.6 1.0
50.81 0.9
Quant 390M
21.5
4366
56.3
Pythia 410M
21:21
60001
560M
54.44
57.94
QuanLM 560M
M 560M
54.24
QuamLM 560M
4634
5244
5884
QuanLL 560M
21.23
42.34
4584
590:
TriLM 560M
25.71
21.01: 12
45.1 1.0
50.211
57.31 0.9
Rinory 560M
24.4 1.2
20.2 1.1
41.9 1.0
47.8
61.51 08
Floot M 830M
2802 1.3
24.52 133
$1.62 1.0
57.34 1.0
61.01
QuanLLM 830M S-Bit
28.24 1.5
25.1
51.7 1.0
57.34 1.0
009 0.9
QuaniLM 830M 6-Biz
27.6
24.7
$1.6 1.0
57.7 1.0
61.3 0.
QuanLM 830M 4 Bix
27.61: 1.3
23.3 12
90.5 1.0
56.2 1.0
58.1 0.9
Quant M 830M 3-Rir
27.12 1.3
22.72 12
4681 1.0
$0.5 1.0
56.11 0.9
TALM X30M
25.34
22.5 12
483 1.0
54:24 1.0
60.4 0.
Pythia 1B
27.0
24 13
40.04 1.0
57.0% 1.0
008 0.
Floutl M 1.18
29.1 1.3
26.1 13
$40 1.0
60.4 1.0
62.9 08
QuamLM L.B Bit
2899
26.1 13
$4.1 1.0
0024 1.0
62.6 08
QuanTM 6-Bii
29.3
25.5
54.1 10
60:1 1.0
6291
QuantLM
5900
61.34
QuanLM
Trll M 1.1B
24.62
$6.1
Biniry 1.1B
$2:74:
Pythia
$4.0
FlostL M
56.4
62.64t
63.24
QuamLM 1.50
29.8
26.01: 13
$6.61: 1.0
62.44
63.34 08
QuamLM
30.14
563-
62.24
63.44
QuanLM L.5 4-Bit
29.41 1.3
26.91 13
$5.21 1.0
60.41 1.0
62.51 08
QuaniLM 1.58 3-Bit
27.8
25.24 13
49.7 1.0
5434 1.0
$3.7 0.
TriLM 1.5B
28.24 1.3
24.7 13
$3.1 1.0
59.0 1.0
54.14 0.9
FloutLM 2.48
32.74 1.4
30.1 13
60.5+: 1.0
63.54 1.0
62. 088
QuantL 2.48 9-Bit
32.64 1.4
30.+ 13
603 1.0
66.7 1.0
62. 08
Q 2.4B Bii
32.71: 1.4
3061 13
60411 1.0
65.41 1.0
62.0:1 0.8
Quant M 2.48 4-Bit
33.34
308 13
50.6 1.0
64.11 1.0
59.0 0.9
QuamLM 2.48 3-Bit
29.74 13
28.4 13
$4.24 1.0
58.44 1.0
55.74 0.9
TrI.M 2.4B
29.91
29.5: 13
58.01: 1.0
63.81 1.0
64.41: 0.8
FloatL.N M
34.6
63.0::
68.34
68.9% 08
QuanLM 3.90
34.61:
31.91:
63.0:: 1.0
68.111
63.111 08
Quant M 3.98 6-Bii
35.12
32.12 14
6333 1.0
63.02 1.0
65.64: 08
QuanLM 3.98 4-Bit
3474 1.4
3294 14
61.24 1.0
68.14 1.0
66.44 08
QuanLMM 3.98 3-Bit
$2.14 1.4
29.3 13
55.5 1.0
62.14 1.0
6004 09
TriLM 3.9B
35.31: 1.4
31.91: 144
608-1: 1.0
66.01 1.0
66.5.1 08
Table 7: Spectra Suite Performance (Part 1): Are Challenge, Are Easy, and BoolQ, Additionally, we also include scores on the Pythia LLM suite.
32

---

Published as a conference paper at ICLR 2025
Models
HellaSwag
WinsCiande
Avg. (HellaSwng, PIQA, WinoGrande, Fay, Are Challengo, and Baol())
Apc Norm.
Ape
Norm.
Pythia 70M
25. 0.4
25. 1:: 0.4
49.8 1.2
49.9 1.2
49.1 1.4
$4.9
MoztLM 99M
$1.6:t 0.5
29.1 0.5
62.8 1.1
63.24 1.1
50.2 1.4
44.3
Quaart M 99M 8-Rit
51.7# 0.5
20.0 0.5
62.6 1.1
63.0 1.1
500: 1.4
44.3
Q 99M Bit
31.7-4 0.5
29.24 0.5
6288 L1
63.14 1.1
5024 1.4
44.3
Quinl M 4-Bit
31.0:1:0 0.5
28.91 05
63.22 111
60.91: 1.1
5041: 144
42.6
QuasrL 3-Bit
29:244
21.74 04
57.25 1.2
58.24
49.24
TriLM 99M
044
60.1
60.4:
50.74
Rinary 99M
27.7
27.24
59.22
59.23
4881
30.8
Pythia 160M
25.14
25.0+ 0.4
53.1
53.1
47.3
35.7
FlouLM 190M
36.61: 0.5
31.41
65.622
64.21
5194
QuearL M. 190M 8-Rir
36.5:t
31.42 0.
65.6
6482
51.4
46.8
QuinLIM 190M Bit
36.3:4 0.5
31.54 0.5
65.6
64.34
5.
46.4
QuIILMM 190M 4-Bit
26.0+t
25.74 0.4
49.3
51.74
510+
36.5
QuanLMM 190M 3-Bit
32,0:1:0 0.5
28.8: 0.5
58.1
58.74
50
42.7
TriLM 190M
31.6+t 0.5
29.0 0.5
62.0= 111
62.3 111
517+ 1.4
42.4
Plouri M 390M
44:4::0.5 0.5
35.7 0.5
68.7 1.1
68.4 111
F185 1.4
48.5
QuantLM 390M Bit
44.5+t 0.5
35.7+ 0.5
68.8 L1
68.6 111
526+ 1.4
48.6
QuaaeLM 390M 6-Bit
4.5
35.6+t 0.5
09.0 L1
68.4 111
550+ 1.4
48.9
Q 390M 4-Bit
0.5
35.1:2
68.1 L1
68.3 I1
1.
47.7
Quaart M 390M 3-Rir
0.5
$2.9 0.5
632 1.1
63.2 11
530 14
46.5
TriLM 390M
0..9++
32.0+ 0.5
64.7 L1
65.0+ 111
522+
46.4
Pythix 410M
0.3
33.8:1 0.5
67.22 1.I
66.3 11
5351
48.6
HoztLM 560M
47.6:t
$7.75 0.5
64 L.I
09.0 111
14
50.5
Quand M 560M
47.6:1
31.7:1 0.5
68.9 L1I
68.91 111
13
50.1
Quaer M 560M 6-Bir
47.6:1 0.5
37.74 0.5
68.7 1.1
688 111
5355
50.1
QuasLLM 560M 4-Bit
46.7:
37.0th 05
618 L1
67.1 111
581
49.8
QuanLMM 560M
0..
33.41
60.52
63.21
4971
46.7
TrILM 560M
41.5: 0.5
33.82
67.2
67.5
48.4
Rinary 560M
36.4:
31.21
64.6
64.22
52X1
44.5
Flost 830M
0..
40.12
71.42
71.72
5641
QuaarLM M 850M S-Hit
51.4 0.5
40.1 05
71.2
71.7+
15
53.2
QuanLMM 830 Bit
0.5
40.2:1 05
71.32
71.81
5621
53.2
QuanLIM 830 4-Bit
50.2:: 0.5
39.211 0.5
70.6
7111 1.1
56011
52.2
QuanI M 830M 3-Bir
45.5::.0.5
35.92 0.5
66.1 111
66.6
53.5 14
49.2
TrLL 830M
46.0 0.5
36.8+ 0.5
68.2 L11
68.4 1.1
55.6 1.4
50.7
Pythic 18
4..24:0.5
37.74 0.5
09.3 1.1
70.8 1.1
532 1.4
51.1
PlostLM LJB
0..2
42.6:t 0.5
72.2 1.0
71.34 111
56.3
54.9
QuaneLM 1.1B Bit
55.24+
42.6 0.5
72.1 1.0
71.24 1.1
562 1.4
54.8
QuanI M 1.1B 6-i
0.59:1:0.5
42.6:2 0.5
71.92 1.0
71.21 111
16
55.0
Quaart M 1.18 4-Hi
54.9: 0.5
42.0:t 0.5
71.6 1.1
70.4 111
$48
54.4
QuanLM 1.1B Bit
51.3:1 0.5
39.4:1 0.5
69.1: L11
68.411 111
5481114
52.6
Tvll 1.
49:1:
38.8 0.5
60.8 1.1
69.11 111
14
51.6
Binary
43.4:
35.1 0.4
66.9 L1
68.3
5534
47.1
Pythix 1.4B
52.0:1 0.5
40.4: 0.5
70.8: L.I
70.6:1 1.1
17
543
Mo 1.5
57.8:
44.32 0.5
73.9 1.0
23.1
5944
56.7
QuanLMM
0.5
0.5
73.7
23.1
59.4
56.8
Quint M
57.5:1 0.5
0.
74.022
73.0
59.71
QuaarL 1.58 4-Bit
56.9:t
43.24
72.7
72
57.1
Quaart M 1,5B 3-Bi
53.7:t
41.0
70.0
09.4
550+
51.6
TriLM 1,50
53.14 0.5
40.9:1 0.5
70.1
20.3+
56.1
52.5
HoutLM 2.48
62.7: 0.5
43.1 0.5
75.2
74,94
6184
59.2
QuinLMM 2,4B B-Bit
62.711 0.5
47.1 0.5
75:42 1.0
74,9 1.0
6141
59.1
QuaarI AM 2,4B 6-Biz
62.9: 0.5
47.0 0.5
75.7 1.0
74.7 1.0
61.1 1.4
59.1
QuaarI AM 2,48 4-Biz
62.24 0.5
46.5 0.5
7544 1.0
74.5 1.0
61.7 1.4
58.5
QuantLM 2.4B B-B
58.6-t 0.5
43.5+ 0.5
72.7 L.
20.8+ 1.1
57.4 1.4
54.7
THi.M 2:4B
59:0:1: 0.5
45.3:1 0.5
72.6 1.0
71,42 1.1
59.7.1 1.4
57.3
PlostLM 398
66.11
49.74 0.5
75.8: 1.0
75.4
6284
61.4
QuanLM
49.7:t 0.5
15.9th
75.5
62.91
QuantT M
0.
78.52
75.62:
62:21
QuaarLM M 3:9B 4-Hir
68.0:t:
49.0:t:
75.5
25.6
6274
60.7
QuaILM 3:9B Dix
61.211 0.5
45.9:1 0.5
72.6:2 1.0
72.34:
5931 1.4
56.8
TriLM 3.9B
61.7:1 0.5
48.3:1 0.
74.6:
1.0
74.4::
62.1 1.4
60.7
Table 8: Spectra Suite Performance (Part 2): HellaSwag, PIQA, WinoGrande, and Average Scores (including Arc Easy, Arc Challenge, and BoolQ). Additionally, we include scores from the Pythia L.M suite.
33

---

Published as a conference paper at ICLR 2025
Chillenge Norm.
Easy
HellaSwag
PIJA
Avg
Norm.
Norm.
BirNet T00M
21.4
51.8
58.2
35.1
68.1
55.2
48.3
BitNet 1.3
24.2
$4.9
36.7
37.7
68.8
55.8
49.7
BiiNet 3B
28.3
61.4
61.5
42.9
71.5
59.3
54.2
BitNet 3,98
28.7
64.2
63.5
44.2
78.2
60.5
55.7
Table 9: Performance of BitNet b1.58 on ARC Challenge, ARC Easy, BoolQ. HellaSwag, PIQA, and Wino- Grande. The scores taken from (Ma et al. 2024).
D.3 TOXICITY
report toxicity-based evaluation in 15 Each considered a zero-shot setting.
The toxicity-based based evaluation included the following tasks:
BBQ (Parrish et al. 2022): The Bias Benchmark for QA (BBQ) dataset, comprises of questions developed by its authors, focusing on documented social biases directed towards individuals from protected classes across nine distinct social dimensions pertinent to U.S. English-speaking environments.
Crows Pairs (Nangia et al. 2020); proposed a challenging dataset aimed at quantifying stereotypical biases embedded within language models, with specific emphasis on U.S. contexts. Hosted on GitHub, this dataset serves as a crucial resource for assessing and addressing biases through paired sentences that illuminate societal stereotypes.
TruthfulQA (Lin et al.) 2021): A benchmark designed to evaluate the truthfulness of language models in generating responses questions. This benchmark includes 817 questions across 38 categories, such as health, law, finance, and politics.
D.4 PERPLEXITY ON OTHER DATASETS
TriLM vs FloalLM Perplextyy Across Othar Corpora
4.0
Froit t/3
Croes Erimpyy (Log Perpeally)
3.0
2.5
Openil Lambadia
Poe Benk
Counspeeda
Daina
$20rc
Inlepedia
RebnedWeb
Figure 18: Cross-entrops (log perplexity) comparison between TriLM and FloatLM (both 3.9B parameters) across various datasets apart from SlimPajama.
We measure perplexity using TriLM 3.9B and FloatL.M 3.9B across various other corpora than SlimPajama, which was used for training. These corpora include OpenAl Lambada, Penn Tree Bank, C4, Cosmopedia, Dolma, S20rc, Wikipedia, and RefinedWeb. A portion of Wikipedia, C4 is included in Slim Pajama. Some other corpora like Dolma and RefinedWeb, may also have overlaps from C4, Wikipedia well Common Crawl.
Figure 18 demonstrates that while TriLM 3.9B is similar or better than FloatLM 3.9B on PTB and Lambada, across the other datasets, with potential overlaps with SlimPajama, it's performance consistently worse indicating lower capability to memorize training data as well as worse in-distribution performance, despite competitive out of distribution performance.
34

---

Published as a conference paper at ICLR 2025
Models
LAMBADA
SeiQ
LogiQA
Perp.
Acc
Acc Norm.
Ace
Ace Norm.
Acc
26.53
73.61:
27.61
21.24:
QuanLMM 99M B-Bi
85.8= 7.0
26.6
62.8#
73.7 1.4
27.8+ 1.8
21.01
Quandl M 99M 6-Bir
89.9 74
26.1: 0.6
61.82 1.5
73.9 1.4
28.1
20.3: 1.6
Q 99M 4 Bit
211.6 17.3
16.7# 0.5
61.2# 1.5
70.7# 1.4
24.9 1.7
20.7: 1.6
QuancL M 99M 3-Bir
$765.42 413.0
4.5 0.3
$1.92 1.6
$7.02 1.6
25.34
19.8 1.6
TRLM 99M
172.0::
20.0::
60.4
67.61: 1.5
25.5 1.7
21.54 166
Binory
14:0=
27.04
22.34
FlosiLM 190M
50.3= 2.7
31.12
65.1 1.5
77.3# 1.3
27.24
22.1
Quand 190M 8-Bit
48.7
31.51: 0.6
65.5 1.5
77.1 1.3
27.0 1.7
22.31 1.6
QuanLM 190M 6-Bit
3..3=
30.0 0.6
64.2 1.5
71.0 1.3
26.1 1.7
22,12 1.6
Quanc MI 190M 4-Bit
72479077.3
0.00 0.0
25.6 1.4
22.92 1.3
23.34 1.7
20.7 1.6
QuanLM 190M 3-Ba
664.5= 41.1
12.4#
58.5#
66.42 1.5
26.3 1.7
21.0 1.6
TeLM 190M
130.7::
23.74
61.02
72.61
25.53
21.51
390M
21.9=009
42.22
75.62:
84.2 1:2
28.14
23
Quunt 390M 8-Bit
21.7=
42.3
75.7 1.4
84.1 1.2
28.3 1.8
24.11 1.7
QuantLM 390M 6-Bit
24.3= 1.0
40.6 0.7
75.5 1,4
83.7 1.2
27.6
23.2 1.7
Quani.M M 390M 4.Bit
30.2 1.3
39.1 0.7
77.1 1.3
84.13 1.2
25.8 1.7
23.34 1.7
QuanLLM 390M 3-Bit
115.0# 5.6
23.0=
67.4#
76.7# 1.3
25.7 1.7
21.8+ 1.6
TrLMM 390M
77.7 3.8
28.0%
68.63L
76.91: 1.3
26.44:
21.81
FlostLM 560M
20.8= 0.9
44.1
74.7 1.4
83.5 1.2
27.0
20.74
Quantl M 560M 8-Rit
20.9=
44.2: 0.7
74.7
83.61 1.2
27.34 1.7
20.7. 1.6
QuanLM 560M 6-Bit
21.7=
42.8 0.7
74.4 1.4
83.6 1.2
25.8 1.7
20.9 1.6
Quanel M 560M 4-Rit
24.9 1.1
40.8 0.7
73.6 1.4
82.02 1.2
27.0 1.7
21,71 1.6
QuanLM 560M 3-B
146.3:1
20.1
71.1
75.91 1.4
25.0::: 1.7
21.81 1.6
TrLMM 560M
2.7
78.72
19.8
Dinary
31.0%
26.7
M 830M
13.3 0.5
49.6 0.7
78.4 1.3
85.91 L.1
26.3 1.7
20.1 1.6
QuanLLM 830M 8-Bit
13.3=
49.4 0.7
78.5 1.3
86.1 1.1
26.61 1.7
20.0 1.6
Quantl . 830M 6-Bit
13.3= 0.5
49.1 0.7
77.8 1.3
85.41 1.1
26.31: 1.7
20.1 1.6
QuanLMM 830M 4-Ba
15.4=
47.3#
78.8#
85.1 L.1
25.5 1.7
21.24
QuantI 830M 3-Hit
47.7= 2.0
30.52
74.1
80.11 1.3
28.11
21.24
830M
FloutLM I.IB
11.7= 0.4
51.2
82.2:: 1.2
88.1 1.0
27.3 1.7
20,91
QuantL M 1.18 8-Bit
11.7004
$1.22 0.7
82.1 1.2
88.1 1.0
27.8
21.24 1.6
QuanLMM I.IB 6iB
11.7 0.4
$1.0 0.7
82.3 1.2
88.1 1.0
27.5 1.8
21.5 1.6
QuantLM 1.IB 4-Bit
13.9=
49.3
81.2
87.6 1.0
28.4
20.3+
QuaniLM I. 3-Bit
26.9 1.1
39.1
78.7
85.0 L.1
25.8 1.7
20.7.1 1.6
0.7
16.2
Rinary
33.4
37.62
1.3
28.41 1.7
23.24
FlosiLM 1.5B
9.4 0.3
$5.5 0.7
80.9:: 1.2
87.4 1.0
26.1 L.7
20.9:t 1.6
Quancl M 1.5B 8-Bit
9.5 03
55.52 0.7
81.3 1.2
81.51 1.0
25.7 1.7
20.64 1.6
QuanLLM 1.5B 6-Bit
9.5 0.3
55.4
81.42:
87.61: 1.0
25.71:
20.34
QuantL M 1.58
10.4:04
$3.02
20.31
QuanLM 1.5
0.7
28.4
TrILM 1,58
16.4= 0.7
46.2
80.7 1.2
87.3 1.1
27.8
21.5
FlosiLM 2.48
1.75 0.3
$9.32 0.7
87.22: 1.1
91.02 0.9
29.5
21.5 1.6
Quant M 2.4B 8-Bit
1.74 0.3
$9.23 0.7
87.1 1.1
91.01:: 0.9
29.5 1.8
21.5 1.6
QuantLM 2.4B 6-Bit
7.9 0.3
58.92
87.3
90.9: 0.9
29.61: 1.8
20.9 1.6
QuantI 2.4B 4-Bit
8.9
$6.12
84.8
89.71: 1.0
29.61 1.8
20.9:1
QuanLLM 2.4B
0.6 8.62
28.61::
TnLM 2.48
58.72
84.2::
88.7 1.0
28.61
24.3::
FlostLM 3.98
6.72
61.12
86.5
90.91: 0.9
26.9
20.9:
QuaniLM
61.12
0.9
26.6.1
QuanLLM 3.98 6-Bit
688
60.82
86.62 1.1
91.34 0.9
25.8 1.7
20.4::
Quantl 3.9 4-Bit
7.41: 0.2
58.5.1 0.7
86.1 1.1
90.8:1 0.9
28.611 1.8
20.1.1 1.6
QuantLM 3.9B B-Bit
14.0=
47.1 0.7
83.1 1.2
88.62 1.0
27.0 1.7
21.5 1.6
M 3,98
633 0.2
61.62 0.7
87.4 1.0
90.81 0.9
27.64 1.8
22.74: 1.6
Table 10: Spectra Suite Performance (Part 3): LAMBADA OpenAI, SciQ, LogiQA. We additionally also include Pythia's performance scores.
35

---

Published as a conference paper at ICLR 2025
D.5 OMNIQUANT: 3-BIT POST- TRAINING QUANTIZED MODELS
We present additional 3-bit quantized (QuantLM 3-bit) models, which were trained using OmniQuant (Shao et al. 2024) for iterations with group size of one row. It important to highlight that the
performance benchmarks for these models are consistent with those of GPT-Q. as reported in Tables
15 and 16. Therefore, these results do not affect our findings in the main paper.
Tasks
Metric
99M
190M
390M
560M
1.1B
ARC Challenge
Acc.
0.19 0.01
0.21 = 0.01
0.22 0.01
0.22 + 0.01
0.24 = 0.01
Acc. (Norm.)
ARC Easy
Acc.
0.37 0.01
0.41 1 0.01
0.45 0.01
0.48 1: 0.01
0.55 1: 0.01
(Norm.)
0.34 0.01
0.38 0.01
0.41 + 0.01
0.42 0.01
0.48 = 0.01
BoolQ
Acc.
0.55 = 0.01
0.54 = 0.01
0.62 = 0.01
0.57 # 0.01
0.63 = 0.01
HellaSwag
Acc.
0.28 0.00
0.29 0.00
0.32 0.00
0.34 1 0.00
0.38 1 0.00
Acc. (Norm.)
0.29 = 0.00
0.32 0.00
0.00
0.41 + 0.00
0.48 = 0.01
LAMBADA
Acc.
0.05 0.00
0.07 = 0.00
0.23 + 0.01
0.25 + 0.01
0.31 = 0.01
LogiQA
Acc.
0.16 = 0.01
0.02
0.23 0.02
0.23 + 0.02
0.21 = 0.02
Acc. (Norm.)
0.23 0.02
0.25 + 0.02
0.29 0.02
0.28 1 0.02
0.27 + 0.02
PIQA
0.58
0.65
0.69 +
Acc. (Norm.)
0.58 = 0.01
0.59 + 0.01
0.63 0.01
0.66 + 0.01
0.69 + 0.01
SciQ
Acc.
0.55 0.02
0.64 + 0.02
0.74 0.01
0.77 + 0.01
0.82 + 0.01
Acc. (Norm.)
0.49 0.02
0.02
0.02
0.68 +
0.01
TriviaQA
Exact Match
0.00 0.00
0.00 + 0.00
0.01 0.00
0.01 + 0.00
0.05 + 0.00
Winogrande
=
0.01
0.53
+ 0.01
MMLUU
Acc.
0.25 0.00
0.25 + 0.00
0.26 0.00
0.27 +: 0.00
0.29 + 0.00
Table 11: Model performance 3-bit post- post-training quantized models using OmniQuant across various tasks, from 99M to 1.IB parameters, evaluated on different metrics.
Tasks
Metric
1,5B
2.4B
ARC Challenge
Acc.
0.26 0.01
0.27 0.01
Acc. (Norm.)
0.29 + 0.01
0.30 0.01
Easy
Acc.
- 0.01
0.60 0.01
Acc. (Norm.)
0.51 + 0.01
0.53 1 0.01
BoolQ
0.01
0.63
HellaSwag
Aoc.
0.40 - 0.00
0.44 1 0.00
Acc. (Nomm.)
0.51 + 0.01
0.57 0.00
LAMBADA
Acc.
0.35 + 0.01
0.44 0.01
LogiQA
Acc.
0.23 - 0.02
0.19 0.02
(Norm.)
0.30
PIQA
Acc.
0.69 1: 0.01
0.71 1: 0.01
(Norm.)
0.70 0.01
0.72
SciQ
Acc.
0.82 + 0.01
0.86 + 0.01
Acc. (Norm.)
0.76 0.01
0.80 0.01
TriviaQA
Exact Match
0.07 - 0.00
0.07 0.00
Winogrande
Acc.
0.56 + 0.01
0.58 = 0.01
MMLU
Acc.
0.28 0.00
0.30 0.00
Table 12: Performance of 3-bit Post-Training Quantized Models using OmniQuant across Various Tasks, from 1.5B 2.4B parameters, across different tasks and metries.
36

---

Published as a conference paper at ICLR 2025
MMLU Stem Acc Across Size (Bits)
MMLU Stem Acc Across Paramelers
29
28
FlostLM
28
QuantLM
27
TrLMM
28
Aoo (og s08e)
28
RoaiLM
Quanti 4-13
QuantLM 3-Bit TriLM
23
10
20
30
40
50
80
22
560M
(10*)
MMLU Social Sciences Acc Across Size (Bits)
MMLU Social Sciences Acc Across Parameters
FloatLM
Quantt M S-Rt
34
34
TrLLM
Acc (0g (cale)
42
RloaILM
QuantLM Bit
QuariLM 3-Bil
T/LM
190M
560M Parameters
1.58
3.98
(Bits)
Humanities Parameters
30
FloatLM
QuantLM 4-Bit
QuantLM
29
Aoc 10g pcal))
28
FloelL
24
Quantt, M 4-Bit
TriLM
20
40
50
60
22
Size bits (1.0))
Paremetees
MMLU Other Acc Across Size (Bits)
MMLU Other Ace Across Parameters
FloatLM
QuaniLM 3-Bit
TriLM
35
Acc (log scale)
30
Quanti QuantLM 3-BI
TrILM
25
190M
660M
1.5B
3.98
(10*)
Parameters
Figure 19: Model performance on MMLU subsets: STEM, Humanities. Social Sciences, and others. Plot accuracy scores against model size in bits (left) and number of parameters (right), ranging from 560M to 3.9B parameters for TriLM (ternary), FloatL. (FP16), and Quantl M (3-bit & 4-bit).
37

---

Published as a conference paper at ICLR 2025
E WEIGHT DISTRIBUTION OF LINEAR LAYERS
FloatiM 90M
PlottEM 190M
FlnasIM 900M
175
17.5
17.5
15.0
15.0
15.0
12.5
12.5
10.0
50.0
7.
3.0
5.0
2.5
0.0
-0.4
-03
5.5
53
9.4
-0.4
-23
0.0
0.2
0.
-0.4
-03
5.0
02
0.4
Woightobu
We:ght
Vot
FloatLM 560M
ReatLM 830M
FluatUM 1.38
17.
15.0
22.5
20.0
10.0
7.3
7.5
0.0
-064
-0.2
0,0
22
0.4
-2.4
-522
0.0
0.2
0.
-2.4
-0.2
0.0
0.
0.4
RoatEM 1.58
FloatLM 2.48
FluatEM 3:08
30
-0.4
-2.2
2.0
0.4
-0.4
-22
0.0
0.2
0.
-0.4
-0.2
5.0
0.2
0.4
Weight
Figure 20: Weight distribution in the linear layers of FloatLM models across various model sizes, ranging from
99M to 3.9B parameters
The observed Gaussian distribution in the weights of our final trained FloatLM models across various scales is supported by both theoretical foundations and empirical evidence. We outline the rationale for the normality of weight distributions as follows:
Empirical Consistency: Across all model scales, ranging from 99 million 3.9 billion parameters, the weight distributions consistently exhibit Gaussian characteristics, as illustrated in Figure 20 This consistency is visually evident in distribution plots and quantitatively confirmed by fitting Gaussian functions to the weight histograms, demonstrating good fit across different model sizes.
Theoretical Underpinning: Neural network weight initialization typically follows Gaussian distribution to facilitate balanced learning dynamics. As training progresses, despite non-linear transformations and complex interactions within the network, the Central Limit Theorem suggests that the aggregation of numerous independent random variables (such updates during backpropa- gation) tends toward normal distribution. This tendency particularly pronounced given the high dimensionality and extensive data processing involved in training large language models.
Stabilization through Regularization and Optimization: Techniques such as I.2 regularization constrain weight magnitudes, encouraging them towards smaller values and contributing to a peak around zero-a characteristic feature of Gaussian distributions. Additionally, optimization algorithms like Adam, which adjust learning rates based on moving averages of recent gradients, promote smoother updates. This approach maintains the Gaussian form by mitigating the impact of outlier gradients.
Given these, we can assert that the weight distribution of our trained models closely follows a Gaussian distribution which is crucial for understanding the weight variance across different scales.
38

---

Published as a conference paper at ICLR 2025
F LEARNING DYNAMICS ANALYSIS OF PEAK LR DROP AND DECAY STAGE
In this section, we present a concise analysis our learning dynamics, inspired by Appendix D of MiniCPM (Hu al. 2024). Specifically, we examine the impact of our two key interventions: the peak learning rate reduction at the halfway point of training and the removal of weight decay regularization two-thirds of the training stage. This analysis explores loss dynamics through the lens of checkpoint updates and gradient behavior.
We analyze both the latent weights and ternary weights, calculating the maximum weight element update, max;y max;, (j ij), across all weight matrices in the TriLM-1.5B model for each training step. Our findings, consistent with observations from MiniCPM, reveal a strong correlation between weight updates and the learning rate decay, as depicted in Figure 21] Notably, after the weight decay regularization is removed at approximately 160,000 training iterations, the model checkpoints undergo smaller updates. However, surprisingly during this stage, despite smaller weight changes, the loss decreases at an accelerated pace.
Max Difference vs lttraaioo for attention. quary. key valuue weight
Max Difference vs Iteration for attention.query key value weight
carry www.wllt.wajf.
diinttn curry Joy waph,
61
Max Difference
0.03
6:-
50002
100900
15-2006
200002
290200
50032
300032 Peration Checkpoint
15-000
200003
290565
Iteration Checkpoint
Iateni weight updates for shard
(b) Ternry weight upshies for shard
Max Difference vs lteration atlention Bueettion.uuery....m. kee__ lue weight
Max Difference vs lberation for atlention query. key. weight
5.05
0.35
Difference
Max Difference
6.25
6.15
6.05
501
50002
10000S Ioeraioo
150000 Checkpoint
200000 250000
50002
100005
150020
200000
250300
ottoonn Chockpoint
(c) weight updates for
(d) Ternary weight updates shard
Figure 21: Max Difference Checkpoints for Layer 2
examine gradient data by training 190M parameter TriLM model. We record the gradients at every step to obtain fine-grained values for deriving second-order gradient information. In Figures 22 and 23] plot the maximum, Lo norm, first and second-order directional derivatives of the gradient, along with the cosine of the gradient angle and the loss curvature along the training trajectory.
Following the methodology of Hu et al. (2024),
While the L2 norm and maximum of the gradient do not provide notable insights, the loss curvature and second-order directional derivative show significant correlations with both the peak learning rate reduction at 5000 steps and the removal of weight decay at 7500 steps. Interestingly, we also observe an increase in the cosine of the gradients and decrease in the first-order directional derivative at both 5000 and 7500 steps, with trends becoming more pronounced after the removal of weight regularization. Moreover, the trends for loss curvature along the parameter trajectory and the second- order directional derivative are opposite for the latent and ternary gradient statistics. A more in dept understanding these trends in the learning dynamics under our current training strategy for TriLM offers promising direction for future exploration.
39

---

Published as a conference paper at ICLR 2025
Norm
Durvature Parameter Trajectory
Order Derivatiee
2.10
0:10
0.35
0.50
2.45
0.20
0.15
2.40
0.10
0.33
-0
020
2005
6032
0002
5002
10320
2003
4320
5003
0320
13200
2003
10:0
6320
0320
12000
Gradieet
Secord Directional Dedivalive
Gradient Angle
4.40
0.25
0.0022
2.
0.2
0.20
0.001
1.000
0.65
-26
-14
2002
RE00
T0C00
2302
0003
4220
8300
10000
2002
0002
2000
1000
Itcratiion
Iteration
Itaration
Figure 22: Latent gradient statistics over the training of 190M model. We drop the peaked lcarning rate at step 5000 and reduce the weight decay to at step 7500
Temary Gradient 12 Norm
Temany Less Curvature On Parameter Trajectory
Temary Frrt: Order Directional Derivattve
0.55
2.00275
3.00250
0.005
000022
0.003
3.00550
ottion
Itrratoo
Iteraton
Gradient
Socond Order Directional Darivative
Gradicnt
0.0023
2.8
3.5
0 0004
0.03
0003
200
0002
-2.
0002
Figure 23: Ternary gradient statistics over the training of 190M model. We drop the peaked learning step 5000 reduce weight decay to at step 7500
40

---

Published as a conference paper at ICLR 2025
G MEMORY BOTTLENECKS AND LOW-BITWIDTH LANGUAGE MODELLING
Recent observations (Gholami et al. 2024) suggest that, given the slower pace of improvements in memory and communication compared to compute (FLOPs), the bottleneck continues shift away from computation towards the memory-related characteristics of hardware for deploying large language models. This shift underscores the importance of exploring solutions that directly address memory constraints, Below, we formally analyze this trend and the impact of low- bitwiitttthh language models on addressing memory bottlenecks during inference.
G.1 OVERVIEW RECENT DATACENTER GPUs AND ACCELERATORS.
We begin our analysis by surveying a broad range of recent datacenter General Purpose GPUs (GPG- PUs) employed for neural network development and research since 2018. This includes hardware from multiple providers, covering various configurations across the latest microarchitectures.
From Nvidia, we consider the following:
Volta: V100 (SXM/PCIe) (Nvidia Team 2018).
Ampere: A100 (40GB/80GB SXM/PCIe) (Nvidia Team 2020).
Hopper: H100 (SXM/PCIe) and H200 (Nvidia Team 2022 2023).
Blackwell: This includes preliminary data for Blackwell microarchitectures, which at the time of access were subject change (Nvidia Team
2024).
From AMD, we analyze the following models:
MI200 Series: MI210, MI250, MI250X (AMD Team 2022a b),
MI100 Series: MI300A, MI300X, MI325X (AMD Team 2023a
2024
Additionally, we include hardware from Intel and Google:
From Intel, the Gaudi Series: Gaudi 2 and Gaudi IIntll Gaudi Team 2024).
From Google, the Tensor Processing Units (TPUs): TPUv3 (Google TPU Team 2018). TPUv4 (Google TPU Team 2021), and TPUv5 (TPUvSe, TPUv5p) (Google TPU Team
2023a b).
All data sourced from the respective datasheets, technical documentation, or press releases of the cited hardware. Over the past several years, each of these four accelerator families has improved in three areas FLOPS, memory capacity, and bandwidth.
G.2 MEMORY TRENDS AND SPEEDUP OPPORTUNITIES LOW-BITWIDTH LANGUAGE MODELING
Trand of sing Memory/FLOPS
Trand Decreasing Bytos/FLOPS
000100
0.35
0.30
0.25
0.006
0.20
0.004
0.15
1030
4500
0.002
0.000
2018
2019
2020
2021
2022
2023
2024
2025
2010
2019
2020
2021
2022
2023
2014
2025
(a) Memory Capacity vs Peak FLOPS
(b) Peak Memory Bandwidth vs Peak FLOPS
Figure 24: Trends of Memory/FLOP Bandwidth/FLOP across different (datacenter) GPGPUs.
41

---

Published as a conference paper at ICLR 2025
Memory Capacity and Bandwidth of GPGPUs Relative to Peak TFLOPs. In Figure 24a show the trends of Memory Capacity over Peak TFLOPS (Half Precision FP16/BF16) for various accelerators over the years. We also perform linear fit for each family of accelerators separately. The linear fit for all the families has a downward slope, showing that memory capacity is improving slower pace than computation capability. This trend holds true even for the most recent hardware, such as Blackwell, MI325X, and Gaudi3. Though we consider Half-Precision TFLOPs, the slope is expected to become steeper when considering peak TFLOPS over Ampere sparse or FP8. Similarly, in Figure 24b we present the trends of Memory Bandwidth (specifically for DRAM or its equivalent memory) over FLOPs for the accelerators over the years, along with the linear fit for each family. We observe downward slope here as well, indicating the trend that memory bandwidth is growing much slower than computation.
Comparison of model size and maximum speed up for quantized models: In this analysis, we include transformer configurations from the LLaMa family (Touvron al.) 2023a b). As larger vocabularies in LLMs are becoming increasingly common for efficient multilingual modeling, use vocabulary size of 128k from LLaMa (AT@ Meta 2024) for our analysis. We assume the Embedding and LM Head weights are retained in Pref-Precisoon across all bitwidths. In Figure 25 we provide the scaling plots of Mss QuantLMs and TriLMs in terms of model size (in GB) and maximum speedup compared the FloatLM baseline.
Models Size Growth across BilWidths
Maximum Speedup across
10
RoatLM
---- QuentLM 4-8
QuantLM 4-Btt
400G3
TrLL
8
Tri.M!
39203
8008
2
138
342
1826
1428
Parameters [Lng Scale)
(Log
(a) Model Size growth at various BitWisths
(b) Maximum Possible Speedup at different BitWidths
Figure 25: Expected gains from low bitwidth modeling. TriLMs over 300B parameters single H100 and achieve up a theoretical maximum of faster autoregressive decoding compared to FloatLM.
42

---

Published as a conference paper at ICLR 2025
H ABLATIONS
Table a4d 13 shows the performance ablation 100B token training runs over the six commonsense benchmarks from BitNet b1.58 at I.IB parameters. The first rows show the performance of TriLM I.IB and Float I.IB at this token count, followed by our replication of BitNet b1.58 (Ours) as well as the scores from BitNet b1.58 over 700M and 1.3B parameters. We observe that at this scale, TriLM does not come close to matching the performance of FloatLM, but it outperforms much larger BitNets. The next two rows show the performance of TriLM 1.IB and FloatL 1.IB when trained on 100B tokens of FineWeht instead of SlimPajama. While the performance of both the models improves on Fihe the average difference in their performance across datasets remains the same. Lastly, show the performances across various optimization schedules. A significant drop in averaged performance is noticed when the baseline schedule of linear decay with constant weight decay is used. The gains from dropping 12 regularization in the schedule are more than that of dropping the peak learning rate, however, not enough to match that of TriLM 1.1B's ' schedule.
HellaSwag
PIQA
FloaLLM 1.IB
50.0 1 0.5
39.3 1: 0.5
70,9 +:: 1.0
70.1 :: L.0
55A 1: 1.4
TrLM 1.10
46.8 1 0.5
37.1 1: 0.4
69.4 +:: 1.0
69.4 L.0
$3.8 1: 1.4
BitNec (Ours)
BiNNer 61.58 700M
61.58
TriLM 1.10 Fin
50.0 :: 0.5
39.2 1: 0.4
70.2 :: 1.0
70.1 1.0
$6.6 1: 1.3
FlossLM 1.IB FineWeb
52.7 + 0.5
41.1 + 0.4
73.0 + 1.0
71.3 1.0
$6.7 à: 1.3
TrLM Only LK Drogped
+:
+
TrLLM Only Reg. Drupped
47.1
68.6
$5.2
TrILM Schedale
46.0:1
Table 13: Ablation Common Sense Task Performance: HellaSwag, PIQA, WinoGrande, Are Easy, Are Challenge, BoolQ (Contd.). BitNet b1.58's scores from Maet al. (222). All runs for 100B tokens on Slim Pajama, except those explicitly stated as FineWeb
Models
Are Challenge
Are Fasy
BoolQ
Aug. HellaSang, PIQA, WinoGrande
Ace
Acc
Ace
Acc
Aec
Arc Easy, Are Challenge, and Bool()
FesiLM
26.3
20.5
503 1.0
36.8
60.6
$2.2
THLM 1.18
26.7 +
493 + 1.0
55.0
54.9 +
50.2
BitNer b1.58 1.IB (Ours)
26.1 + 12
23.6 1: 12
47.7 +: 1.0
55.3 1:: 1.0
49.7 :1: 0.8
48.9
RitNer bl. 18 700M
21.4
$1.8
58.2
48.3
BitNer 61.58 1.38
24.2
54.9
56.7
49.5
TriLM 1.ID Wen
31.7
31
63.1
66.8
58.3
$4.9
FlessLMM M FineWeh
344
65.7 +
702 09
59.3 +
56.9
TriLM 1.1 Only Peak LR Drogped
27.4 d: 1.
23.6 d: 12
483 4: 1.0
55.1 +: 1.0
$1.6 4:
49.7
Tri. 1B Only 1.2 Reg. Dropee
27.611 1.3
24.8 1.2
40.3 + 1.0
55.1 1.0
53.1 4:08
$0.1
TiLM 1.18 Baseline Schedule
262 1.2
23.2 122
48.0
$4. 1.0
49.4 +
49.1
Table 14: Ablation Common Sense Task Performance: HellaSwag, PIQA, WinoGrande, Arc Fasy, Arc Challenge. BoolQ. BitNet b1.58's scores from/Ma et al. (2024). All runs are for 100B tokens on Slim Pajama, except those explicitly stated as Fib
43

---

Published as a conference paper at ICLR 2025
Models
TriviaQA Exact Match
CrowsPairs
Big
TrutffuQQA
Likelihood dill.
Pet sterevlppe
Ace
Acc
M 99M
0.62 0.1
372.4 14.6
55.42:
30.81 0.4
24.1 1.5
Quand 99M 8-Bit
0.62
570.9: 14.6
55.1
26.51:
24.1 1.5
Quant 6-Hit
0.62
148
56.91
26.71
24.24
Q
22.94
QuantLM
TriLM 99M
0.1:
362.4: 10.8
$4.24 1.2
30.8
24.24 1.5
Bisary 99M
0.2: 0.0
353.54 11.1
53.32 1.2
31.5
25.7+ 1.5
Floarl 190M
0.62:
348.21 11.3
$5.9
27.34
22.4 1.5
Q
Q
QuantMM 190M 4-Bit
0.:
961.9: 25.4
43.8
35.0
1.5
QuanLLM 190M 3-Bit
0.1:0
482.74 15.2
53.74 1.2
0.
25.04 1.5
TALM 190M
0.2:
343.5% 10.9
55.53 1.2
29.71: 0.4
23.94 1.5
Floarl 390M
0
355.5 10.4
99.6
25.41
22.4 1.5
QuanLM
QuansM
QuantLM 390M 4-Bit
1.34 0.1
368.24 10.2
59.4
03
22.8 1.5
QuantLM 390M 3-Bit
0,8 0.1
444.41 12.2
$4.34 1.2
2.
23.0H 1.5
TrLM 390M
1.3d 0.1
344.5% 10.3
58.3: 1.2
26.9 0.3
24.4 1.5
35181
QuanLM
4.73
Quant .M 560M 6-8it
352 0.1
353.71: 9.9
59.3
0.
15
QuantLM 560M 4-Bit
2.12
372.74 10.7
59.2
27.0 0.4
22.24 155
QuantLM 560M 3-Bit
1.5 0.1
411.24 11.3
$7.9 122
29.0 0.4
22.4 1.5
TrL. 560M
2.4
345.11 10.1
58.71 1.2
2.
23.6 1.5
Bisary 560M
0.2: 0.0
356.341 10.4
58.51 1.2
26.3
23.1 1.4
354.6::
62.61t
Quant .M 830M 8-Bit
8.5
9.6
62.14
25.6
230 15
QuantLM 830M 6-Bit
8.5
354.6::
62.74
03
15
QuantLM 830M 4-Bit
10.5: 0.2
364.24:: 9.8
59.9 1.2
25.91
21.8 1.4
Quant 830M 3-Bit
3.13
389.54 10.9
59.9 1.2
30.5 0.4
24.4 1.5
TriLM 830M
4.3 0.2
341.91 10.0
60.74 122
0.
22 1.5
FloarIM
12.92
9.7
61.24
QuantLM 1.18 8-Bit
12.72 0.2
61.12
03
21.74 14
QuantLM 1.IB 6-Bit
12.4 0.2
349.7 9.6
59.9 122
25.5
21.94 1.4
QuantLM 1.1B 4-Bit
9.3: 0.2
359.1: 10.1
60.9 122
2..4
21.34 1.4
QuanLLM 1.1B 3-Bit
6.8: 0.2
422.4:: 11.5
$8.74 1.2
29.9 0.4
21.2 1.5
TrLM 1.I
1.9 0.1
343.11 9.9
61.42 122
25.81 0.3
21.54 1.4
Rinary 1.18
2.24 0.1
381. 9.8
$8.34 1.2
26.4 0.3
23.24 1.4
HootLM 1.5B
12:2: 0.2
351.9 351.99:99 9.6
61.62
1.4
QunntLM 1.5B B-Bit
12.5 0.2
352.4::: 9.6
61.64 1.2
26.8 0.3
21.8 1.4
QuantLM 1.5B 6-Bit
11.3s 0.2
350.94: 9.7
61.9 1.2
27.1
21.54 1.4
QuanLLM 1.5B 4-Bit
9.02 0.2
357.9:: 9.8
60.72 1.2
23.9::03
20.8 14
QuantLM 1.5B 3-Bit
4.2: 0.1
400.0:: 10.6
60.9: 1.2
06.82:0.3
20.8 14
THi. 1,5B
5.95: 0.1
348.94: 9.9
$9.9 1.2
25.24 0.3
21.74 1.4
MouuLM 2.4B
20.7:t
360.41: 9.4
64.21 1.2
26.74 0.3
21.74 1.4
QuantLM B-Bit
20.7: 0.3
360.5 9.4
64.21 122
26.5
21.94 1.4
QuaniLM 2:4B 6-Bit
20.4:
360.8::
63/4: 1.2
26.4+:
2184 1.4
QuaniLM 2-1B 4-Bit
21.1:2 0.3
358.74
63/12 1.2
26.0 0.3
21.74 1.4
Quandi M 2.48 3-Bir
10.9 0.2
360.2.1: 9.5
$9.94 1.2
25.81: 0.3
21.54 1.4
Tril W.I.M.. 2.48
12.32 0.1
353.04: 10.0
64.1 1.2
25
23.0 1.5
FlouLM 3.9B
21.5a:
359.211: 9.6
64.74 1.2
25.411
23.64 1.5
QuanLL M 3.98 8-Bit
21.72 0.3
359.8:1:10 9.6
64.62 1.2
25.44:
23.64 1.5
QuanLM 3.98 6-Bit
21.0= 0.3
359.5:::96
63.9: 1.2
23.4
23.54 1.5
QuanLLM 3.9B 4-Bit
17.9 0.3
365.5
61.81 1.2
25.34 0.3
21.24 1.5
Quanei M 3.9R 3-Bir
8.24: 0.2
365.94: 9.8
64.34 1.2
25.54 0.3
21.94 1.4
THi. M. 3.98
21.32 0.3
362.44: 9.6
65.4 1.2
25.94
24.14
Table 15: Spectra Suite Performance (Part TriviaQA, CrowsPairs, Big Bench BBQ Lite, TruthQA. We additionally also include Pythia's performance scores.
44

---

Published as a conference paper at ICLR 2025
Models
MMLU Accuracy
Stem
Humanities
Social Seiences
Other
Avg.
Ploar M 99M
22.8 0.7
24.0: 0.6
27.0:t 0.8
28.01: D.
25.3d: 0.4
QuanLL 99M 8-Bit
22.91 0.7
24.24 0.6
26.9: 0.8
27.91 0.8
25.34 0.4
Quand 99M 6-Rit
22.7. 0.7
24.1: 0.6
26.6:t 0.8
28.24
25.24 04
QuantLM 99M 3-Bin
23.5
23.94 0.6
26.24 0.8
25.9:
0.4
TriLM 99M
23.91: 0.8
23.6.1 0.6
26.74 0.8
26.6.1: 0.8
25.0:1 0.4
Binary 99M
21.64t: 0.7
24.3 0.6
21.82t 0.7
24.0: 0.7
23.1
FloaLLM 190M
24.0
24.4 0.6
0.8
30.1
26.5+ 0.4
QuanLM 190M 8-Bir
24.1 0.8
24.5: 0.6
28.91 0.8
30.01 088
26.6:t 044
Quantl 190M 6-Bin
24.1:: 0.8
24.5: 0.6
28.31 0.8
29.8: 088
2644 04
QuaniLM 190M 4-Bit
22.9:t: 0.7
22.9: 0.6
24.5:t
23.4:: 0.8
23.4: 0.4
QuantL M 190M 3-8in
23.9 0.8
23.24:
25.4
27.54 D.8
24.84 0,4
TriLM 190M
22.5. 0.7
23.8.1 0.6
26.7.1 0.8
28.41:
25.24l 0.4
FloaLLM 390M
25.8.1: 0.8
25.9. 0.6
30.3. 0.8
32.81 0.8
28.33 0.4
QuaniLM 390M 8-Bit
25.7 0.8
25.9::0.6
30.2 0.8
32.42: 0.8
28.2# 0.4
QuantLM M 390M 6-Bit
26.0d: 0.8
25.8: 0.6
30.24 0.8
32.34
28.33 0.4
Quanti 4-Bit
0.6
0.8
TrILM
24.1
0.6
0.8
29.01
26 0.4
HoaiLM 560M
26.71
30.53 0.8
32.34
QuantLM
0.6
QuantLM 560M 6-Bit
24.6
2.674
0.8
31.34
28.1 044
QuantL 560M 4-Bit
24.7 0.8
25.9::
29.91 0.8
31.11 0.8
27.71: 0.4
QuaniLM 560M 3-Bit
24.5 0.8
24.2: 0.6
28.14
28.24 0.8
26.0: 0.4
Tril M 500M
25.0:t: 0.8
28.12: 0.6
29.0: 0.8
30.2 D88
27.0: 0.4
Binary 560M
21.4:t 0.7
24.2: 0.6
21.6:t: 0.7
23.91
22.9 0.3
HeatLM M 830M
23.8:: 0.8
27.3+
32.34 0.8
34.6 88
29.7 0.4
M 830M B-Bit
25.8.1: 0.8
27.4:1 0.6
32.11 0.8
34.74: 0.8
29.7 0.4
QuaniLM 830M 6-Bit
25.6 0.8
2.63
32.12
34.24 0.8
29.5 0.4
QuantLM 830M 4.Bit
25.9:1:
26.8:1 0.6
31.24 0.8
33.61
29.1: 0.4
25.1:t
28.9:
0.6
M 1.IB
26.41: 0.8
27.6:1 0.6
32.51 0.8
34.81 0.8
30.0:t 0.4
QuantLM 8-8it
26.241
27.4:
32.53 0.8
34,91
QuanLM
QuantLM 4-8it
26.0t:
26.6
32.44: 0.8
33.84
29.3+ 0.4
QuantLM I.IB 3-Bit
25.9:1 0.8
26.1:: 0.6
30.0+1 0.8
33.01 0.8
28.4: 0.4
TriLM 1.BB
25.24: 0.8
26.1 0.6
30.6+t 0.8
32.24 0.8
28.3 0.4
Binary 1.1B
21.0+t 0.7
24.2 0.6
21.74 0.8
24.41 0.7
23.0+t 0.3
FlouLM
26.1::
28.0:t 0.7
33.0:t 0.8
35.61 0.8
30.4: 0.4
Quant M 1,58 8-Hit
26.1 0.8
28.14
32.91 0.8
$5.54 D8
30.3: 044
M 1.5B 6-
26.3 0.8
28.0:t: 0.7
33.0:1: 0.8
35.41: 0.8
30.4:t 0.4
QuantLM M 1,58 4-Bit
26.24: 0.8
28.12:
32.44 0.8
3484 0.8
30.1 0.4
Q 1,5B 3-Bit
25.5 0.8
26.74 0.6
31.21 0.8
33.41
28.9.1 0.4
TriLM
34.61
FloatLM M 2:4B
26.91: 0.8
29.4: 0.7
34.24 0.8
38.1
31.8: 0.4
Quant 2.4B 8-Bit
27.0::
29.41
34.13 0.8
38.05
31.81 0.4
2.4B
0.7
QuantM 2.4B
QuantL 2,48 3-Bit
25.5+t:
27.14
0.8
36.44
0.4
THiLM 2.4B
27.411 0.8
27.8.1 0.6
34.6.1 0.9
35.1.1: 0.8
30.8 0.4
QuaniLM 3.98 8-Bit
27.64t
30.74 0.7
37.0+ 0.9
39.74
33.44 0.4
Quant 3.98 6-Bit
27.34 0.8
30.3:1 0.7
36.9 0.9
39.34 0.9
33.1 04
QuaniLM 3.9B 4-Bit
27.11: 0.8
30.3 0.7
36.3 0.9
38.84: 0.9
32.8: 0.4
Quanst M 3.98 3-Hir
27.34 0.8
28.44: 0.7
34.34 0.9
37.24 0.9
31 11.44:10 0.4
TrILM 3.9B
28.3 0.8
29.5:t 0.7
35.41: 0.9
39.611:
32.8:t: 0.4
Table 16: Spectra Suite Performance (Part 5): MMLU- STEM, Humanities, Social Sciences, Others.
45

---

Published as a conference paper at ICLR 2025
1 ILLUSTRATIVE EXAMPLES OF TRILM 3.9B's COMPLETION CAPABILITIES
showease instances of outputs produced by TriLM (3.9B) across diverse tasks, highlighting its proficiency in tasks such as comprehension, prompt completion, and creative composition.
Generated Output on Reading Comprehension by TriLM (3.9B)
The Blitz Background: From the German point of view, March 1941 saw an im- provement. The Luftwaffe flew 4,000 sorties that month, including 12 major and three heavy attacks. The electronic war intensified but the Luftwaffe flew major inland missions only moonlit nights. Ports were easier find and made better targets. To confuse the British, radio silence was observed until the bombs fell. X- and Y-Gerät heams were placed over false targets and switched only the last minute. Rapid frequency changes were introduced for X- X-Gerät, whose wider band of frequencies and greater tactical flexibility ensured it remained effective time when British selective jamming was degrading the effectiveness of Y-Gerät.
Title:
Q: How many sorties were flown in March 1941?
A: 4,000
Q: When did the Luftwaffe fly inland missions?
A: Only during moonlit nights
Title: Helsinki
Background: Helsinki the capital and largest city of Finland. Is in the region of Uusimaa, in southern Finland, on the shore of the Gulf of Finland. Helsinki has population of an urban population of, and a metropolitan population of over million, making the most populous municipality and urban area in Finland. Helsinki is some north of Tallinn, Estonia, east of Stockholm, Sweden, and west of Saint Petersburg, Russia. Helsinki has close historical connections with these three cities. The Helsinki metropolitan of Helsinki, Espoo, Vantaa, Kauniainen, and surrounding commuter towns. It is the world's northernmost metro of over one million people, and the city is the northernmost capital of an EU member state. The Helsinki metropolitan area is the third largest metropolitan area in the Nordic countries after Stockholm and Copenhagen, and the City of Helsinki is the third largest after Stockholm and Oslo. Helsinki is Finland's 's major political, educational, financial, cultural, and research center as well as one of northern Europe's 's major cities. Approximately 75% of foreign companies that operate in Finland have settled in the Helsinki region. The nearby municipality of Vantaa is the location of Helsinki Airport, with frequent service to various destinations in Europe and Asia.
includes the urban
Q: What is the most populous municipality in Finland?
A: Helsinki
Q: How many people live there?
A: 1.4 million the metropolitan area
Q: What percent of the foreign companies that operate in Finland are in Helsinki?
A: 75%
Q: What towns are a part of the metropolitan area?
A: Espoo, Vantaa, Kauniainen, and surrounding commuter towns
46

---

Published as a conference paper at ICLR 2025
Auto-complete on a creative writing task by TriLM (3.9B)
Essay Writing
In the year 2150, humanity had established colonies on Mars. Captain Ramirez was on routine mission to explore an uncharted region of the planet when he and his crew were attacked by Martian natives who wanted them dead for their technology. The captain's ship crashed into the surface of Mars where it exploded in flames.
The survivors managed to escape from the wreckage but they soon found themselves stranded with way out or back home. They would have survive alone until rescue came. But what there nothing waiting for them? What if they are trapped forever?
Poem Writing
a garden where the roses bloom, Underneath the silver moon, gentle breeze begins to sway, And whispers secrets of the day.
The stars above start to appear, While shadows dance and disappear, The night unfolds in its own time, The sun rises over the horizon,
As it does every morning,
But this is ordinary dawn. It's a new beginning for all who are here today. For those who have come before us,
And those yet to be born.
We celebrate life with music and laughter,
With friends old and new.
This our moment to shine!
Auto-complete various prompts by TriLM (3.9B)
Making cake: Several cake pops shown display. A woman and girl are shown making the cake pops in a kitchen. They use a mixer to mix ingredients, then knead dough by hand. The women roll out the dough with rolling pin, cut it into shapes using cookie cutters, bake them, and decorate them with frosting or sprinkles.
The energy given up by electrons as they move through the electron transport chain is used to produce ATP. In this process, oxygen and water are produced reaction called cellular respiration.
When talking about sensitive topics, is crucial to be aware of the fact that there many different types of people. Some will react negatively and others positively. It's important not to take offense when someone says something you don't like.
someone disagrees with you, is best to ignore them. The most important thing in life not what we have but who we are and how we treat others.
47

---

Published as a conference paper at ICLR 2025
J BROADER IMPACT
Interpretability Beyond Neuron Level: While several efforts have been made to understand how language models work and the means to steer them without training, these methods have mostly focussed on intervening at the neuron level. TriLMs open a new degree of interpretability at the connection level. Here, the connections between any two neurons in a layer are in one of the three states 0 (no connection), -1 (negative connection) and +1 (positive connection), each with equal strength. This is in sharp contrast to FloatL where these connections can be of varying strengths, making it harder to study interpretability beyond the neuron level. By releasing the checkpoints across our training runs, we facilitate research along these directions.
Environmental Benefits and Resource Efficiency: The open release of our models mitigates future emissions by allowing others to bypass the need for pretraining models from seratch. Moreover, TriLMs need much fewer resources during deployment and can perform the autoregressive generation at a faster pace making them critical to scenarios demanding striet latency. Additionally, TriL.Ms represent substantial advancement in enhancing performance on resource-constrained edge devices, including smartphones, laptops, and automobiles.
Impact on Specialised Hardware: While TriLMs offer significant memory reduction and latency improvements on General Purpose GPUs like H100 and RTX4090, certain specialized hardware benefits more from ternrry modeling. Hardware (like Cerabras 3 that supports high byte-to-flop ratio computations, can leverage the sparsity stemming from ternarization for speedup in both training as well as inference. On the other hand, hardware with limited Memory/SRAM (like Grod benefits from the reduction in the number of chips needed to deploy LLM.
Reduced Training Costs: The Chinchilla scaling laws established that for training compute optimality, may be recommended train larger LLMs for fewer tokens than smaller LLMs for more tokens to achieve the desired model performance. However, memory requirements and latency associated with deployment of larger models, have motivated costlier training runs that go far beyond Chinchilla optimality. For example, LLaMa model with only 8B parameters was trained for 15T tokens. Since TriLM and ternary models, in general, can reduce the memory requirements and latency, this can motivate shift inparameter-tole tradeoff for efficient training runs towards Chinchilla's compute-optimal regime.
The open suite of TriLM, FloatLM, and QuantLM families aims to empower researchers to explore the nuanced impacts of precision levels on model performance and efficiency, thereby catalyzing ongoing advancements in the development and deployment of language models, as well as enhancing their interpretability and safety Lij (2024). By providing a range of publicly accessible models trained on openly available data, the suite offers unprecedented transparency in the training process. Intermediate checkpoints are available for all models, accompanied by detailed documentation of training procedures and hyperparameters.
Advancing Research through Open Access:
"hups://groq.com/
48


Reviews:
[{"review_id": "zHNlrR3Glr", "reviewer": "Reviewer_gRaf", "summary": "This paper addressed the limitation of post-training quantization by investigating the pretraining of low-bitwidth models specifically Ternary Language Models as an alternative to traditional floating-point models and their post-training quantized versions (QuantLMs). The authors conducted extensive experiments to evaluate the model's performance in different aspects. The experiment results and analysis revealed valuable insights into the feasibility and scalability of low-bandwidth language models.", "questions": "1. Could the authors clarify what the \"validation loss\" in Figure.7 means? Is it \"Log Perplexity\" just as in the y-axis in Figure.19 in Appendix D.4? I am curious about the model's generation performance on commonly used datasets such as WikiText-2, C4, PTB.\n2. Some fonts in the line chart are difficult to recognize, such as legends in Figure 6(a). In addition, many figures seem incorrectly scaled, making the label of the x/y-axis twisted. Authors should check all figures to improve the clarity.\n3. This paper only provides the maximal speed-up compared with FP16. Authors are recommended to benchmark the end-to-end inference performance, such as throughput, first token time, and average latency.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Authors comprehensively evaluate the model's commonsense reasoning performance, knowledge capacity, and toxicity. The experiment part is sufficient and convincing.\n2. This work is well presented and it offers valuable insights into the pertaining of low-bitwidth models.", "weaknesses": "1. The pretraining cost of QuantLMs is not revealed (e.g., hardware, GPU hours).\n2. The paper does not discuss how the hyperparameters are selected and tuned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ebWR0XURqQ", "reviewer": "Reviewer_fojc", "summary": "In this paper, the authors present the Spectra LLM suite, i.e a suite of ternary (1.5-bit) language models (TriLLMs), ranging from 99M to 3.9B parameters. \n\nSpecifically, the authors first extensively pretrain TriLLMs in various size and the corresponding LLM in FP16 (FloatLLM) with exactly the same recipe, and study their differences. The findings are quite fruitful:\n1. **Convergence behavior**: TriLLM in various scales can converge normally. With some techniques, like decreasing peak learning rate and removing weight decay at some points, they can converge faster.\n2. **Scaling law**: Overall, TriLLM demonstrate a similar scaling law as FloatLLM. With a similar model size in GB, TriLLM achives lower perplexity than FloatLLM. With a similar parameter count, TriLLM is worse than FloatLLM.\n3. **Benchmark accuracy**: The authors compare TriLLM to FloatLLM and the quantized version of FloatLLM with GPTQ (QuantLLM), the experimental results show: (1) With a similar model size, TriLLM outperforms FloatLLM and QuantLLM <= 4-bit; (2) With a similar parameter count, TriLLM is worse than FloatLLM and QuantLLM-4bit, but outperforms QuantLLM-3bit.", "questions": "1. In Table 5, why isn't the number of skipped tokens proportional to the number of skipped batch?\n2. In Figure 6b, what's the motivation for the comparison between TriLLM 2.4B and FloatLLM 1.1B and 1.5B, instead of FloatLLM 2.4B? I think it's better to compare with a similar number of paramters or with a similar model size.\n3. Could you also offer some few-shot results on some benchmarks, like MMLU and Lambda? It would be interesting to see the in-context learning ability of TriLLM.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written, and the structure of the paper is clear.\n2. The experiments are extensive and the settings are fair, with all claims being well supported. \n3. The findings are interesting. And I believe that the release of all models (including TriLLM and FloatLLM) will further advance the study of TriLLM.", "weaknesses": "1. The only weakness is the applied quantization method in this paper, i.e. GPTQ. GPTQ performs well for bit-level >= 4-bit. However, with lower bit-level, its performance degrades significantly. It would be interesting to see the comparison with more advanced post-training quantization methods, like OmniQuant [1], BiLLM [2], AffineQaunt [3] and so on. \n\n\n[1] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models\n[2] AffineQuant: Affine Transformation Quantization for Large Language Models\n[3] BiLLM: Pushing the Limit of Post-Training Quantization for LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JRmxTS3iWZ", "reviewer": "Reviewer_Z2nY", "summary": "This paper presents an extensive study on the scaling law of low-bit models, specifically 4-bit, 3-bit, and ternary models. Ternary models (TriLMs) demonstrate impressive effectiveness when scaling model size, with the authors concluding that TriLMs achieve higher performance gains more rapidly during scaling compared to FloatLMs and QuantLMs. A 3.9B TriLM model exhibits performance similar to a 3.9B FloatLM but has a size smaller than that of an 830M FloatLM. The paper first introduces the theoretical advantages of scaling low-bit LMs using information theory. Following this, the authors propose the Spectra LLM suite, which facilitates training experiments on models of varying sizes and bit widths. Finally, scaling experiments conducted on the Spectra LLM platform reveal that scaling a TriLM is more efficient than scaling QuantLMs or FloatLMs. Detailed experimental results are provided, covering the training dynamics of low-bit LMs, pretraining outcomes, and evaluations on downstream tasks. The experiments show that, when scaled to 3.9B, the TriLM achieves comparable performance to the original FloatLM on most aspects, except in areas like toxicity and stereotyping. Further studies on BiLMs are also included in the appendix, where BiLMs show scaling effectiveness, although the performance gap reduction between BiLMs and FloatLMs is slower than with TriLMs.", "questions": "1. Section 2.2 suggests that low-bit models should theoretically provide a better approach for capturing weight variance. In the experiments, TriLMs—where each parameter is restricted to {-1, 0, 1}—demonstrate better scalability than 4-bit models, which in turn scale better than floating-point models. However, in Section 5, the results show that 3-bit models exhibit lower scalability compared to 4-bit models and TriLMs. What might be the underlying reason for this? Additionally, why could TriLMs be more scalable than BiLMs?\n\n2. Based on the discussion in Section 2.2, there may be a relationship between the number of states in each numeric representation and scaling efficiency. What might this relationship look like? Could there be an optimal number representation? This is intended as a discussion question, and a concrete answer is not required during the rebuttal phase. Personally, I believe this could be an interesting direction for future research.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper provides an exceptional perspective on the scaling law of low-bit models, with a particular focus on Ternary Models (TriLMs). The strengths of this study are summarized below:\n\n1. Extensive and thorough workload. This paper comprehensively addresses major concerns regarding the scaling law of low-bit models, including training dynamics, loss curves during pretraining, and evaluation results on downstream tasks. The authors conducted numerous model training sessions and extensive evaluations to support their conclusions. Additionally, the paper provides well-articulated theoretical explanations for why low-bit models scale more effectively and examines how these models align with modern GPU architectures. The paper is inclusive and well-scaled in its coverage of relevant aspects.\n\n2. High novelty and significant contribution. To the best of my knowledge, research on the scaling law of low-bit models is limited, making this work a valuable contribution with well-supported, insightful conclusions. By demonstrating the efficiency of scaling low-bit models, this paper encourages more effective model designs for scaling.\n\n3. Valuable software contribution. As described in the paper, the Spectra LLM Suite serves as a platform for studying the scaling law of low-bit models. This platform is beneficial for the research community, facilitating future work on quantization and low-bit model training.", "weaknesses": "Although this paper provides valuable contributions, there are some areas for improvement. The overall score could be raised if the following issues are fixed and good discussion is made during the rebuttal phase.\n\n1. Presentation needs refinement. While the figures in this paper offer essential information, some are poorly displayed. For instance, Figures 6 and 13 appear compressed, with text that is has low readability. Figure 5 is also confusing due to the similar colors used for arrows. A detailed explanation should guide readers through each part of Figure 5. Additionally, presenting Table 1 alongside Figure 5 could clarify the calculation details.\n\n2. Content organization for conciseness. To fit within the constraints of a 10-page conference paper, the paper could better prioritize content. Since the main contribution is the scaling law of TriLMs versus FloatLMs, related content should be given more prominence. Sections 2.1 and 2.3 are to some extent redundant and could be moved to the Appendix. Section 4.3 could be split into separate sections on scaling law and training dynamics, with each elaborated more thoroughly. Moving parts of Appendix C and findings in Appendix A.5 to the main text would improve the discussion of the corresponding aspect. Moving some BiLM discussions to the main text would also help clarify the outline.\n\n3. Deeper analysis of training dynamics. The paper notes a loss reduction halfway through training but could benefit from further examination of this phenomenon, as it may relate to efficient convergence and scaling. A detailed discussion of changes in the model at this stage would be valuable. Incorporating experiments similar to Appendix D \"Analysis of the decay stage\" of the MiniCPM paper [1] (COLM 2024 official version) might be helpful to explain this effect. Additionally, a deeper exploration of why low-bit models scale more efficiently than FloatLMs in terms of training dynamics would be insightful. \n\n[1] MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (COLM 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ta25rZvTRC", "reviewer": "Reviewer_AzmB", "summary": "The author developed a series of LLMs with parameters ranging from 99M to 3.9B. These models include a 16-bit floating-point model and a ternary model, both trained from scratch using 300B tokens. Additionally, the author created 3, 4, 6, and 8-bit PTQ variants by applying GPTQ to quantize the floating-point model. Detailed evaluations and comparisons were conducted for each model. The results indicate that the ternary variant outperforms others at the same bit size, and the 3.9B parameter ternary model performs comparably to the 3.9B floating-point model on several evaluation datasets.", "questions": "1. Can the author provide the actual inference speedup numbers for the quantized model compared to the FP model?\n\n2. Train the FP models with more tokens and compare them with the Ternary models.\n\n3. For the PTQ quantized model comparison versus QAT, can the author compare with QAT variant or use more recent PTQ methods such as SpinQuant/QuaRot?\n\n4. What is the training cost of full QAT training compared to the FP baseline? The paper mentions model parallelism at scale. Why is it even necessary, given that the largest model is just 3.9B parameters?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The author released a family of a fp and quantized model variant, as well as all the train details/loss and training techniques. Further more, the paper offers detailed evaluations results. Together, it provide good reference for the commnity for lower bit models.", "weaknesses": "The paper extensively discusses the theoretical or maximum possible speedup of the Ternary model, but it lacks results on actual inference speed. It would be beneficial to test the model in a real kernel environment, as many factors could influence speedup, such as activation quantization, KV cache, and kernel implementations.\n\n*FP vs Ternary*: Both models were trained with 300B tokens, and the training loss does not appear to have fully converged yet. It remains to be seen how the models compare once both have converged, and the FP model may require more tokens due to its larger model capacity.\n\n*Ternary vs Quant 3/4-bit Variant*: The comparison seems unfair, as the Ternary model is trained from scratch with QAT, while the Quant models are derived from GPTQ. The conclusion in the paper may not hold when using QAT for 3/4 bit variant.\n\nThe novelty is also limited. All components, such as the Ternary network, QAT with STE training, GPTQ, and model architecture, are well-known. The released pretrained models do not have much practical use, as there is a significant performance gap compared to state-of-the-art models of the same size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqDnHGzyEj", "reviewer": "Reviewer_DqjN", "summary": "This paper investigates the scaling law of low bit-width models, specifically ternary language models (TriLMs). The authors present Spectra LLM, an open suite for the quantization-aware training of LLMs. They conduct detailed analysis about TriLM and FloatLM in terms of reasoning, knowledge and toxicity. 3.9B parameter TriLM matches the performance of the FloatLM 3.9B across various benchmarks.", "questions": "See Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The authors provide detailed results of LLMs in various sizes and bits.\n2) The authors conduct extensive evaluations in terms of reasoning, knowledge and toxicity.", "weaknesses": "The novelty of the proposed work is unclear. The architecture and training approach of TriLM appear quite similar to BitNet b1.58 [1], including methods such as two-stage weight decay and learning rate scheduling. Previous work [1] has already shown that a 3B ternary LLM can match half-precision LLMs with similar parameter counts and training costs. What additional contributions does this paper offer beyond BitNet b1.58?\n\n[1] Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., ... & Wei, F. (2024). The era of 1-bit llms: All large language models are in 1.58 bits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)