Surprising Effectiveness of pretraining Ternary  Language Model at Scale

Authors: Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, Irina Rish
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, low-bit language models, quantization-aware training, pretraining of large language models, and scaling laws

Abstract:
Rapid advancements in GPU computational power has outpaced memory capacity and bandwidth growth, creating bottlenecks in Large Language Model (LLM) inference. Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but it suffers from significant performance degradation below 4-bit precision. This paper addresses these challenges by investigating the pretraining of low-bitwidth models specifically Ternary Language Models (TriLMs) as an alternative to traditional floating-point models (FloatLMs) and their post-training quantized versions (QuantLMs). We present Spectra LLM suite, the first open suite of LLMs spanning multiple bit-widths, including FloatLMs, QuantLMs, and TriLMs, ranging from 99M to 3.9B parameters trained on 300B tokens. Our comprehensive evaluation demonstrates that TriLMs offer superior scaling behavior in terms of model size (in bits). Surprisingly, at scales exceeding one billion parameters, TriLMs consistently outperform their QuantLM and FloatLM counterparts for a given bit size across various benchmarks. Notably, the 3.9B parameter TriLM matches the performance of the FloatLM 3.9B across all benchmarks, despite having fewer bits than FloatLM 830M. Overall, this research provides valuable insights into the feasibility and scalability of low-bitwidth language models, paving the way for the development of more efficient LLMs.

Decision: accept (spotlight)
Reviewer scores: 8, 10, 10, 5, 5

Reviews and discussion:
<thread>
creation date:
1730279032136

thread_type:
official_review

<thread_object>
creator: reviewer_dqjn

<content>
summary:
This paper investigates the scaling law of low bit-width models, specifically ternary language models (TriLMs). The authors present Spectra LLM, an open suite for the quantization-aware training of LLMs. They conduct detailed analysis about TriLM and FloatLM in terms of reasoning, knowledge and toxicity. 3.9B parameter TriLM matches the performance of the FloatLM 3.9B across various benchmarks.

strengths:
1) The authors provide detailed results of LLMs in various sizes and bits.
2) The authors conduct extensive evaluations in terms of reasoning, knowledge and toxicity.

weaknesses:
The novelty of the proposed work is unclear. The architecture and training approach of TriLM appear quite similar to BitNet b1.58 [1], including methods such as two-stage weight decay and learning rate scheduling. Previous work [1] has already shown that a 3B ternary LLM can match half-precision LLMs with similar parameter counts and training costs. What additional contributions does this paper offer beyond BitNet b1.58?

[1] Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., ... & Wei, F. (2024). The era of 1-bit llms: All large language models are in 1.58 bits.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730451551322

thread_type:
official_review

<thread_object>
creator: reviewer_azmb

<content>
summary:
The author developed a series of LLMs with parameters ranging from 99M to 3.9B. These models include a 16-bit floating-point model and a ternary model, both trained from scratch using 300B tokens. Additionally, the author created 3, 4, 6, and 8-bit PTQ variants by applying GPTQ to quantize the floating-point model. Detailed evaluations and comparisons were conducted for each model. The results indicate that the ternary variant outperforms others at the same bit size, and the 3.9B parameter ternary model performs comparably to the 3.9B floating-point model on several evaluation datasets.

strengths:
The author released a family of a fp and quantized model variant, as well as all the train details/loss and training techniques. Further more, the paper offers detailed evaluations results. Together, it provide good reference for the commnity for lower bit models.

weaknesses:
The paper extensively discusses the theoretical or maximum possible speedup of the Ternary model, but it lacks results on actual inference speed. It would be beneficial to test the model in a real kernel environment, as many factors could influence speedup, such as activation quantization, KV cache, and kernel implementations.

*FP vs Ternary*: Both models were trained with 300B tokens, and the training loss does not appear to have fully converged yet. It remains to be seen how the models compare once both have converged, and the FP model may require more tokens due to its larger model capacity.

*Ternary vs Quant 3/4-bit Variant*: The comparison seems unfair, as the Ternary model is trained from scratch with QAT, while the Quant models are derived from GPTQ. The conclusion in the paper may not hold when using QAT for 3/4 bit variant.

The novelty is also limited. All components, such as the Ternary network, QAT with STE training, GPTQ, and model architecture, are well-known. The released pretrained models do not have much practical use, as there is a significant performance gap compared to state-of-the-art models of the same size.

questions:
1. Can the author provide the actual inference speedup numbers for the quantized model compared to the FP model?

2. Train the FP models with more tokens and compare them with the Ternary models.

3. For the PTQ quantized model comparison versus QAT, can the author compare with QAT variant or use more recent PTQ methods such as SpinQuant/QuaRot?

4. What is the training cost of full QAT training compared to the FP baseline? The paper mentions model parallelism at scale. Why is it even necessary, given that the largest model is just 3.9B parameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730559921811

thread_type:
official_review

<thread_object>
creator: reviewer_z2ny

<content>
summary:
This paper presents an extensive study on the scaling law of low-bit models, specifically 4-bit, 3-bit, and ternary models. Ternary models (TriLMs) demonstrate impressive effectiveness when scaling model size, with the authors concluding that TriLMs achieve higher performance gains more rapidly during scaling compared to FloatLMs and QuantLMs. A 3.9B TriLM model exhibits performance similar to a 3.9B FloatLM but has a size smaller than that of an 830M FloatLM. The paper first introduces the theoretical advantages of scaling low-bit LMs using information theory. Following this, the authors propose the Spectra LLM suite, which facilitates training experiments on models of varying sizes and bit widths. Finally, scaling experiments conducted on the Spectra LLM platform reveal that scaling a TriLM is more efficient than scaling QuantLMs or FloatLMs. Detailed experimental results are provided, covering the training dynamics of low-bit LMs, pretraining outcomes, and evaluations on downstream tasks. The experiments show that, when scaled to 3.9B, the TriLM achieves comparable performance to the original FloatLM on most aspects, except in areas like toxicity and stereotyping. Further studies on BiLMs are also included in the appendix, where BiLMs show scaling effectiveness, although the performance gap reduction between BiLMs and FloatLMs is slower than with TriLMs.

strengths:
This paper provides an exceptional perspective on the scaling law of low-bit models, with a particular focus on Ternary Models (TriLMs). The strengths of this study are summarized below:

1. Extensive and thorough workload. This paper comprehensively addresses major concerns regarding the scaling law of low-bit models, including training dynamics, loss curves during pretraining, and evaluation results on downstream tasks. The authors conducted numerous model training sessions and extensive evaluations to support their conclusions. Additionally, the paper provides well-articulated theoretical explanations for why low-bit models scale more effectively and examines how these models align with modern GPU architectures. The paper is inclusive and well-scaled in its coverage of relevant aspects.

2. High novelty and significant contribution. To the best of my knowledge, research on the scaling law of low-bit models is limited, making this work a valuable contribution with well-supported, insightful conclusions. By demonstrating the efficiency of scaling low-bit models, this paper encourages more effective model designs for scaling.

3. Valuable software contribution. As described in the paper, the Spectra LLM Suite serves as a platform for studying the scaling law of low-bit models. This platform is beneficial for the research community, facilitating future work on quantization and low-bit model training.

weaknesses:
Although this paper provides valuable contributions, there are some areas for improvement. The overall score could be raised if the following issues are fixed and good discussion is made during the rebuttal phase.

1. Presentation needs refinement. While the figures in this paper offer essential information, some are poorly displayed. For instance, Figures 6 and 13 appear compressed, with text that is has low readability. Figure 5 is also confusing due to the similar colors used for arrows. A detailed explanation should guide readers through each part of Figure 5. Additionally, presenting Table 1 alongside Figure 5 could clarify the calculation details.

2. Content organization for conciseness. To fit within the constraints of a 10-page conference paper, the paper could better prioritize content. Since the main contribution is the scaling law of TriLMs versus FloatLMs, related content should be given more prominence. Sections 2.1 and 2.3 are to some extent redundant and could be moved to the Appendix. Section 4.3 could be split into separate sections on scaling law and training dynamics, with each elaborated more thoroughly. Moving parts of Appendix C and findings in Appendix A.5 to the main text would improve the discussion of the corresponding aspect. Moving some BiLM discussions to the main text would also help clarify the outline.

3. Deeper analysis of training dynamics. The paper notes a loss reduction halfway through training but could benefit from further examination of this phenomenon, as it may relate to efficient convergence and scaling. A detailed discussion of changes in the model at this stage would be valuable. Incorporating experiments similar to Appendix D "Analysis of the decay stage" of the MiniCPM paper [1] (COLM 2024 official version) might be helpful to explain this effect. Additionally, a deeper exploration of why low-bit models scale more efficiently than FloatLMs in terms of training dynamics would be insightful. 

[1] MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (COLM 2024)

questions:
1. Section 2.2 suggests that low-bit models should theoretically provide a better approach for capturing weight variance. In the experiments, TriLMs—where each parameter is restricted to {-1, 0, 1}—demonstrate better scalability than 4-bit models, which in turn scale better than floating-point models. However, in Section 5, the results show that 3-bit models exhibit lower scalability compared to 4-bit models and TriLMs. What might be the underlying reason for this? Additionally, why could TriLMs be more scalable than BiLMs?

2. Based on the discussion in Section 2.2, there may be a relationship between the number of states in each numeric representation and scaling efficiency. What might this relationship look like? Could there be an optimal number representation? This is intended as a discussion question, and a concrete answer is not required during the rebuttal phase. Personally, I believe this could be an interesting direction for future research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565500430

thread_type:
official_review

<thread_object>
creator: reviewer_fojc

<content>
summary:
In this paper, the authors present the Spectra LLM suite, i.e a suite of ternary (1.5-bit) language models (TriLLMs), ranging from 99M to 3.9B parameters. 

Specifically, the authors first extensively pretrain TriLLMs in various size and the corresponding LLM in FP16 (FloatLLM) with exactly the same recipe, and study their differences. The findings are quite fruitful:
1. **Convergence behavior**: TriLLM in various scales can converge normally. With some techniques, like decreasing peak learning rate and removing weight decay at some points, they can converge faster.
2. **Scaling law**: Overall, TriLLM demonstrate a similar scaling law as FloatLLM. With a similar model size in GB, TriLLM achives lower perplexity than FloatLLM. With a similar parameter count, TriLLM is worse than FloatLLM.
3. **Benchmark accuracy**: The authors compare TriLLM to FloatLLM and the quantized version of FloatLLM with GPTQ (QuantLLM), the experimental results show: (1) With a similar model size, TriLLM outperforms FloatLLM and QuantLLM <= 4-bit; (2) With a similar parameter count, TriLLM is worse than FloatLLM and QuantLLM-4bit, but outperforms QuantLLM-3bit.

strengths:
1. The paper is well-written, and the structure of the paper is clear.
2. The experiments are extensive and the settings are fair, with all claims being well supported. 
3. The findings are interesting. And I believe that the release of all models (including TriLLM and FloatLLM) will further advance the study of TriLLM.

weaknesses:
1. The only weakness is the applied quantization method in this paper, i.e. GPTQ. GPTQ performs well for bit-level >= 4-bit. However, with lower bit-level, its performance degrades significantly. It would be interesting to see the comparison with more advanced post-training quantization methods, like OmniQuant [1], BiLLM [2], AffineQaunt [3] and so on. 


[1] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
[2] AffineQuant: Affine Transformation Quantization for Large Language Models
[3] BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

questions:
1. In Table 5, why isn't the number of skipped tokens proportional to the number of skipped batch?
2. In Figure 6b, what's the motivation for the comparison between TriLLM 2.4B and FloatLLM 1.1B and 1.5B, instead of FloatLLM 2.4B? I think it's better to compare with a similar number of paramters or with a similar model size.
3. Could you also offer some few-shot results on some benchmarks, like MMLU and Lambda? It would be interesting to see the in-context learning ability of TriLLM.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567204754

thread_type:
official_review

<thread_object>
creator: reviewer_graf

<content>
summary:
This paper addressed the limitation of post-training quantization by investigating the pretraining of low-bitwidth models specifically Ternary Language Models as an alternative to traditional floating-point models and their post-training quantized versions (QuantLMs). The authors conducted extensive experiments to evaluate the model's performance in different aspects. The experiment results and analysis revealed valuable insights into the feasibility and scalability of low-bandwidth language models.

strengths:
1. Authors comprehensively evaluate the model's commonsense reasoning performance, knowledge capacity, and toxicity. The experiment part is sufficient and convincing.
2. This work is well presented and it offers valuable insights into the pertaining of low-bitwidth models.

weaknesses:
1. The pretraining cost of QuantLMs is not revealed (e.g., hardware, GPU hours).
2. The paper does not discuss how the hyperparameters are selected and tuned.

questions:
1. Could the authors clarify what the "validation loss" in Figure.7 means? Is it "Log Perplexity" just as in the y-axis in Figure.19 in Appendix D.4? I am curious about the model's generation performance on commonly used datasets such as WikiText-2, C4, PTB.
2. Some fonts in the line chart are difficult to recognize, such as legends in Figure 6(a). In addition, many figures seem incorrectly scaled, making the label of the x/y-axis twisted. Authors should check all figures to improve the clarity.
3. This paper only provides the maximal speed-up compared with FP16. Authors are recommended to benchmark the end-to-end inference performance, such as throughput, first token time, and average latency.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732043177831

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comments on inference efficiency for all the reviewers

comment:
We gratefully acknowledge the constructive feedback provided by reviewers gRaf02 and AzmB. In response, we present additional details regarding the inference speedup achieved by our ternary models through the use of llama.cpp kernels [1], an open-source project specifically optimized for efficient inference.

## Packing Schemes for Ternary Weights
Below, we provide details on the packing of our weights. 

### TQ1 Packing: 1.6875 Bits per Weight

The packing structure used in TQ1 is as follows:  
- Packing: The first 240 elements are packed with 5 elements per byte, and the remaining 16 elements with 4 elements per byte.
- Block structure: Each block of 256 ternary values is packed into 52 bytes, consisting of 48 bytes for 240 elements and 4 bytes for the remaining 16 elements, along with a float16 scale per block. This results in a 1.6875 bpw representation.
- Ternary value mapping: The ternary values are stored as unsigned integers, where the values {-1, 0, 1} are mapped to {0, 1, 2}, respectively.  

### TQ2 Packing : 2.0625 Bits per Weight


The TQ2 format was developed as a faster alternative to TQ1, particularly for compute-bound hardware.  
- Packing: Each byte contains 4 ternary values, with 2 bits used to represent each value.  
- Block structure: Each block consists of 256 ternary values, which are packed into 64 bytes, with a float16 scale per block. This results in a 2.0625 bpw representation. 
- Ternary value mapping: Same as TQ1.

It is crucial to highlight that the choice of 256 elements is deliberate, as all of our models are designed with row sizes that are multiples of 256, ensuring compatibility and optimal performance.


## Experimental Results

The performance of TQ1 and TQ2 was evaluated in terms of throughput and latency. **The results indicate that TQ2 is significantly faster than other quantization formats.**   
A comparison of the throughput of various quantization formats is presented in the table below:

| Configuration         | Model Size      | Apple M3 Pro (GB/s)       | Intel Core m3-8100Y (AVX2) (GB/s) |
|-----------------------|-----------|---------------------------|------------------------------------|
| **f16**               | 7.99 GB  | 93.47                     | 30.60                              |
| **q8**              | 4.24 GB  | 131.86                    | 67.03                              |
| **q4**              | 2.42 GB  | 135.33                    | 64.17                              |
| **TQ1**             | 994 MB   | 124.70                    | 70.31                              |
| **TQ2**             | 1.17 GB  | **176.43**                   | **141.83**                             |

A latency comparison on the MacBook M3 Pro across various configurations is shown in the table below:
| **Configuration**  | **Total Time (ms)** | **Tokens Processed** | **Time per Token (ms)** | **Tokens per Second** |
|--------------------|---------------------|----------------------|-------------------------|-----------------------|
| **F16**            | 12574.22           | 260                  | 48.44                   | 20.64                 |
| **q8**           | 4660.77            | 260                  | 17.69                   | 56.53                 |
| **q4**           | 4927.69            | 261                  | 18.88                   | 52.97                 |
| **TQ1**           | 7022.70            | 260                  | 27.01                   | 37.02                 |
| **TQ2**            | **3990.42**           | **260**                  | **15.27**                   | **65.47**                 |


Throughput is measured for the quantized vector operations in llama.cpp, providing a comparison across different quantization types. q8 and q4 refer to the 8-bit and 4-bit quantized formats, respectively. Additionally, it should be noted that the TQ1 kernels are not fully optimized for Metal.

The primary focus of our paper is to explore the scalability and feasibility of pre-training extreme-low bit models, such as binary and ternary LLMs, and to compare their performance with FloatLMs. To demonstrate actual speedup numbers, we will include these results in the appendix. While the CPU benchmarking results are presented here, we acknowledge that there is room for further optimization of both CPU and GPU kernel implementations. This optimization remains an important area for future work.

[1] Gerganov, G. et al (2023). llama.cpp: Port of Facebook’s LLaMA model in C++. Retrieved from https://github.com/ggerganov/llama.cpp.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732514552749

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments on Benchmarking 3-bit Post-Training Quantized Models with OmniQuant

comment:
We provide additional 3-bit quantized models, which were trained using Omniquant for 5 iterations with a group size of 1 row.

# Model Performance Table 

| Tasks                | Metric      | 99M     | 190M    | 390M    | 560M    | 1.1B   | 1.5B   | 2.4B   |
|----------------------|-------------|---------|---------|---------|---------|--------|--------|--------|
| arc_challenge        | acc         | 0.19 ± 0.01 | 0.21 ± 0.01 | 0.22 ± 0.01 | 0.22 ± 0.01 | 0.24 ± 0.01 | 0.26 ± 0.01 | 0.27 ± 0.01 |
|                      | acc_norm    | 0.23 ± 0.01 | 0.24 ± 0.01 | 0.26 ± 0.01 | 0.24 ± 0.01 | 0.26 ± 0.01 | 0.29 ± 0.01 | 0.30 ± 0.01 |
| arc_easy             | acc         | 0.37 ± 0.01 | 0.41 ± 0.01 | 0.45 ± 0.01 | 0.48 ± 0.01 | 0.55 ± 0.01 | 0.56 ± 0.01 | 0.60 ± 0.01 |
|                      | acc_norm    | 0.34 ± 0.01 | 0.38 ± 0.01 | 0.41 ± 0.01 | 0.42 ± 0.01 | 0.48 ± 0.01 | 0.51 ± 0.01 | 0.53 ± 0.01 |
| boolq                | acc         | 0.55 ± 0.01 | 0.54 ± 0.01 | 0.62 ± 0.01 | 0.57 ± 0.01 | 0.63 ± 0.01 | 0.59 ± 0.01 | 0.63 ± 0.01 |
| hellaswag            | acc         | 0.28 ± 0.00 | 0.29 ± 0.00 | 0.32 ± 0.00 | 0.34 ± 0.00 | 0.38 ± 0.00 | 0.40 ± 0.00 | 0.44 ± 0.00 |
|                      | acc_norm    | 0.29 ± 0.00 | 0.32 ± 0.00 | 0.39 ± 0.00 | 0.41 ± 0.00 | 0.48 ± 0.01 | 0.51 ± 0.01 | 0.57 ± 0.00 |
| lambada_openai       | acc         | 0.05 ± 0.00 | 0.07 ± 0.00 | 0.23 ± 0.01 | 0.25 ± 0.01 | 0.31 ± 0.01 | 0.35 ± 0.01 | 0.44 ± 0.01 |
| logiqa               | acc         | 0.16 ± 0.01 | 0.20 ± 0.02 | 0.23 ± 0.02 | 0.23 ± 0.02 | 0.21 ± 0.02 | 0.23 ± 0.02 | 0.19 ± 0.02 |
|                      | acc_norm    | 0.23 ± 0.02 | 0.25 ± 0.02 | 0.29 ± 0.02 | 0.28 ± 0.02 | 0.27 ± 0.02 | 0.26 ± 0.02 | 0.30 ± 0.02 |
| piqa                 | acc         | 0.58 ± 0.01 | 0.59 ± 0.01 | 0.63 ± 0.01 | 0.65 ± 0.01 | 0.69 ± 0.01 | 0.69 ± 0.01 | 0.71 ± 0.01 |
|                      | acc_norm    | 0.58 ± 0.01 | 0.59 ± 0.01 | 0.63 ± 0.01 | 0.66 ± 0.01 | 0.69 ± 0.01 | 0.70 ± 0.01 | 0.72 ± 0.01 |
| sciq                 | acc         | 0.55 ± 0.02 | 0.64 ± 0.02 | 0.74 ± 0.01 | 0.77 ± 0.01 | 0.82 ± 0.01 | 0.82 ± 0.01 | 0.86 ± 0.01 |
|                      | acc_norm    | 0.49 ± 0.02 | 0.59 ± 0.02 | 0.61 ± 0.02 | 0.68 ± 0.01 | 0.73 ± 0.01 | 0.76 ± 0.01 | 0.80 ± 0.01 |
| triviaqa             | exact_match | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.01 ± 0.00 | 0.01 ± 0.00 | 0.05 ± 0.00 | 0.07 ± 0.00 | 0.07 ± 0.00 |
| winogrande           | acc         | 0.51 ± 0.01 | 0.50 ± 0.01 | 0.51 ± 0.01 | 0.53 ± 0.01 | 0.56 ± 0.01 | 0.56 ± 0.01 | 0.58 ± 0.01 |
| mmlu (continuation)  | acc         | 0.25 ± 0.00 | 0.25 ± 0.00 | 0.26 ± 0.00 | 0.27 ± 0.00 | 0.29 ± 0.00 | 0.28 ± 0.00 | 0.30 ± 0.00 |
| - humanities         | acc         | 0.24 ± 0.01 | 0.23 ± 0.01 | 0.24 ± 0.01 | 0.25 ± 0.01 | 0.26 ± 0.01 | 0.26 ± 0.01 | 0.27 ± 0.01 |
| - other              | acc         | 0.27 ± 0.01 | 0.27 ± 0.01 | 0.29 ± 0.01 | 0.29 ± 0.01 | 0.33 ± 0.01 | 0.32 ± 0.01 | 0.36 ± 0.01 |
| - social sciences    | acc         | 0.26 ± 0.01 | 0.27 ± 0.01 | 0.29 ± 0.01 | 0.29 ± 0.01 | 0.32 ± 0.01 | 0.30 ± 0.01 | 0.33 ± 0.01 |
| - stem               | acc         | 0.22 ± 0.01 | 0.23 ± 0.01 | 0.25 ± 0.01 | 0.24 ± 0.01 | 0.25 ± 0.01 | 0.25 ± 0.01 | 0.27 ± 0.01 |

It is important to note that the benchmarks are comparable to GPT-Q (values provided as percentages in Tables 6, 7, 9, 14, and 15 of the main paper) and, therefore, do not affect our findings. Furthermore, we followed best practices for GPT-Q [1]

Reference:
[1] Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., & Richtárik, P. (2024). PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression. arXiv preprint arXiv:2405.1485216.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732582615888

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Changes to the manuscript

comment:
We have made the following revisions to the manuscript based on the reviewer’s feedback:

- Figure 4 has been added, along with the forward pass equation for all models, to enhance clarity.
- Section 2 has been streamlined for improved conciseness.
- The section on training dynamics and scaling laws has been separated, with the training dynamics section expanded for further detail.
- Two new sections have been added to Appendix A: “Details on the Selection of Hyperparameters” and “Perplexity for Increasing Training Tokens in TriLM Models”.
- Figures 2, 3, 5, 6, and 12 have been updated to correct scaling issues and ensure consistent color usage.
- Appendix F has been added, presenting an analysis of the "learning dynamics during the peak learning rate drop and decay stage".
- Appendix H has been added, presenting inference benchmarks titled “Latency and Throughput Analysis of Ternary Quantization Formats”.
- An additional QuantLM variant using OmniQuant has been added as Appendix D.5.
- Added a table of contents for the appendix section.
</content>
</thread_object>

</thread>


Metareview:
metareview:
TriLMs offers a more efficient alternative to traditional floating-point models and post-training quantized models in terms of scaling behavior and performance at larger model sizes. The paper presents a thorough evaluation of TriLMs across various aspects, including model size scaling, training dynamics, commonsense reasoning, knowledge capacity, and toxicity. The paper is well-organized. The authors provide benchmarks for various quantization formats, including inference speed and memory usage. The paper compares their TriLM models with existing methods like BitNet b1.58. The authors have actively addressed reviewer concerns during the rebuttal, improving the paper's clarity, providing further clarifications.

Reviewer DqjN felt the work was too similar to BitNet b1.58 in terms of architecture, training approach, and some conclusions. They questioned what additional contributions the paper offered beyond this prior work. The claim that Spectra was the "first" to demonstrate the feasibility of pretraining ternary language models was challenged, given the existence of BitNet b1.58. Reviewer AzmB was concerned that the models were not fully converged after training on 300B tokens, especially the larger models. They questioned how the performance of FloatLMs and TriLMs would compare after full convergence. Reviewer AzmB argued that comparing a pretrained TriLM with PTQ models was unfair, as QAT during fine-tuning could significantly improve the performance of 3/4-bit models.

Two reviewers with high scores gave low confidence scores (3), while two reviewers who leans to reject the paper had high confidence scores (5 and 4). Given the mixed reviews and the detailed responses, the paper can be accepted for a poster presentation.