PT$^2$-LLM: Post-Training Ternarization for Large Language Models
Authors: Xianglong Yan
ChengZhu Bao
Zhiteng Li
Tianao Zhang
Kaicheng Yang
Haotong Qin
Ruobing Xie
Xingwu Sun
Yulun Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Post-Training Quantization
Ternarization
LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: PT$^2$-LLM is a post-training ternarization framework for efficient and effective large language model compression.

Abstract: Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment. Ternarization has gained attention as a promising compression technique, delivering substantial size reduction and high computational efficiency. However, its potential in the post-training quantization (PTQ) setting remains underexplored, due to the challenge of training-free parameter optimization and the quantization difficulty posed by outliers and dispersed weights. To address these issues, we propose PT$^2$-LLM, a post-training ternarization framework tailored for LLMs. At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline: (1) Iterative Ternary Fitting (ITF), which alternates between optimal ternary grid construction and flexible rounding to minimize quantization error, and (2) Activation-aware Grid Alignment (AGA), which further refines the ternary grid to better match full-precision outputs. In addition, we propose a plug-and-play Structural Similarity-based Reordering (SSR) strategy that leverages inter-column structural similarity to ease quantization and mitigate outlier effects, further enhancing overall performance. Extensive experiments demonstrate that PT$^2$-LLM delivers competitive performance against state-of-the-art (SOTA) 2-bit PTQ methods with lower memory cost, while also accelerating both prefill and decoding to achieve end-to-end speedup. The code and models will be available at \url{https://github.com/XIANGLONGYAN/PT2-LLM}.

Published as a conference paper at ICLR 2026
PL2 POST-TRAINING TERNARIZATION FOR LARGE LANGUAGE MODELS
Xianglong Yan'; Chengzhu Bao"; Zhiteng Li", Tianao Zhang', Kaicheng Yang', Haotong Qin?, Ruobing Xie Xingwu Sun Yulun Zhang' 1 Shanghai Jiao Tong University, 2ETH Zürich, Teencett Hunyuan
ABSTRACT
Large Language Models (LLMs) have shown impressive capabilities across di- verse tasks, but their large memory and compute demands hinder deployment. Ternarization has gained attention as a promising compression techmique, deliv- ering substantial size reduction and high computational efficiency. However, its potential in the post-training quantization (PTQ) setting remains underexplored, due to the challenge of training-free parameter optimization and the quantization difficulty posed by outliers and dispersed weights. To address these issues, we propose PL2-LLM, post-training ternarization framework tailored for LLMs. At its core is an Asymmetric Ternary Quantizer equipped with a two-stage finement pipeline: (1) Iterative Ternary Fitting (ITF), which alternates between optimal ternary grid construction and flexible rounding minimize quantization error, and (2) Activation-aware Grid Alignment (AGA), which further refines the terary grid to better match full-precision outputs. In addition, we propose plug-and-play Structural Similarity- Reordering (SSR) strategy that leverages inter-column structural similarity ease quantization and mitigate outlier effects, further enhancing overall performance. Extensive experiments demonstrate that PT2-LLM delivers competitive performance against state-of offtt--o--he-rr art (SOTA) 2-b PTQ methods with lower memory cost, while also accelerating both prefill and decoding to achieve end-to-end speedup. The code and models will be available at https: / /i thub. com http///AANTTOGGNNAAAAOA
LLM.
INTRODUCTION
Large Language Models (LLMs) (Touvron et al., 2023a;b; Dubey et al., 2024; Yang et al., 2025; Zhang et al., 2022) have achieved remarkable progress in language understanding, reasoning, and generation. They many real-world applications and remain at the forefront of Al research. However, these achieve- ments are largely enabled by the massive scale of model parameters. Modern LLMs often contain tens or even hundreds of billions of parameters (e.g., DeepSeck-RI (Guo al., 2025) has 671 bil- lion), leading to substantial memory consumption and intensive computational demands. Running such models demands powerful GPUs, large mem- ory, and high energy, which hinders deployment resource- limited latency-sensitive platforms.
55
658
508
Avg. Accuracy (%)
500
as the foundation for
P138
45.
308
30a
PT
40
1188
Slim-LLM
GPTQ
PB-LLM
10 14 16 18
Model Size (GB)
Figure 1: LLaMA performance on 7 zero-shot Question Answering (QA) datasets. PT yields the best accuracy equal memory cost.
Weight-only quantization (Frantar et al., 2023) reduces weight precision to save memory and ac- celerate inference. Among various schemes, ternarization (Li al., 2016) constrains weights to (-1,0,+1 +1), enabling high compression ratios and efficient computation. Compared to low-bit quan- tization (e.g.. 2-4 bit) (Lin et al., 2024b), eliminates most floating-pois point multiplications by using simple additions, reducing both computational and energy costs. Compared to binarization (Raste- gari et al., 2016), ternriization better fits the unimodal distribution of LLM weights and offers stronger representational capacity, yielding higher accuracy. Balancing efficiency and expressiveness, ternarization suits seourree-limiter LLM deployment (Wang et al., 2025; Yin al., 2025).
"qqual contribution Corresponding author; Yulun Zhang, yulun 100 gmail.com

---

Published as a conference paper at ICLR 2026
Recent studies (Lu al., 2024; Zhang al., 2020) on ternarization primarily focus quantization- aware training (QAT), where models are trained under ternary constraints. Such methods are mainly explored sedderate-sized architectures like BERT or DIT, where training remains affordable. While attempts have been made to extend QAT-hased ternarization to LLMs (e.g. BitNet b1.58 (Ma al., 2024)), such approaches are highly impractical due to the immense parameter scale and the prohibitive demands on training resources, computational budget, and full access to training data. In contrast, post-training quantization (PTQ) offers far more practical and efficient alternative: enables rapid conversion from full- full-precisson models to compact ternary versions without retraining or access to full training data, making it more suitable for real world LLM deployment scenarios.
However, PTQ-based ternarization remains underexplored, as its direct application often causes severe performance degradation, making models unusable. Through analysis, we identify two main challenges: (i) Unlike QAT, which optimizes ternary parameters through gradient-based updates on large-scale training data, PTQ must efficiently refine them without any training, which poses core challenge. (ii) As extreme low-bit quantization scheme, ternarization struggles to represent weights with dispersed or outlier-hevyy distributions, making it particularly prone to large quantization error.
In this paper, we propose PT-- a post-training ternarization framework tailored for LLMs. To tackle the challenge of training-free free ternary parameter optimization, we propose an Asymmetric Terary Quantizer (ATQ), refined through two stages: Iterative Temary Fitting (ITF) and Activation- aware Grid Alignment (AGA). ITF alternates between optimal ternary grid construction and flexible rounding to minimize quantization error, while AGA leverages calibration data further align lernary outputs with full-precision ones. To handle dispersed weights and outliers, we propose a plug-and-play Structural Similarity-based Reordering (SSR) strategy, which reorganizes columns based on inter-column structural correlation to ease quantization. Equipped with ATQ and SSR, -LL enables accurate and robust post-training ternarization. As shown Fig. outperforms state-of-ther the-art (SOTA) 2-bit PTQ methods in zero-shot QA accuracy under the same memory budget.
Our key contributions can be summarized as follows:
We propose PT--LLM, a novel ternarization framework that efficiently compresses pre- trained LLMs into a ternary grid without any retraining, addressing the unexplored challenges of post-training ternarization in LLMs.
We design an Asymmetric Ternary Quantizer for post-training ternarization. It is optimized through two training-free free stages: Iterative Ternary Fitting (ITF) and Activation- awware Grid Alignment (AGA). These components enable effective refinement of ternary parameters, reducing quantization error and improving alignment with full-precision outputs.
We develop plug-and-pll Structural Similarity- based Reordering (SSR) strategy. It reorders weight columns based on structural similarity, which helps reduce quantization difficulty and suppress the influence of outliers.
Extensive experiments demonstrate the competitive performance of P compared to SOTA 2-bit PTQ methods, with reduced memory consumption and faster inference.
RELATED WORKS
2.1 NETWORK TERNARIZATION
Ternarization compresses neural networks by constraining parameters to (-1,0,+11) making them memory and computation efficient while preserving strong representational capacity. Ternary weight networks (TWN) (Li al., 2016) introduced scale-aware ternary quantization minimize the Eu- clidean distance from full precision, achieving up to 16x compression. Trained ternary quantization (TTQ) (Zhu et al., 2016) improves accuracy by jointly learning both ternary weight values and their assignments during training. Further works (Wang et al., 2018; Alemdar et al., 2017) extended ternar- ization to activations, enabling fully ternary networks with greater efficiency. More recently, efforts have aimed to apply ternarization to larger and more complex models. TernaryBERT (Zhang et al., 2020) achieved 14.9x BERT compression via loss-aware ternarization and distillation. TerDiT (Lu et al., 2024) scaled ternarization to 4.2B diffusion transformers. TernaryLLM (Chen et al., 2024) introduced learnable scaling and feature distillation, outperforming prior low-bit LLMs. BitNet b1.58 (Ma al., 2024) proposed ternary- training framework for LLMs, delivering near full-precision accuracy with reduced latency and energy consumption. However, most existing ternarization methods rely training, limiting their practicality in real-world deployment.
2

---

Published as a conference paper at ICLR 2026
(a) Input
(c)
Asymmetric Ternary Quantizer
Original Weight Matris W
Calibration Set
Initialization
Iterative Ternary Fitting
Activation-aware Grid Alignment
W g
Eor= IW
(cT + aI
O E
|WX (AT MQKI3
Optimal
W.
1'
m1
Clastering
(b) SSR
Optimal Grid
W
wPP.wPP.
0.35
Optimal
IW II
Tomortees
Cuipptt alligmeet
Temarbedd
Reordering
(d) Output
Represent black
Wij
Reordered Weight Macris
Flestble rounding
Overview of - Structural Similarity- bbsed Reordering (SSR): reorders columns based structural similarity. Asymmetric Ternary Quantizer: enhanced by Iterative Ternary Fitting (ITF) and Activation-awes aware Grid Alignment (AGA) for refined ternary parameter optimization.
Figure 2:
2.2 QUANTIZATION FOR LARGE LANGUAGE MODELS
Quantization reduces the memory footprint and inference cost of large language models, and is typically categorized into quantization-awae aware training (QAT) and post-training quantization (PTQ).
Quantization-Awan Training (QAT). QAT incorporates quantization during training, enabling LLMs to learn robust low-bit representations through backpropagation. Works such as LLM-QAT (Liu et al., 2024a) and BitDistiller (Du et al., 2024) leverage knowledge distillation to preserve accuracy under low-bit quantization. EfficientQAT (Chen et al., 2025) reduces QAT's overhead via a two-stage training scheme. Recent efforts like Onebit (Xu et al., 2024) and BinaryMoS (Jo et al., 2024) further extend QAT to the 1-bit regime. While QAT can effectively preserve performance under low-bit quantization, its high computational and memory cost remains major limitation.
Post- DestnTrainin Quantization (PTQ). Unlike QAT, PTQ directly quantizes pretrained models without retraining, making more efficient and deployment-friendly for LLMs. Early methods (Dettmers et al., 2022: Yao et al., 2022; Li et al., 2021) improve quantization by introducing grouping labels. Techniques like AWQ (Lin al., 2024b) and OWQ (Lee al.. 2024) introduced transforma- tions that scale salient weights, aiming to preserve activation expressiveness and overall model capacity. GPTQ (Frantar et al., 2023) leverages Hessian-guided error compensation, with GP- TAQ (Li et al., 2025a) extending it via asymmetric calibration. OmniQuant (Shao et al., 2023) and SmoothQuant (Xiao al., 2023) address activation outliers through scale redistribution. More recent work (Lin al., 2024a; Ashkboos et al., 2024; Liu et al., 2024b) adopts saation-bbsed transformations for improved low-bit quantization. For ultra-low-bit settings, QulP (Chee al., 2025) and QulP# (Tseng al., 2025) use incoherence processing to boost performance, while Slim-LLM (Huang et al., 2025) employs salience-aware mixed- precision schemes. In the binarization domain, 1-bit methods such as PB-LLM (Shang et al., 2024). BiLLM (Huang et al., 2024), and ARB-LLM (Li et al., 2025b) show competitive results. Sub-1-bi approaches (Dong et al., 2025; Yan al., 2025; Gu et al., 2025) further advance compression by reducing average bitwidths while retaining strong accuracy. Our method falls under the category of post-training ternary quantization.
mixed-precision
3 METHOD
Overview. Fig. 2 illustrates the overall workflow PT-- We first review the standard symmetric ternarization formulation and basic notations in Section 3.1. Building on this foundation, Section 3.2 introduces the Asymmetric Ternary Quantizer, which features two training Ternrry Fitting (ITF) and Activation-aware Grid Alignment (AGA). Section 3.3 then presents the Structural Similarity-based Reordering (SSR), demonstrating how column clustering by structural similarity can be effectively combined within the GPTQ framework.
stages: Iterative
3.1 PRELIMINARY
Symmetric Ternarization. Symmetric ternarization compresses full full-precision weights into the ternary set (,,, +1} by minimizing the discrepancy between the original weight matrix and its
3

---

Published as a conference paper at ICLR 2026
ternary approximation under appropriate scaling:
a', T arg min |W aTI
a,
where W E R is the full-precision weight matrix, a E R is a row-wise scaling factor, and T {-1,0, +1 ) is the ternary matrix. Since jointly optimizing and TT causes parameter coupling, TWN (Li et al., 2016) proposes threshold-basee solution. Specifically, for each element W ij. row-wise threshold A [gxx1 used determine the corresponding ternary value T.j as:
1,
if W ij > A:;
Tij
0, if i y S A,
(2)
-1, if Wi <-
Given fixed threshold A, the ternary matrix T is deterministically defined, enabling closed-form solution for the optimal scaling factor a. Since directly optimizing A is difficult in practice, TWN approximates A based assumed weight distributions. Assuming uniform normal priors, A is approximated by scaled mean of absolute weights, and the optimal a follows:
0.75
W... a >== T W.
A
RJ
Σ 2=1
m
This approximation enables fast and training-free free ternarization by decoupling a and T. providing a practical solution to ternary parameters initialization in PTQ settings.
3.2 ASYMMETRIC TERNARY QUANTIZER
Asymmetric Ternary Initialization. Empirical observations reveal that the weight distributions in LLMs are not always symmetric, as many layers exhibit non-zero means. We provide visualizations in the supplementary file to further support this observation. While symmetric ternarization (as discussed in Section 3.1) performs well under QAT due to its ability to reshape the weight distribution through backpropagation, this assumption no longer holds in PTQ, where pre-trained weights remain fixed. To better capture the bias in pre- trained weights, we follow prior work (Chen et al., 2024) and adopt an asymmetric ternarization scheme by introducing a row-wise offset E Rax initialized as the mean of each row. The dequantized weight W is then computed as:
W = aT H4
W :
j=1
For the initialization of a and the ternary matrix T. we follow the same strategy described in Section 3.1, applying it to the centered weight matrix W = W - to remove bias:
0.75 m
T T W.
A
Σ W.s. J=1
a
(5)
m
Ž T
T is still initialized using Eq. 2, with A applied W. This asymmetric initialization offers stable and expressive foundation for post-training ternarization under non-zero-mean weight distributions.
Iterative Ternary Fitting. After initialization, we obtain the three key components of ternarization: the scaling factor a, the shift parameter μu, and the ternary matrix T. a and fr together define a ternary grid with only three possible quantized values for each row i, namely {-a; + 14, Pi; a + u4). How to refine this ternary grid so that it better fits the underlying weight distribution is crucial for improving quantization quality. We first define the quantization error of weights Ene as:
(6) Our current optimization objective is to minimize the quantization error En,, which is achieved by optimizing the ternarization parameters a, nt, and T. Since and μu together determine the discrete grid values of ternarization, we refer to them as the ternary grid parameters. A well-constructed grid essential to provide reliable basis for subsequent optimization of the ternary matrix T. Therefore, we first focus on establishing a high-quality temary grid. By differentiating the quantization error & with respect to a, and μ4, we obtain the following gradients:
= W where =
i.,.
2(a,t, 1,11 -
Op: 2(aat, +111 -w.)1,
da:
where t, E R Ixmm denotes the i-th row of T, and w, E R. xm is the corresponding row of W. I E mmx1 is column vector of all ones. The parameters as and l4 serve the scaling factor and
4

---

Published as a conference paper at ICLR 2026
shift associated with the i-th row, respectively. To obtain the optimal ternary grid parameters, set the partial derivatives to zero and solve for a; and 14,. This leads to a system of linear equations for the optimal grid parameters a; and μ4; (see supplementary file for detailed derivation):
ge 0a;
8r =
t,
1 1
w.t! w
0, 3p4
t,1
=
(8)
which can be efficiently solved to obtain the optimal ternary grid for the i-th row. To enable efficient batched computation across rows, we further reformulate the optimal solutions of a" and p* into more compact vectorized form (see supplementary file for detailed derivation):
a² m (W o T)1 (T1) (W1) (T (T1)²
oT [TI (WI) (T1) [( o T)1
(9)
(To (T1)²
where denotes element-wiss multiplication, with all divisions also element-wist, m is the number of elements per row. This vectorized form enables parallel closed-form solutions across rows, ensuring optimal a" and μ under fixed T and thus the best ternary grid the current stage. After obtaining the current optimal ternary grid, we update T by mapping the full-precision weights onto it. Instead of fixed threshold, which rigid and often suboptimal for diverse weight distributions, we adopt a more flexible element- wise ternary rounding to minimize the quantization error &w (Eq. 6). Given a² and n=, the optimal value each entry T is determined by the following rule:
W 14
T mi Zzg- t|,
where Zij
(10)
Of
This guarantees that, under fixed a" and μ=, the updated T yields the minimal quantization error Ew. making it the optimal ternary assign- ment for the current grid. We observe that ob- taining the optimal ternary grid and the optimal ternary matrix naturally forms an iterative opti- mization scheme. By alternating between Eq. and Eq. 10, the algorithm greedily reduces the quantization error E at each step. Convergence reached when the update in Eq. 10 longer changes the ternary matrix T, indicating that the temarized structure has stabilized. In practice, converges within about 10 iterations.
Algorithm 1 Pseudocode of the Asymmetric Ternary Quantizer. See supp. file for details.
func ATQ(W, X)
Input: W E RRax.. weight matrix
X R Bxxxm calibration data
Output: W E Raxm
1: a, p,, T =
Initary_Iit((
2: T pree ==
3: while T + T Trree do
4:
T Prree <- T
5: a <-
COptinal_Grrd(( W) T + Flexible RoalleeRonnWWW a)
6:
7: end while
Activation-aware Grid Alignment. While Iter- ative Ternrry Fitting effectively minimizes the weight quantization error Ew, the actual output of LLMs depends on the interaction between weights and activations. To address this issue, we introduce the activation-aware output error E
8: a. μ + AGA(WA (E,,W,,TTX T,X)
9: W < --TTT +
10: return W
(11) Here, X E RBxxx denotes the calibration data with batch size B, sequence length L, and embedding dimension This formulation directly couples quantization with the model outputs, ensuring that the optimization better reflects the real scenario. In line with the Iterative Ternary Fitting, we again differentiate & with respect to a; and μ4, set the derivatives to zero, and obtain system of equations that gives the optimal temarr grid under the current objective. As before, the solution is expressed row-wise for the i-th component:
= ||X- W where W aT + p.
OE.
OE
Ct
C
as ps
w: Ct,
da:
= -0, 0p:
0
t1
1 c
w,C1
(12)
where E. E. X X,, By solving this system, we obtain the closed-form solutions:
a' d. (W o T)S1 vo (WS1) d T2 v2
1 T o (WS1) - [(W T)S1]
(13)
d T' 22
where d - 1T a scalar, and TS1; 2 and v2 denote clement-wise squares. This activation- aware alignment of grid parameters significantly improves the consistency between quantized and full-precision outputs. Ideally, we would update T to further reduce the output error &,, but unlike ITF, no optimal solution exists. Greedy search is possible, yet in practice we observe that updating T leads to severe overfitting on the calibration set. Therefore, we freeze T and update only (a, μ)) once, which already yields accurate approximations. See supplementary file for details overfitting.
5

---

Published as a conference paper at ICLR 2026
L1aMIA-3-8B Layer Q Projection Matris
Quantization Error Exe
Initial Quantization Error 4
Initial Outpor Errir
214
14:80
Contaciin
Output Error
218
Quantization Error Reduction
Output Faror Reduction
Varisane
Arthoris Gnd Algginent
Gral
Oprimization Step
Optimization Step
Indei
Figure 3: Visualization of the proposed Asymmetric Ternary Quantizer (ATQ) and Structural Similarity- Reordering (SSR) effects. Left: Quantization error En across optimization steps during ATQ. Middle: Output error & across optimization steps during ATQ Right: After column reordering, the block-wise variance becomes smaller, showing a more compact weight distribution.
Overall ATQ Workflow. As shown in Algorithm I, ATQ refines the ternary parameters via ITF for a more accurate T, and then applies AGA to further align the ternary grid parameters with the output, yielding the final quantized weights. In Fig. 3 (left and middle), we plot quantization error & and output error & across steps. & steadily decreases during ITF and slightly rises after the AGA update as the optimization objective shifts, while & drops modestly during ITF and sharply after AGA. Overall, the results show ATQ reduces both quantization and output errors without retraining.
3.3 STRUCTURAL EIMILARITY-BASER REORDERING
Motivation. Following GPTQ (Frantar et al., 2023), our ternarization blockwise: large weight matrices are split into fixed-size blocks and quantized independently. While this improves accuracy over quantizing the entire matrix at once, we still find that naive blockwise ternarization causes severe performance degradation. To investigate, we analyze the weight distributions and identify two key issues. (i) weights within a block often exhibit high variance, making ternarization too coarse and leading to large quantization error; (ii) many layers exhibit column-wise bias, where outlier columns distort the ternarization range and degrade fidelity.
Etructure-Aware Column Clustering. To address these issues, we revisit column reordering, which in GPTQ (Frantar et al., 2023) is offered as an optional technique: while GPTQ can quantize weights in fixed order, reordering columns by Hessian-derived importance has been shown to improve performance. Formally, reordering can be expressed using a permutation matrix P:
W WP, X' XP, X'W'T = -W
(14)
Here P simply permutes columns, so the result of matrix multiplication remains unchanged. Since applying P is just index reordering rather than actual multiplications, the computational overhead during inference is negligible. Building on this formulation, we observe that the potential of reordering remains underexplored. Placing structurally similar and numerically close columns in the same block yields a more compact distribution, improving row-wise wwse ternarization. Similarly, grouping outliers together prevents them from distorting normal polumns-outlicns among outliers cease to be outliers. To this end, we propose a simple yet effective structure-aware column clustering method. Specifically, we compute pairwise cosine similarities between weight columns to capture their structural similarity:
W. W.j
Sij
(15)
W. w
where W.., denotes the i-th column of W. Based on the similarity matrix S, we cluster columns with aligned directions to form more homogeneous blocks for ternarization. As shown in Fig. 3 (right), reordering reduces block variances, indicating more compact weight distributions within blocks.
Efficient Integration with GPTQ, GPTQ quantizes weights block by block, applying error compen- sation after each step. This inter-block dependency makes a one-time clustering- based reordering ineffective, while re-clustering after every update is too costly. To balance accuracy and efficiency, we adopt a lightweight strategy: after each update, we compute a mean reference from the residual and select the top-k similar columns, where is the quantization block size:
B = Top-k ( T., 1T (
W.
with w
Σ
W 5
(16)
Here, is the mean vector of the remaining submatrix. and B contains the top-k most similar columns, forming the next quantization block. We term this lightweight strategy Structural Similarity- based Reordering (SSR), which retains the benefits of reordering with minimal overhead.

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation on Multiple LLM Backbones. We report perplexity (PPL) on WikiText2 and C4, and accuracy (%) on seven zero-shot tasks. All quantized models use a block size of 128. Best and second-best -eet results (excluding FP16) are marked in bold and underlined, respectively.
Model
Method
#W
[1
(4(J)
PIQA
Are E
Arc C
Hella.
Wino.
OBQA
BoolQ
Avg(
FP16
16
5.68
7.34
78.67
75.34
41.89
56.93
70.01
34,20
75.05
61.73
AWQ
2.60e5
2.86c5
52.83
25.25
22,44
25.27
49.88
14.00
37.83
32.50
GPTQ
129.19
79.06
55.39
30.64
19.62
27.36
48.30
14.40
44.74
34.35
1.78
QulP
29.74
33.74
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Slim-LLM
2
14.58
30.71
$7.83
33.46
25.09
36.70
52.64
16.40
56.05
39.74
PB-LLM
177
82.76
76.63
55.17
29.42
18.94
27.68
47.83
12.20
PT
24.55
52.48
24.32
34.04
18.40
45.07
FP16
16
5.09
6,80
79.16
77.40
46,42
59.91
72.69
33,20
77.89
63.81
AWQ
2
2.76e5
2.30c5
53.37
26.22
22.87
25.54
49.96
15.60
62.17
36.33
GPTQ
20.46
18.97
61.9 97
41.67
24.06
38.13
54.85
19.20
47.13
41.00
L-13B
QulP
38.82
28.62
N/A
N/A
N/A
N/A
N/A
NIA
N/A
Slim-LLM
2
9.12
19.59
66,59
55.22
25.00
38.25
62.83
24.40
64.01
48.04
PB-LLM
1.7
44.93
40.64
60.12
36.53
19.28
30.78
30.67
14.60
62.75
39.25
PT
1,58
9.11
17.32
67.19
58.50
26.19
39.17
63.54
22.20
63.67
48.64
FP16
16
3.53
81.28
$1.36
52.82
64.55
77.43
38,20
84.83
68.
AWQ
7.40p4
7.50ke
53.16
25.13
22.27
25.48
49,80
11,20
37.83
32.12
GPTQ
2
8.66
10,23
73.12
64.94
35.07
47.95
63.93
26.60
67.31
54.13
L-65B
QulP
7.83
13,99
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Slim-LLM
6.15
11.11
75.20
53.72
35.10
45.91
70.21
25.80
76.91
54.69
PB-LLM
1.7
12.81
15.30
72.18
68.43
35.18
46.35
70.16
26.40
62.75
54.49
PT-LLLM
1,58
6.62
9.17
73.01
70.08
35.58
46.52
70.48
28.00
67.98
55.95
FP16
16
5.47
7,26
78.07
76.30
43.34
57.16
68.95
31.40
77.68
61.85
AWQ
2.2265
1.70c5
52.39
25.00
21.16
GPTQ
52,22
35.27
58.05
33.16
21.42
32.65
49.88
15.60
55.47
38.03
L2-7B
QulP
2
39.73
31.94
N/A
N/A
N/A
N/A
N/A
NIA
N/A
N/A
Slim-LLM
15.84
84,92
63.82
47.81
23.38
33.76
56,91
17.80
59.97
43.35
PB-LLM
1.7
66.41
80.69
53,59
27.82
18.69
26.91
48.54
13.20
41.25
32.86
PT--LLM
1,58
11.56
24.38
62,95
47.01
21.08
33.82
56.75
18.80
62.91
43.33
FP16
16
3,32
5.71
82.15
82.79
54.44
64.78
77.98
37,20
83.76
69.01
AWQ
7.25e4
7.30c4
$2.50
25.76
22.35
25.33
49 49
14.20
62.17
35.97
GPTQ
8.18
19.55
72.52
62.67
34.56
47.66
25.00
66.76
53.76
L2-70B
QulP
2
N/A
N/A
N/A
NIA
N/A
N/A
N/A
N/A
N/A
N/A
Slim-LLM
2
6.28
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
PB-LLM
177
28.37
N/A
N/A
N/A
N/A
N/A
N/A
PT
1.58
6.27
12,00
72.96
71,00
46.17
71.35
25.60
55,87
FP16
16
6.14
9,45
79.54
80.13
50.34
60.13
73.40
34,60
ST.01
65.59
AWQ
1,7085
2.10c5
$2.72
24.16
21.50
25.58
49.33
14,60
62.17
35.72
GPTQ
1480.43
394,74
52.12
25.72
21.59
26.72
49.17
13.60
44.16
33.30
L3-8B
QulP
84.97
130.00
N/A
N/A
N/A
N/A
N/A
NIA
N/A
N/A
Slim-LLM
2
38.21
390.02
$5.77
32.15
19.11
27.83
48.78
13,20
44.83
27.83
PB-LIM
1.7
73.08
104.15
56.64
33.08
17.15
27.98
51.07
12.40
55.44
36.25
PT
1.58
32.19
129.83
56.86
34,22
18.43
30.36
53.28
13.80
$7.58
37.79
FP16
6.38
74.20
54.27
35.00
86.50
AWQ
2.68e7
2.18e7
$3.00
24.60
23.00
25.30
50.70
20.00
20
34.69
Qwen3
GPTQ
2
37.90
74,50
56.31
34.64
20.65
33.30
52.72
17.20
46.33
37.31
QulP
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Hase
Slim-LLM
22.85
68.38
61.33
$2.54
29.35
31.52
52.04
20.40
61.20
44.13
PB-LLM
2.8964
2.44e4
54.08
25.93
20.73
25.76
47.99
15,00
38.04
32.50
PT-M
1,58
16.48
68.13
62.95
53.03
23.63
33.65
59.75
20.60
62.17
45.11
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Implementation Details. All experiments are conducted using PyTorch (Paszke et al., 2019b) and Huggingface (Paszke et al., 2019a) on single NVIDIA A800-80GB GPU. As PT is PTQ framework, it requires no training or gradient backpropagation. Following Li et al. (2025a) and Huang al. (2025), we use 128 calibration samples from the Wikitext2 (Merity al., 2017) dataset, each with sequence length of 2048. All quantized models use fixed block size of 128.
Models and Evaluation. We conduct comprehensive experiments on the LLaMA (Touvron et al., 2023a), LLaMA-2 (00-2((Toon) (Touvron et al., 2023b), and LLaMA-3 families (Dubey et al., 2024), as well as the more recent Qwen3 series (Yang et al., 2025). Following prior work (Frantar et al., 2023; Lin et al., 2024b), we evaluate model performance in terms of both perplexity and accuracy. We report perplexity on WikiText2 (Merity et al., 2017) and C4 (Raffel et al., 2020) using a sequence length of 2048 tokens, and assess zero-shot accuracy seven widely-used QA benchmarks: ARC-c (Clark et al., 2018), ARC-e (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OBQA (Mihaylov et al., 101 PIQA (Bisk et al., 2020), and Winogrande (Sakaguchi et al., 2020).
7

---

Published as a conference paper at ICLR 2026
Table Ablation studies conducted on LLaMA-2-7B and LLaMA-3-8B, We report perplexity on Wikitext2 and C4, as well as average accuracy across seven zero-shot tasks. (b) Effectiveness of SSR
(a) Effectiveness of ITF and AGA
Model
AGA Wikitext2
Avg.
Model
Reorder
Method
Wikitext2
C4
Avg.
x
X
22.88
222.17
37.11
13.06
27.66
41.37
HLMM--27BB
x
x
15.47
12:25
26.17
42. 42.86
38.12
ILMMA-2-7R
Hessian-hased
Random
12.35
12.84
28.24
25.44 25.44
40.86
39.15
<
11.56
24.38
43.33
SSR
11,56
24.38
43,33
x
247.75
1227.94
33,26
112.83
599.19
33.08
LLaMA-3-8B
/
x
83.76
1039,80
33,90
LLaMA-3-8B
Random
113.42
466.07
33.37
47.83
$20. 14
35,29
Hessian-based
35.86
131.33
37,31
32.19
129.83
37.79
SSR
32.19
129.83
37.79
(c) Ablation study on calibration set size
(d) Ablation study on calibration set type
Model
Calib, Set Size
Wikitext2
C4
Avg. Ace
Model
Callb. Data Type Wikitext2
C4
Arg. Acc
11.92
25.27
43.31
Wikiiex12 C4
11.56
24.38
43.33
LLaMA-2-7B
128
11.56
24.38
43.33
LLaMA-2-7B
18.94
20.15
43,32
256
11.35
23.52
43.55
PTB
27.52
35.15
41.15
38,90
252.19
35.20
Wikitext2
32.19
129.83
37.79
MaMM-3-8B
128
32.19
129.83
37.79
LLaMA-3-RR
C4
168.81
72.00
37.80
256
32.25
167.48
37.95
PTB
428.97
$79.57
35,21
Baselines, We compare PT--LLM against a diverse set of representative PTQ methods operating in the 2-bit and sub-2-bit regimes. Slim-LLM (Huang al., 2025) serves strong baseline for mixed-precision quantization, achieving high performance with an average of 2 bits. PB-LLMM (Shang al., 2024) targets the sub-2-bit regime and is closest to ours in average bitwidth, making it a relevant baseline. We further include GPTQ (Frantar et al., 2023) and AWQ (Lin et al., 2024h) as widely used baselines, along with QulP (Chee al., 2025), which targets 2-bit quantization.
4.2 COMPARISONS WITH STATE-OF-THE-ART METHODS
Table summarizes the results of PT2-LLM and baselines on LLaMA, LLaMA-2, LLaMA-3, and Qwen3-base reporting WikiText2/C4 perplexity, zero-shot accuracy on seven tasks (with average), and average bitwidth. Despite operating at the lowest bitwidth (1.58), PT consistently ranks among the top two in both perplexity and average accuracy across all model sizes. It clearly outperforms 2-bit baselines such as GPTQ, AWQ, and QulP. Compared to Slim-I.L.M, the current SOTA 2-bit method, PT--LLM achieves higher average accuracy on all models except L.LMMA-2-7B, where it performs comparably. Relative to PB-LLM with comparable bitwidth, PT delivers significant gains: on LLaMA-7B, it improves average accuracy from 33.44 to 45.07, reduces WikiText2 perplexity by 86%, and requires less memory. Additional results are provided the supplementary file.
4.3 ABLATION STUDY
Effectiveness of ITF and AGA. We conduct a breakdown ablation to validate our Asymmetric Ternary Quantizer, focusing its two components: Iterative Ternry Fitting (ITF) and Activation- aware Grid Alignment (AGA). As shown in Table 2a, ITF and AGA each provide gains over the baseline, for example, I.LaMA-2-7B, ITF reduces Wikitext2 perplexity from 22.88 to 15.47, while AGA improves average accuracy from 37.11 1 to 42.86. When applied together, they deliver the best overall performance, underscoring their complementary benefits and validating our design.
Effectiveness of SSR. We evaluate the impact of Structural Similarity-based Reordering (SSR) on quantization performance. As shown Table 2b, omitting reordering yields suboptimal results, as ternrizaatio is highly sensitive to scattered weights and outliers. Random reordering has minimal effect, while Hessian- based reordering, though occasionally effective, often overlooks the structural challenges of block-wise ternarization. In contrast, our proposed SSR consistently yields superior performance by promoting column- wise similarity, enabling tighter block-wise weight distributions and reducing outlier sensitivity in block-wise ternarization.
blation for Calibration Set Size. We assess the impact of calibration set size on PT performance. As shown in Table 2c, larger sets yield modest gains in perplexity and accuracy. While 64 samples result in slightly lower performance, using 128 or 256 samples produces nearly identical results, indicating strong robustness to calibration size once modest threshold is reached. Considering the trade-off between performance and memory efficiency, we adopt 128 samples for all experiments, demonstrating the practicality of P1--LLM in resource-constrained settings.
8

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of model size, reduction ratio, and compression time across various quantization methods on LLaMA-7B, highlighting the efficiency of PT over existing approaches.
Method
FP16
GPTQ
Slim-LLM
PB-LLM
BiLLM
ARB-LLMx
PL
Model Size
13.48 GB
2.19 GB
2.30 GB
2,91 GB
2.93 GB
3.23 GB
1.88 GB
Size Reduction
6.16x
5.86x
4.63x
4.60x
4.17x
7.17x
Run Time
2
182min
22min
45 min
88 min
32 min
Throughput (token/s)
Prefill
Decode
End-to-End Generation
160
1.28%
160
160
2-bit
1.25x
2-bit
1.25K
2-bit
120
1.58-bit
120
1.58-bit
120
1.58-bit
1.39x
1.38
1.37*
80
80
80
2.48%
1,60%
1,60x
40
2.20x
40
2.16x
40
2.11X
7B
13B
30B
65B
7B
13B
30B
65B
7B
13B
30B
65B
Model Size
Model Size
Model Size
Figure 4: Throughput comparison between ternary (1.58-bit) and 2-bit quantized models across
LLaMA 7B-65B. We evaluate performance on prefill, decode, and end-to-end generation stages.
Ablation for Calibration Set Type. We study how calibration dataset choice affects quantization by ablating over three datasets: WikiText2, C4, and PTB. As shown in Table 2d, using WikiText22 C4 leads to comparable average accuracy, while PTB performs significantly worse, likely due to its lower data quality. Furthermore, calibrating on WikiText2 C4 improves perplexity on their respective datasets, showing clear in-domain benefits. Considering all metrics and following prior work such as SliM-LLM (Huang et al., 2025), we adopt WikiTex12 as our calibration set.
4.4 COMPRESSION TIME AND MODEL SIZE ANALYSES
We compare the compression time and model size of PT with GPTQ, Slim-LLM, PB-LLM, and two binarization methods, BiLLM (Huang et al., 2024) and ARB-LLM (Li et al., 2025b).
Compression Time. As shown Table PT offers strong trade-off between quality and efficiency. Compressing LLaMA-7B takes only 32 minutes, which is significantly faster than Slim-LLM, (182 min), BiLLM (45 min), and ARB-LLMx (88 min). While slightly slower than GPTQ and PB-LLM, the overhead remains modest and acceptable given the improved performance.
Model Size. As shown in Table 3, PT2-LLM achieves the smallest model size (1.88 GB for LLaMA- 7B), yielding 7.17x x reduction over FP1 It surpasses GPTQ (6. 16x), Bim (5.86x and PB-LLM (4.63 ), and even outperforms binarization methods like BiLLM (4.60 and ARB-LLMX (4.17x), whose compression ratios are limited by overhead from complex bitmap designs. A detailed discussion on memory composition and storage breakdown provided in the supplementary file.
4.5 INFERENCE SPEED EVALUATION
As shown in Figure 4, we evaluate inference throughput on NVIDIA A800 GPUs across four model scales (LLaMA-7B to LLaMA-65B) using llama. cpp', with sequence lengths of 128 (prefill), 256 (decode), and 128+256 (end-to-end). Compared to the standard 2-bit quantization, our 1.58-bit 2T LLM consistently improves throughput across all three stages and achieves up to 2x acceleration on LLaMA-65B for end-to-end generation. Experimental details are provided in the supplementary file.
CONCLUSION
In this paper, we propose PT2-LLM post-training ternarization framework tailored for LLMs. Our method proposes an Asymmetric Ternary Quantizer (ATQ) with a two-stage pipeline, where Iterative Ternary Fitting (ITF) reduces quantization error in training-free manner and Activation- aware Grid Alignment (AGA) aligns ternary outputs more closely with full-precision ones. We further propose Structural Similarity-basee Reordering (SSR). a plug-and and-alay strategy that reduces quantization difficulty and alleviates the impact of outliers. Experimental results show that -L attains competitive accuracy compared SOTA 2-bit PTQ methods while reducing model size and accelerating inference. This work establishes a strong baseline for ternarization in the PTQ setting of LLMs, pushing the boundary of sub-2-bit compression and laying the foundation for future research.
httppa: / http//gttt/uu cot.gg.ommmmmmmmm org/1 / ama . cpp

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (62501386, 625B1025, 625B1024) and also sponsored by CCF-Tencent Rhino-Bird Open Research Fund.
ETHICS STATEMENT
The research conducted in the paper conforms, in every respect, with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have provided implementation details Section 4. We will also release all the code and models.
LLM USAGE STATEMENT
Large Language Models (LLMs) were used solely for polishing writing. They did not contribute to the research content or scientific findings this work.
REFERENCES
Hande Alemdar, Vincent Leroy, Adrien Prost-Boucle, and Frédéric Pétrot. Teraryy neural networks for resource-efficient applications. IJCNN, 2017.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms. In NeurIPS, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. In NeurIPS, 2025.
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientgat: Efficient quantization-awanl training for large language models. In ACL, 2025.
Tianqi Chen, Zhe Li, Weixiang Xu, Zeyu Zhu, Dong Li, Lu Tian, Emad Barsoum, Peisong Wang, and Jian Cheng. Ternaryllm: Ternarized large language model. arXiv preprint 47X17:2406.07177. 2024.
Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolg: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1rXhv:1803.05457. 2018
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O): 8-bit matrix multiplication for transformers scale. In NewrIPS, 2022.
Peijie Dong, Lujun Li, Yuedong Zhong. Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang. Qiang Wang, Wei Xue, Yike Guo, et Stbllm: Breaking the 1-bit barrier with structured binary llms. In ICLR, 2025.
Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit lims via self-distillation. In ACI, 2024.
10

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.21783 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ICLR, 2023.
Hao Gu, Lujun Li, Zheyu Wang, Bei Liu, Qiyuan Zhu, Sirui Han, and Yike Guo. Btc-Ilm: Efficient l-bit IIm quantization via learnable transformation and binary codebook. arXiv preprint arXiv:2506. 12040, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint urXiv:2501. 2025.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post- post-training quantization for llms. In ICML, 2024.
Wei Huang. Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. Slim-llm: Salience-driver mixed-precision quantization for large language models. In ICML, 2025.
Dongwon Jo, Taesu Kim, Yulhwa Kim, and Jae-Joon Kim. Mixture of scales: Memory-efficient token-adaptive binarization for large language models. arXiv preprint (xXiv2240.1231/1 2024.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI, 2024.
Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Ternary weight networks. arXiv preprint arXiv:1605.04711 04771,, 2016.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. In ICLR, 2021.
Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. Gptaq: Efficient finetuning- quantization for asymmetric calibration. In JCML. 2025a.
Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, Linghe Kong, Yulun Zhang, Xiaokang Yang, et al. Arb-llm: Alternating refined binarizations for large language models. In ICLR, 2025b.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song. Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. In NeurlPS, 2024a.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Il compression and acceleration. MLSys, 2024b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. In ACL, 2024a.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantization with learned rotations. arXiv preprint arXiv:2405.
2024b.
Xudong Lu, Aojun Zhou, Ziyi Lin, Qi Liu, Yuhui Xu, Renrui Zhang, Xue Yang, Junchi Yan, Peng Gao, and Hongsheng Li. Terdit: Ternary diffusion models with transformers. arXiv preprint arXiv:2405. /4854, 2024.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Lifeng Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.1 17764, 1(4), 2024.
11

---

Published as a conference paper at ICLR 2026
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Paszke, Gross, F Massa, A Lerer, JP Bradbury, Chanan, Killeen, Z. Lin, Gimelshein, L Antiga, et al. An imperative style, high-performance deep learning library. In NeurlPS, 2019a.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, 2020.
Yuzhang Shang. Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. In ICLR, 2024.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXi::2307.092: 09288, 2023b.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. In ICML, 2025.
Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, and Furu Wei. Bitnet. cpp: Efficient edge inference for ternary llms. arXiv preprini arXiv:2502.11880, 2025.
Peisong Wang. Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two-step quantization for low-bit neural networks. In CVPR, 2018.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. ICML, 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang. Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In NeurIPS, 2024.
Xianglong Yan, Tianao Zhang, Zhiteng Li, and Yulun Zhang. Progressive binarization with semi- structured pruning for llms. arXiv preprint arXiv:2502.01705. 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lvv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.099 093388 2025.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for scage-scale transformers. In NeurlPS, 2022.
12

---

Published as a conference paper at ICLR 2026
Chenyang Yin, Zhenyu Bai, Pranav Venkatram, Shivam Aggarwal, Zhaoying Li, and Tulika Mitra. Tereffic: Highly efficient ternary Ilm inference on fpga. arXiv preprint arXiv:2502.16473 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? ACL, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009. /28/2, 2020.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:6612.01064 2016.
13


Reviews:
[{"review_id": "PmB9GAihF2", "reviewer": "Reviewer_V5DX", "summary": "The paper proposes PT2-LLM, a post training quantization framework for ternary quantization. Model weights follow iterative ternary filtering to minimize the layerwise weight-level L2 error. Following this, a single round of activation-aware grid alignment is run to minimize the layerwise activation L2 error. The paper also proposes column reordering to minimize the effects outliers for block level quantization. PT2-LLM improves over several methods in both perplexity and downstream evals.", "questions": "* When applied without ITF, does AGA still overfit with multiple steps?\n* Has blockwise thresholding been tried for ITF?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The ternary quantized model gets a significant speedup over the 2-bit model.\n* Quantization runtime remains comparable to GPTQ.\n* The paper also evaluates frontier open source models like Qwen3.", "weaknesses": "The paper does not compare with vector quantization methods like QuIP# or AQLM that allows for ultra-low precision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mP5jDsDoHX", "reviewer": "Reviewer_ofNJ", "summary": "This paper tackles the challenge of performing ternary (3-value) quantization on large language models in a post-training (PTQ) setting, where weight distributions and outliers make naive ternarization highly error-prone.\nThe authors propose PT2‑LLM, combining an Asymmetric Ternary Quantizer (ATQ) (with row-wise offset and scale) and a Structural Similarity-based Reordering (SSR) step to rearrange columns before quantization to reduce variance.\nExperiments on multiple LLMs (e.g. LLaMA variants) and benchmark tasks show that PT2‑LLM outperforms existing quantization baselines, achieving lower perplexity and higher accuracy under the same compression constraints.\nThe results suggest that careful adaptation of ternary quantization (non-symmetric, order-aware) can make extreme low-bit quantization feasible for LLMs in PTQ settings.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method explicitly models asymmetry in weight distributions and block-level structure, making the ternarization process more compatible with real LLM architectures.\n\n- The combination of the Asymmetric Ternary Quantizer (ATQ) with a structural similarity-based reordering (SSR) step is clever and helps reduce quantization error.\n\n- The authors evaluate on multiple large language models and a range of NLP tasks, showing consistent gains over state-of-the-art quantization baselines under comparable compression constraints. \n\n- In addition to accuracy metrics, they analyze memory, latency, and inference speed, demonstrating PT2‑LLM’s utility in real-world deployment scenarios.", "weaknesses": "- Although the paper achieves strong final results, the three claimed innovations lack originality and are quite similar to prior methods such as AWQ, RPTQ, and ARB-LLM, appearing more as a combination of existing core techniques rather than truly novel contributions.\n\n- SSR and the calibration steps (ITF / AGA) add additional computation and implementation complexity in the PTQ pipeline, which could be nontrivial for very large models or constrained environments.\n\n- The quantization accuracy tends to degrade more significantly on newer models (e.g., LLaMA3-8B and Qwen3-14B-Base), suggesting that the method may not scale robustly to newer models.\n\n- The paper could provide more comprehensive evaluation on challenging benchmarks such as MMLU, GPQA, and GSM8K to better demonstrate its effectiveness across diverse reasoning tasks.\n\n[1] Lin, Ji, et al. \"Awq: Activation-aware weight quantization for on-device llm compression and acceleration.\" Proceedings of machine learning and systems 6 (2024): 87-100.\n\n[2] Yuan, Zhihang, et al. \"Rptq: Reorder-based post-training quantization for large language models.\" arXiv preprint arXiv:2304.01089 (2023).\n\n[3] Li, Zhiteng, et al. \"Arb-llm: Alternating refined binarizations for large language models.\" arXiv preprint arXiv:2410.03129 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITgg8mrK3P", "reviewer": "Reviewer_qmRu", "summary": "This paper presents a novel post-training ternarization (PTQ) framework tailored for Large Language Models (LLMs), aiming for efficient compression without retraining. It reduces the average bitwidth to 1.58, and also achieve better performance against state-of-the-art (SOTA) 2-bit PTQ methods.", "questions": "1. Can the authors provide a more formal justification for the SSR strategy, explaining its connection to quantization error minimization beyond the intuition of variance reduction? Furthermore, please clarify the algorithmic details of its integration and provide an analysis of its computational overhead.\n2. Could the authors provide a direct comparison against another 1.58-bit PTQ method to substantiate their claim of state-of-the-art performance? If no such public method exists, could they construct a strong baseline (e.g., by adapting GPTQ or other frameworks to a ternary setting) for a more rigorous comparison?\n3. Why were the evaluations on the LLaMA-3 and Qwen3 families limited to a single model each? To support the claim of generalizability, please provide results on larger models like LLaMA-3-70B and on MoE variants like Qwen3-MoE.\n4. Given that Qwen3 is known for its reasoning abilities, can the authors provide evaluation results on more challenging reasoning benchmarks like MATH or AIME to give a more transparent assessment of how 1.58-bit quantization affects these critical capabilities?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Focus on Extreme Low-Bit PTQ: The paper explores the challenging but important domain of sub-2-bit post-training quantization, an area with significant potential for deploying LLMs on resource-constrained devices.\n2. Systematic Optimization Pipeline: The proposed two-stage optimization process (ITF for weights, AGA for outputs) is a structured approach to a complex problem, attempting to balance local reconstruction error with global functional preservation.\n3. Favorable Reported Results on Standard Benchmarks: The reported results in Table 1 indicate that the proposed 1.58-bit method achieves lower perplexity and higher average accuracy on several standard zero-shot QA tasks compared to the selected 2-bit PTQ baselines across various model scales.", "weaknesses": "1. Weak Theoretical Grounding and Unclear Practicality of the SSR Strategy. The proposed Structural Similarity-based Reordering (SSR) strategy lacks a strong theoretical connection to the primary objective of minimizing quantization error. Unlike Hessian-based methods... its objective (maximizing cosine similarity) is decoupled from the actual error minimization process. This weak theoretical grounding makes it difficult to understand why it works or to guarantee its effectiveness. This ambiguity is compounded by a lack of implementation clarity and cost analysis. The paper claims SSR is a \"lightweight strategy\" but fails to provide a detailed algorithm or an analysis of its computational overhead, making it difficult to assess its true practicality and reproduce the results.\n2. Unsubstantiated Claim of SOTA due to Lack of Direct Comparison: The paper's central claim of state-of-the-art performance at 1.58 bits is not adequately supported. The evaluation primarily relies on comparisons against higher-bitwidth (2-bit) methods. While outperforming them is a positive signal, a direct, apple-to-apples comparison against another 1.58-bit PTQ baseline is necessary to validate the true efficacy of the proposed techniques over other potential ternarization strategies. The absence of such a comparison represents a significant gap in the evaluation, leaving the paper's primary claim unsubstantiated.\n3. Insufficient Validation on Recent and Diverse Architectures: The paper's claims of general applicability are undermined by an incomplete evaluation on recent model families and a complete lack of testing on Mixture-of-Experts (MoE) architectures. While thorough on older LLaMA/LLaMA-2 models, the validation on the more recent LLaMA-3 and Qwen3 families is limited to a single small-scale model. This raises questions about the method's robustness on larger variants like LLaMA-3-70B. Furthermore, the absence of any experiments on MoE models (e.g., Qwen3-MoE) is a significant omission, as these architectures present unique quantization challenges that are not addressed.\n4. Mismatch Between Model Capabilities and Evaluation Benchmarks: The choice of evaluation benchmarks fails to adequately probe the impact of extreme quantization on the core competencies of the tested models. Specifically, while the paper includes modern, reasoning-capable models like Qwen3, it restricts evaluation to older, simpler zero-shot QA benchmarks (e.g., ARC, BoolQ). These benchmarks do not effectively measure the multi-step, complex reasoning abilities that are a hallmark of such models. Consequently, the evaluation may be masking significant performance degradation on these crucial capabilities. A comprehensive assessment requires reporting performance on advanced reasoning benchmarks (e.g., MATH, AIME).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQJSmNT89A", "reviewer": "Reviewer_TTyp", "summary": "This paper introduces PT2-LLM, a novel post-training ternarization framework for large language models (LLMs) that compresses weights to ternary values $\\{-1, 0, +1\\}$ without retraining. The authors address two key challenges in post-training quantization (PTQ): training-free parameter optimization and handling outliers/dispersed weights. Their solution comprises an Asymmetric Ternary Quantizer (ATQ) with a two-stage pipeline—Iterative Ternary Fitting (ITF) for grid construction and rounding, and Activation-aware Grid Alignment (AGA) for output calibration—alongside a Structural Similarity-based Reordering (SSR) strategy.", "questions": "See the weakness above.\n\nAn interesting job!  And I would be willing to raise my ratings if the above concerns are well solved.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents a technically innovative approach to an underexplored problem—ternarization in PTQ settings for LLMs. The ATQ framework with its ITF-AGA pipeline is well-motivated, addressing asymmetric weight distributions through mathematically grounded grid optimization (Equations 9 and 13). The SSR strategy offers a clever solution to outlier handling by leveraging structural correlations, demonstrating significant error reduction in Figure 3. \nEmpirical evaluation is comprehensive, spanning multiple model architectures (7B–65B parameters) and tasks (perplexity, zero-shot QA), with rigorous ablation studies validating component contributions (Table 2). The 7.17× compression ratio and end-to-end speedup highlight practical value, while the commitment to code/model release enhances reproducibility.", "weaknesses": "Block size sensitivity is inadequately explored. All experiments use fixed block size 128 (Section 4.1), but no analysis justifies this choice or examines its impact on quantization error. \nModel diversity is restricted to decoder-only transformers (LLaMA/Qwen3), excluding encoder-decoder or hybrid architectures. Scalability analysis is absent: no computational complexity bounds are provided for ATQ/SSR.\n\nKey baselines are missing. The paper compares to 2-bit PTQ methods (GPTQ, AWQ) but omits other ternarization approaches in PTQ settings (e.g., TernaryLLM [Chen et al., 2024] without training). Parameter sensitivity is superficial: only calibration set size is ablated (Table 2c), while critical hyperparameters (e.g., threshold $\\Delta \\approx 0.75/m \\sum |\\tilde{W}_{:,j}|$ in Equation 3) lack sensitivity analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)