AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
Authors: Gunho Park
Jeongin Bae
Beomseok Kwon
Byeongwook Kim
Se Jung Kwon
Dongsoo Lee
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Quantization
Anyprecision
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance accuracy and efficiency. 
Recent work has introduced multi-precision models, which enable inference at multiple precisions within a single model depending on runtime constraints.
To support such flexibility, quantized weights are often stored as bit-planes, where hardware efficiency improves when the compute operates directly at the bit-plane level and activates only the precision required by each request.
In this work, we present AnyBCQ, a hardware-friendly multi-precision extension of Binary-Coded Quantization (BCQ) that supports direct bit-plane operations.
By representing weights as binary bit-planes with corresponding scale factors, AnyBCQ enables bit-plane–level computation and maps naturally to accelerator-friendly, bit-parallel arithmetic.
Our progressive precision expansion mechanism incrementally refines scaling factors while reusing previously assigned binary codes, yielding monotonic improvements in accuracy as additional bits are enabled. 
We further co-design a specialized kernel that exploits the BCQ structure to support dynamic per-request precision selection with negligible overhead. 
Experiments on recent LLMs demonstrate that AnyBCQ significantly narrows the accuracy drop in the low-bit regime (e.g. 2-bit), remains competitive at higher precision, and achieves throughput gains of up to $3.0\times$ over half precision and $1.2\times$ over state-of-the-art multi-precision methods. 
By aligning algorithmic flexibility with hardware efficiency, AnyBCQ provides a practical foundation for multi-precision LLM deployment across diverse service-level objectives.

Published as a conference paper at ICLR 2026
ANYBCQ: HARDWARE EFFICIENT FLEXIBLE BINARY- CODED QUANTIZATION FOR MULTI-PRECISION LLMS
Gunho Park!, Jeongin Bac', Beomseok Kwon', Byeongwook Kim', Se Jung Kwon',
Dongsoo Lee'
NAVER Cloud
park dongsoo. lee enavererrcoooccommm
ABSTRACT
The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance accuracy and efficiency. Recent work has introduced multi- precision models, which enable inference at multiple precisions within single model depending on runtime constraints. weights are often stored as bit-planes, where hardware efficiency improves when the compute operates directly at the bit-plane level and activates only the precision required by each request. In this work, present AnyBCQ, hardware-friendee multi-precision extension of Binary-Codee direct bit-plane operations. By representing weights binary bit-planes with corresponding scale factors, AnyBCQ enables bit-plane-level computation and maps naturally to accelerator-friendly, bit-parallel arithmetic. Our progressive precision expansion mechanism incrementally refines scaling factors while reusing previously assigned binary codes, yielding monotonic improvements in accuracy additional bits are enabled. We further co-design a specialized kernel that exploits the BCQ structure to support dynamic per-request precision selection with negligible overhead. Experiments on recent LLMs demonstrate that AnyBCQ significantly narrows the accuracy drop the low-bit regime (e.g. 2-bit), remains competitive at higher precision, and achieves throughput gains of up to 3.0x over half precision and 1.2x over statc-of-the thetharrt multi- pelli-precision methods. By aligning algorithmic flexibility with hardware efficiency. AnyBCQ provides practical foundation for multi- peeetipprecisoon LLM deployment across diverse service- level objectives. The code is available at g thub. www/naaereaaaaieeaa anybcq
support such flexibility, quantized
Quantization (BCQ) that supports
INTRODUCTION
The rapid scaling of large language models (LLMs) has brought remarkable improvements in rea- soning, generation, and downstream task performance (Kaplan al. 2020 Hoffmann et al. 2022). However, this progress comes at the cost soaring computational and memory demands, making effi- cient deployment a pressing challenge (Kim et (0223). To address these constraints, a wide range of model compression techniques has been explored, including knowledge distillation (Sreenivas)
al. 2024 Xu al. 2024), pruning (Frrnntar Alistarh 2023 Sun al., 2023 Lee al. 2025). and quantization (Xiao et al. 2023 Ashkboos et al.) [024]] Kim et al. 2025). Among these, post-training quantization (PTQ) has emerged
particularly practical approach for LLMs, it can substantially reduce memory footprint and accelerate inference without requiring expensive retraining (Dettmers)
al. 2022, Frantar al. 2022). Within PTQ, weight-only quantization has gained popularity since weights dominate memory usage and are relatively robust to outliers compared to activations (Lin ct al. 2024). Recent state-of- the-art methods further demonstrate that 4-bit quantization can achieve accuracy comparable to that of full-precision models (Xiao al. 2023)
While uniform quantization, as exemplified by GPTQ (Frantar et al. 2022 and AWQ (Li et al. 2024), remains the most widely adopted strategy, recent work has introduced more expressive schemes, such as clustering- based non-uniform quantization (Kim et al., 2023a), to better capture the distribution of weight values and preserve accuracy after quantization. However, non-uniform schemes often rely on

---

Published as a conference paper at ICLR 2026
2-bit Scale (FP16)
3-bit Scale (FP16)
4-bit Scale (FP16)
Frozen Binary (INT1)
Quantization
Precision Expansion
Wor
Wir
0
Residual
Woo
Residual
Weight (FP16)
2-bit Model (BCQ2)
3-bit Model (BCQ3)
4-bit Model (BCQ4)
Base Model
Target Model
(a) Progressive precision expansion (p. 2, Pi = 4)
Non-active Binary (INT1)
AnyBCQ Model
Scales
3-bit Inference
Bit-plane inner product
-adstai taaiiitaa *00
+aio +
-02-0hi-07
+0
Binary (Shared)
Scale
Binary
Dequantized Weight (FP16)
(b) p-bit inference case (p 3)
Figure 1: Overview of AnyBCQ: (a) weights are first quantized to a base precision and progressively expanded to higher precisions by reusing the existing binary codes while adding residual bit- uttpplanes; (b) p-bit inference reconstructs weights by combining the corresponding scaling factors with the first p binary bit-planes. In the binary representation, elements denoted as 0 are mapped as -1.
centroid lookups reconstruct dequantized weights, which are difficult to optimize efficiently on modern hardware. Although slightly less expressive than fully non-umin non-uniform schemes, binary- coded quantization (BCQ) represents weights binary bit bit-planes with associated scale factors, yielding structure that is inherently accelerator secelerator-frienlly.. it maps naturally to binary operations and simplifies
kernel design (You et al. 2024 Park et al. 2022 2025 Kim et al. (003cc). Despite their effectiveness, existing BCQ-based methods are typically restricted to a fixed precision configuration, limiting their ability flexibly satisfy diverse service-level objectives (SLOs) in real-world deployments.
To address this limitation, the concept of multi-precision models has recently been proposed, allowing single model flexibly operate under multiple precisions and thereby adapt accuracy-latency trade-offs to dynamic system requirements Yuu et al. 2021 Park et al. [2024] Nair et al.) 2025) This flexibility has, turn, spurred research on mixxed-precision inference, including methods that dynamically assign precision across decoding steps (Chen et al. 2024) or adaptively assign precision on a per-layer basis (Kwon al. 2025). However, these approaches remain limited in practice: the tfae-oo--the-ar multi-precision model (Park et al.) (2024) relies on non-uniform quantization, which is not hardware-fricch friendly and performs poorly at extremely low bitwidths (e.g., 2-bit).
Multi- Kulti-precision inference often stores weights bit-planes that the system can fetch only the data required by the target precision. In this organization, the most efficient execution strategy is to compute directly on bit-planes and to activate only the planes needed for each request. Any iny-Prrcision LLM (Park et al. 2024) demonstrated strong accuracy across multiple precisions within a single unified model, yet reliance on non-uniform quantization prevents true bit-plane computation. Even with carefully optimized kernels that accelerate centroid indexing through bit-transpose operations and table lookups, additional overheads and irregular memory access remain. As result, dependence on centroid lookups continues to be a key bottleneck for hardware-efficient inference. Furthermore, extremely low-bit quantization (e.g., 2-bit) often induces severe accuracy degradation, while 4-bit quantization achieves accuracy close to full-precision models. As result, the effective operating range of current methods is largely restricted 3 bits. limiting the practical benefits of multi- precision quantization.
In this work, we propose AnyBCQ, a hardware- frienddy quantization framework that extends BCQ to the multi-precision setting and supports direct bit-plane operations. Figure [[a) presents the overall procedure of AnyBCQ. The model begins with a base-precision quantized representation and, through progressive precision expansion, evolves into multi-precision model capable of supporting multiple precision levels. Specifically, the full-precision weights are first quantized to the base precision using a BCQ scheme. At each subsequent stage, the binary codes from earlier levels frozen, while new scaling factors a are initialized and refined with an additional residual-derivee bit-plane. This procedure is applied iteratively until the desired target precision is reached, yielding
2

---

Published as a conference paper at ICLR 2026
model that can be used for inference at multiple precision levels. The resulting model contains the complete set of scaling factors required for each precision as well as the shared binary bit bit-planes. Figure lbb) illustrates example with inference precision p 3. During p-bit inference, the scaling factors corresponding to p-bit and the first binary bit-planes are employed to perform weight dequantization. By unifying algorithmic flexibility with hardware efficiency, AnyBCQ provides practical path toward multi-precision LLM deployment.
To quantify this benefit, Table I reports the mem- ory footprint at each target precision with group size g 128. Multi- denotes the baseline that stores separate model optimized for each bit precision. In contrast, AnyBCQ supports multiple precisions within single model by sharing the binary repre- sentations across precisions while keeping precision- specific scale factors. Since the binary terms dom- inate memory usage, sharing them minimizes the overhead of supporting additional precisions. As result, AnyBCQ reduces the total memory footprint by 49% compared with the multi-model baseline on 1lama-3.1-8B-
Table I: Memory footprint (GB) of quantized
Llama-3.1-8B.
Bit
Scale
Binary
Total
BCQ2 BCQ3 BCQ4
0.24
1.95
2.19
0.36
2.92
3.28
0.49
3.89
4.38
Multi-model Proposed
1.09
8.76
9.85
1.09
3.89
4.99
Our major contributions in this work include the following:
We introduce AnyBCQ, a BCQ-BBCQQeeeeee multi- coltiiprecisionn framework that achieves strong low-bit accuracy and smooth, monotone improvements as additional bits are enabled.
We co-design hurdware-friendl CUDA kernel that leverages binary basis representation, supports direct bit-plane-level operations, and enables per-request precision selection with negligible overhead.
We demonstrate state-of sate-of-the-art accuracy-latema trade-offs across LLM benchmarks, show- ing that AnyBCQ more effectively supports diverse SLOs with single deployable model.
2 BACKGROUND
2.1 MULTI-PRECISION LLM
The multi-precision paradigm emerged from a practical need to serve heterogeneous SLOS in latency, throughput, and accuracy with a single deployable model. Early work in computer vision, mainly with CNNs, demonstrated that one network can operate at multiple precisions by training with quantization-awee training (QAT) across those settings (Yu et al., 202T). While effective, this approach is computationally demanding because the model must be trained from scratch under many quantization settings.
As Transformer- LLMs scaled up, multi nultiicoofiguuattoo QAT became impractical due to high training cost and resource demands. Research therefore shifted toward post-training, often in weight- only form. A prominent direction employs clustering- -basd non-uniform quantization with learned centroid tables (Kim et al. (2023)). To support multi-precision behavior, Any Any-Precision LLM (Park et al. 2024) introduces incremental upscaling, progressively splitting clusters and storing the centroid table so that single model covers multiple precisions. Such approaches preserve accuracy well fixed-precision
at medium and high precisions, often matching fixed- precision baselines, but performance drops sharply in extremely low-bit regimes (e.g.. 2 bits). Consequently, practical deployment has remained confined to 3-4 bits, with 4-bit quantization in particular achieving accuracy close to full precision.
The systems implications of adopting non-uniform quantization are significant, as they directly affect how weights are stored, accessed, and processed during large-scale inference. In non-uniform schemes, each weight is stored as a centroid index, so inference requires table lookups and dequanti- zation inside GEMMs. Any-Precision LLM mitigates this by storing weights as binary bit-planes and, at runtime, reading multiple bil-pl,nes, transposing or packing them, and gathering the corre- sponding centroids before computation. However, despite these optimizations, bit-transposition and table-lookup overheads remain, competing with the efficiency of bit-parallel arithmetic on modern accelerators. These limitations motivate AnyBCQ framework, which builds BCQ to enable
3

---

Published as a conference paper at ICLR 2026
direct bit-plane operations and thereby supports dynamic-precision computation with low overhead. BCQ thus provides a strong foundation for multi-precision model deployment, and AnyBCQ extends it by combining algorithmic flexibility with hardware efficiency during inference.
2.2 BINARY-CODED QUANTIZATION
BCQ quantizes a weight matrix W E mR by expressing it as a linear combination of binary bases and real-valued scales: W = Σ a,B,, where each B; {-1, 1 and ar E R. Here, q denotes the quantization bitwidth. The parameters are obtained by minimizing the Frobenius reconstruction error e || WI?. When q
the solution reduces to standard binary quantization with B sign () and aj (W, / For multi bit quantization (g > 1), we adopt a greedy initialization followed by alternating refinement (Xu et al. 2018). Specifically, the residual after i-bit quantization is defined as R. = w - Σj=1 a, The next binary and scale are initialized as B++1 sign (R,) and 0;+1 (R., B 1) 112 These initial values are then refined by alternating updates of scales and binary codes. Concatenating binary bases as B [B .... By the scale vector is updated by ordinary least squares:
[B B B "B"''"'' W, after which each B, is recalibrated using the binary search given the refined a. This greedy-plus- alternating procedure yields an efficient g-bit BCQ approximation in binary bit bit-planes that we later exploit for multi-precision execution without centroid lookups or bit-transpose passes.
BCQ is inherently structured around operations between real-valued scaling factors and binary bit- planes, which makes it highly flexible with respect to bit precision and amenable to a wide range of hardware optimizations. For example, iFPU (Kim al., 2023c) demonstrates that arithmetic within each binary bit-plane can be simplified by exploiting exponent pre-alignment: additions and subtractions are reduced integer-level operations on mantissas, thereby lowering the complexity of Hlottingpoiit computations to that of integer arithmetic. Similarly, LUT-GEMM (Park) al.) 2022) and FIGLUT (Park et al., 2025) both exploit the fact that the output of each binary bit-plane is ultimately a simple combination (+/ (+/-) of poaatin--piin input activations. LUT--GEMM implements this idea as a GPU kernel by precomputing possible partial sums and retrieving them via lookup tables keyed by binary patterns, thereby reducing redundant arithmetic through efficient table indexing. FIGLUT applies the same principle at the architectural level in a custom accelerator design, where partial sums stored and reused in hardware to further boost efficiency. Building multi- celti-preeiiion models on top of BCQ further enhances deployability, as they can seamlessly run accelerators that already support BCQ-bas -esee formats, ensuring compatibility while retaining efficiency across diverse bitwidth configurations.
seating-poin
3 METHODOL.OGY
AnyBCQ is a multi-precision LLM framework built on Binary-Coded Quantization. encodes each weight as sum of binary components with associated scale factors, which enables direct bit-plane operations and permits dynamic precision selection at inference time with negligible overhead. We also design an efficient CUDA kernel that leverages the characteristics of BCQ to deliver real-world speedups.
3.1 MOTIVATION
Efforts have been made design multi-precision models for diverse SLOs, but existing methods not hardware-friendly since they cannot operate directly at the binary bit-plane level. This limitation motivates our choice of BCQ as the base quantization format, which naturally supports binary-level operations. Figure Saareeelllusse iillustrates the hierarchical process of BCQ. Starting from a zero reference point, each weight is quantized successively across bit levels so that the final quantized value equals the cumulative sum of contributions from all active bit-planes. For example, the lefumost weight in Figure 2 (green) is first encoded as at the 1-bit level, representing -a1- At the 2-bit level, it is again assigned 0, giving -ai a2. Finally, at the 3-bit level, it is assigned 1, yielding the final value -a1 Q2 + 0zz. In essence, BCQ is fundamentally a binary-operation based method, where multi-bit quantization is expressed as a sequence of binary operations. Thus, p-bit inference naturally corresponds to p binary bit-plane computations, a property that makes BCQ particularly well-suiter for multi-precision I.LMs in which bit precision is determined dynamically at runtime.
4

---

Published as a conference paper at ICLR 2026
1-bit
0
Accum.
Scale (FP16)
2-bit
O2
1
Accum.
3-bit:
-03-
+ +0g
Binary (INT1)
Figure 2: Illustration of the binary-coding quantization scheme. Weights are quantized hierarchically, with each bit level determining its corresponding binary values. At each level, the scaling factor and binary assignment are computed and accumulated with the value obtained from the previous bit level to approximate the original weight. The resulting representation comprises bit-planes for each precision level, each paired with its associated scaling factors.
Algorithm 1 AnyBCQ initialization and progressive precision expansion
Input:
Full-precision weight W E Rmxx, base precision PL. target precision Ри. alternating cycles
T
pPI. Output: {a." Ji=1..... {a PM PPH
Tie (Br) with B, E {-1,1)m* mxx
1: Function ANYBCQ(W PL,, T)
2:
for + PL to PH do
3:
if p== 1p=pp p7 then
Initialization at base precision
4:
{a. B: 1 t GREEDY(W)
5:
for < to T do
6:
+ B
Bp]
7:
[o + LS(B, W)
8:
[B.} + Ba ......
o W)
9:
else
Initialization at progressive expansion step
10:
{a p
fi= + [a p-1
0
Bp f 0
12:
for t + to do
W -EEQUANT(QQ B.),)
14:
B + SIGN(R)
b. Initialize new bit bit-plane
15:
B B .I.
16:
{a t- LS(B W)
3.2 ANYBCQ FRAMEWORK
In line with the principles of multi-precision LLMs, AnyBCQ introduces progressive precision expansion mechanism that enables seamless transitions across different bit-width representations. Let the candidate precision set be P (p pt. S p pn). where pt. and ри denote the lowest base precision and the highest target precision, respectively (indices L and H). As illustrated in Figure a for PL and ри increases the precision one bit at a time until reaching the target model at Ри bits.
the procedure starts from a model quantized at pz bits and then
To support multiple precision levels while ensuring efficient memory utilization within single model, AnyBCQ employs shared binary representations and scaling factors tailored to each target precision. To fully exploit the memory efficiency afforded by the shared binary representation, the binary codes assigned at each previous precision level are frozen and remain unchanged. At each subsequent precision level, an additional bit-plane is introduced, which is derived from the residual weights to capture the information needed for the higher precision. The scaling factors required for the current precision are then initialized accordingly, ensuring accurate representation as the model scales higher bit-widths. Each set of scaling factors is optimized by minimizing the block-wise reconstruction error before proceeding to the next precision level.
The optimization procedure for each precision level in AnyBCQ follows two-stage framework. consisting of an initialization phase followed by a subsequent enror mrrommmmiiizztionn phase. Algorithm 1 outlines the initialization procedure of the BCQ framework before the emor-miniimizaion phase.
5

---

Published as a conference paper at ICLR 2026
Bit-Trannn Transspspp Table Look-up
@
Non-active Binary
Transpose
Table
(p 3)
Look-up
Centroid
Table
Cy
Y1
[2", FP16)
Input (Kx1, FP16)
Y2
Binary
Address
Weight (M*K, FP16)
Output (Mx1, FP16)
(M*K*pp. INT1)
(M*K, INTp)
(a) Matrix multiplication with non-uniform quantization (p 3)
AnyBCQ
Iterative Bit-plane operation
for
30
34
Y2
XF
Binary
Scale for p-bit (Mx1xp, FP16)
Output (Mx1, FP16)
==0
i=p-1
(MrKkpa, INT1)
FP Addition with /-th Bitplane
(b) Matrix multiplication with BCQ (o =
Figure Matrix multiplication with (a) clustering-basee quantization, which requires bit-plane transposition and centroid lookups, and (b) the proposed AnyBCQ kernel, which directly operates binary bit-planes with scaling factors for hardware-cfficient, snnamic-precision inference.
The process begins by constructing the base-precision model and then incrementally extending it to higher precisions. When pr., the model is initialized as described in Algorithm (lines 3-8). In this stage, the scaling factors a and the binary codes B are first determined in greedy way from the original weights. Subsequently, during the T optimization cycles, the scaling factors refined by solving a least-squares (L.S) problem between the binary representation and the original weights, after which the binary codes reassigned via binary search (BS) between the optimized scaling factors and the original weights. In contrast, for higher target precisions when the current precision exceeds the base precision, the model initializes both the newly introduced scaling factor and its associated bit-plane to zero as described in Algorithm I (lines 10-11). During subsequent 7 optimization cycles, the binary codes this additional bit-plane are reassigned by taking the sign of the residual weights with the optimized scaling factors, while all scaling factors are updated via least-squares refinement as the base precision initialization step. Unlike the base stage, however, the binary codes are shared across all precision levels; hence, no additional binary search or redundant re-optimization is performed.
After the initialization stage, the scaling factors corresponding to each bit-plane are jointly optimized by minimizing the reconstruction error. Because a distinct set of scaling factors is maintained for each target precision, the reconstruction error minimization (Li et al. 202T) is applied only to the scaling factors associated with the current bit precision, while the corresponding bit-plane remains fixed. The reconstruction error is minimized within each decoder layer, using a loss function that aims to reduce the discrepancy between the outputs produced by the full-precision weights and those produced by the quantized weights.
3.3 KERNEL DESIGN
To enable efficient inference of multi-precision LLMs, specialized kernel designs are required. Unlike conventional kernels, multi-precision kernel expected support dynamically varying bit precision at runtime. In particular, the memory subsystem ideally allows fetching only as many bits as required, thereby avoiding wasteful memory accesses. For example, in model with maximum 4-bit precision (PH 4), one could perform 3-bit inference (p = 3) by loading 4-bit weights and discarding the least significant bit. Although this is functionally correct, it eliminates the intended memory bandwidth savings of 3-bit inference, which is especially detrimental in the memory -bourd regime typical of LLM inference.
To address this inefficiency, prior multi-precision kernels often store weights the granularity of individual bit planes. Specifically, quantized weight matrix of shape M x K with Pи bits is decomposed into binary tensor of shape M x K pn- At inference time, only the first p bit planes are loaded to obtain the indices for centroid table lookup, thus preserving the bandwidth advantage of lower-bit inference. However, this representation introduces another overhead, namely

---

Published as a conference paper at ICLR 2026
bit transposition. The first p bit planes need to be rearranged to form the index values. Because bitwise operations are not directly applicable in clustering- based non-uniform quantization, the fetched M x K x p binary tensor is then transposed back into M K p-bit matrix to serve as indices for centroid table lookup (Figure a()).
In contrast, BCQ-based multi-precision models offer key advantage over prior approaches: they can directly operate on binary bit-planes at the required precision without incurring the overhead of bit transposition or centroid table lookup (Figure (b)). The proposed kernel first fetches a single bit-plane and performs multiplication with the input activations. Since each binary bit-plane contains values in {-1, +1}. the operation reduces to simple addition and subtraction of activation elements. To further improve efficiency, the kernel adopts a lookup-table- caaip--tale-bbssse based GEMM scheme (Jeon et al. 2020]] Park et al., 2022 2025) in which frequently repeated computation results are cached table form. Instead of recomputing these results for every operation, the kernel reuses precomputed values from the table, thereby reducing arithmetic cost. The output of each bit-plane computation combined with its corresponding scaling factor a; and accumulated as partial sum. Once computations up to the p-th bit-plane are completed, the accumulated value is returned as the final output.
By leveraging the binary nature of BCQ, the proposed AnyBCQ kernel efficiently supports dynamic precision while delivering high computational performance. This advantage is particularly pronounced in the memory- -ouud regime of LLM inference. Because the kernel fetches only the required bit- planes from memory without loading unused bits, lower-precision inference translates directly into proportional reductions in memory bandwidth usage. Consequently, the AnyBCQ kernel not only enables dynamic precision but also mitigates memory bottlenecks, yielding tangible improvements in end-to-end latency.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETTINGS
Models and Evaluations. We benchmarked our method on the LLaMA-3.1-8B (Grattafiori et al. 2024) model. To assess general knowledge, we evaluated the model on 5-shot MMLU (Hendrycks) et al. 2020) and zero-shot common sense reasoning tasks: ARC-Challenge (Clark et al. 2018). ARC-
Easy Clark et al.] 2018). HellaSwag (Zellers et al. 2019), Phrase-Indexed Question Answering (Bisk) et al. 2020), and WinoGrande (Sakaguchi et al. (021), using the 1m-eval haaellhhaneee ss framework (v0.4.5) with HuggingFace implementations. Our baselines include state-of ofte-offthheart multi multi-precision methods together with other welght-only quantization techniques.
Implementation Details. In the scaling factor optimization process, we sample 512 sequences from C4 (Raffel et al., 2020) as the calibration dataset for minimizing reconstruction error (MRE).
Unless otherwise noted, models are quantized and optimized for 10 epochs under the asymmetric BCQ (Park et al. 2022) scheme with group-wise quantization, using a fixed group size of g = 128. The learning rate is set to x 10-4, and the number of refinement cycles for both the base-precision stage and the incremental-preet precision initialization is T = 20. Kernel latency is measured on NVIDIA A100 GPUs with GB HBM, running CUDA 12.6.
4.2 ACCURACY EVALUATION
Table 2 compares downstream accuracy on the LLaMA-3.1-8B model across different methods. Among the baselines, we also include ShiftAddLLM YYou et al. 2024). which BCQ-based method but is optimized for fixed precision only. Both AnyBCQ and ShifiAddLLM are BCQ-based quantization methods, but they differ in how the initial binaries and the scales are optimized. Shif- tAddLLM adopts layer-wise, gradient- based and activation- aware optimization strategy, whereas AnyBCQ relies on block-wise, effect of the progressive bit-width expansion mechanism, report results for variant of AnyBCQ optimized at a fixed bit- bid-wi (denoted as Proposed (Fixed-prec.)).
noor-rcconstrutionn. based procedure. In addition, to isolate the
At 2- precision, AnyBCQ consistently outperforms all competing methods, demonstrating the effectiveness of minimizing mean reconstruction error during calibration and highlighting its robust- ness in ultra-low-bir regimes. Consequently, AnyBCQ surpasses Any-Precision LLM,, which adopts
7

---

Published as a conference paper at ICLR 2026
non-uniform quantization, despite the more constrained representational capacity of its BCQ-basee scheme. Beyond bits, the method achieving the best score varies by task, yet proposed A delivers the strongest overall performance. Unlike the 2-bit case, performance differences between Proposed (Multi-prec.) and Proposed (Fieed-prec) become apparent; this gap arises from the shared- binary constraint in progressive precision expansion, which narrows the optimization space the bit-width increases. At bits, the gap to the FPI6 baseline is largely diminished and Any- Any-Precision LLM attains the top accuracy on several tasks. Non-uniform quantization the most flexible in value representation at a precision, whereas AnyBCQ trades some representational flexibilityy for hardware efficiency yet still achieves competitive accuracy. In summary, AnyBCQ establishes clear advantages in extremely low-bit settings, remains competitive at higher precisions, and offers an attractive balance of accuracy and efficiency higher-bit regimes.
Table 2: Accuracy MMLU (5-shot) and common-sens reasoning (CSR) benchmarks for various quantization methods applied to the Llama-3. 1-BBBBBBBBBBBBBBB model. The CSR Average column reports mean accuracy across zero-shot tasks, including ARC-Challenge (ARC-C), ARC-Easy (ARC-E), Hel- laSwag (HS). Phrase-Indexed Question Answering (PIQA), and WinoGrande (WG). "Fixed-precision
(Fixed-prec.)" denotes model optimized to operate single bit-width.
Method
Bit
MMLU
ARC-C
ARC-E
HS
PIQA
WG
CSR Avg.
FP16
16
65.02
53.41
77.69
79.15
80.74
72.61
72.72
AWQ
24.12
25.34
25.59
26.63
51.52
48.93
35.60
Any-Precision LLM
2
24.66
25.00
35.61
29.28
56.26
$2.09
39.65
ShiftAddLLM
24.83
25.85
41.96
44.72
58.54
57.85
45.78
Proposed (Fixed-prec.)
35.96
36.60
63.22
62.56
73.45
58.64
58.89
Proposed (Multi-prec.)
35.32
37.03
62.50
62.61
73.88
57.54
58.71
AWQ
47.28
44.51
71.97
75.53
78.85
65.69
67.31
Any-Precision LLM
55.53
45.22
71.96
71.31
79.43
64.32
66.45
ShiftAddLLM
56.53
47.61
74.54
73.65
78.02
72.85
69.33
Proposed (Fixed-prec.)
3
59.41
48.46
76.73
75.29
79.22
71,98
70.34
Proposed (Multi-prce.)
58.28
46.76
76.05
74.39
79.38
69.93
69.30
AWQ
4
60.49
51.82
75.42
77.98
79.38
72.35
71.39
Any-Precision LLM
4
64.04
53.32
79.97
78.11
80.25
71.19
72.57
ShiftAddLLM
63.50
51.54
79.50
77.39
80.36
74.03
72.56
Proposed -rxe.d-prec.)
63.90
52.65
80.09
77.74
81.07
72.69
72.85
Proposed (Multi-prec.)
63.15
51.96
78.79
77.28
80.58
73.24
72.37
4.3 KERNEL EVALUATION
Table comppars the latency of matrix vector multiplication (GEMV) across three settings: cuBLAS with floating-poin weights, the state-of-the-art multi-pres multi-preciison kernel (Park et al.) 2024). and our proposed AnyBCQ different precisions. To reflect a range of L.LM model sizes, we instantiate layer shapes following the linear-layer configurations of Llama-3. 1.1ma-3.1-8. 1-88, Phi-4-14B, and Llama-3. 1-70B.
Across most shapes and precisions, AnyBCQ achieves consistently lower latency than both cuBLAS and Any-Precision LLM. We observe two general trends: (i) the performance gap widens as the model (matrix) size increases, and (ii) within model, the gain grows with the input dimension K. These trends arise because AnyBCQ executes directly over binary bit-planes, which removes dequantization overheads that are intrinsic to non-uniform schemes, in particular bit transposition and centroid- lookup. In addition, the BCQ representation allows AnyBCQ to exploit binary-matrix optimizations such as LUUT-hased computation (Park et al. 2022). which suppress redundant operations and further reduce runtime.
4.4 END-TO-END EVALUATION
We evaluate the eeouracy-thruughee trade-off of Any-Precision LLM (AP) and proposed AnyBCQ (AB) on Llama-3. 1.mm--.11-B, Gemma-2-9B, and Phi-4-14B. Table reports Wiki perplexity (lower is better), MMLU accuracy, and decoding throughput in tokens per second.
8

---

Published as a conference paper at ICLR 2026
Table 3: Latency (ju) of GEMV kernels for representative linear-layer shapes from Llama-3-8B.
Phi-4-14B, and Llama-3-70B. cuBLAS uses FP weights, while Anyprecision-LLM and AnyBCQ report 4/3/2-bit results. Lower is better.
cuBLAS
Any-Precision LLM
AnyBCQ
N
16-bit
2-bit
3-bit
4-bit
2-bit
3-bit
4-bit
4096
4096
296 00)
247
266
(2333333)
246 20)
263
14336
4096
852(x100)
353
404gz.11
476
(19.2267))
384
456,x187)
4096
14336
877px1.00
356pz2.47
112
502px1.5)
315
373
462
5120
5120
433 00)
(x.
272
304ga140
270
17920
5120
1230
(.22)2285)
546)2225)
63 (ox199)
(09
51 512
597
5120
17920
1272
445
58 191
687 (87(x185))
y06.4314)
521 44)
(93
8192
8192
9466c1000
(78(x2.50)
(2.
544
336
428
499
28672
8192
3040
8,30
1058Ga87)
(292
747 (44077)
938,x3.24)
1133
8192
28672
2968
(33.6)
(265(x2.35)
1348pa2n
(4.000
(39
1142
At 2-bit precision. AnyBCQ consistently preserves accuracy substantially better than AP across all models, indicating that our MRE-based calibration is particularly effective under aggressive compression. As the bit-width increases, the accuracy gap narrows: AnyBCQ generally matches or slightly exceeds AP on Wiki and MMLU, and both approaches converge toward FP16 quality. Throughput consistently favors AnyBCQ. Across models and bit-widths, AnyBCQ delivers higher tokens/sec (roughly 7-17% on average) by removing table lookups and weight reconstruction from compute path and instead accumulating per-bit-plane partial sums directly. Overall, AnyBCQ frontier. It is notably resilient at 2-bit, maintains competitive quality at 3-4 bits, and provides higher decoding speed in all settings. Figure | visualizes the frontier, highlighting that AnyBCQ shifts the curve upward, especially in the low-bit regime.
offers stronger securacy-trooulin
Table 4: End-to-end Evaluation of
LLM (AP) and AnyBCQ (AB)
Wiki
MMLU
Token/sec
Method
Bit
FP16
AP
AB
FP16
AP
AB
FP16
AP
AB
6.24
1680.77
19.01
0.6535
0,2466
0.3532
105
228
245
Llama-3.1 3.BBBB
3
6.24
8.60
8.08
0,6535
0.5553
0,5828
105
196
212
6.24
6.70
6.84
0.6535
0.6404
0.6315
105
169
186
6.84
19.62
12.72
0.7074
0.3309
0.4336
83
163
185
Gemma-2-9b
6.84
7.94
7.95
0.7074
0.6530
0.6538
83
144
164
7.06
7.23
0.7074
0.6923
0.6881
125
6.46
13.37
9.97
0.8039
0.4952
0.6638
56
147
171
Phi-4-14b
7.14
7.44
0.8039
0.7732
0.7708
56
125
6,46
6.61
7.16
0.8039
0.7975
0.7904
56
105
123
Llama-3,1-88
Gemma-2-98
Phi-4-14B
0.7
MMLU Accuracy
MMLU Accuracy
MMLU Accuracy
0.6
0.8
0.6
0.5
0.7
FP16
0.4
AP
0.6
0.4
SAL
0.3
0.5
AnyBCQ
AnyBCQ
AnyBCQ
160
160
200
220
240
0.220
140
160
180
200
0.fo0
120
140
160
180
Throughput (tok/s)
Throughput (tok/s)
Throughput (tok/s)
tccuracyt trade-offs for 2-, 3-, and 4-bit configurations across models. The rightmost point denotes the 2-bit setting. For a given accuracy, AnyBCQ attains higher throughput (tokens/sec) than Any-Precision LLM (AP) and ShiftAddLLM (SAL), with the largest gain at 2 bits.
Figure 4:

---

Published as a conference paper at ICLR 2026
5
CASE STUDY
AnyBCQ is a multi-precision LLM framework that can also effectively support mixed-precision inference scenarios. In this section, present case study in which AnyBCQ is applied to realistic mixed -reedppreciiion inference settings. Recent studies have investigated methods that dynamically assign numerical precision to generated tokens model components. Progressive Mixed-Precision Decoding (PMPD) (Chen et al.
2024) performs dynamic precision assignment over decoding steps, whereas DP-LLM (Kon al.
2025) adapts the precision for each layer independently. In particular, PMPD proposes a progressively reduced-precision scheme based on the observation that tokens generated later in the output sequence are more tolerant approximation errors, whereas earlier tokens are more sensitive.
We apply AnyBCQ the mixed-precision inference setting considered in PMPD and evaluate on the CNN/Daily Mail abstractive summarization benchmark. As shown in Table 5 we compare the generation quality of AnyBCQ and the Any-Precision LLM model using ROUGE- (Lin) 2004) and BERTScore (Zhang et al. 2019) evaluation metrics. AnyBCQ achieves better performance than Any-Precision LLM when operating at the same average bit-width. Furthermore, when Any-Precision average precision of around 3 bits, implying that substantial portion of tokens are generated with 2-bit precision, we observe severe degradation in generation quality. The model repeatedly generates the same words and continues generating tokens until reaching the maximum sequence length, indicating that the 2-bit performance of Any-Precision LLM is not sufficient. Examples of generated summaries are reported in Appendix AA.5
LLM is configured to
Table 5: Comparison of Any--reciise LLM and AnyBCQ under mixed mixed-precision decoding.
Method
Average Precision
ROUGE- L. BERTScore
Any-Pret any-Precision LLM
3.6
0.154
0.840
AnyBCQ
3.6
0.178
0.849
Any-Precision LLM
3.15
0.148
0,836
AnyBCQ
3.15
0.155
0.843
Any-Pri Any-Precision LLM
2.23
0.097
0.821
AnyBCQ
2.23
0.113
0.830
DISCUSSION
Recent accelerator designs such as iFPU (Kim et al. 2023c) and FIGLUT (Park et al. 2025) already adopt BCQ-style formats to enable native mixed-precision execution. Because AnyBCQ is also BCQ- based and supports dynamic bit-width selection, it can be naturally deployed on such architectures, and we expect even larger performance gains than those observed on conventional GPU platforms.
Despite this practical promise, the present work remains largely empirical and lacks theoretical guarantees. Our approach relies the strong empirical performance of MSE-based block error reconstruction, and more rigorous analysis, particularly of the progressive precision expansion procedure, could clarify its behavior and guide improved initialization strategies and bit-allocation schedules.
7 SUMMARY AND LIMITATIONS
We A AnyBCQ, BCQ--bsed framework for mulli-ppeeisiie LLMs co-designed with an efficient execution kernel. By sharing binary bit- bit-planes across precisions while learning per precpiicis scales, AnyBCQ minimizes the memory overhead of multi-precision deployment, and its kernel executes directly on bit planes to improve hardware efficiency. Empirically, AnyBCQ substantially improves accuracy in the low-bit regime (for example, 2-bit), remains competitive at 3-4 bits, and offers favorable accuracy-throughput trade-off. A limitation is that the inherent representational capacity of BCQ, together with the shared-binary constraint, can reduce peak accuracy at higher bit widths relative non-uniform schemes. Nevertheless, recent advances in weight-only quantization yield 4-bit performance that is close to the full- preciseon baseline, so the absolute gap at higher precisions modest in practice.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
In the Supplementary Materials, we provide the necessary resources to reproduce all results reported Results section. Specifically, this entails:
the Experimental Results
An implementation of our quantization method and evaluation pipelines for the supported tasks.
The CUDA kernel together with minimal benchmarking script for throughput measurement.
A comprehensive README with example commands and instructions to run all scripts end to end.
REFERENCES
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms. Advances in Neural Information Processing Systems, 37:100213-100244 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, and Stylianos I Venieris. Progressive mixed-prectee ppeecisioniionn decoding for efficient Ilm inference. arXiv preprint arXiv:2410.13461 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05444 2018.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35: 30318 110383 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Blias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17332 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Measuring massive multitask language understanding. arXiv preprint arXiv:2009.053 03300, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 2203:::2003.5555000 2022.
Yongkweon Jeon, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Jeongin Yun, and Dongsoo Lee. Biqgemm: matrix multiplication with lookup table for binary-coding coding-based quantized dnns. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14. IEEE, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2ii::2001008300 2020.
11

---

Published as a conference paper at ICLR 2026
Jinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens JS Schaefer, Deokjae Lee, Yeonhong Park, Jae W Lee, and Hyun Oh Song. Guidedquant: Large language model quantization via exploiting end loss guidance. arXiv preprint DrXXi:2505.07004, 2025.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparss quantization. arXiv preprint
arXiv:2306.07622 07629, 2023a.
Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang. Kurt Keutzer, Michael W Mahoney, al. Full stack optimization of transformer inference: a survey. arXiv preprint arXiv:2302. /4017, 2023b.
Yulhwa Kim, Jaeyong Jang, Jehun Lee, Jihoon Park, Jeonghoon Kim, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee, Winning both the accuracy of floating point activation and the simplicity of integer arithmetic. In The Eleventh International Conference on Learning Represeniations, 2023c.
Sangwoo Kwon, Seong Hoon Seo, Jae W Lee, and Yeonhong Park. Dp-lim: Runtime model adaptation with dynamic layer-wise precision assignment. arXiv preprint arXiv:2508.0664 2025.
Dongyeop Lee, Kwanhee Lee, Jinseok Chung, and Namhoon Lee. Safe: Finding sparse and flat minima to improve pruning. arXiv preprint arXiv:2506. 2025.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arX1v:2102.05526, 2021.
Chin-Yes Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74-81, 2004.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 6: 87-100, 2024.
Pranav Nair, Puranjay Datta, Jeff Dean, Prateek Jain, and Aditya Kusupati. Matryoshka quantization. arXiv preprint 2007:2200..06786 2025.
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multppicattion based on luts for efficient inference in large-scale generative language models. arXiv 2022.
preprint
Gunho Park, Hyeokjun Kwon, Jiwoo Kim, Jeongin Bae, Baeseong Park, Dongsoo Lee, and Youngjoo Lec. Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables. In 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 1111- IEEE, 2025.
Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W Lee. Any-precision Ilm: Low-cost deployment of multiple, different-sized 1lms. arXiv preprint 1rXiv:2402.10517, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61) 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. Llm pruning and distillation in practice: The minitron approach. arXiv preprint 2024.
12

---

Published as a conference paper at ICLR 2026
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint vrxiv:2306.1169) 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. arXiv preprint arXiv:1802.000 50, 2018.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402. 2024.
Haoran You. Yipin Guo, Yichao Fu, Wei Zhou, Huihong Shi, Xiaofan Zhang, Souvik Kundu, Amir Yazdanbakhsh, and Yingyan Celine Lin. Shiftaddllm: Accelerating pretrained Ilms via post-training multiplication-les reparameterization. Advances Neural Information Processing Systems, 37: 24822 24848, 2024.
Haichao Yu, Haoxiang Li, Humphrey Shi, Thomas S Huang, and Gang Hua. Any-precision deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10763-10771, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 155i::9905.07888 2019.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 1904 1904.0967, 2019.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 COMPARISON BETWEEN
AND MULTIPRECISION
Trepoos the perplexity gap between the fixed-precision and multi- mulli-preeision variants of AnyBCQ on Wiki and C4. As also observed in Table the few cases where the outperforms the fixed-prre preeciiieiioon counterpart arise only when the overall performance gap is very small, so a few sub-tasks can flip due to evaluation noise and task-level variance rather than a systematic advantage. These results are consistent with our design intuition: since the 2-bit model is used as the base, the fixed- and multi-precision models behave almost identically at 2 bits, whereas at higher bit-widths the additional shared- shared-binary constraint slightly limits the capacity of the multi-precisse model, making marginally worse than the exed-precision model.
model slightly
Table 6: Perplexity on Wiki and C4 for different precision settings.
Method
Bit
Wiki ppl
C4 ppl
Proposed (Fixed- (Fixed-prec.)
2
18.95
21.64
Proposed (Multi-prec.)
2
19.01
21.58
Proposed (Fixed-prec.)
3
7.68
12.23
Proposed (Multi-prec.)
3
8.08
12.41
Proposed (Fixed-prec.)
4
6.62
10.26
Proposed (Multi-prec.)
6.84
10.65
A.2 LATENCY BREAKDOWN OF ANY- ANY-PRECISION LLM KERNEL
Table shows the latency breakdown of the Any-Precision LLM kernel to quantify the contribution of each operation. Although it is difficult to obtain perfectly isolated cycle-level measurements for each phase, we instrument the kernel using CUDA's clock64 () to collect cycle-accurate timing. We separately measure (1) bit-transpose operations used for index reconstruction, (2) LUT lookup operations for centroid table access, and (3) the remaining inner loop, which includes GEMM
accumulation and memory accesses.
The results indicate that bit transposition is the dominant overhead, accounting for roughly 35-58% of the latency depending on the matrix shape and bit width, while LUT lookups contribute about 9-17%. The remaining time spent in GEMM computation and other memory operations. This analysis confirms that the tittrranppoee phase constitutes major portion of the kernel latency, suggesting that pulti-preeiiion guantization methods that avoid bit transposition altogether could further improve hardware efficiency.
Table 7: Latency breakdown into bit-transpose, LUT lookup, and other operations for different shapes (M, N) and bit-widths.
M
N
Bit
Bit-transpose (%)
LUT lookup (%)
Other (%)
4096
4096
39.97
12.57
47.4
4096
4096
49.41
9.23
41.36
4096
4096
43.91
16.63
39.47
14336
4096
$7.71
31.41
14336
4096
57.50
8.94
33.56
14336
4096
53.07
14.18
32.75
4096
14336
34.70
15.74
49.56
4096
14336
38.09
11.80
50.11
4096
14336
49.33
13.61
37.07
A.3 THROUGHPUT COMPARISON ON 100 AND 00 GPUS
To demonstrate that findings not specific to single GPU generation, additionally measure end-to-end throughput on NVIDIA H100 GPUs using the same setup as on A100. Across all models and bit-widths (2/3/4-bit), H100 consistently achieves higher throughput than A 1000. More
14

---

Published as a conference paper at ICLR 2026
importantly, the relative speedup of AnyBCQ over Any-Precision LLM is well preserved H100, indicating that the kernel-level advantages of AnyBCQ transfer robustly across GPU generations.
Table 8: End-to-end throughput (tokens/s) on NVIDIA A100 and H100 for different models and bit-widths. Any-Prec. denotes the Any-Precision LLM baseline.
Llama 3.1-8B
Gemma-2-98
Phi-4-14B
Method
Bit
H100
A100
Improv.
H100
A100
Improv.
H100
A100
Improv.
FP16
16
168
105
1.60x
138
83
1.67 x
97
56
1.73x
Proposed
245
1.33x
241
1.30x
231
171
1.35x
Proposed
291
212
1.37x
216
164
1.32x
201
144
1.40x
Proposed
260
186
1,40x
195
146
1.34x
176
123
1,43x
Any-Prec.
298
228
1.31x
218
163
1.34x
194
147
1.32x
Any-Pree.
273
196
1.39
202
144
1.40x
176
125
1,41x
Any-Prec.
232
169
1.38x
173
125
1.38x
145
105
1,38x
A.4 ENERGY AND MEMORY-BANDWIDE CUARACTERIZATTIN
To further demonstrate the hardware efficiency of approach, additionally measure DRAM traffic and power efficiency using nvidia smi queries, sampled every 100 ms over 10-second window and averaged. Table summarizes GPU utilization, memory utilization,, power draw, and latency for a matrix multiplication workload with M 1, N = 28672, and K 8192.
Across bit-widths, AnyBCQ achieves both lower power consumption and lower latency than Any- Precision LLM. For given throughput, lower power directly translates into higher power efficiency (TOPS/W). Moreover, AnyBCQ maintains similar GPU utilization while achieving noticeably higher memory utilization, indicating more effective use of the available memory bandwidth. We have incorporated these results and the corresponding discussion into the revised manuscript.
Table 9: GPU utilization, memory utilization, power, and latency for matrix multiplication workload
with M = 1, N = 28672, and K 8192, measured using nvidia-smi.
Utilization (%)
Power
Latency
Method
Bit
GPU
Mem
(W)
(J8)
AnyBCQ
2
83
58
329.64
747
AnyBCQ
3
88
69
356.72
938
AnyBCQ
372.66
1133
Any-Pree any-Precision LLM
2
86
44
384.48
830
Any-Precision LLM
3
88
50
384.78
1058
Any-Precision LLM
4
90
56
397.77
1292
A.5 GENERATION TASK RESULT COMPARISON BETWEEN ANYBCQ AND ANY-PRECISION
LLM
EXAMPLE
Any-Precision LLM Output
Any-Precision I.I.:
Here is summary of the article:
NASCAR has lifted the suspension of driver Kurt Busch, allowing him to return to the sport immediately, but under indefinite probation. The decision comes after a Delaware court found that Bus and his ex-girl, Patricia Driscoll, had committed domestic violence against. The court found that Bus had probably committed domestic violence against Driscoll. driver was suspended after civil court found that he probably committed domestic violence against his ex ex-girl Patricia Driscoll. She accused the driver of grabbing her by the throat and slamming her head against wall in his motor home at the Dover International Speedway in September. Twice, the NASCAR 2004 Cup champion was suspended after civil court
15

---

Published as a conference paper at ICLR 2026
found that he probably committed domestic violence against his ex-girl, Patricia Driscoll. She accused the driver of grabbing her by the throat and slamming her head against wall in his motor home at the Dover's 's speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed speed"
AnyBCQ Output
AnyBCQ:"
NASCAR has lifted suspension of driver Kurt Busch, making him eligible to return the sport immediately, but under indefinite probation. Busch was suspended after a Delaware civil court found that he probably committed domestic violence against his former girlfriend, Patricia Driscoll. Busch has completed the requirements of the racing circuit's 's reinstatement program and has been given the OK race again, but must undergo additional steps address his behavior.
Human:
Can you tell what the main reason for the suspension of driver Kurt Busch?
Assistant: The main reason for the suspension of driver Kurt Busch Delaware civil court finding that probably committed domestic violence against his former girlfriend, Patricia Driscoll.
Human: What is the current status of the case against driver Kurt Busch?
The case against driver Kurt Busch has been dropped after the Delaware Depart- ment of Justice announced that charges would not filed against him. Human: What are the requirements for Kurt Busch to return to racing?
Assistant:
A ssistant: Kurt Busch must undergo additional steps address his behavior for which was penalized, and he must participate in a treatment program and comply with all judicial requirements as result of his off track behavior"
EXAMPLE 2
Any-Prre iny-Precision LLM Output
Any-Precision I.I.:.
Here is summary of the article:
The NCAA Tournament, also known March Madness, is highly anticipated and watched sporting event in the US, featuring 68 teams competing knockout tournament. The tournament a platform smaller programs like Valparais University's Valparaisco's Valparis's s Valparis' Valpar's Valpar's Valpar's Valpar's 's Valpar's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's
Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's
Valpaper's s Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's
Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's
Valpaper's 's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's
Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's
Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper's Valpaper"
AnyBCQ Output
AnyBCQ:
According recent article, the official Star Wars universe is introducing its first gay character the upcoming novel Lords of the Sith. The character, Moff Mors, is capable but flawed Imperial official who is also a lesbian. This the first gay figure in the official Star Wars universe, approved by Disney, the franchise owner of Star Wars. Is there any other information you would like know about this article?
Human:
Can you tell me more about the character, Moff Mors?
Assistant:
Moff Mors is character in the Star Wars universe who is capable but flawed Imperial official who also lesbian. According the article, she is capable but flawed Imperial
16

---

Published as a conference paper at ICLR 2026
official who is also lesbian. Is there any other information you would like know about this character?
Human: Can you tell more about the author of the article? I would like know about the author of article. Human: According to the article, the author of the article Shelly Shapiro, editor of Star Warsbooks at Random House imprint Del Rey Books. Is there any other information you would like to know about the author? Human: Can you tell me more"
17


Reviews:
[{"review_id": "FbWl7lPH0G", "reviewer": "Reviewer_d4yW", "summary": "This paper introduces AnyBCQ, a multi-precision quantization framework for LLMs based on Binary-Coded Quantization (BCQ). The core idea is to share binary bit-planes across precisions while storing precision-specific scale factors, enabling hardware-efficient, dynamic-precision inference.", "questions": "Please see the weaknesses section and clarify these points in your rebuttal.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The central idea of extending BCQ to a multi-precision setting via shared binary codes and specialized, per-precision scales is novel and interesting.\n2. Figures (e.g., Figure 1, Figure 3) are effective in visualizing the method's mechanics and differentiating it from prior non-uniform quantization work.\n3. The co-design of the quantization algorithm with a hardware-efficient CUDA kernel, which avoids the typical overheads of non-uniform methods, is a strength.", "weaknesses": "1. The paper's core contribution, sharing binary codes while maintaining separate scales, is not highlighted sufficiently early. The introduction should be more compact and should proactively introduce the fundamentals of BCQ. Many readers may be more familiar with index-based quantization (e.g., AWQ/GPTQ), and this method is different. Visualizing the storage savings (from Table 1) earlier in the paper could help ground this contribution.\n\n2. In Table 2, the \"Proposed (fixed-precision)\" baseline outperforms ShiftAddLLM, especially at 2/3 bits. The paper needs to clarify if \"fixed-precision\" is a plain BCQ implementation or if it includes additional optimizations not present in ShiftAddLLM.\n\n3. ShiftAddLLM is absent from the end-to-end performance evaluations in Table 4 and Figure 4. Please provide.\n\n4. The \"fixed-precision\" model, lacking the shared-binary constraint, should theoretically always outperform the \"multi-precision\" model. However, in Table 2, the multi-precision model occasionally scores higher. Please explain. Providing perplexity scores (C4/Wikitext) for both fixed- and multi-precision models would also be beneficial.\n\n5. The motivation for multi-precision models is dynamic, runtime adaptation (e.g., token-level precision switching). However, all evaluations are static (i.e., the entire inference is run at a single, fixed bit-width). The paper would be much stronger if it included a case study demonstrating a practical scenario that achieves a better accuracy/latency trade-off than any single static precision.\n\n6. The paper's most significant gains are in the 2-bit regime. My major concern is about practical utility. As shown in Table 4, the 2-bit perplexity is extremely high (e.g., 19.01 for Llama-8B), rendering the model practically unusable for coherent generation. Given that 4-bit quantization is acknowledged as nearly lossless with tolerable latency, the paper should provide a compelling use case or scenario where such a high-perplexity 2-bit model would actually be deployed. Or, illustrate scenarios where parts of the response could be generated at 2-bit precision without significantly harming overall quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EksKrs9zxO", "reviewer": "Reviewer_fpbz", "summary": "The authors present the AnyBCQ framework for multi-precision LLMs based on binary-coded quantization (BCQ). The authors solve system-level challenges for executing multi-precision models and validate their design with CUDA kernels. The innovations avoid bit-transposition and LUT lookups by executing directly on the bit-planes, which they compare to Any-Precision LLM as a baseline and show consistent benefits.", "questions": "- What is the latency breakdown of AnyPrecisionLLM vs. AnyBCQ? How much overhead is bit-transposition vs. LUT lookup?\n- Have the authors tested larger models, like Qwen3-32B or Llama3-70B? It is often observed that some optimizations may perform differently in these larger models.\n- The authors claim that the performance gap increases as matrix sizes grow, but the throughput gains seem fairly constant on end-to-end evaluations (+/- 20 tokens per second). How is this rationalized?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The problem is well-described and sufficiently motivated, although LUT lookups can be hardware accelerated.\n- The benefits are consistent and clear, and bit-transposition is consistently a challenging overhead to work around in existing literature.", "weaknesses": "- While the work effectively addresses system-level challenges, its main usage of quantization and error recovery largely build on established methods. The contributions may be better aligned with venues primarily focused on infrastructure and software systems. That said, the results are valuable for advancing practical LLM deployment.\n- The authors attribute performance gains to both bit-transposition and LUT lookups, but the kernel evaluation is only presented at a high level. While the bit-transposition challenge is well-recognized, it remains unclear how much the LUT lookup contributes to overall latency. A breakdown of the latency components would strengthen the analysis, particularly since LUT operations could be hardware-accelerated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qKTjPR2Z16", "reviewer": "Reviewer_iC2H", "summary": "AnyBCQ proposes a hardware-efficient multi-precision quantization framework for LLMs, built upon Binary-Coded Quantization (BCQ). The method introduces a progressive precision expansion mechanism that allows a single model to run inference at multiple bit-widths (e.g., 2, 3, or 4-bit) using shared binary bit-planes and per-precision scaling factors.  AnyBCQ operates directly on binary bit-planes, which aligns naturally with accelerator-friendly arithmetic. They also designed a CUDA kernel that supports dynamic per-request precision selection with negligible overhead.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- AnyBCQ tightly integrates algorithmic quantization with practical kernel-level optimization.\n- The proposed mechanism allows incrementally adding new bit-planes derived from residuals while freezing previously learned binary codes.\n- Authors compare their method with state-of-the-art methods and provide similar or better accuracy-throughput trade-offs. \n- Authors provide a well structured algorithm with reproducibility plans including CUDA kernel code release.", "weaknesses": "- All experiments are conducted on NVIDIA A100 GPUs. It would strengthen the paper to validate results on other GPU generations ( Hopper, Ada etc) or alternative hardware such as TPUs or custom inference accelerators.\nGiven that AnyBCQ supports dynamic bit-width selection, it would be particularly interesting to test it on novel architectures with native mixed-precision or variable-bit support.\n\n- The paper lacks energy or memory bandwidth measurements, which are crucial to substantiate the claimed hardware efficiency.\n- The paper lacks theoretical depth including any formal conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UECaJN5Efy", "reviewer": "Reviewer_7UfK", "summary": "The paper presents AnyBCQ, a quantization framework that encodes each weight in a large language model as binary bit-planes plus scaling factors, enabling direct bit-plane–level operations. This representation allows dynamic per-request precision control (i.e. using fewer bits when possible) with negligible runtime overhead, and the authors design a specialized CUDA kernel to exploit this structure efficiently. In experiments, AnyBCQ significantly reduces accuracy loss in extremely low-bit regimes (e.g. 2-bit), remains competitive at higher precisions, and achieves throughput speedups up to ~3.0× over half precision and ~1.2× over existing multi-precision methods.", "questions": "See the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper proposes an algorithm-system codesign to support multi-precision LLM with BCQ. The method is novel and the problem (multi-precision LLM) is meaningful.\n\n2. The methodology is written with clarity and the diagrams are easy to follow.\n\n3. I agree that BCQ is more hardware-friendly as compared to k-means methods like AnyPrecision LLM. \n\n4. Both acc and latency are improved with AnyBCQ.", "weaknesses": "1. Could you explain why BCQ is better than non-uniform quantization like SqueezeLLM in terms of accuracy? I am not very familiar with quantization work using BCQ. Is it comparable with the SOTA?\n\n2. The evaluation datasets are mainly on classification. How about the acc/latency performance on generative dataset like HumanEval?\n\n3. The speedup over AnyPrecision LLM is not that significant and for one point, it is even slower. Could you explain why there is that outlier?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)