Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models
Authors: Seungcheol Park, Jeongin Bae, Beomseok Kwon, Minjun Kim, Byeongwook Kim, Se Jung Kwon, U Kang, Dongsoo Lee
Accepted at: acl25

Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models

Seungcheol Park1, Jeongin Bae2, Beomseok Kwon2, Minjun Kim1, Byeongwook Kim2, Se Jung Kwon2, U Kang1*, Dongsoo Lee2
Seoul National University1    NAVER Cloud2
{ant6si, minjun.kim, ukang}@snu.ac.kr

Abstract
How can we quantize large language models while preserving accuracy? Quantization is essential for deploying large language models (LLMs) efficiently. Binary-coding quantization (BCQ) and uniform quantization (UQ) are promising quantization schemes that have strong expressiveness and optimizability, respectively. However, neither scheme leverages both advantages. In this paper, we propose UniQuan_F (Unified Quantization with Flexible Mapping), an accurate quantization method for LLMs. UniQuan_F harnesses both strong expressiveness and optimizability by unifying the flexible mapping technique in UQ and BCQ’s non-uniform quantization levels. We propose unified initialization, and local and periodic mapping techniques to optimize the parameters in UniQuan_F precisely. After optimization, our unification theorem removes computational and memory overhead, allowing us to utilize the superior accuracy of UniQuan_F without extra deployment costs induced by the unification. Experimental results demonstrate that UniQuan_F outperforms existing UQ and BCQ methods, achieving up to 4.60% higher accuracy on GSM8K benchmark.

1 Introduction
How can we compress large language models without compromising accuracy? Reducing the size of large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023; Dubey et al., 2024) is crucial for deploying them in real-world applications since they require expensive computational and memory costs. Quantization algorithms (Xu et al., 2018; Kwon et al., 2022; Dettmers et al., 2022; Xiao et al., 2023; Lin et al., 2024) efficiently compress LLMs via bit-width reduction of weights by encoding them with a small set of values, namely, quantization levels. Each quantization scheme has its own quantization parameters that determine the quantization levels; accurately optimizing these parameters is important because it guarantees that the quantization levels are well aligned with the model’s weight distribution.

Uniform quantization (UQ) (Lee et al., 2023; Lin et al., 2024; Shao et al., 2024) and binary-coding quantization (BCQ) (Xu et al., 2018; Kwon et al., 2022) are promising quantization schemes that ensure fast inference of quantized LLMs. While UQ divides its quantization levels uniformly, BCQ results in non-uniform quantization levels through the addition and subtraction of values assigned per each bit. Research on quantizing LLMs with UQ is actively ongoing, but no study has yet explored BCQ on LLMs as fast acceleration kernels (Park et al., 2024a; You et al., 2024) that support BCQ scheme have been released recently.

We compare the expressiveness and optimizability of quantization schemes for LLMs in Figure 1. UQ demonstrates strong optimizability, efficiently reducing quantization errors, by leveraging various advanced methods such as FlexRound (Lee et al., 2023) and OmniQuant (Shao et al., 2024). However, its expressiveness is constrained, failing to adapt its quantization levels to the weight distribution due to their uniform spacing. Conversely, BCQ offers strong expressiveness through non-uniform quantization levels, but its optimizability is limited due to the absence of precise quantization techniques. ALTERNATING (Xu et al., 2018) is the only BCQ method applicable to LLMs, but its major drawback is that it does not consider the input distribution. In summary, neither scheme fully achieves both strong expressiveness and optimizability.

In this paper, we propose UniQuan_F (Unified Quantization with Flexible Mapping), an accurate quantization method for LLMs. We find that the strong optimizability of UQ originates from its transformation process, and the strong expressiveness of BCQ originates from its generalized
(a) Uniform Quantization (UQ) (FlexRound, OmniQuant)
Weak expressiveness
Strong optimizability
Steep decrement
Steps
Uniform
Error
q1 q2 q3 q4^w

(b) Binary-coding Quantization (BCQ) (ALTERNATING)
Strong expressiveness
Weak optimizability
Slow decrement
Steps
Dense Sparse \ Sparse
Error
q1 q2 q3 q4^w

(c) Unified Quantization (UniQuan) (UniQuan_F)
Strong expressiveness
Strong optimizability
Steep decrement
Steps
Dense Sparse \ Sparse
Error
q1 q2 q3 q4^w

Figure 1: A comparison of (a) uniform, (b) binary-coding, and (c) unified quantization schemes. UQ has strong optimizability swiftly reducing errors during optimization, and BCQ has strong expressiveness adapting its non-linear quantization levels (\( q_i \), bar) according to the distribution of weights (\( w \), circle). UniQuan combines the advantages of both schemes by unifying UQ and BCQ.

mapping process that maps weights to non-uniform quantization levels (see Section 3.1). Building on this observation, we define UniQuan (Unified Quantization) which unifies UQ and BCQ schemes by integrating UQ’s transformation process into BCQ’s mapping process, thereby harnessing both strong optimizability and expressiveness as shown in Figure 1(c). In UniQuan_F, we unify FlexRound and ALTERNATING, the best-performing UQ and BCQ methods, respectively, following the quantization process of UniQuan. We improve the accuracy of UniQuan_F with two main ideas: 1) unified initialization for joint initialization of quantization parameters from FlexRound and ALTERNATING, and 2) local and periodic mapping to accelerate the slow mapping process of BCQ. We further remove the extra deployment cost caused from the unification by integrating the two-step inference process into a single step with unification theorem (see Theorem 1). As a result, UniQuan_F exhibits the best performance without any additional memory and computational costs at deployment.

Our main contributions are as follows:

• Algorithm. We propose UniQuan_F, an accurate quantization method for LLMs. UniQuan_F unifies the best-performing UQ and BCQ methods to leverage both of their advantages.

• Analysis. We analyze that UniQuan_F exhibits both strong expressiveness and optimizability.

• Experiments. We show that UniQuan_F outperforms UQ and BCQ methods on various benchmarks, showing up to 4.60% higher accuracy. The source code for implementing UniQuan_F is publicly available at https://github.com/snuedm-starlab/UniQuanF.

The rest of this paper is organized as follows. Section 2 defines LLM quantization problem and provides preliminaries. We propose UniQuan_F in Section 3. Section 4 presents the experimental results, followed by a discussion of related works

![Illustrations of quantization levels (q) assigned for a weight group w under (a) UQ and (b) BCQ schemes, where wm and wM are the minimum and the maximum weights in w, respectively. UQ has evenly-spaced quantization levels within a clipping range while BCQ has non-uniform quantization levels determined by its scale factors α and a shifting factor z_B.](page_784_563_627_312.png)

Figure 2: Illustrations of quantization levels (\( q \)) assigned for a weight group \( w \) under (a) UQ and (b) BCQ schemes, where \( w_m \) and \( w_M \) are the minimum and the maximum weights in \( w \), respectively. UQ has evenly-spaced quantization levels within a clipping range while BCQ has non-uniform quantization levels determined by its scale factors \( \alpha \) and a shifting factor \( z_B \).

in Section 5. Finally, we conclude the paper with a summary of our findings.

2 Preliminary

We introduce the LLM quantization problem and describe preliminaries. We describe the frequently used terminologies in Appendix A.

2.1 LLM Quantization Problem

We have a pre-trained LLM \( F \), a desired bit-width \( k \), and a sample dataset \( \mathbb{D} \). Our objective is to find an accurate \( k \)-bit quantized model \( \hat{F} \). The target model is a Transformer-based (Vaswani et al., 2017) LLM (Jiang et al., 2023; Dubey et al., 2024) with \( N \) blocks, where quantization is applied to the weight matrices in each block. We divide each weight matrix into multiple weight groups that share quantization parameters. Given a group \( w \in \mathbb{R}^g \) of \( g \) weight values and a desired bit-width \( k \), a quantizer \( Q \) quantizes \( w \) into \( \tilde{w} = Q(w, k; \Theta) \) where \( \Theta \) is a set of quantization parameters. In this paper, we focus on two quantization schemes: uni-
Figure 3: Quantization processes for a weight \( w \) in (a) UQ, (b) BCQ, and (c) UniQuan schemes. Blue-colored processes are parameterized functions which are the source of UQ’s strong optimizability and BCQ’s powerful expressiveness, respectively. UniQuan takes the advantages of both UQ and BCQ schemes by combining the parameterized functions in both schemes. See Section 3.1 for details.

form quantization (UQ) and binary-coding quantization (BCQ) which are supported by the state-of-the-art inference kernel (Park et al., 2024a). We directly compare their accuracy since they require the same memory and computational costs.

2.2 Uniform Quantization (UQ)

Uniform quantization (UQ) (Lee et al., 2023; Shao et al., 2024) is a quantization scheme that has uniformly spaced quantization levels. We explain UQ using Round-to-nearest (RTN) (Gupta et al., 2015), a representative method of UQ. Figure 2(a) illustrates the quantization levels in UQ that are evenly distributed in a clipping range \([w_{m,c}, w_{M,c}]\), where \(w_{m,c}\) and \(w_{M,c}\) are the minimum and maximum values in the clipping range, respectively. RTN begins with a grid search to find the proper \(w_{m,c}\) and \(w_{M,c}\) to precisely approximate the original weights. The quantization parameters \(\Theta_R = \{\Delta, z_U\}\) of RTN are derived based on its clipping range; scale factor \(\Delta = (w_{M,c} - w_{m,c})/(2^k - 1)\) and zero-point \(z_U = [-w_{m,c}/\Delta]\), where \([\cdot]\) is the rounding function. Then, RTN quantizer \(Q_R\) quantizes \(w \in \mathbb{R}^g\) into \(\hat{w} = Q_R(w; k; \Theta_R)\) as follows:

\[
\hat{w} = Clip \left( [w/\Delta + z_U 1_g], 0, 2^k - 1 \right),
\]
\[
\hat{w} = \Delta (\hat{w} - z_U 1_g),
\]

where \(Clip(\cdot, m, M)\) is an element-wise clipping function with a min-max range \([m, M]\), and \(\hat{w}\) is a vector of low-bit integers assigned for each weight. \(1_g\) is a vector of size \(g\) filled with ones.

2.3 Binary-coding Quantization (BCQ)

Binary-coding quantization (BCQ) is a non-uniform quantization scheme with adaptive quantization levels. Its quantization parameters \(\Theta_B = \{\alpha, z_B\}\) consists of a vector \(\alpha \in \mathbb{R}^k\) of scale factors and a shifting factor \(z_B \in \mathbb{R}\). Figure 2(b) illustrates the quantization levels in BCQ that are determined through the summation and subtraction of scale factors \(\alpha\) after shifting with \(z_B\). In this example, BCQ quantizer defines the set \(\{z_B + \alpha_1 + \alpha_2, z_B + \alpha_1 - \alpha_2, z_B - \alpha_1 + \alpha_2, z_B - \alpha_1 - \alpha_2\}\) of four quantization levels forming a binary tree centered at \(z_B\) with widths \(\alpha_1\) and \(\alpha_2\).

We assign a binary code \(c \in \{-1, +1\}^k\) for each weight \(w\), indicating whether the weight selects the positive or negative value for each scale factor in \(\alpha\). Then, BCQ quantizer \(Q_B\) quantizes a weight group \(w \in \mathbb{R}^g\) into \(\hat{w} = Q_B(w; \Theta_B)\) as follows:

\[
\hat{w} = C \alpha + z_B 1_g,
\]
\[
\textit{where } C = \arg \min_C ||w - (C' \alpha + z_B 1_g)||_2^2.
\]

\(C \in \{-1, +1\}^{g \times k}\) is the binary code matrix for \(w\) where its ith row contains the binary code for the ith weight in \(w\). Existing works (Xu et al., 2018) first set \(z_B\) as 0, then search the scale factors \(\alpha\) and binary codes \(C\) through an alternating update process (see Algorithm 2 in Xu et al. (2018)).

The main advantage of BCQ is its strong expressiveness; any UQ is representable in the form of BCQ (see Appendix C in Park et al. (2024a)). However, BCQ methods are far less accurate in quantizing LLMs than UQ methods due to the lack of accurate BCQ-based quantization methods. In this paper, we propose UniQuan_F which transfers UQ’s accurate quantization methods to BCQ to fully exploit BCQ’s strong expressiveness.

3 Proposed Method

We propose UniQuan_F (Unified Quantization with Flexible Mapping), an accurate quantization method for LLMs. We first propose UniQuan (Unified Quantization), a framework unifying the quantization processes of UQ and BCQ schemes to exploit their strong expressiveness and optimizability. Then, we present UniQuan_F-0, a naïve method that unifies FlexRound and ALTERNATING following UniQuan, but with several limitations. Finally,
Table 1: Comparison of quantization methods. UniQuan_F incorporates the advantages of FlexRound and ALTERNATING by exploiting \( \mathcal{T}_F, \mathcal{D}_R, \) and \( \mathcal{M}_B^* \), overcoming the challenges in UniQuan_F-0. \( \circ \) is a function composition operator. See Sections 3.2 for details.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Optimization</th>
    <th colspan="2">Deployment</th>
  </tr>
  <tr>
    <th>Initialization</th>
    <th>Quantization</th>
    <th>Parameters</th>
    <th>Inference</th>
  </tr>
  <tr>
    <td>FlexRound</td>
    <td>Grid search</td>
    <td>\( \mathcal{D}_R \circ \mathcal{M}_U \circ \mathcal{T}_F \)</td>
    <td>\( \tilde{w}, \Theta_R \)</td>
    <td>\( \mathcal{D}_R(\tilde{w}; \Theta_R) \)</td>
  </tr>
  <tr>
    <td>ALTERNATING</td>
    <td>Alternating update</td>
    <td>\( \mathcal{M}_B \)</td>
    <td>\( C, \Theta_B \)</td>
    <td>\( \mathcal{R}_B(C; \Theta_B) \)</td>
  </tr>
  <tr>
    <td>UniQuan_F-0</td>
    <td>-</td>
    <td>\( \mathcal{D}_R \circ \mathcal{M}_B \circ \mathcal{T}_F \)</td>
    <td>\( C, \Theta_B, \Theta_R \)</td>
    <td>\( \mathcal{D}_R(\mathcal{R}_B(C; \Theta_B); \Theta_R) \)</td>
  </tr>
  <tr>
    <th>UniQuan_F (Proposed)</th>
    <th>Unified initialization</th>
    <th>\( \mathcal{D}_R \circ \mathcal{M}_B^* \circ \mathcal{T}_F \)</th>
    <th>\( C, \Theta_B^* \)</th>
    <th>\( \mathcal{R}_B(C; \Theta_B^*) \)</th>
  </tr>
</table>

we propose UniQuan_F which achieves high accuracy without requiring extra costs at deployment, by addressing the limitations in UniQuan_F-0.

3.1 Unification of UQ and BCQ

We define a general representation of the quantization processes in UQ and BCQ schemes to clarify the source of their superior capabilities. Given a quantizer \( Q \) with quantization parameters \( \Theta \), the general representation is defined as follows:

\[
\tilde{w} = Q(w; \Theta) = \mathcal{D}(\mathcal{M}(\mathcal{T}(w; \Theta); \Theta); \Theta),
\]

where \( \mathcal{T} \) is a transformation process which transforms weights \( w \) to transformed weights \( \tilde{w} \), \( \mathcal{M} \) is a mapping process which maps transformed weights \( \tilde{w} \) to the corresponding quantization levels, and \( \mathcal{D} \) is a detransformation process which reverts mapped weights \( \tilde{w} \) to the original weight space, yielding quantized weights \( \hat{w} \). We illustrate the quantization processes in diverse quantization schemes in Figure 3 following this general representation.

Optimizability of UQ. As shown in Figure 3(a), we formulate the quantizer \( Q_U \) of UQ as follows:

\[
Q_U(w, k; \Theta_U) = \mathcal{D}_U(\mathcal{M}_U(\mathcal{T}_U(w; \Theta_U), k); \Theta_U),
\]

where \( \mathcal{M}_U(w, k) = \mathrm{Clip}(|w|, 0, 2^k - 1) \) is a uniform mapping function that maps transformed weights \( \tilde{w} \) to uniform quantization levels. \( \mathcal{T}_U \) and \( \mathcal{D}_U \) are affine transformation functions with quantization parameters \( \Theta_U \). There are two main methods belonging to UQ: RTN and FlexRound. In RTN, the quantizer \( Q_R \) is formulated as follows:

\[
Q_R(w, k; \Theta_R) = \mathcal{D}_R(\mathcal{M}_U(\mathcal{T}_R(w; \Theta_R), k); \Theta_R),
\]

where \( \Theta_R = \{\Delta, z_U\} \), \( \mathcal{T}_R(w; \Theta_R) = w / \Delta + z_U 1_g \) and \( \mathcal{D}_R(w; \Theta_R) = \Delta (w - z_U 1_g) \). In FlexRound, the main idea is to allow each weight to explore diverse quantization levels to find the optimal one. This is achieved by replacing the transformation function \( \mathcal{T}_R \) with \( \mathcal{T}_F \) defined as follows:

\[
\mathcal{T}_F(w; \Theta_F) = w \odot (\Delta \cdot s \cdot s_r) + z_U 1_g,
\]

where \( \Theta_F = \{\Delta, z_U, s, s_r\} \) is a set of quantization parameters of FlexRound, and \( \odot \) denotes an element-wise division. \( s \in \mathbb{R}^g \) and \( s_r \in \mathbb{R} \) are element-wise and row-wise scale factors to promote the exploration of weights, respectively.

In summary, each quantization method has its own \( \Theta_U, \mathcal{T}_U, \) and \( \mathcal{D}_U \), and they are the origin of strong optimizability. We formulate the quantizers of various UQ methods in Appendix E.6.

Expressiveness of BCQ. As shown in Figure 3(b), we formulate the quantizer \( Q_B \) of BCQ as follows:

\[
Q_B(w; \Theta_B) = \mathcal{D}_B(\mathcal{M}_B(\mathcal{T}_B(w); \Theta_B)),
\]

where \( \mathcal{T}_B(w) = \mathcal{D}_B(w) = w \) are identity functions. \( \mathcal{M}_B(w; \Theta_B) = Q_B(w; \Theta_B) \) is a non-uniform mapping function that maps transformed weights \( \tilde{w} \) to the nearest non-uniform quantization levels. \( \mathcal{M}_B \) enables the mapping between weights and non-uniform quantization levels, enabling strong expressiveness. ALTERNATING (Xu et al., 2018) is the only quantization method in BCQ scheme applicable for LLMs with the same quantizer \( Q_B \) defined above.

UniQuan (Unified Quantization). We propose UniQuan, a framework unifying the quantization processes of UQ and BCQ schemes. UniQuan formulates the unified quantizer \( Q_I \) with quantization parameters \( \Theta_I = \{\Theta_B, \Theta_U\} \) as follows (see Figure 3(c)):

\[
Q_I(w; \Theta_I) = \mathcal{D}_U(\mathcal{M}_B(\mathcal{T}_U(w; \Theta_U); \Theta_B); \Theta_U).
\]

\( Q_I \) has strong expressiveness by mapping weights to the non-uniform quantization levels using \( \mathcal{M}_B \). Also, \( Q_I \) has strong optimizability by incorporating useful methods in UQ by replacing \( \Theta_U, \mathcal{T}_U, \) and
\( D_U \) with those of the method to incorporate. We elaborate on incorporating FlexRound (Lee et al., 2023), the best-performing UQ method, into BCQ in Sections 3.2 and 3.3. We cover the incorporation of AWQ (Lin et al., 2024), OmniQuant (Shao et al., 2024), and GPTQ (Frantar et al., 2023) into BCQ in Section E.6.

3.2 UniQuan\(_F\)-0
We introduce a basic method UniQuan\(_F\)-0 which naively unifies the best-performing UQ and BCQ methods, FlexRound (Lee et al., 2023) and ALTERNATING (Xu et al., 2018), respectively. The quantizer \( Q_{I_F} \) of UniQuan\(_F\)-0 is defined as follows:

\[
Q_{I_F}(\mathbf{w}; \Theta_{I_F}) \\
= \mathcal{D}_R(\mathcal{M}_B(\mathcal{T}_F(\mathbf{w}; \Theta_F); \Theta_B); \Theta_R).
\] (2)

Note that FlexRound begins its optimization process by initializing its quantization parameters with a grid search process to find a proper clipping range. FlexRound enhances its optimizability by utilizing \( \mathcal{T}_F \) in Equation 1 and optimizes its quantization parameters \( \Theta_F \) using stochastic gradient descent (SGD). After optimization, FlexRound stores low-bit weights \( \tilde{\mathbf{w}} = \mathcal{M}_U(\mathcal{T}_F(\mathbf{w}; \Theta_F), k) \) and quantization parameters \( \Theta_R = \{ \Delta, z_U \} \), and approximates the original weights \( \mathbf{w} \) as \( \hat{\mathbf{w}} = \mathcal{D}_R(\tilde{\mathbf{w}}; \Theta_R) \) in the detransformation step; \( s \) and \( s_r \) in \( \Theta_F \) are used only for transformation. On the other hand, ALTERNATING initializes its quantization parameters in \( \Theta_B = \{ \alpha, z_B \} \) with alternating update (Xu et al., 2018). Then, it stores the binary code matrix \( C \) and quantization parameters \( \Theta_B \). It approximates weights \( \mathbf{w} \) as \( \hat{\mathbf{w}} = \mathcal{R}_B(C; \Theta_B) = C \alpha + z_B 1_g \) for inference.

UniQuan\(_F\)-0 inherits the strong optimizability from FlexRound’s \( \mathcal{T}_F \) and \( \mathcal{D}_R \), and the strong expressiveness from ALTERNATING’s \( \mathcal{M}_B \). However, UniQuan\(_F\)-0 has three points of improvements as follows:

C1. Joint initialization. FlexRound and ALTERNATING are based on different initialization methods which seem incompatible. How can we jointly initialize the quantization parameters considering their dependency?

C2. Slow mapping process. The quantization levels of UniQuan\(_F\)-0 are changed during optimization process where its quantization parameters are updated. Therefore, it is crucial to update the mappings between weights and quantization levels during optimization using \( \mathcal{M}_B \).

Algorithm 1 UniQuan\(_F\) for a block \( f \)

Input: A Transformer block \( f \), \( f \)'s input \( X \), a set \( \Phi \) of weight groups in \( f \), the input \( \hat{X} \) of a quantized \( f \) obtained by preceding quantized blocks, number \( E \) of epochs, and number \( |\mathbb{D}| \) of data points

Output: Optimized quantization parameters \( \Theta_B^* \) and a binary-code matrix \( C \) for each group
1: Initialize \( \Theta_{I_F} \) for each weight group
   ▷ (II) Unified Initialization
2: for epoch \( e \) in 1, 2, ..., \( E \) do
3:   for \( s \) in 1, 2, ..., \( |\mathbb{D}| \) do
4:     Quantize \( \Phi \) into \( \hat{\Phi} \) using Equation 2
        ▷ (II) Local and periodic mapping
5:     \( \mathcal{L} \leftarrow \| f(\hat{X}_s; \hat{\Phi}) - f(\hat{X}_s; \hat{\Phi}) \|_F^2 \)
6:     Perform backpropagation using \( \mathcal{L} \)
7:     Update \( \Theta_{I_F} \) for each group
8:   end for
9: end for
10: Convert each \( \Theta_{I_F} \) into \( \Theta_B^* = \{ \alpha^*, z_B^* \} \)
    ▷ (III) Unification Theorem
11: Find \( C \) using Equation 4 for each group

However, existing method (Xu et al., 2018) exploits a binary search tree (BST) to implement \( \mathcal{M}_B \) which finds the mappings through a GPU-unfriendly sequential process, and it requires intractable time for updating mappings during optimization. How can we expedite the mapping process?

C3. Expensive deployment cost. UniQuan\(_F\)-0 suffers from memory and computational overheads at deployment, since UniQuan\(_F\)-0 require executing both inference procedures \( \mathcal{D}_R \) and \( \mathcal{R}_B \) (see Table 1). How can we decrease the inference cost?

3.3 UniQuan\(_F\)
We propose UniQuan\(_F\) (Unified Quantization with Flexible Mapping), which improves UniQuan\(_F\)-0 with the following main ideas.

II. Unified initialization. We unify the initialization algorithms in FlexRound and ALTERNATING to jointly initialize the quantization parameters, considering their dependency.

I2. Local and periodic mapping. We efficiently update mappings by exploiting the locality and the temporal sparsity of changes in mappings.

I3. Unification theorem. We unify the inference process of UniQuan\(_F\)-0 into a single BCQ’s inference process, removing additional memory and computational costs at deployment.
UniQuan_F quantizes an LLM through a block-wise optimization strategy, quantizing sequentially from the bottom-most Transformer block upwards. Algorithm 1 details the process of quantizing a Transformer block \( f \) with a set \( \Phi \) of weight groups, to obtain a binary-code \( C \) and optimized quantization parameters \( \Theta_B \) for each group. We provide inputs \( X \) and \( \tilde{X} \) of unquantized and quantized blocks, respectively. We iterate the optimization process for \( E \) epochs using \( |\mathbb{D}| \) data points.

First, for each weight group, we initialize the quantization parameters \( \Theta_{I_F} = \Theta_B \cup \Theta_F \) using the unified initialization technique (line 1) which we describe later in this subsection. Subsequently, we iteratively optimize the quantization parameters to minimize the distance between the outputs of the quantized and unquantized blocks (lines 2-9). In each optimization step, we quantize the weight groups in \( \Phi \) using UniQuan_F’s quantizer \( Q_{I_F}^* \) with its quantization parameters \( \Theta_{I_F} \) as follows:

\[
Q_{I_F}^*(w; \Theta_{I_F}) \\
= \mathcal{D}_R(\mathcal{M}_B^*(\mathcal{T}_F(w; \Theta_F); \Theta_B); \Theta_R),
\]

where \( \mathcal{M}_B^* \) represents the local and periodic mapping technique, which we describe later in this subsection, to accelerate mappings to non-uniform quantization levels. \( \mathcal{T}_F \) is the transformation function of FlexRound in Equation 1. We quantize each weight group in parallel, and the quantized weight groups are stored in \( \tilde{\Phi} \) (line 4). We measure the reconstruction loss \( \mathcal{L} \) which represents the distance between the outputs of the original and quantized blocks, and then optimize the quantization parameters \( \Theta_{I_F} \) via stochastic gradient descent (lines 5-7). After the iterative optimization process, we have the optimized quantization parameters \( \Theta_{I_F} = \Theta_B \cup \Theta_F \) which generate the outputs closely resembling those of the original block. We leverage the unification theorem (Theorem 1) to merge the optimized quantization parameters into a single set of BCQ’s quantization parameters \( \Theta_B^* = \{\alpha^*, z_B^*\} \) (line 10). Finally, we obtain a binary-code matrix \( C \) using \( \Theta_B^* \) for each weight group as follows:

\[
C = \arg\min_{C'} ||w - (C'\alpha^* + z_B^*1_g)||_2^2.
\]

Then, the quantization process for a block is completed (line 11). We obtain an entirely quantized model by sequentially applying Algorithm 1 from the bottom to the top blocks.

<table>
  <tr>
    <th rowspan="2">Step size</th>
    <th colspan="4">Index change</th>
  </tr>
  <tr>
    <th>0</th>
    <th>1</th>
    <th>2</th>
    <th>>2</th>
  </tr>
  <tr>
    <td>Single step</td>
    <td>99.97%</td>
    <td>0.01%</td>
    <td>0.00%</td>
    <td>0.01%</td>
  </tr>
  <tr>
    <td>All steps (2,560)</td>
    <td>89.30%</td>
    <td>8.37%</td>
    <td>1.36%</td>
    <td>0.97%</td>
  </tr>
</table>

Table 2: The percentage of index changes per optimization step and throughout all 2,560 steps. Minimal changes in each step add up to substantial shifts overall.

Unified Initialization. How can we jointly initialize the quantization parameters of FlexRound and ALTERNATING? FlexRound initializes \( \Delta \) and \( z_U \) through a grid search process, and ALTERNATING utilizes an alternating update process to find \( \alpha \), fixing \( z_B \) as 0. We propose a unified initialization process which initializes the quantization parameters \( \Theta_{I_F} = \{\Delta, z_U, s, s_r, \alpha, z_B\} \) by unifying both techniques, considering their dependency.

In unified initialization, we perform alternating updates to find \( \alpha \) and \( z_B \) in each iteration of a grid search process for \( \Delta \) and \( z_U \). As a result, we find \( \alpha \) and \( z_B \) according to the \( \Delta \) and \( z_U \) during the grid search process. As in FlexRound, we initialize \( s \) and \( s_r \) as \( 1_g \) and 1, respectively. We detail the unified initialization algorithm in Appendix E.1.

Local and Periodic Mapping. How can the mappings between weights and quantization levels be swiftly updated during optimization? Table 2 shows that while over 8% of weights change indices of the mapped quantization levels during the full optimization, 99.97% remain stable at each step, and most changes involve adjacent levels. Hence, we propose a local and periodic mapping which evaluates only neighboring levels and remaps periodically. We set a hyperparameter \( p \) as a remapping period, and in every \( p \) steps, we find the closest quantization level among the previous level and its two neighbors. We detail the process of local and periodic mapping in Appendix E.4.

Unification Theorem. How can we eliminate the memory and computational overheads at deployment induced by the unification? The deployment cost of UniQuan is expensive since the unified quantization process in Equation 3 requires two-step inference procedures with \( \mathcal{D}_R \) and \( \mathcal{R}_B \). We propose a unification theorem that integrates the two-step inference process of UniQuan_F-0 into a single BCQ inference process as follows:

Theorem 1 (Unification Theorem). Given a reconstruction function \( \mathcal{R}_B(C; \Theta_B) \) and a detransformation function \( \mathcal{D}_R(\tilde{w}; \Theta_R) \) where \( \tilde{w} =
Table 3: Comparison of the average accuracy of 0-shot and 5-shot on MMLU, and the perplexity on WikiText2 (Wiki) benchmarks. Higher accuracies and lower perplexities indicate better performance. Bold and underlined texts indicate the best and second-best performance, respectively. UniQuan_F shows the best performance in most cases.

<table>
  <tr>
    <th rowspan="2"># Bits</th>
    <th rowspan="2">Scheme†</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Mistral 7B</th>
    <th colspan="2">Llama-3 8B</th>
    <th colspan="2">Llama-3 70B</th>
  </tr>
  <tr>
    <th>MMLU Avg. (↑)</th>
    <th>Wiki Avg. (↓)</th>
    <th>MMLU Avg. (↑)</th>
    <th>Wiki Avg. (↓)</th>
    <th>MMLU Avg. (↑)</th>
    <th>Wiki Avg. (↓)</th>
  </tr>
  <tr>
    <td colspan="8">Full precision</td>
    <td>61.40</td>
    <td>5.25</td>
    <td>62.77</td>
    <td>6.14</td>
    <td>77.52</td>
    <td>2.86</td>
  </tr>
  <tr>
    <td rowspan="4">4</td>
    <td rowspan="4">UQ</td>
    <td>RTN</td>
    <td>55.20</td>
    <td>6.04</td>
    <td>54.67</td>
    <td>7.80</td>
    <td>67.74</td>
    <td>3.85</td>
  </tr>
  <tr>
    <td>OmniQuant</td>
    <td>56.28</td>
    <td>5.67</td>
    <td>56.02</td>
    <td>7.37</td>
    <td>76.62</td>
    <td>3.34</td>
  </tr>
  <tr>
    <td>FlexRound</td>
    <td>58.60</td>
    <td>5.52</td>
    <td>58.59</td>
    <td>8.22</td>
    <td><b>77.24</b></td>
    <td><b>3.31</b></td>
  </tr>
  <tr>
    <td>ALTERNATING</td>
    <td>24.65</td>
    <td>9.8e4</td>
    <td>22.95</td>
    <td>1.3e5</td>
    <td>44.07</td>
    <td>11.06</td>
  </tr>
  <tr>
    <td rowspan="4">3</td>
    <td rowspan="4">BCQ</td>
    <td>RTN</td>
    <td>28.59</td>
    <td>36.07</td>
    <td>24.35</td>
    <td>120.50</td>
    <td>40.61</td>
    <td>24.42</td>
  </tr>
  <tr>
    <td>OmniQuant</td>
    <td>32.82</td>
    <td>9.24</td>
    <td>26.34</td>
    <td>42.06</td>
    <td>71.13</td>
    <td>5.23</td>
  </tr>
  <tr>
    <td>FlexRound</td>
    <td>53.45</td>
    <td>6.41</td>
    <td>50.22</td>
    <td>10.11</td>
    <td><b>72.44</b></td>
    <td><b>5.81</b></td>
  </tr>
  <tr>
    <td>ALTERNATING</td>
    <td>24.19</td>
    <td>1.1e4</td>
    <td>25.51</td>
    <td>8.4e4</td>
    <td>24.06</td>
    <td>1.5e3</td>
  </tr>
  <tr>
    <td rowspan="2">3</td>
    <td rowspan="2">UniQuan_F</td>
    <td><b>59.22</b></td>
    <td><b>5.51</b></td>
    <td><b>61.43</b></td>
    <td><b>7.01</b></td>
    <td><b>76.97</b></td>
    <td><b>3.19</b></td>
  </tr>
  <tr>
    <td><b>53.68</b></td>
    <td><b>6.20</b></td>
    <td><b>53.46</b></td>
    <td><b>8.75</b></td>
    <td><b>74.79</b></td>
    <td><b>4.24</b></td>
  </tr>
</table>

† UQ and BCQ methods have the same costs when using BCQ kernels (Park et al., 2024a)

\( \mathcal{R}_B(C; \Theta_B) \), there is a set \( \Theta_B^* \) of unified quantization parameters such that \( \mathcal{R}_B(C; \Theta_B^*) = \mathcal{D}_R(\mathcal{R}_B(C; \Theta_B); \Theta_R) \) for any \( C, \Theta_B, \) and \( \Theta_R \).

Proof. See Appendix E.5.

As a result, UniQuan_F exploits the UQ’s optimizability without any extra memory and computational costs at deployment.

Discussion. UniQuan_F is interpreted as a quantization framework that augments the BCQ’s quantization process with the linear transformation process of UQ. By defining the transformation function in UniQuan_F as a linear function, several key advantages emerge. First, it enables the seamless integration of well-established UQ methods (Lee et al., 2023; Lin et al., 2024; Shao et al., 2024) into the BCQ framework, thereby reducing the time and effort required to develop new methods. Second, through the unification theorem, the inference process is consolidated into a single step, allowing the system to benefit from the improved accuracy of the unified approach without incurring additional memory or computational costs at deployment. One might consider introducing non-linear transformation functions to further enhance the expressiveness of the unified quantization process. However, this approach presents notable drawbacks since the aforementioned two key advantages do not hold for non-linear transformation functions; we cannot exploit existing UQ methods and unification theorem. Therefore, we incorporate linear transformation processes into the BCQ’s quantization process, rather than non-linear transformation processes, to enhance efficiencies in method development and deployment.

4 Experiments

We perform experiments to answer the following questions. Additional analyses on UniQuan_F are discussed in Appendix F.

Q1. General knowledge evaluation. How accurately does UniQuan_F quantize LLMs on general knowledge benchmarks?

Q2. Task-specific knowledge evaluation. How accurately does UniQuan_F quantize LLMs on task-specific knowledge benchmarks?

Q3. Ablation study. Do all ideas in UniQuan_F improve the accuracy of quantized LLMs?

Q4. Case study. Does UniQuan_F effectively utilize BCQ’s expressiveness and UQ’s optimizability as we intended?

4.1 Experimental Setup

We briefly introduce the experimental setup. Refer to Appendix C for further details.

Setup. We evaluate the quantized performance of Llama-3 8B, Llama-3 70B (Dubey et al., 2024), and Mistral 7B (Jiang et al., 2023) models on MMLU (Hendrycks et al., 2021) and WikiText2 (Merity et al., 2017) benchmarks. For task-specific experiment, we quantize Llama-3 8B Instruct (Dubey et al., 2024) model and evaluate on
Table 4: Accuracies (%) of quantized Llama-3 Instruct 8B models on GSM8K benchmark. UniQuan_F outperforms the competitors in all cases.

<table>
  <tr>
    <th>Scheme</th>
    <th>Method</th>
    <th>4bit</th>
    <th>3bit</th>
  </tr>
  <tr>
    <td colspan="2">Full precision</td>
    <td>76.12</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">UQ</td>
    <td>RTN</td>
    <td>49.43</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>OmniQuant</td>
    <td>59.16</td>
    <td>2.78</td>
  </tr>
  <tr>
    <td>FlexRound</td>
    <td>70.66</td>
    <td>54.13</td>
  </tr>
  <tr>
    <td rowspan="2">BCQ</td>
    <td>ALTERNATING</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>UniQuan_F</td>
    <td>72.38</td>
    <td>58.73</td>
  </tr>
</table>

GSM8K (Cobbe et al., 2021). We sample 128 token sequences of length 2048 from C4 (Raffel et al., 2020) and GSM8K (Cobbe et al., 2021) for general and task-specific benchmarks, respectively. All experiments are done with a single A100 GPU.

Baselines. We compare UniQuan_F with the four UQ and BCQ methods: RTN (Gupta et al., 2015), OmniQuant (Shao et al., 2024), FlexRound (Lee et al., 2023), and ALTERNATING (Xu et al., 2018).

Hyperparameters. We set each row of weight matrices as a single weight group except for Llama-3 70B where we divide each row into small weight groups of size 128 to prevent severe accuracy loss. Note that each row has at least 1024 weights. Hyperparameters of UniQuan_F is in Appendix C.1. We provide a thorough and detailed analysis of UniQuan_F by altering its hyperparameters to guide hyperparameter search processes in Section F.

4.2 General Knowledge Evaluation

We evaluate the amount of general knowledge retained in quantized models. Table 3 summarizes the average accuracies of 5-shot and 0-shot MMLU, and perplexity on WikiText2 (Wiki). UniQuan_F achieves the highest performance in almost all cases, outperforming the second-best method by up to 3.24% in average MMLU accuracy. In Llama-3 70B, a small group size of 128 makes a small number of weights sharing their quantization parameters, reducing the difficulty of quantization; competitors show comparative performance to UniQuan_F in both 3- and 4-bit cases.

4.3 Task-specific Knowledge Evaluation

Table 4 summarizes the accuracies of a quantized Llama-3 8B Instruct model evaluated on GSM8K benchmark, which focuses exclusively on mathematical problems. Note that GSM8K is a difficult task in which models should generate the exact answer, while general knowledge benchmarks require only selecting correct answers among multiple choices, or evaluating given texts. UniQuan_F achieves the highest performance, outperforming the second-best competitor by 4.60% and 1.72% in the 3-bit and 4-bit experiments, respectively. Notably, UniQuan_F significantly improves the performance of FlexRound by utilizing ALTERNATING even when ALTERNATING exhibits zero accuracy. This demonstrates that UniQuan_F effectively unifies UQ and BCQ methods, leading to achieving previously unattainable performance.

Table 5: Ablation study results on MMLU benchmark. All ideas contribute to improving accuracies.

<table>
  <tr>
    <th>Method</th>
    <th>4bit</th>
    <th>3bit</th>
  </tr>
  <tr>
    <td><b>UniQuan_F</b></td>
    <td>61.43</td>
    <td>53.46</td>
  </tr>
  <tr>
    <td>- remapping</td>
    <td>56.89</td>
    <td>47.83</td>
  </tr>
  <tr>
    <td>- unified init. (\( \Theta_F \))</td>
    <td>24.70</td>
    <td>22.87</td>
  </tr>
  <tr>
    <td>- unified init. (\( \Theta_B \))</td>
    <td>60.84</td>
    <td>52.14</td>
  </tr>
</table>

4.4 Ablation Study

Table 5 summarizes the result of an ablation study using a Llama-3 8B model to show the effectiveness of our main ideas. “-remapping” refers to the case where we do not apply local and periodic mapping, thus mappings are not updated during optimization. “-unified init. (\( \Theta_F \))” and “-unified init. (\( \Theta_B \))” denote the cases when we do not use unified initialization. In “-unified init. (\( \Theta_F \))”, we do not perform a grid search process to initialize quantization parameters in \( \Theta_F \), i.e. we initialize \( \Delta, z_U, s, \) and \( s_r \) in \( \Theta_F \) as 1, 0, 1_g, and 1, respectively. In “-unified init. (\( \Theta_B \))”, we do not perform an alternating update process to initialize quantization parameters in \( \Theta_B \), i.e. we initialize \( \alpha \) and \( z_B \) in \( \Theta_B \) to have uniformly spaced quantization levels in the space transformed by \( T_F \). As a result, UniQuan_F outperforms all of its variants, showing the effectiveness of our main ideas. Specifically, precisely initializing \( \Theta_F \) is crucial to exploit the strong optimizability of \( \Theta_F \).

4.5 Case Study

We conduct a case study on a Llama-3 8B model quantized with UniQuan_F to verify whether the intended mechanisms function properly.

Expressiveness. Figure 4(a) visualizes the distribution of weights (green bars) within a weight group and the assigned quantization levels (red triangles) following UniQuan_F. As shown in the
figure, UniQuan_F assigns dense quantization levels near zero, where most weights are concentrated, and sparse levels in regions with fewer weights. This demonstrates that UniQuan_F effectively learns non-linear quantization levels aligned to the weight distribution, harnessing the strong expressiveness of BCQ scheme.

Optimizability. Figure 4(b) shows the changes of reconstruction error during optimization for a block when using FlexRound (green), ALTERNATING* (brown), and UniQuan_F (blue); ALTERNATING* is a modification of ALTERNATING that optimizes \( \alpha \) using a block-wise output reconstruction process in UniQuan_F devised for comparing optimizability. Compared to the baselines, UniQuan_F achieves the least error at the end with a steep decrease at the early stage of optimization. This demonstrates that UniQuan_F effectively minimizes the reconstruction error, harnessing the strong optimizability of UQ scheme.

In summary, UniQuan_F successfully unifies UQ and BCQ schemes, harnessing their strong expressiveness and optimizability, as we intended. We further verify that UniQuan_F effectively leverages flexible mapping, the main optimization technique of FlexRoud, in Appendix F.6.

5 Related Work

Quantization. Quantization (Lin et al., 2024; Ashkboos et al., 2024; Piao et al., 2022; Liu et al., 2024b) has gained great attention since it effectively speeds up LLM inference by minimizing the cost of memory operations. The majority of quantization approaches rely on uniform quantization (UQ) (Shao et al., 2024; Lee et al., 2023) due to its hardware-friendly nature, benefiting from pre-existing acceleration kernels. Some works (Ashkboos et al., 2024; Liu et al., 2024b) even quantize activation by mitigating outliers through weight rotation, which is orthogonal to our approach. Other approaches, such as vector (Kim et al., 2024; Tseng et al., 2024; Liu et al., 2024a) and additive quantization (Egiazarian et al., 2024) suffer from slow inference since they lack of acceleration kernels. Binary-coding quantization (BCQ) (Xu et al., 2018; Kwon et al., 2022) also has been underexplored due to its missing kernels even with broader expressiveness over UQ (see Section 2.3). However, recent studies (Park et al., 2024a; You et al., 2024) have proposed kernels leveraging look-up tables, enabling BCQ to achieve the same speed as UQ. Taking advantage of them, UniQuan unlocks the superior expressiveness of BCQ by unifying it with the UQ scheme.

Other compression methods. Pruning (Men et al., 2024; Frantar and Alistarh, 2023; Ma et al., 2023; Park et al., 2025, 2024b; Zhong et al., 2024) and knowledge distillation (Gu et al., 2024; Wang et al., 2020; Jeon et al., 2023; Jang et al., 2023; Lee et al., 2021; Cho and Kang, 2022; Yoo et al., 2019; Kim et al., 2021) are compatible techniques that improve the accuracy of compressed models when used with quantization (Park et al., 2024c). Pruning improves the accuracy of quantized models by allowing us to utilize more bit-widths for the necessary weights under memory constraints through explicitly removing unnecessary weights. Knowledge distillation boosts the accuracy of quantized models by transferring the useful knowledge of uncompressed models to quantized models during quantization.

6 Conclusion

We propose UniQuan_F, an accurate quantization method for LLMs. We unify the best-performing uniform quantization (UQ) and binary-coding quantization (BCQ) methods to leverage their strong optimizability and expressiveness concurrently. We propose local and periodic mapping, unified initialization, and unification theorem to improve the accuracies of quantized models without introducing additional memory and computational costs at deployment. As a result, UniQuan_F achieves the best performance, demonstrating the effectiveness of the unification. Incorporating diverse UQ methods into BCQ, and integrating UQ methods with other non-uniform quantization schemes beyond BCQ are our promising future works.

Figure 4: (a) Quantization levels learned by UniQuan_F and (b) changes of reconstruction error during optimization. UniQuan_F assigns quantization levels that align closely with the weight distribution, and effectively reduces the reconstruction error during optimization.
Acknowledgements

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.RS-2020-II200894, Flexible and Efficient Model Compression Method for Various Applications and Environments], [No.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)], and [No.RS-2021-II212068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)]. This work was supported by Youlchon Foundation. The Institute of Engineering Research at Seoul National University provided research facilities for this work. The ICT at Seoul National University provides research facilities for this study. U Kang is the corresponding author. This work was partly done while Seungcheol Park was an intern at NAVER Cloud. This work was improved by the dedicated input and collaboration of researchers from NAVER Cloud.

Limitations

In this paper, we define UniQuan which is a quantization scheme that unifies uniform quantization (UQ) and binary-coding quantization (BCQ) schemes. We verify the effectiveness of UniQuan with only UniQuan_F which unifies the best-performing UQ and BCQ methods. We cover the unification of extended quantization methods in Appendix E.6 in theoretical aspects. The experimental validation of diverse pairs of quantization methods in extended quantization schemes is one of our promising future works.

References

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. CoRR, abs/2404.00456.

Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Ikhyun Cho and U Kang. 2022. Pea-kd: Parameter-efficient and accurate knowledge distillation on bert. PLOS ONE, 17(2).

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukas Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024. Extreme compression of large language models via additive quantization. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation.

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minilmll: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737–1746. PMLR.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).

Jun-Gi Jang, Chun Quan, Hyun Dong Lee, and U Kang. 2023. Falcon: lightweight and accurate convolution based on depthwise separable convolution. Knowl. Inf. Syst., 65(5):2225–2249.

Hyojin Jeon, Seungcheol Park, Jin-Gee Kim, and U. Kang. 2023. Pet: Parameter-efficient knowledge distillation on transformer. PLOS ONE, 18(7).

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.

Junghun Kim, Jinhong Jung, and U. Kang. 2021. Compressing deep graph convolution network with multi-staged knowledge distillation. PLOS ONE, 16.

Minjun Kim, Jaehyeon Choi, Jongkeun Lee, Wonjin Cho, and U Kang. 2025a. Zero-shot quantization: A comprehensive survey. In IJCAI.

Minjun Kim, Jongjin Kim, and U Kang. 2025b. Synq: Accurate zero-shot quantization by synthesis-aware fine-tuning. In ICLR.

Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. SqueezeLm: Dense-and-sparse quantization. In Forty-first International Conference on Machine Learning.

Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, and Dongsoo Lee. 2022. Alphatuning: Quantization-aware parameter-efficient adaptation of large-scale pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3288–3305. Association for Computational Linguistics.

Hyun Dong Lee, Seongmin Lee, and U. Kang. 2021-Auber: Automated bert regularization. PLoS ONE, 16(6).

Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. 2023. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 18913–18939. PMLR.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Weiming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. 2024a. VPTQ: extreme low-bit vector post-training quantization for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 8181–8196. Association for Computational Linguistics.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuaram Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024b. Spinquant: LLM quantization with learned rotations. CoRR, abs/2405.16406.

Xinyin Ma, Gongfan Fang, and Xinzhao Wang. 2023. Llm-pruner: On the structural pruning of large language models. In NeurIPS.

Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2024a. LUT-GEMM: quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Seungcheol Park, Hojun Choi, and U Kang. 2024b. Accurate retraining-free pruning for pretrained encoder-based language models. In ICLR.

Seungcheol Park, Jaehyeon Choi, Sojin Lee, and U Kang. 2024c. A comprehensive survey of compression algorithms for language models. arXiv preprint arXiv:2401.15347.

Seungcheol Park, Sojin Lee, Jongjin Kim, Jinsik Lee, Hyunjik Jo, and U Kang. 2025. Accurate sublayer pruning for large language models by exploiting latency and tunability information. In IJCAI.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style,
high-performance deep learning library. Advances in neural information processing systems, 32.

Tairen Piao, Ikhyun Cho, and U Kang. 2022. Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression. PloS one, 17(4):e0265621.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip#: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrette Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.

Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. 2018.

Alternating multi-bit quantization for recurrent neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.

Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. 2019. Knowledge extraction with no observable data. In NeurIPS.

Haoran You, Yipin Guo, Yichao Fu, Wei Zhou, Hui-hong Shi, Xiaofan Zhang, Souvik Kundu, Amir Yazdanbakhsh, and Yingyan Celine Lin. 2024. Shiftaddllm: Accelerating pretrained llms via post-training multiplication-less reparameterization. arXiv preprint arXiv:2405.04532.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. 2024. Blockpruner: Fine-grained pruning for large language models. arXiv.
A Terminology

We summarize the definitions of terminologies frequently used in this paper to promote clarity.

A.1 Units in LLMs

We summarize the definitions of the units in Large Language Models (LLMs) from a weight to a model. Figure 5 illustrates an example of a Transformer-based LLM with \( N \) blocks.

![An illustration of a Transformer architecture with L blocks.](page_349_627_388_246.png)

Figure 5: An illustration of a Transformer architecture with \( L \) blocks.

• Weight: the smallest unit, representing an individual numerical weight value. The quantization bit width represents the number of bits to represent each weight.

• Weight group: a collection of weights grouped by a specified group size, all of which share the same quantization parameters. It is easy to quantize models when we have small group sizes since we have plenty of quantization parameters.

• Weight matrix: a two-dimensional matrix composed of weights, containing multiple weight groups.

• Layer: a component that performs affine transformations with a weight matrix and a bias vector.

• Module: a collection of layers that performs a specific functionality. In Transformers, modules include Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP).

• Block: a fundamental unit of a Transformer, consisting of one MHA module and one MLP module. FlexRound (Lee et al., 2023) and UniQuanF sequentially quantize each block from the bottom to top to reduce the cost of quantization.

• Model: a complete language model consisting of multiple blocks. An LLM refers to a model.

<table>
  <tr>
    <th>Symbol</th>
    <th>Definition</th>
  </tr>
  <tr>
    <td>\( w \)</td>
    <td>Weight</td>
  </tr>
  <tr>
    <td>\( \mathbf{w} \)</td>
    <td>Group of weights</td>
  </tr>
  <tr>
    <td>\( \tilde{\mathbf{w}} \)</td>
    <td>Group of transformed weights</td>
  </tr>
  <tr>
    <td>\( \hat{\mathbf{w}} \)</td>
    <td>Group of mapped weights</td>
  </tr>
  <tr>
    <td>\( \bar{\mathbf{w}} \)</td>
    <td>Group of quantized weights</td>
  </tr>
  <tr>
    <td>\( \Phi \)</td>
    <td>Set of weight groups</td>
  </tr>
  <tr>
    <td>\( g \)</td>
    <td>The size of a weight group</td>
  </tr>
  <tr>
    <td>\( k \)</td>
    <td>Quantization bit-width</td>
  </tr>
  <tr>
    <td>\( f \)</td>
    <td>Transformer block</td>
  </tr>
  <tr>
    <td>\( |\mathbb{D}| \)</td>
    <td>Number of data points</td>
  </tr>
  <tr>
    <td>\( \mathbf{1}_g \)</td>
    <td>Vector of size \( g \) filled with ones</td>
  </tr>
  <tr>
    <td>\( \lfloor \cdot \rfloor \)</td>
    <td>Rounding function</td>
  </tr>
  <tr>
    <td>\( Clip(\cdot, m, M) \)</td>
    <td>Clipping function with range \([m, M]\)</td>
  </tr>
  <tr>
    <td>\( \odot \)</td>
    <td>Element-wise division</td>
  </tr>
  <tr>
    <td>\( \Delta \)</td>
    <td>Scale factor of UQ</td>
  </tr>
  <tr>
    <td>\( z_U \)</td>
    <td>Zero-point of UQ</td>
  </tr>
  <tr>
    <td>\( C \)</td>
    <td>Binary-code matrix of BCQ</td>
  </tr>
  <tr>
    <td>\( \alpha \)</td>
    <td>Scale factors of BCQ</td>
  </tr>
  <tr>
    <td>\( z_B \)</td>
    <td>Shifting factor of BCQ</td>
  </tr>
  <tr>
    <td>\( s \)</td>
    <td>Element-wise scale factor of FlexRound</td>
  </tr>
  <tr>
    <td>\( s_r \)</td>
    <td>Row-wise scale factor of FlexRound</td>
  </tr>
  <tr>
    <td>\( G \)</td>
    <td>Grid search iterations</td>
  </tr>
  <tr>
    <td>\( T \)</td>
    <td>Alternating update iterations</td>
  </tr>
  <tr>
    <td>\( p \)</td>
    <td>Remapping period</td>
  </tr>
</table>

Table 6: Symbols and their definitions.

A.2 Error Types

In this paper, we mention two types of errors: quantization error and output reconstruction error. Quantization error represents the error before and after quantization at the weight level without considering the model’s input. If we quantize a weight group \( \mathbf{w} \) into \( \bar{\mathbf{w}} \), then the quantization error is \( \| \mathbf{w} - \bar{\mathbf{w}} \|_2^2 \). On the other hand, output reconstruction error measures how the quantized model’s outputs are close to those before quantization. We measure the reconstruction error at block level. Assume that we have a block \( f \) with parameters \( \Phi \), quantized into \( \hat{f} \) with parameters \( \hat{\Phi} \). The reconstruction error is \( \| f(\mathbf{X}; \Phi) - \hat{f}(\hat{\mathbf{X}}; \hat{\Phi}) \|_F^2 \), where \( \mathbf{X} \) and \( \hat{\mathbf{X}} \) are the inputs of the block \( f \) and the quantized block \( \hat{f} \), respectively.

We first initialize the quantization parameters to minimize quantization errors at the weight level. Then, we optimize the quantization parameters to minimize the output reconstruction errors at the block level to take account of the input distribution.
Table 7: A summary of terminologies regarding different quantization methods. \( \Rightarrow \) represents the application of the unification theorem (see Theorem 1). UniQuan’s quantization parameters \( \Theta_U \), transformation function \( \mathcal{T}_U \) and detransformation function \( \mathcal{D}_U \) are determined according to the UQ method unified into the BCQ scheme.

<table>
  <tr>
    <th>Method</th>
    <th>Quantizer</th>
    <th>Quantization Parameters</th>
    <th>Transformation Function</th>
    <th>Mapping Function</th>
    <th>Detransformation Function</th>
    <th>Reconstruction Function</th>
  </tr>
  <tr>
    <td>General form</td>
    <td>\( Q \)</td>
    <td>\( \Theta \)</td>
    <td>\( \mathcal{T}(w; \Theta) \)</td>
    <td>\( M(w; \Theta) \)</td>
    <td>\( \mathcal{D}(w; \Theta) \)</td>
    <td>\( \mathcal{R}_B(C; \Theta) \)</td>
  </tr>
  <tr>
    <td>RTN</td>
    <td>\( Q_R \)</td>
    <td>\( \Theta_R = \{\Delta, z_U\} \)</td>
    <td>\( \mathcal{T}_R(w; \Theta_R) \)</td>
    <td>\( M_U(w, k) \)</td>
    <td>\( \mathcal{D}_R(w; \Theta_R) \)</td>
    <td>-</td>
  </tr>
  <tr>
    <td>FlexRound</td>
    <td>\( Q_F \)</td>
    <td>\( \Theta_F = \{\Delta, z_U, s_r\} \)</td>
    <td>\( \mathcal{T}_F(w; \Theta_F) \)</td>
    <td>\( M_U(w, k) \)</td>
    <td>\( \mathcal{D}_R(w; \Theta_R) \)</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ALTERNATING</td>
    <td>\( Q_B \)</td>
    <td>\( \Theta_B = \{\alpha, z_B\} \)</td>
    <td>\( \mathcal{T}_B(w) \)</td>
    <td>\( M_B(w; \Theta_B) \)</td>
    <td>\( \mathcal{D}_B(w) \)</td>
    <td>\( \mathcal{R}_B(C; \Theta_B) \)</td>
  </tr>
  <tr>
    <td>UniQuan</td>
    <td>\( Q_I \)</td>
    <td>\( \Theta_I = \Theta_U \cup \Theta_B \Rightarrow \Theta_B \)</td>
    <td>\( \mathcal{T}_U(w; \Theta_U) \)</td>
    <td>\( M_U^*(w; \Theta_B) \)</td>
    <td>\( \mathcal{D}_U(w; \Theta_U) \)</td>
    <td>\( \mathcal{R}_B(C; \Theta_B') \)</td>
  </tr>
  <tr>
    <td>UniQuan<sub>F</sub></td>
    <td>\( Q_{I_F} \)</td>
    <td>\( \Theta_{I_F} = \Theta_U \cup \Theta_B \Rightarrow \Theta_B' \)</td>
    <td>\( \mathcal{T}_F(w; \Theta_F) \)</td>
    <td>\( M_U^*(w; \Theta_B') \)</td>
    <td>\( \mathcal{D}_F(w; \Theta_F) \)</td>
    <td>\( \mathcal{R}_B(C; \Theta_B') \)</td>
  </tr>
</table>

A.3 Symbols and Definitions

We summarize the frequently used symbols and their definitions in Tables 6 and 7.

B Inference Speed of BCQ Kernels

In this paper, we propose UniQuan which unifies UQ (Lee et al., 2023; Lin et al., 2024; Shao et al., 2024; Kim et al., 2025a,b) and BCQ (Xu et al., 2018; Kwon et al., 2022) schemes, and the resulting models are represented in a BCQ scheme. The effectiveness of UniQuan depends on the inference speed of BCQ kernels (Park et al., 2024a; You et al., 2024). Thus, we summarize the inference speed of LUT-GEMM (Park et al., 2024a), the state-of-the-art BCQ kernel, for completeness.

Table 8: Latency comparison of the first FFN layer on OPT-175B model with various precision and kernels on A100-80GB-GPU.

<table>
  <tr>
    <th>Kernel</th>
    <th>Schemes</th>
    <th># Bits</th>
    <th>Latency (ms)</th>
  </tr>
  <tr>
    <td>cuBLAS</td>
    <td>-</td>
    <td>16</td>
    <td>0.7256</td>
  </tr>
  <tr>
    <td>GPTQ</td>
    <td>UQ</td>
    <td>3</td>
    <td>0.3599</td>
  </tr>
  <tr>
    <td>AWQ</td>
    <td>UQ</td>
    <td>4</td>
    <td>0.3238</td>
  </tr>
  <tr>
    <td>LUT-GEMM</td>
    <td>UQ, BCQ</td>
    <td>4</td>
    <td>0.2688</td>
  </tr>
  <tr>
    <td>LUT-GEMM</td>
    <td>UQ, BCQ</td>
    <td>3</td>
    <td>0.2250</td>
  </tr>
</table>

Table 8 summarizes the main results of Table 1 in LUT-GEMM (Park et al., 2024a). In this table, latency represents the time for inferencing the first FFN layer in OPT-175B (Zhang et al., 2022) with various precision, and LUT-GEMM shows faster inference speeds than GPTQ (Frantar et al., 2023) and AWQ (Lin et al., 2024) kernels, which support only a UQ scheme.

Table 9 summarizes the main results of Table 6 in LUT-GEMM (Park et al., 2024a), reporting the end-to-end latency per token for OPT family models. \( k \) and \( g \) represent the bit width and group size, respectively. As summarized in the table, LUT-GEMM provides end-to-end inference speedup with diverse bit-widths and group sizes.

In summary, LUT-GEMM provides faster inference speed for quantized models than existing UQ kernels and speeds up end-to-end inference. Therefore, UniQuan<sub>F</sub>, which is supported by LUT-GEMM, is an essential method for quantizing LLMs. Note that we need to convert quantized models using a UQ scheme into a BCQ scheme to use LUT-GEMM which provides a faster inference speed than existing UQ kernels as shown in Table 8. Therefore, quantized models in both schemes require the same memory and computational costs when they have the same bit width.

Table 9: Comparison of end-to-end latency per token for OPT-30B models on a A100-80GB-GPU.

<table>
  <tr>
    <th>Model</th>
    <th>Kernel-\( k-g \)</th>
    <th>Latency (ms)</th>
  </tr>
  <tr>
    <td>cuBLAS-16-N/A</td>
    <td></td>
    <td>40.5</td>
  </tr>
  <tr>
    <td>OPT</td>
    <td>LUT-GEMM-4-32</td>
    <td>18.5</td>
  </tr>
  <tr>
    <td>-30B</td>
    <td>LUT-GEMM-4-64</td>
    <td>17.8</td>
  </tr>
  <tr>
    <td></td>
    <td>LUT-GEMM-3-32</td>
    <td>16.7</td>
  </tr>
  <tr>
    <td></td>
    <td>LUT-GEMM-3-64</td>
    <td>15.7</td>
  </tr>
</table>

C Implementation Details

We use PyTorch (Paszke et al., 2019) and Hugging Face (Wolf et al., 2019) libraries for implementation. We use the pretrained weights of Llama-3 (Dubey et al., 2024) and Mistral (Jiang et al., 2023) models from the Hugging Face library. We discuss the implementation details of UniQuan<sub>F</sub> and competitors to reproduce the performance reported in this paper.

C.1 Implementation Details of UniQuan<sub>F</sub>.

Hyperparameter settings. Our objective is to demonstrate that the outstanding performance of
Table 10: Hyperparameter settings of UniQuan_F

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Setting</th>
  </tr>
  <tr>
    <td>Learning rate for \( \Theta_F \)</td>
    <td>0.005</td>
  </tr>
  <tr>
    <td>Learning rate for \( \Theta_B \)</td>
    <td>0.0005</td>
  </tr>
  <tr>
    <td>Grid search iterations (\( G \))</td>
    <td>1, 30</td>
  </tr>
  <tr>
    <td>Alternating update iterations (\( T \))</td>
    <td>15</td>
  </tr>
  <tr>
    <td>Remapping Period (\( p \))</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>20</td>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Clipping strategy</td>
    <td>Fixed-minimum</td>
  </tr>
</table>

UniQuan_F is achieved without expensive hyperparameter tuning although UniQuan_F employs a block-wise output reconstruction process that requires many hyperparameters. To this end, we fix all hyperparameters except for the grid size \( G \) for unified initialization and utilize only two combinations of hyperparameters across all cases. We outline these combinations in Table 10.

Gradient filtering. We utilize a straight-through estimator (STE) (Bengio et al., 2013) to update UniQuan_F’s quantization parameters since its mapping function \( M_B^* \) is not differentiable. We filter the gradient of the weights that have large mapping errors \( |\bar{w} - \tilde{w}| \) to stabilize the optimization process of UniQuan_F, where \( \bar{w} \) and \( \tilde{w} \) represent transformed and mapped weights, respectively. STE hypothesizes that the gradient of a transformed weight \( \bar{w} \) and mapped weight \( \tilde{w} \) have the same gradients. However, if the difference between transformed and mapped weight is significant, the hypothesis does not hold which degrades the accuracy of quantized models. Therefore, we set the hyperparameter \( \tau \) as a gradient filtering threshold, and zero out the gradients of weights whose mapping error is larger than \( \tau \).

We mimic the quantization process in the UQ scheme to determine \( \tau \). In UQ, weights within the clipping range \([w_{m,c}, w_{M,c}]\) are transformed to the values in the range \([0, 2^{k-1}]\), while weights outside the clipping range, such as \( w_m \) and \( w_M \), are transformed into values outside the range \([0, 2^{k-1}]\). These out-of-range weights cause significant mapping errors. In the transformed space, the interval between quantization levels is 1, and each level has a range of 0.5 on either side. Thus, it is reasonable to allow a margin of 0.5 even for the smallest and largest quantization levels 0 and \( 2^{k-1} \).

For UniQuan_F, every quantization level has a potential risk of significant mapping errors, and the value of 0.5 is replaced with the smallest scale factor \( min(\alpha) \) which is the smallest width in the binary tree constructed by BCQ’s quantization levels (see Figure 2 (b)). Therefore, we set the gradient filtering threshold \( \tau \) as \( min(\alpha) \).

C.2 Implementation Details of Competitors

RTN (Gupta et al., 2015). We search the clipping range with 100 iterations, which is larger than the number of iterations used in unified initialization. We execute the same source code for clipping range search in UniQuan_F.

ALTERNATING (Xu et al., 2018). We implement ALTERNATING based on the original paper (Xu et al., 2018). We use an alternating update with 15 iterations which is the same as the number of iterations used in unified initialization. We use the same source code for the alternating update in UniQuan_F.

FlexRound (Lee et al., 2023). We implement the FlexRound following the original paper (Lee et al., 2023).

OmniQuant (Shao et al., 2024). We refer to the official implementation\footnote{https://github.com/hendrycks/test} of OmniQuant and report the results following the best hyperparameter settings reported in the paper.

D Evaluation Protocol

We report the average performance using random seeds 0, 1, and 2 except for Llama-3 70B which uses only 0 because of its long quantization time. We sample 128 token sequences of length 2048 from C4 (Raffel et al., 2020) and GSM8K (Cobbe et al., 2021) for general and task-specific knowledge evaluations, respectively. We detail the evaluation protocols for general and task-specific knowledge evaluation as follows.

General knowledge evaluation. We use MMLU (Hendrycks et al., 2021) and WikiText2 (Merity et al., 2017) benchmarks for general knowledge evaluation. MMLU consists of multiple-choice problems across 57 subjects. We evaluate the 0-shot and 5-shot accuracies on MMLU to evaluate the amount of general knowledge in the quantized models. 0-shot and 5-shot refer to settings with 0 and 5 examples provided in the prompt, respectively. We follow the evaluation protocol in the official code repository\footnote{https://github.com/hendrycks/test} for the MMLU benchmark. WikiText2 consists of tokens within a set of
verified articles from Wikipedia. We report the perplexity of quantized models on WikiText2 benchmark to evaluate the language modeling capabilities of quantized models. We follow the evaluation protocol used in OmniQuant (Shao et al., 2024) according to its official implementation2.

Task-specific knowledge evaluation. We use GSM8K (Cobbe et al., 2021) benchmark for task-specific knowledge evaluation. GSM8K consists of high-quality grade school math problems that require multi-step reasoning, and we use GSM8K to evaluate the amount of Mathematical knowledge in the quantized models. We implement our evaluation code using the language model evaluation harness (Gao et al., 2023) package3.

We summarize the properties of benchmarks in Table 11.

<table>
  <tr>
    <th>Benchmark</th>
    <th>Subject</th>
    <th>Instance</th>
    <th>Metric</th>
  </tr>
  <tr>
    <td>MMLU</td>
    <td>General</td>
    <td>14,042</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>WikiText2</td>
    <td>General</td>
    <td>141</td>
    <td>Perplexity</td>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td>Math</td>
    <td>1,319</td>
    <td>Accuracy</td>
  </tr>
</table>

E Theoretical Details

We explain the theoretical details on clipping strategy, general alternating update, local and periodic mapping, unification theorem, and extensibility of UniQuan.

E.1 Algorithm of Unified Initialization

Algorithm 2 outlines the overall process of unified initialization which initializes the quantization parameters in the set \( \Theta_{I_F} = \{ \Delta, z_U, s, s_r, \alpha, z_B \} \) for each weight group. Following FlexRound (Lee et al., 2023), we initialize \( s \) and \( s_r \) as 1_g and 1, respectively. We initialize the center \( z_B \) of BCQ’s quantization levels to \( (2^k - 1)/2 \) since \( T_F \) transforms weights \( w \) into the range \([0, 2^k - 1]\) (line 2). Then, we conduct a grid search by adjusting the scale ratio \( \gamma \) to find the optimal quantization parameters (lines 3-13). We explore diverse values for the center of BCQ’s quantization levels in the original weight space, by adjusting candidate scale factor \( \Delta' \) and candidate zero-point \( z_U' \) according to \( \gamma \); the weight transformed into \( z_B \) changes as the adjusted

Algorithm 2 Unified Initialization

Input: Weights \( w \), grid search iterations \( G \), alternating update iterations \( T \), and a bit-width \( k \)
Output: Initialized quantization parameters \( \Theta_{I_F} \)
1: \( w_m, w_M \leftarrow min(w), max(w) \)
2: \( z_B, s, s_r, e \leftarrow (2^k - 1)/2, 1_g, 1, \text{MAX\_NUM} \)
3: for \( \gamma \) in \( 1/G, 2/G, ..., 1 \) do
4: \( \Delta', z_U' \leftarrow \text{adjust-clipping}(w_m, w_M, \gamma, k) \) \( \triangleright \) Appendix E.2
5: \( \bar{w} \leftarrow T_F(w; \Delta', z_U', s, s_r) \) \( \triangleright \) Equation 1
6: \( \alpha', z_B' \leftarrow \text{general-alternating}(\bar{w}, z_B, G, T) \) \( \triangleright \) Algorithm 3 in Appendix E.3
7: \( \Theta_{I_F}' \leftarrow \{ \Delta', z_U', s, s_r, \alpha', z_B' \} \)
8: \( \hat{w} \leftarrow Q_{I_F}^*(w; \Theta_{I_F}') \) \( \triangleright \) Equation 2
9: \( e' \leftarrow \| w - \hat{w} \|_2^2 \) \( \triangleright \) Quantization error
10: if \( e' < e \) then
11: \( \Delta, z_U, \alpha, z_B, e \leftarrow \Delta', z_U', \alpha', z_B', e' \)
12: end if
13: end for
14: \( \Theta_{I_F} \leftarrow \{ \Delta, z_U, s, s_r, \alpha, z_B \} \)

\( \Delta' \) and \( z_U' \) modify the transformation function \( T_F \). There are Fixed-minimum, Fixed-maximum, and Balanced strategies for adjusting \( \Delta' \) and \( z_U' \). The selection of a clipping strategy is a hyperparameter and we detail those strategies in Appendix E.2 (line 4). Then, we transform \( w \) into \( \bar{w} \) using Equation 1 (line 5). We find \( \alpha' \) and \( z_B' \) through a general alternating update in Algorithm 3 which is the improved version of alternating update (Xu et al., 2018) to find \( z_B \) (line 6). After obtaining the candidate quantization parameters in \( \Theta_{I_F}' \), we quantize \( w \) into \( \hat{w} \) and evaluate the quantization error \( e' \) (lines 7-9). We update the quantization parameters using the candidate ones only if the evaluated error \( e' \) is lower than the previous minimum error \( e \) (lines 10-12). After the grid search process over \( G \) iterations, we obtain the initialized quantization parameters (line 14).

E.2 Details of Clipping Strategies

In unified initialization, we find the optimal quantization parameters by adjusting the clipping range \([w_{m,c}, w_{M,c}]\) from altering a hyperparameter \( \gamma \). The length of the clipping range is adjusted as \( \gamma(w_M - w_m) \) by setting scale factor \( \Delta' = \gamma(w_M - w_m)/(2^k - 1) \), and the clipping range is shifted according to the definition of the zero-point \( z_U' \). There are three strategies to define \( z_U' \) as follows:

• Fixed-minimum. \( w_m \) is fixed at \( w_{m,c} \) so that the \( w_m \) is always included in the clipping range. In this case, we find \( z_U' = -w_m/\Delta' \) by solving
Algorithm 3 General alternating update

Input: A weight group \( w \) of size \( g \). Numbers \( G \) and \( T \) of iterations for grid search and alternating update, respectively
Output: Updated \( \alpha \) and \( z_B \)
1: Greedily initialize \( \alpha \) and \( C \)
   ▷ Equation 4 in Xu et al. (2018)
2: for \( t \) in 1, 2, ..., \( T \) do
3:   \( \alpha \leftarrow (C^T C)^{-1} C^T (w - z_B 1_g) \)
4:   \( C \leftarrow \arg\min_C \|w - (C' \alpha + z_B 1_g)\|_2^2 \)
5:   if \( G = 1 \) then
         ▷ Update \( z_B \) only if not grid search
6:     \( z_B \leftarrow \sum_{i=1}^g (w_i - C_{i,:} \alpha)/g \)
7:   end if
8: end for

\( \mathcal{T}_F(w_m) = 0 \).

• Fixed-maximum. \( w_M \) is fixed at \( w_{M,c} \) so that the \( w_M \) is always included in the clipping range. In this case, we find \( z_U' = 2^k - 1 - w_M/\Delta' \) by solving \( \mathcal{T}_F(w_M) = 2^k - 1 \).

• Balanced. \( w_{m,c} \) and \( w_{M,c} \) are adjusted as \( \gamma w_m \) and \( \gamma w_M \), respectively, so that the minimum and maximum clipped values are adjusted in balance. In this case, we find \( z_U' = -\gamma w_m/\Delta' \) by solving \( \mathcal{T}_F(\gamma w_m) = 0 \).
The center of BCQ’s quantization levels explores more diverse values during grid search when we use Fixed-maximum or Fixed-minimum strategies than Balanced strategy; the clipping range is asymmetrically adjusted in Fixed-maximum and Fixed-minimum strategies.

E.3 Algorithm of General Alternating Update

Algorithm 3 details the general alternating update process employed in line 6 of Algorithm 2 which is an improved version of alternating update in Xu et al. (2018) to find proper \( z_B \). We initialize \( \alpha \) and \( C \) using a greedy initialization strategy (Equation (4) in Xu et al. (2018)), where we determine \( \alpha \) and \( C \) for each bit-width one by one (line 1). Then, we iteratively minimize the quantization error \( e = \|w - (C' \alpha + z_B 1_g)\|_2^2 \) for \( T \) iterations to find the optimal quantization parameters (lines 2-8). In each iteration, we optimize for \( \alpha, C, \) and \( z_B \) sequentially while keeping the others fixed. For \( \alpha \) and \( z_B \), we find the values where the derivatives \( \frac{\partial e}{\partial \alpha} \) and \( \frac{\partial e}{\partial z_B} \) equal to zero, respectively, where \( C_{i,:} \) represents the \( i \)th row of \( C \) (lines 3 and 6). For \( C \), we find the optimal binary codes for each weight by comparing errors obtained using all possible binary codes (line 4). After completing these iterations, we obtain the optimized values for \( \alpha \) and \( z_B \). We do not update \( z_B \) when \( G \) is greater than 1 since diverse values for \( z_B \) are explored during the iterative grid search process (line 5).

Algorithm 4 Local and periodic mapping (\( \mathcal{M}_B^* \))

Input: A transformed weight group \( \tilde{w} \) of size \( g \), iteration step \( s \), a remapping period \( p \), BCQ’s quantization parameter \( \Theta_B = \{\alpha, z_B\} \), and mapping indices \( d \) and mapped weights \( \tilde{w} \) in the previous step
Output: Updated mapping indices \( d \) and mapped weights \( \tilde{w} \)
1: if \( s \equiv 0 \pmod{p} \) then   ▷ Periodic mapping
2:   \( q \leftarrow \text{compute-quantization-levels}(\alpha, z_B) \)
3:   for \( i \) in \( \{1, 2, ..., g\} \) do
4:     \( l, r \leftarrow \max(d_i - 1, 1), \min(d_i + 1, 2^k) \)
5:     \( \mathcal{N} \leftarrow \{l, d_i, r\} \)
         ▷ Neighbor quantization levels’ indices
6:     \( d_i \leftarrow \arg\min_{d'\in\mathcal{N}} |\tilde{w}_i - q_{d'}| \)   ▷ Local mapping
7:     \( \tilde{w}_i \leftarrow q_{d_i} \)
8:   end for
9: end if

E.4 Algorithm of Local and Periodic Mapping

Algorithm 4 describes the overall process of local and periodic mapping which swiftly updates the mapping between weights and quantization levels during optimization. In the beginning, we initialize the mapping by calculating distances to all quantization levels. Then, we begin local and periodic mapping using the resulting mapping \( d \) and mapped weights \( \tilde{w} \) where \( d \) contains the indices of quantization levels that the weights are mapped. We periodically update the mappings between weights and quantization levels in every \( p \) optimization step (line 1). We compute quantization levels using \( \alpha \) and \( z_B \), and store them in \( q \in \mathbb{R}^{2^k} \) (line 2). Then, we find the closest quantization level among its previously mapped level and its neighboring levels (lines 4-6). We also update the mapped weight value to use them for the successive steps before the next update (line 7). Local and periodic mapping is efficient, updating \( 1/p \) times less often than non-periodic algorithms and using only 3 among \( 2^k \) quantization levels for computing the distance between weights.
E.5 Proof of Theorem 1

Proof. By the definitions of \( \mathcal{R}_B(C; \Theta_B) \) and \( \mathcal{D}_R(\hat{w}; \Theta_R) \), we reduce \( \mathcal{D}_R(R_B(C; \Theta_B); \Theta_R) \) as follows.

\[
\begin{align*}
\mathcal{D}_R(\mathcal{R}_B(C; \Theta_B); \Theta_R) \\
&= \Delta((C\alpha + z_B 1_g) - z_U 1_g) \\
&= C(\Delta \alpha) + \Delta(z_B - z_U) 1_g.
\end{align*}
\]

Thus, \( \alpha^* = \Delta \alpha, z_B^* = \Delta(z_B - z_U) \), and \( R_B(C; \Theta_B^*) = C \alpha^* + z_B^* 1_g \) where \( \Theta_B^* = \{\alpha^*, z_B^*\} \). \( \square \)

E.6 Extensibility of UniQuan

In the main text, we employ UniQuan_F which unifies FlexRound (Lee et al., 2023) and ALTERNATING, the best-performing UQ and BCQ methods, respectively. We cover the unification between other quantization methods to explain the extensibility of UniQuan. The main idea of UniQuan is to utilize the unified quantization process with quantization parameters \( \Theta_I = \Theta_U \cup \Theta_B \) as follows:

\[
Q_I(w; \Theta_I) = \mathcal{D}_U(\mathcal{M}_B(\mathcal{T}_U(w; \Theta_U); \Theta_B); \Theta_U).
\]

Considering the lack of accurate BCQ methods, we focus on replacing \( \mathcal{T}_U \) and \( \mathcal{D}_U \) with method-specific ones to unify UQ methods into the BCQ scheme.

AWQ (Lin et al., 2024) reduces output reconstruction error by scaling weights based on their importance before quantization. This is accomplished by using \( \mathcal{T}_A \) and \( \mathcal{D}_A \) defined as follows:

\[
\begin{align*}
\mathcal{T}_A(w; \Theta_A) &= (w \odot s_A)/\Delta_A + z_A 1_g, \\
\mathcal{D}_A(\tilde{w}; \Theta_A) &= \Delta_A(\tilde{w} - z_0 1_g) \oslash s_A,
\end{align*}
\]

where \( s_A \in \mathbb{R}^g \) is a column-wise scale factor to reflect the importance of each column of the weight matrix where the weight group \( w \) is located. \( \Delta_A \in \mathbb{R} \) and \( z_A \in \mathbb{R} \) represent the new scale factor and zero-point after applying \( s_A \), respectively. \( \Theta_A = \{\Delta_A, z_A, s_A\} \) is a set of quantization parameters for AWQ. \( \odot \) and \( \oslash \) represent element-wise multiplication and division, respectively.

OmniQuant (Shao et al., 2024) parameterizes the clipping range with parameters \( \beta \) and \( \gamma \) to find the best one through optimization. This is accomplished by using \( \mathcal{T}_O \) and \( \mathcal{D}_O \) as follows:

\[
\begin{align*}
\mathcal{T}_O(w; \Theta_O) &= w/\Delta_O + z_O 1_g, \\
\mathcal{D}_O(\tilde{w}; \Theta_O) &= \Delta_O(\tilde{w} - z_0 1_g),
\end{align*}
\]

where \( \Delta_O = (\gamma w_M - \beta w_m)/(2^k - 1) \) and \( z_O = -|\beta w_m/\Delta_O| \) are the parameterized scale factor and zero-point, respectively. \( w_m \) and \( w_M \) are the minimum and maximum weights in a weight group \( w \). \( \Theta_O = \{\beta, \gamma\} \) is the set of quantization parameters of OmniQuant.

GPTQ (Frantar et al., 2023) is widely used in a UQ scheme, but it is a general error compensation technique which covers even pruning (Frantar and Alistarh, 2023). Thus, GPTQ is able to be unified into a BCQ scheme by applying GPTQ’s error compensation strategy to BCQ methods.

In summary, it is able to unify diverse UQ methods into a BCQ scheme following UniQuan’s framework in theory. Extending the coverage of unification to other quantization schemes beyond UQ and BCQ, and validating with experiments is one of our promising future works.

F Additional Analyses

We analyze the memory usage, quantization time, hyperparameter sensitivity, and the effect of sample dataset size on UniQuan_F. We also validate that UniQuan_F effectively leverages the optimization technique in FlexRound.

F.1 Memory Usage of UniQuan_F

We analyze the memory usage of quantized models using UniQuan_F. As summarized in Table 1, UniQuan_F requires the same amount of memory at inference time as existing BCQ methods (Xu et al., 2018; Kwon et al., 2022) since we convert the inference process of the unified quantization process into a single BCQ’s inference process using the unification theorem. When we quantize a weight group \( w \in \mathbb{R}^g \) of size \( g \) into \( k \) bits, we need to store a binary code matrix \( C \in \{-1, +1\}_{g \times k} \), BCQ’s scale factors \( \alpha \in \mathbb{R}^k \), and BCQ’s shifting factor \( z_B \in \mathbb{R} \). Each binary code requires 1 bit and each real number requires 16 bits for saving, thus we require \( gk + 16(k+1) \) bits in total. The memory overhead due to the quantization parameters is \( 16(k+1)/g \) bits per weight, and if we assume channel-wise quantization where \( g \) is larger than 4000 for LLMs (Jiang et al., 2023; Dubey et al., 2024), the memory overhead is negligible.

F.2 Quantization Time of UniQuan_F

We compare the running time for quantizing Llama-3 8B into 3 bits and the accuracy of the quantized model, using UniQuan_F and FlexRound to evaluate the efficiency of UniQuan_F. We compare them
across different epoch settings since the quantization time varies depending on the number of epochs. We do not include the time for unified initialization in UniQuan_F since it is performed as a preprocessing before quantization; the initialized quantization parameters are reused in different hyperparameter settings when the alternating update iteration T and the grid search iteration G are not changed.

Table 12: Comparison of average accuracies (%) on 0-shot and 5-shot MMLU benchmarks, and quantization time (s) when quantizing Llama-3 8B into 3bit using UniQuan_F and FlexRound, respectively. Bold and underlined texts represent the best and second-best results in each method, respectively.

<table>
  <tr>
    <th rowspan="2">Epochs</th>
    <th colspan="2">UniQuan_F</th>
    <th colspan="2">FlexRound</th>
  </tr>
  <tr>
    <th>Accuracy</th>
    <th>Time</th>
    <th>Accuracy</th>
    <th>Time</th>
  </tr>
  <tr>
    <td>5</td>
    <td>47.97</td>
    <td>5,089</td>
    <td>46.71</td>
    <td>4,239</td>
  </tr>
  <tr>
    <td>10</td>
    <td>51.04</td>
    <td>10,022</td>
    <td>47.97</td>
    <td>8,350</td>
  </tr>
  <tr>
    <td>15</td>
    <td>52.08</td>
    <td>15,905</td>
    <td>51.08</td>
    <td>12,405</td>
  </tr>
  <tr>
    <td>20</td>
    <td>53.46</td>
    <td>20,403</td>
    <td><b>51.19</b></td>
    <td>16,868</td>
  </tr>
  <tr>
    <td>25</td>
    <td><b>54.23</b></td>
    <td>25,884</td>
    <td>48.19</td>
    <td>21,970</td>
  </tr>
  <tr>
    <td>30</td>
    <td>53.86</td>
    <td>28,521</td>
    <td><u>51.09</u></td>
    <td>25,139</td>
  </tr>
</table>

Table 12 shows that UniQuan_F requires about 20% longer time for quantization than FlexRound, but it achieves higher accuracy than FlexRound across all epoch settings. Especially, UniQuan_F requires a shorter time of 15,905 seconds to outperform the highest accuracy of FlexRound which requires 16,868 seconds, demonstrating its efficiency.

The running time for the unified initialization process depends on G and T. When G = 30 and T = 15, as used in this experiment, it takes approximately 6,700 seconds and UniQuan_F takes longer than FlexRound if we include this. However, considering that the quantization time only needs to be performed once for deployment, the initialization results are reusable, and FlexRound cannot reach the high performance of UniQuan_F even though we invest more time.

F.3 Sensitivity Analysis on p, G, and T

We report the change in the performance of quantized models according to the variation of UniQuan_F’s hyperparameters G, T, and p to promote reproducibility. We report the average accuracy of 0-shot and 5-shot MMLU benchmarks of the 3-bit quantized Llama-3 8B models.

Remapping period (p). Figure 6(a) shows the change in the quantized models’ accuracy with regard to the change of remapping period p. We use \( p \in \{1, 2, 4, 8, 16, 32\} \) where a higher p represents the sparse update of mapping between weights and quantization levels. As a result, we find that the quantized model achieves the highest accuracy when \( p \in \{2, 4\} \), outperforming the case of \( p = 1 \) where we update the mapping in every step. This result indicates that periodic mapping not only improves the efficiency of UniQuan_F but also improves the accuracy of the quantized models. We recommend using \( p = \{2, 4\} \) and we use only \( p = 2 \) in this paper since they exhibit similar accuracies.

Grid search iterations (G). Figure 6(b) shows the change in the quantized models’ accuracy with regard to the change of grid search iterations G. We use \( G \in \{1, 10, 20, 30, 40, 50\} \) where a higher G represents the exhaustive search for unified initialization. We also include \( G = 1 \) which indicates the case that we find the center of BCQ’s quantization levels using the general alternating update in Algorithm 3, unlike the other cases that use grid search. As a result, we find that \( G \in \{1, 30, 40\} \) shows the highest accuracy and we recommend using \( G = \{1, 30\} \) to efficiently explore two different
Table 13: The average accuracies on 0-shot and 5-shot MMLU benchmarks of 3-bit Llama-3 8B models with various clipping strategies.

<table>
  <tr>
    <th>Strategy</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td>Fixed-minimum</td>
    <td>53.46</td>
  </tr>
  <tr>
    <td>Fixed-maximum</td>
    <td>54.34</td>
  </tr>
  <tr>
    <td>Balanced</td>
    <td>53.21</td>
  </tr>
</table>

Table 14: The average accuracies on 0-shot and 5-shot MMLU benchmarks of 3-bit Llama-3 8B models on various sizes of sample datasets.

<table>
  <tr>
    <th>Sample Size</th>
    <th>32</th>
    <th>64</th>
    <th>128</th>
    <th>256</th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>47.72</td>
    <td>50.71</td>
    <td>53.46</td>
    <td>53.86</td>
  </tr>
</table>

strategies for searching the BCQ’s center.

Alternating update iterations (\( T \)). Figure 6(c) shows the change in the quantized models’ accuracy with regard to the change of alternating update iterations \( G \). We use \( T \in \{5, 10, 15, 20\} \) where a higher \( T \) represents the more iterations for alternating update. As a result, we find that UniQuan\(_F\) achieves high accuracy when \( T \) is equal to or higher than 10. Thus, we recommend using \( T = \{10, 15\} \) which shows the highest accuracy.

F.4 Sensitivity on the Clipping Strategies

We compare the performance of three clipping strategies in unified initialization. Table 13 summarizes the comparison of the performance of Llama-3 8B models quantized using the three clipping strategies.

Fixed-maximum and Fixed-minimum strategies outperform the Balanced strategy. This is because the center of BCQ’s quantization level explores more diverse values in Fixed minimum and Fixed maximum strategies than Balanced strategy, resulting in better quantization levels. Therefore, we recommend evaluating Fixed minimum and Fixed maximum strategies and using the better one according to the experimental settings.

F.5 Sensitivity on the Size of Sample Dataset

To illustrate the effect of sample dataset size on the performance of UniQuan\(_F\), we quantize the Llama-3 8B model to 3 bits using sample datasets of varying sizes. We evaluate the performance of quantized models on 0-shot and 5-shot MMLU benchmarks. We use sample datasets ranging in size from 32 to 256, where 128 is the size used in the main text. We summarize the result in Table 14.

![Index difference of mapped quantization levels per weight before and after applying s and s_r. Flexible mappings occur sufficiently in UniQuan_F compared to FlexRound.](page_1012_120_393_246.png)

Figure 7: Index difference of mapped quantization levels per weight before and after applying \( s \) and \( s_r \). Flexible mappings occur sufficiently in UniQuan\(_F\) compared to FlexRound.

Experimental results show that the accuracy of the quantized models increases as the sample size grows. This is attributed to the fact a large number of data points promote the distillation of the general knowledge in the unquantized model to the quantized model. Therefore, UniQuan\(_F\) achieves higher performance than that reported in the main text by providing enough data points.

F.6 Flexible Mappings in UniQuan\(_F\)

UniQuan\(_F\) unifies FlexRound (Lee et al., 2023) and ALTERNATING. We perform an in-depth analysis to verify that UniQuan\(_F\) effectively leverages the main optimization technique of FlexRound. The main idea of FlexRound is “flexible mapping” which introduces additional scale factors \( s \) and \( s_r \) to make weights explore diverse quantization levels, and selects the best one. We compare the amount of flexible mapping occurred in UniQuan\(_F\) and FlexRound for validation. We use a 3-bit quantized Llama-3 8B model for analysis.

Case study. We analyze the amount of flexible mappings in a weight group of quantized models and visualize the results. Figure 7 represents the index difference of mapped quantization levels for each weight before and after applying scale factors \( s \) and \( s_r \) when quantizing Llama-3 8B models into 3-bit using UniQuan\(_F\) and FlexRound. An index difference equal to or larger than 1 represents that the weight experiences the flexible mapping through \( s \) and \( s_r \). Note that plenty of flexible mappings occur in the quantization process of UniQuan\(_F\) similarly in FlexRound.

Global pattern. We analyze the proportion of flexible mapping applied across all model weights to examine that flexible mappings occur globally. Table 15 illustrates the proportion of the flexibly mapped weights in the entire model. The column names in the tables indicate the amount of changes
Table 15: Proportion of weights exhibiting flexible mapping across the entire model. Each column indicates the index difference of the mapped quantization level resulting from flexible mapping.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Index Difference</th>
  </tr>
  <tr>
    <th>0</th>
    <th>1</th>
    <th>2</th>
    <th>>2</th>
  </tr>
  <tr>
    <td>FlexRound</td>
    <td>95.4132</td>
    <td>4.5862</td>
    <td>5.00e-04</td>
    <td>1.30e-05</td>
  </tr>
  <tr>
    <td>UniQuan<sub>F</sub></td>
    <td>96.0395</td>
    <td>3.8619</td>
    <td>9.85e-02</td>
    <td>7.62e-05</td>
  </tr>
</table>

in indices of weights’ mapped quantization levels resulting from flexible mapping. Across the entire model, UniQuan<sub>F</sub> demonstrates a similar level of flexible mapping as FlexRound. Therefore, UniQuan<sub>F</sub> effectively utilizes flexible mapping across the entire model as we intended.

G Use of AI Assistant

We use ChatGPT<sup>4</sup> and Gemini<sup>5</sup> only for grammar checking and sentence re-wording purposes. We do not use them for research purposes such as developing our main ideas or analyzing our experimental results.

H Potential Risks

In this paper, we propose UniQuan<sub>F</sub>, a quantization method for large language models (LLMs), and there is a potential risk of losing the models’ knowledge during quantization. In experiments, we rigorously validate the amount of knowledge loss in both general and task-specific aspects to verify whether the risk occurs. As a result, we demonstrate that UniQuan<sub>F</sub> results in significantly less knowledge loss than other methods, confirming its low risk.

<sup>4</sup>https://chatgpt.com/
<sup>5</sup>https://gemini.google.com/

Reviewer paper summaries: This paper proposes  a unified quantization framework for large language models (LLMs) that combines the strengths of binary-coding quantization (BCQ) and uniform quantization (UQ). While BCQ offers high expressiveness and UQ ensures optimizability, existing methods fail to integrate these benefits. UniQuan addresses this by merging BCQ’s non-uniform quantization levels with UQ’s flexible mapping, enabling both precision and efficiency. The method employs **unified initialization** and **local-periodic mapping** to optimize parameters effectively. A unification theorem ensures the removal of computational/memory overhead post-optimization, allowing deployment without added costs.

This paper proposes $UniQuan_F$, a unified quantization method that combines the optimization capability of Uniform Quantization (UQ) with the representational flexibility of Binary Code Quantization (BCQ) to achieve efficient compression for LLMs. By integrating unified initialization, a local periodic mapping technique, and a unification theorem, $UniQuan_F$ merges UQ's transformation process with BCQ's non-uniform quantization levels, preserving UQ's fast optimization while leveraging BCQ's adaptability to weight distributions. Experiments demonstrate that UniQuan significantly outperforms existing methods across benchmarks (e.g., MMLU, WikiText2, GSM8K) with up to 4.60% accuracy improvement, incurring no additional computational or memory overhead during deployment.
Summary of strengths: 1. The method proposed in the article ultimately does not increase the inference delay.

2. BCQ is actually a non-uniform quantization method, which is very suitable for quantization with uneven weight distribution. The application of this method can significantly improve the quantization accuracy.

1. **Innovative Framework**: Creative work to unify UQ and BCQ, resolving the trade-off between optimization efficiency (UQ) and representational capacity (BCQ) in traditional quantization methods.

2. **Efficient Optimization Techniques**: Novel components like unified initialization and local periodic mapping accelerate quantization parameter search, while the unification theorem ensures deployment feasibility by eliminating overhead.

3. **Comprehensive Evaluation**: Validated on multiple models (Llama-3, Mistral) across diverse scenarios like general knowledge (MMLU), language modeling (WikiText2), and task-specific reasoning (GSM8K) with thorough and convincing results.

4. **Theoretical Roubustness**: The unification theorem formally proves inference equivalence, complemented by technical descriptions and open-sourced code for reproducibility.
Summary of weaknesses: 1. I would like to ask if the author uses truncation and rounding in the UQ part in actual application. I think the UQ part of UniQuan is actually a linear transformation. If so, regardless of the story of the article, can a more effective equivalent transformation method be explored?

2. The article actually uses weight-only quantization. Can you compare it with some weight-only quantization methods? For example, AQLM[1].

   [1]Extreme Compression of Large Language Models via Additive Quantization. ICML 2024.

1. **Limited Generalization Validation**: The experiments seems only validate the combination of UniQuanF with FlexRound and Alternating, without exploring its compatibility with other UQ/BCQ methods (e.g., AWQ, GPTQ), which limits the extensibility of the method.

2. **Lack of Discussion on Computational Costs**: The unified initialization involves grid search and alternating updates, which may introduce high computational overhead. However, the paper does not quantify the time cost or propose optimization strategies.

3. **Ambiguity in Task Adaptability**: While the GSM8K experiments show significant improvements in task-specific knowledge, the paper does not explain how to adjust hyperparameters (e.g., sample size, mapping period) for different tasks, which may affect practical applications.
Comments, suggestions and typos: See Summary Of Weaknesses.

1. **Theoretical Basis for Unified Initialization**: How does the joint initialization process avoid parameter conflicts between UQ's grid search and BCQ's alternating updates? Is there a formal proof of convergence?

2. **Method Scalability**: Can UniQuan be directly applied to mixed-precision quantization (e.g., simultaneous weight and activation quantization)? If modifications are needed, what are the core challenges?

3. **Task Adaptability**: For tasks requiring high dynamic ranges(e.g., long-text generation), could non-uniform quantization levels introduce cumulative errors? What mitigation strategies exist?

4. **Expecting More Details on Large-Scale Models**: The experiments on Llama-3 70B show performance close to baselines, but the paper does not deeply analyze the impact of group size on quantization error or explore the stability of larger models.
Overall assessment scores: 3

4