<figure><img src="image_1.png" /></figure>

HAL
open science

# Training for Mixed-Precision Integer Weights, Activations and Embeddings in BERT

Cédric Gernigon, Xavier Pillet, Anastasia Volkova, Richard Dufour

► To cite this version:

Cédric Gernigon, Xavier Pillet, Anastasia Volkova, Richard Dufour. Training for Mixed-Precision Integer Weights, Activations and Embeddings in BERT. DASIP 2026: Workshop on Design and Architectures for Signal and Image Processing, Jan 2026, Krakow (Cracovie), Poland. hal-05322672v2

HAL Id: hal-05322672

https://inria.hal.science/hal-05322672v2

Submitted on 19 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Training for Mixed-Precision Integer Weights, Activations and Embeddings in BERT

Cédric Gernigon¹, Xavier Pillet¹,², Anastasia Volkova³, and Richard Dufour¹

¹ Nantes Université, LS2N, UMR 6004

{cedric.gernigon, richard.dufour, xavier.pillet}@univ-nantes.fr

² Valeuriad

³ Inria, INSA Lyon, CITI, UMR 3720

anastasia.volkova@inria.fr

**Abstract.** Pre-trained language models (LMs) deliver strong performance across a wide range of Natural Language Processing (NLP) tasks but remain costly to deploy on embedded devices due to their high memory and compute requirements. A widely used strategy for adapting LMs to resource-constrained devices is aggressive quantization. At low bit-widths, mixed-precision schemes, where different components of the model use different numerical precisions, offer an effective balance between compression and accuracy. In this work, we evaluate the impact of mixed-precision quantization for inference on the BERT language model. Unlike prior studies that often overlook activation quantization, our evaluation systematically explores mixed-precision configurations for both weights and activations. We also examine the effects of quantizing the embedding layer, which is commonly limited to token-weight quantization. Evaluated on the SQuAD and GLUE benchmarks, our approach achieves substantial reductions in memory and computational cost without sacrificing accuracy.

**Keywords:** Quantization · Mixed-precision · Inference acceleration · Embedding layer quantization · Quantization-Aware Training

## 1 Introduction

Pre-trained language models such as BERT [8] have achieved significant improvements across a wide range of NLP tasks [28,22], but these major gains come at the cost of massive parameter counts and high compute. For example, BERT<sub>BASE</sub> has 109M parameters (418MB memory size) and requires billions of operations. The need for real-time inference, along with privacy and security concerns, has made the efficient deployment of these models on edge devices a major challenge. Due to the limited resources, including energy, memory, and computational power of these devices, model compression is essential for their implementation. Various model compression methods have been successfully applied to BERT, such as pruning [12], knowledge distillation [24], and quantization [36].
Quantization reduces model size and computational cost by representing
weights and activations at lower bit-widths, potentially altering the underlying arithmetic. Transformer-based models can be uniformly quantized to 8-bit integers [36], or to ternary [37] or binary [1] representations with limited accuracy degradation. Advanced techniques [29,9,13,33] further increase compression by assigning different bit-widths across (sub)layers. Identifying optimal mixed-precision configurations that minimize accuracy loss remains challenging. Optimization-based methods have emerged as an effective approach, providing a favorable trade-off between compression efficiency and computational cost. Previous studies have predominantly focused on weight quantization, as linear layers constitute the majority of model memory. Nevertheless, deployed models execute all parameters and computations in low precision, highlighting a gap between software-modeled performance and actual hardware accuracy.

In this paper, we push mixed-precision quantization beyond weights-only to
include activations and all embedding parameters in BERT using AdaQAT [11].
AdaQAT is a gradient-based optimization framework for mixed-precision quan-
tization that learns bit-widths to jointly quantize weights and activations during
training, utilizing a Quantization-Aware Training (QAT) strategy. Its key lies
in the use of relaxed fractional bit-widths, which are continuously optimized
via gradient descent but discretized during both forward and backward passes.
We propose a per-layer mixed-precision approach permitting a fine-grained opti-
mization. We evaluate our approach on the SQuAD dataset, enabling direct com-
parison with state-of-the-art methods, and on GLUE benchmark to investigate
the impact of mixed-precision quantization over different language tasks. Our
experiments demonstrate that the proposed method achieves substantial reduc-
tions in memory and computational overhead for BERT<sub>BASE</sub> while maintaining
task accuracy. In addition, we examine the effect of quantizing the embedding
layer's weights on model accuracy. While conventional approaches often focus
exclusively on token embeddings, our findings demonstrate that <em>all</em> embedding
weights can be quantized with minimal impact on performance. For instance,
on the SQuAD dataset, quantizing the full embedding layer to 4 bits or below
preserves accuracy while halving the memory requirements of the embedding
layer. The tool and experiments are open-source and available<sup>4</sup>.

This article is structured as follows. Section 2 reviews related work on QAT and mixed-precision methods. Section 3 describes our methodology, including the integration of knowledge distillation into the mixed-precision flow. Finally, Section 4 presents experimental results, analyzing the performance of mixed-precision quantization applied to the BERT<sub>BASE</sub> model and the impact of embedding layer quantization.

## 2 Related Work

There are two primary approaches to quantization of DNNs. The first, Post-Training Quantization (PTQ), applies quantization after training and is fast to

<footnote n="4">⁴ The anonymized code is at https://github.com/cgernigon/AdaQAT</footnote>
deploy, but can incur non-negligible accuracy loss at very low bit-widths [2]. In contrast, Quantization-Aware Training (QAT), while slower and requiring access to training data, generally leads to better results.

Quantization-Aware Training (QAT) has been widely studied to train CNNs with small bit-widths. QAT methods stem from pioneering work on binary neural networks [6,7]. At its core, a QAT method consists of using a quantized version of the network during training in both forward and backward passes, while performing updates on full-precision copies of the network parameters. These full-precision parameters are then quantized to be used in the next iteration. A crucial aspect is how to perform backpropagation through quantized variables (parameters and activations). In the binary case, this was done using a so-called Straight-Through Estimator (STE) [3], and this approach was later extended to cover larger bit-widths [39], while also applying quantization to gradient signals. To further improve the accuracy of quantized networks, the STE idea can also be used to learn the parameters of uniform quantizers, such as scaling factors and bias terms for weight quantization [10,4], and in the case of ReLU-based activations, clipping parameters [5].

QAT has been successfully applied to LLMs, with BERT being trained in 8-bit integer QAT with minimal accuracy loss using STE-based approaches [36]. Quantization with lower bit-widths has also been explored, including ternary [37] and binary [1,21] quantized BERT. [26] proposed to replace the STE-based gradient for updating the quantization scale with an MSE-based gradient strategy. The SQuAT [30] method alternates training between the sharpness objective and the step-size objective, allowing the model to learn the most appropriate parameter update magnitude to reach convergence close to flat minima.

*Mixed-Precision for Inference* aims to reduce bit-widths at a finer granularity, typically at the (sub)layer level, which is challenging without automated configuration methods. Several approaches have been proposed to address this problem.

Search-based methods like HAQ [29] rely on reinforcement learning with hardware (latency & energy) feedback in the agent, whereas neural architecture search works like DNAS [31] uses gradient-based information. The major downside of using them is that they require significant time and computational resources. AQ-BERT [38] adopts a Network Architecture Search approach to set different precision for each neuron sub-group of BERT.

Much faster results can be obtained using metric-based methods. For example, HAWQ [9] uses Hessian spectrum information at each layer to assign precisions; this approach was used in Q-BERT [25]. Methods like in [34,18] rely on linear programming models, while [16] encourages quantization that leads to reduced sharpness in the task loss function. These approaches are fast but can be sub-optimal compared to more global optimization strategies [13].

Optimization-based approaches formulate the bit-width assignment as an optimization problem, with the main challenge being how to handle the fact that the loss is non-differentiable w.r.t. the bit-widths. Methods like FracBits [33] and BitPruning [20] use fractional bit-widths and linear interpolation during the
forward path, whereas SDQ [13] is based on stochastic quantization, but seems limited to weight quantization. These methods work well in fine-tuning scenarios, but are unstable or do not work when training from scratch.

AdaQAT belongs to the optimization-based family and offers a flexible QAT framework that learns relaxed (fractional) bit-widths via gradient-based optimization. This paper extends AdaQAT to LLMs and improves its applicability to BERT, including joint treatment of activations and embeddings.

# 3 Mixed-precision Exploration with AdaQAT

For simplicity, we start by presenting AdaQAT in the context of learning only one bit-width for all weights in the network and similarly for activations. We then extend the method to per-layer bit-width allocation.

## 3.1 Objective Function

To learn the bit-widths of the uniform quantizers for both weights and activations, we use two real-valued variables $N_w$ and $N_a$, respectively. The actual integer bit-widths of the quantized network will be $\lceil N_w \rceil$ and $\lceil N_a \rceil$.

We use a regularization approach to model the loss function that takes into account the cost of a particular bit-width configuration, which corresponds to:

$$ \mathcal{L}_{\text{Total}} = \mathcal{L}_{\text{Task}}(\mathbf{Y}_{U,U}^{(L)}; \mathbf{T}) + \lambda \mathcal{L}_{\text{Hard}}(\lceil N_{\mathbf{w}} \rceil, \lceil N_{\mathbf{a}} \rceil), \quad (1) $$

where $\lambda > 0$ is a regularization hyper-parameter between the main task loss $\mathcal{L}_{\text{Task}}$ (i.e., the usual loss for the task that needs to be addressed) and the regularizing hardware loss $\mathcal{L}_{\text{Hard}}$ that measures the complexity of the model (in this case in terms of hardware cost). Here, $\mathbf{Y}_{U,U}^{(L)}$ represents the output of the L-layer quantized network we want to optimize, where its weights have been quantized to $\lceil N_w \rceil$ bits (the first U in the superscript notation) and its activations to $\lceil N_a \rceil$ bits (the second U in the superscript) on a current minibatch of data and T is the expected output.

FracBits [33] has reviewed various methods used to model the hardware cost of arithmetic precision choices for weights and activations. They argue in favor of memory size if only targeting weight quantization, and BitOPs for joint weight and activation quantization. As an example, for a linear layer, the BitOPs metric corresponds to

$$ \text{BitOPs}(l) = \lceil N_{\mathbf{w}} \rceil \lceil N_{\mathbf{a}} \rceil |w_l|, \quad (2) $$

where $l$ is a linear layer and $|w_l|$ denotes the number of weights in the linear layer $l$. Thus, the overall BitOPs of a model corresponds to the sum of each linear layer's BitOPs.

In our particular case, since we are using one bit-width per weights and one per activations, the overall BitOPs hardware cost will be linear in $\lceil N_w \rceil \lceil N_a \rceil$, and we can simplify the regularization term to

$$ \mathcal{L}_{\text{Hard}}(\lceil N_{\mathbf{w}} \rceil, \lceil N_{\mathbf{a}} \rceil) = \lceil N_{\mathbf{w}}13 - 14 \rceil \lceil N_{\mathbf{a}} \rceil. \quad (3) $$
Using Equation 3 will impact the value we need to consider for $\lambda$, which will be smaller than when integrating the scaling factors specific to each layer (determined by the layer type and shape) directly into $\mathcal{L}_{\text{Hard}}$.

## 3.2 Bit-Width Gradients and Parameter Updates

Since the task loss $\mathcal{L}_{\text{Task}}$ is not directly differentiable with respect to the bit-width parameters, we use finite difference approximations as

$$ 
\begin{aligned}
\frac{\partial \mathcal{L}_{\text{Task}}}{\partial N_{\mathbf{w}}} &\approx \mathcal{L}_{\text{Task}}(\mathbf{Y}_{\text{U,U}}^{(L)}; \mathbf{T}) - \mathcal{L}_{\text{Task}}(\mathbf{Y}_{\text{D,U}}^{(L)}; \mathbf{T}), \\
\frac{\partial \mathcal{L}_{\text{Task}}}{\partial N_{\mathbf{a}}} &\approx \mathcal{L}_{\text{Task}}(\mathbf{Y}_{\text{U,U}}^{(L)}; \mathbf{T}) - \mathcal{L}_{\text{Task}}(\mathbf{Y}_{\text{U,D}}^{(L)}; \mathbf{T}).
\end{aligned}
\quad (4) $$

The U (up) and D (down) subscripts in the $\mathbf{Y}^{(L)}$ outputs denote how the weights (first subscript) and activations (second subscript) have been rounded.

The gradients of the total loss 1 w.r.t. the bit-widths are

$$ \frac{\partial \mathcal{L}_{\text{Total}}}{\partial N_{\mathbf{x}}} \approx \frac{\partial \mathcal{L}_{\text{Task}}}{\partial N_{\mathbf{x}}} + \lambda \frac{\partial \mathcal{L}_{\text{Hard}}}{\partial [N_{\mathbf{x}}]}, \quad (5) $$

which are then used to update the fractional bit-width parameters. The gradient descent rule that does this takes the form

$$ N_{\mathbf{x}}^{+} = N_{\mathbf{x}} - \eta_{\mathbf{x}} \frac{\partial \mathcal{L}_{\text{Total}}}{\partial N_{\mathbf{x}}}, \quad (6) $$

with $\mathbf{x} \in \{\mathbf{w}, \mathbf{a}\}$, $N_{\mathbf{x}}^{+}$ the new bit-width values at the next iteration, and $\eta_{\mathbf{x}} > 0$ the corresponding learning rates.

To summarize, during the forward propagation phase, the weights and activations are quantized according to the learned integer bit-widths, $\lceil N_{\mathbf{w}} \rceil$ and $\lceil N_{\mathbf{a}} \rceil$, respectively, and to the quantization policy. In the gradient estimation phase, the bit-width gradients are approximated according to Equation 4 and Equation 5, while in the parameter update phase, we use Equation 6 to update the bit-width parameters. The rest of the network and quantizer parameters are updated in parallel using standard optimizers, with their separate hyperparameters.

When $N_{\mathbf{w}}$ and $N_{\mathbf{a}}$ reach their optimized values, further decreases cause a steep increase in the task loss $\mathcal{L}_{\text{Task}}$ and consequently $\mathcal{L}_{\text{Total}}$. Their gradient estimates in Equation 5 then become negative, and the update rule in Equation 6 starts increasing $N_{\mathbf{w}}$ and $N_{\mathbf{a}}$, producing an oscillatory pattern observed in several STE-based gradient approximation methods [19,15,17]. We monitor the oscillations, and once they exceed a threshold (empirically set to 10), we fix the bit-widths to $\lceil N_{\mathbf{w}} \rceil$ and $\lceil N_{\mathbf{a}} \rceil$, and continue the quantization process in the standard fashion.

## 3.3 Per-layer Extension

Extending the method to layer-wise precision configurations requires finding appropriate bit-width variables for each layer of the model such that, for an L-layer
model, $[N_{\mathbf{w}}^{(i)}]$ and $[N_{\mathbf{a}}^{(i)}]$ are the bit-widths for layer $i \in [1..L]$. This problem is non-trivial, since the computational cost of the gradient estimation phase would grow at least linearly with the number of bit-widths to learn. Instead, it is more cost-effective to focus on updating the bit-width of only one layer's weights or activations at a time, potentially switching the layer to update at each iteration. The overall approach can thus be viewed as a quantized version of a block coordinate descent [27] update rule. Block coordinate optimization methods update only a group of parameters at a time and have emerged as viable approaches to solve complex nonconvex optimization problems, like those involving "big data" (see [32] for instance and the references therein).

Careful attention must be paid when selecting which bit-widths to update at a given iteration. The most straightforward approach is to take a bit-width parameter at random, but this turns out to be highly suboptimal in our case. Greedy-type approaches offer better results: at each iteration, update (e.g., decrease) the bit-width that is most likely to degrade accuracy the least on the current training batch.

*Weights and Activations* Similarly to the per-network bit-width learning context, one must decide if weight and activation bit-widths should be updated simultaneously or in sequence. Whereas in the per-network setting model accuracy is not sensitive to this choice (simultaneous and in-sequence updates tend to behave analogously), at finer granularities (e.g., layer or channel) this is not the case. Taking bit-widths separately allows more freedom in exploring the search space. It is also generally the case that activations are more sensitive to perturbations than weights, meaning that model accuracy is less impacted by focusing on weights bit-width learning first and then moving on to activation bit-widths. The fact that activation values depend on the weights (and inputs) also makes it more natural to start with weights.

*Bit-width update selection metrics* Choosing to always update the bit-width that leads to the least significant reduction (i.e., with the smallest cost) is a sensible approach. Accordingly, we adopted a Mean Squared Error (MSE) strategy to select which bit-width to update in the subsequent block coordinate descent iteration. For two quantized tensors of the same shape $\tilde{\mathbf{X}}_1$ and $\tilde{\mathbf{X}}_2$ (i.e., either weights or activations, before and after the corresponding bit-width parameter has been updated), the cost is their mean squared error:

$$ \text{MSE}(\tilde{\mathbf{X}}_1, \tilde{\mathbf{X}}_2) := \|\tilde{\mathbf{X}}_1 - \tilde{\mathbf{X}}_2\|_F^2 / \text{size}(\tilde{\mathbf{X}}_1), \quad (7) $$

where $\text{size}(\tilde{\mathbf{X}}_1)$ is the number of elements in $\tilde{\mathbf{X}}_1$ and $\|\cdot\|_F$ is the Frobenius norm.

# 4 Experiments and Discussion

This section provides an experimental evaluation of our PyTorch-based framework, compared to the state-of-the-art. We evaluate model efficiency using memory size and the BitOPs metric (2). As state-of-the-art mixed-precision models do
<table><caption>**Table 1.** Comparison with state-of-the-art quantization methods on SQuAD.</caption><thead><tr><th rowspan="2">Model</th><th>Bit-width</th><th>Size</th><th>BitOPs</th><th colspan="2">SQuAD v1.1</th><th colspan="2">SQuAD v2.0</th></tr><tr><th>W-E-A</th><th>(MB)</th><th>(×10<sup>9</sup>)</th><th>EM</th><th>F<sub>1</sub>-score</th><th>EM</th><th>F<sub>1</sub>-score</th></tr></thead><tbody><tr><td>BERT<sub>BASE</sub></td><td>fp32</td><td>417.6</td><td>/</td><td>81.5</td><td>88.7</td><td>74.5</td><td>77.7</td></tr><tr><td>TernaryBERT</td><td>2-2-8</td><td>28.0</td><td>1.368</td><td>80.1</td><td>87.5</td><td>73.3</td><td>76.6</td></tr><tr><td>BinaryBERT</td><td>1-1-4</td><td>16.5</td><td>0.342</td><td>79.3</td><td>87.2</td><td>72.5</td><td>75.4</td></tr><tr><td>Q-BERT</td><td>MP-8-8</td><td>53.2</td><td>≈ 1.80</td><td>79.85</td><td>87.49</td><td>/</td><td>/</td></tr><tr><td>AQ-BERT</td><td>MP-8-8</td><td>55.0</td><td>≈ 1.92</td><td>79.85</td><td>87.00</td><td>/</td><td>/</td></tr><tr><td>AdaQAT</td><td>2.08-4-3.93</td><td>29.2</td><td>0.677</td><td>/</td><td>/</td><td>75.2</td><td>77.9</td></tr><tr><td></td><td>2.06-4-3.96</td><td>35.9</td><td>0.701</td><td>81.07</td><td>88.71</td><td>/</td><td>/</td></tr></tbody></table>

not report per-layer bit-widths, we approximate BitOPs via memory size. Given that activations are uniform across layers, they can be factorized, reducing the calculation to the memory size of weights (excluding embedding weights).

$$ \text{BitOPs}(M) = N_{\mathbf{a}} \sum_{i=0}^{L-1} \left[ N_{\mathbf{w}}^{(i)} \right] |w_i| = N_{\mathbf{a}} \times \text{memsize}(W) \quad (8) $$

where *M* is an *L*-layer model and memsize(*W*) is computed in MB.

All tables report bit-widths in the W-E-A format, corresponding to linear-layer weights, word embeddings, and activations. When zooming into embedding layers, T-S-P refers to token, segment and position. For state-of-the-art methods without precise configurations, MP denotes variable-length layers. Size includes embedding weights unless stated otherwise. Depending on the task, we report exact match (EM) and F₁ scores.

*Quantization Procedures* We quantize the weights and the inputs (activations) from all Transformer layers. For a fair comparison with state-of-the-art, only word embeddings are quantized in the sections on mixed-precision. The embedding bit-width is also not included in the mixed-precision search space. The embedding layer quantization is explored in detail in Section 4.3. Biases in linear layers, softmax, layer normalization, and the final task-specific layer remain unquantized. LSQ [10] is used to learn quantization ranges for both weights and activations. We use the fp32 BERT fine-tuned on the downstream task to initialize our quantized model.

*Benchmarks* We evaluate our models on SQuAD and GLUE. SQuAD v1.1 [22] contains over 100,000 question-answer pairs based on Wikipedia, and v2.0 [23] adds over 50,000 unanswerable questions. GLUE [28] comprises nine natural language understanding tasks spanning diverse datasets, difficulties, and text genres. These benchmarks assess performance across a broad range of reading comprehension and linguistic phenomena.

## 4.1 Comparison of AdaQAT with SotA

Table 1 shows the results of applying AdaQAT on SQuAD datasets compared to other methods from the literature. The first line shows the floating-point
baseline result, whereas the second group of lines showcases *static* methods, where activations are not quantized and weights are quantized uniformly. The BitOPs for Q-BERT and AQ-BERT were not provided by the authors, hence we approximate them using the given memory size and the model linear weights.

The third group of lines corresponds to mixed-precision methods where the weight bit-width is learned and the activations are quantized uniformly to a manually set bit-width. The last groups of lines present our mixed-precision BERT<sub>BASE</sub> results on SQuAD. EM and F₁-score are similar to or slightly better than the fp32 baseline. Compared to the other mixed-precision models Q-BERT and AQ-BERT, considering activations significantly reduces the computational cost of the model. Our configurations also significantly reduced the size required to store the weights in memory, while achieving better accuracy. Although mixed-precision models require more memory than binary or ternary models, they offer greater accuracy closer to the baseline. The computational cost of activations is also not negligible, with much higher BitOPs for TernaryBERT despite a smaller memory size for weights compared to our mixed-precision models.

## 4.2 Mixed-precision for diverse language tasks with GLUE Dataset

Table 2 presents the state-of-the-art results for quantized BERT<sub>BASE</sub> models on the GLUE benchmark. Note that the mixed-precision approaches, up to our knowledge, have considered only a subset of GLUE datasets/tasks. Table 3 presents our mixed-precision results on the full GLUE benchmark.

Compared to Q-BERT, our approach demonstrates clearly superior results in both accuracy and size. By including the embedding and activation layer quantization into the optimization objective, AdaQAT usually maintains the baseline accuracy drop within 1% while dividing the memory footprint by roughly 11 compared to the fp32 baseline, and by 1.5 compared to Q-BERT. This demonstrates the robustness of BERT towards full model quantization.

Extremely low but uniform precision models, TernaryBERT and Binary-BERT, perform very well on most of the tasks but suffer on datasets that are sensitive to quantization, e.g. CoLa. With AdaQAT one can automatically find mixed-precision configurations that have slightly larger configurations that add the necessary bit-widths at appropriate layers. For instance, for CoLA, we obtained that a configuration 2.41-4-4.30 limits the accuracy drop to only 1%, while the TernaryBERT 2-2-8 configuration suffers a roughly 12% accuracy loss compared to the baseline. This non-trivial solution of decreasing activation bit-widths while non-homogeneously increasing the formats of weights and embeddings is impossible to find by hand but requires an optimization-based approach.

## 4.3 Quantization effects on the embedding layer

The weights of the embedding layer account for 21% of the total parameters in the BERT<sub>BASE</sub> model. Of this 21%, 98% are token weights. For this reason, in SotA approaches, token weights are typically included in quantization exploration, while segment and position weights are not quantized as they are
<table><caption>**Table 2.** State-of-the-Art quantization results on the GLUE benchmark.</caption><thead><tr><th>Model</th><th>Bit-width<br>W-E-A</th><th>Size<br>(MB)</th><th>BitOPs<br>(×10<sup>9</sup>)</th><th>MNLI<br>-m/mm</th><th>QQP</th><th>QNLI</th><th>SST-2</th><th>CoLA</th><th>STS-B</th><th>MRPC</th><th>RTE</th></tr></thead><tbody><tr><td>BERT<sub>BASE</sub></td><td>fp32</td><td>417.6</td><td>/</td><td>84.57/84.46</td><td>91.39</td><td>91.34</td><td>92.32</td><td>59.67</td><td>90.14</td><td>86.28</td><td>69.93</td></tr><tr><td rowspan="3">Q-BERT</td><td>2-8-8</td><td>43.1</td><td>1.368</td><td>76.56/77.02</td><td>/</td><td>/</td><td>84.63</td><td>/</td><td>/</td><td>/</td><td>/</td></tr><tr><td>MP-8-8</td><td>53.2/53.2</td><td>≈ 1.80</td><td>83.51/83.55</td><td>/</td><td>/</td><td>92.55</td><td>/</td><td>/</td><td>/</td><td>/</td></tr><tr><td>MP-8-8</td><td>46.1/48.1</td><td>≈ 1.35/1.48</td><td>81.75/82.29</td><td>/</td><td>/</td><td>92.08</td><td>/</td><td>/</td><td>/</td><td>/</td></tr><tr><td>TernaryBERT</td><td>2-2-8</td><td>28.0</td><td>1.368</td><td>83.0/82.2</td><td>88.4</td><td>90.0</td><td>92.9</td><td>47.8</td><td>84.3</td><td>87.5</td><td>68.4</td></tr><tr><td>BinaryBERT</td><td>1-1-4</td><td>16.5</td><td>0.342</td><td>83.9/84.2</td><td>91.2</td><td>90.9</td><td>92.3</td><td>44.4</td><td>87.2</td><td>83.3</td><td>65.3</td></tr></tbody></table>

<table><caption>**Table 3.** Our mixed-precision results with AdaQAT on the GLUE benchmark.</caption><thead><tr><th></th><th>MNLI<br>-m/mm</th><th>QQP</th><th>QNLI</th><th>SST-2</th><th>CoLA</th><th>STS-B</th><th>MRPC</th><th>RTE</th></tr></thead><tbody><tr><td>Accuracy (%)</td><td>84.66/84.38</td><td>90.96</td><td>91.72</td><td>92.43</td><td>58.02</td><td>89.73</td><td>86.50</td><td>69.10</td></tr><tr><td>Bit-width W-E-A</td><td>2.25-4.4-11</td><td>2.16-4.4-03</td><td>2.18-4-4.14</td><td>2.16-4-4.07</td><td>2.41-4-4.30</td><td>2.27-4-4.21</td><td>2.24-4-4.18</td><td>2.47-4-4.34</td></tr><tr><td>Size (MB)</td><td>36.67</td><td>36.23</td><td>36.75</td><td>37.11</td><td>39.77</td><td>37.70</td><td>37.04</td><td>40.28</td></tr><tr><td>BitOPs (×10<sup>9</sup>)</td><td>0.756</td><td>0.737</td><td>0.773</td><td>0.781</td><td>0.902</td><td>0.815</td><td>0.792</td><td>0.925</td></tr></tbody></table>

considered negligible. While this is a reasonable argument, this perspective is inappropriate when considering compression for model deployment on resource-constrained devices.

Q-BERT [25] examines the impact of quantizing word embeddings and position embeddings in the BERT<sub>BASE</sub> model. The authors argue that the embedding layer is more sensitive to quantization than the linear layer, with performance drops of up to 10% on tasks like SST-2, MNLI, and CoNLL-03, and over 20% on SQuAD v1.1 when using 4-bit embedding quantization. Additionally, they find that position embeddings are more sensitive to quantization, leading to an extra 2% performance degradation compared to quantizing word embeddings alone. GOBO [35] observed different behaviors, suggesting that it is nevertheless possible to lower the bit-width of the embeddings without significantly impacting accuracy, which we confirm with our experiments as well.

Table 4 shows the size of the embedding layer, model accuracy across multiple datasets and different quantization configurations. These results indicate that it is actually possible to quantize all weights in the embedding layer to 2-bit while maintaining the model accuracy drop below 2% on the evaluated datasets. The memory footprint of the embedding layer is reduced by a factor of 16 compared to the single-precision model and by a factor of 4.2 compared to the approach generally used in state-of-the-art work, i.e. quantizing only the token weights.

# 5 Conclusion & Future Work

In this paper, we improve the efficiency of BERT-based models via AdaQAT, a mixed-precision algorithm that considers weights, activations, and embedding layers. Our mixed-precision BERT<sub>BASE</sub> uses low bit-width (< 8) fixed-point representations across linear and activation layers, often preserving baseline accuracy while reducing computational cost by over 2x and memory size by 1.5x com-
<table><caption>**Table 4.** Evaluation of the quantization effects of embedding weights on various tasks. CR is the embedding weights compression ratio w.r.t. fp32 baseline.</caption><thead><tr><th>Model</th><th>Bit-width<br>W-A</th><th>Embedding<br>T-S-P</th><th>Embedding<br>size (MB)</th><th>CR<br>×</th><th>MNLI<br>-m/mm</th><th>SST-2</th><th>SQuAD v1.1<br>EM</th><th>SQuAD v1.1<br>F₁-score</th><th>SQuAD v2.0<br>EM</th><th>SQuAD v2.0<br>F₁-score</th></tr></thead><tbody><tr><td>BERT<sub>BASE</sub></td><td>fp32</td><td>fp32</td><td>95.32</td><td>1.00</td><td>84.57/84.46</td><td>92.32</td><td>81.41</td><td>88.61</td><td>74.45</td><td>77.64</td></tr><tr><td rowspan="4">Q-BERT</td><td rowspan="4">8-8</td><td>8-fp32-8</td><td>23.84</td><td>4.00</td><td>83.83/83.91</td><td>92.88</td><td>/</td><td>88.47</td><td>/</td><td>/</td></tr><tr><td>4-fp32-8</td><td>12.12</td><td>7.86</td><td>82.91/83.67</td><td>91.74</td><td>/</td><td>87.55</td><td>/</td><td>/</td></tr><tr><td>8-fp32-4</td><td>23.64</td><td>4.03</td><td>82.84/82.25</td><td>89.11</td><td>/</td><td>72.38</td><td>/</td><td>/</td></tr><tr><td>4-fp32-4</td><td>11.92</td><td>8.00</td><td>78.08/78.96</td><td>85.55</td><td>/</td><td>61.70</td><td>/</td><td>/</td></tr><tr><td rowspan="4">AdaQAT</td><td rowspan="4">8-8</td><td>8-fp32-fp32</td><td>25.01</td><td>3.81</td><td>84.57/84.46</td><td>92.89</td><td>83.59</td><td>90.56</td><td>75.17</td><td>78.17</td></tr><tr><td>4-fp32-fp32</td><td>13.30</td><td>7.17</td><td>84.95/84.72</td><td>92.66</td><td>83.54</td><td>90.55</td><td>75.20</td><td>78.24</td></tr><tr><td>8-fp32-8</td><td>23.84</td><td>4.00</td><td>85.05/84.78</td><td>92.55</td><td>83.55</td><td>90.57</td><td>75.27</td><td>78.20</td></tr><tr><td>4-4-4</td><td>11.91</td><td>8.00</td><td>85.02/84.43</td><td>92.78</td><td>83.54</td><td>90.56</td><td>75.28</td><td>78.23</td></tr><tr><td rowspan="4">2-4</td><td rowspan="4">2-4</td><td>2-2-2</td><td>5.96</td><td>16.0</td><td>84.97/84.64</td><td>92.66</td><td>83.60</td><td>90.47</td><td>75.38</td><td>78.29</td></tr><tr><td>4-fp32-fp32</td><td>13.30</td><td>7.17</td><td>83.89/84.14</td><td>91.17</td><td>81.58</td><td>89.00</td><td>74.98</td><td>76.86</td></tr><tr><td>4-fp32-4</td><td>11.92</td><td>8.00</td><td>83.76/83.60</td><td>91.17</td><td>81.47</td><td>89.00</td><td>74.80</td><td>77.60</td></tr><tr><td>4-4-4</td><td>11.91</td><td>8.00</td><td>83.78/84.11</td><td>91.17</td><td>81.51</td><td>89.00</td><td>74.43</td><td>77.44</td></tr><tr><td></td><td></td><td>2-2-2</td><td>5.96</td><td>16.0</td><td>83.79/84.30</td><td>90.60</td><td>81.36</td><td>88.91</td><td>74.11</td><td>77.14</td></tr></tbody></table>

pared to weight-only quantization approaches. We further evaluate quantizing all embedding weights, showing that 2-bit representations maintain accuracy within 2% of the baseline while reducing the embedding memory footprint by 16× versus fp32 and 4.2× versus prior work.

Future work includes applying mixed-precision to smaller models such as TinyBERT [14] and DistilBERT [24], which are more sensitive to quantization, as well as exploring domain-specific models and datasets where non-trivial mixed-precision assignments may yield efficient configurations.

## Acknowledgment

This work was performed using HPC resources from GENCI-IDRIS (Grant 2025-AD011016060). This work has received French government support granted to the CominLabs Excellence Laboratory and managed by the National Research Agency in the “Investing for the Future” program under reference ANR-10-LABX-07-01. This work has been partially funded by the French National Research Agency project PEPR AI under the reference ANR-23-PEIA-0010.

## References

1. Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., King, I.: Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701 (2020)

2. Banner, R., Nahshan, Y., Soudry, D.: Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems 32 (2019)
3. Bengio, Y., Léonard, N., Courville, A.: Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432 (2013)

4. Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., Kwak, N.: LSQ+: Improving Low-Bit Quantization Through Learnable Offsets and Better Initialization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 696–697 (2020)

5. Choi, J., Wang, Z., Venkataramani, S., Chuang, P.I.J., Srinivasan, V., Gopalakrishnan, K.: PACT: Parameterized Clipping Activation for Quantized Neural Networks. arXiv preprint arXiv:1805.06085 (2018)

6. Courbariaux, M., Bengio, Y., David, J.P.: BinaryConnect: Training Deep Neural Networks with binary weights during propagations. In: Advances in Neural Information Processing Systems 28 – NIPS 2015. NIPS'15, vol. 2, pp. 3123–3131. MIT Press (2015)

7. Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. arXiv preprint arXiv:1602.02830 (2016)

8. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT. vol. 1, p. 2 (2019)

9. Dong, Z., Yao, Z., Gholami, A., Mahoney, M.W., Keutzer, K.: HAWQ: Hessian Aware Quantization of Neural Networks with Mixed-Precision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 293–302 (2019)

10. Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R., Modha, D.S.: Learned Step Size Quantization. arXiv preprint arXiv:1902.08153 (2019)

11. Gernigon, C., Filip, S.I., Sentieys, O., Coggiola, C., Bruno, M.: Adaqat: Adaptive bit-width quantization-aware training. In: 2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS). pp. 442–446. IEEE (2024)

12. Gordon, M.A., Duh, K., Andrews, N.: Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307 (2020)

13. Huang, X., Shen, Z., Li, S., Liu, Z., Xianghong, H., Wicaksana, J., Xing, E., Cheng, K.T.: SDQ: Stochastic Differentiable Quantization with Mixed Precision. In: International Conference on Machine Learning. pp. 9295–9309. PMLR (2022)

14. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., Liu, Q.: Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 (2019)

15. Kuzmin, A., Van Baalen, M., Ren, Y., Nagel, M., Peters, J., Blankevoort, T.: Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Systems 35, 14651–14662 (2022)

16. Liu, J., Cai, J., Zhuang, B.: Sharpness-Aware Quantization for Deep Neural Networks. arXiv preprint arXiv:2111.12273 (2021)

17. Liu, S.Y., Liu, Z., Cheng, K.T.: Oscillation-free quantization for low-bit vision transformers. In: International Conference on Machine Learning. pp. 21813–21824. PMLR (2023)

18. Ma, Y., Jin, T., Zheng, X., Wang, Y., Li, H., Wu, Y., Jiang, G., Zhang, W., Ji, R.: OMPQ: Orthogonal Mixed Precision Quantization. arXiv preprint arXiv:2109.07865 (2021)

19. Nagel, M., Fournarakis, M., Bondarenko, Y., Blankevoort, T.: Overcoming oscillations in quantization-aware training. In: International Conference on Machine Learning. pp. 16318–16330. PMLR (2022)
20. Nikolić, M., Hacene, G.B., Bannon, C., Lascorz, A.D., Courbariaux, M., Bengio, Y., Gripon, V., Moshovos, A.: BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization. arXiv preprint arXiv:2002.03090 (2020)

21. Qin, H., Ding, Y., Zhang, M., Yan, Q., Liu, A., Dang, Q., Liu, Z., Liu, X.: Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203.06390 (2022)

22. Rajpurkar, P.: Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)

23. Rajpurkar, P., Jia, R., Liang, P.: Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 (2018)

24. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

25. Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M.W., Keutzer, K.: Q-bert: Hessian based ultra low precision quantization of bert. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 8815–8821 (2020)

26. Tang, H., Zhang, X., Liu, K., Zhu, J., Kang, Z.: Mkq-bert: Quantized bert with 4-bits weights and activations. arXiv preprint arXiv:2203.13483 (2022)

27. Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications **109**, 475–494 (2001)

28. Wang, A.: Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 (2018)

29. Wang, K., Liu, Z., Lin, Y., Lin, J., Han, S.: HAQ: Hardware-Aware Automated Quantization With Mixed Precision. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019). https://doi.org/10.1109/CVPR.2019.00881

30. Wang, Z., Li, J.B., Qu, S., Metze, F., Strubell, E.: Squat: Sharpness-and quantization-aware training for bert. arXiv preprint arXiv:2210.07171 (2022)

31. Wu, B., Wang, Y., Zhang, P., Tian, Y., Vajda, P., Keutzer, K.: Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search. arXiv preprint arXiv:1812.00090 (2018)

32. Xu, Y., Yin, W.: A globally convergent algorithm for nonconvex optimization based on block coordinate update. Journal of Scientific Computing **72**(2), 700–734 (2017)

33. Yang, L., Jin, Q.: FracBits: Mixed Precision Quantization via Fractional Bit-Widths. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 10612–10620 (2021)

34. Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., Mahoney, M., et al.: HAWQ-V3: Dyadic Neural Network Quantization. In: International Conference on Machine Learning. pp. 11875–11886. PMLR (2021)

35. Zadeh, A.H., Edo, I., Awad, O.M., Moshovos, A.: Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In: 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). pp. 811–824. IEEE (2020)

36. Zafrir, O., Boudoukh, G., Izsak, P., Wasserblat, M.: Q8bert: Quantized 8bit bert. In: 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS). pp. 36–39. IEEE (2019)

37. Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., Liu, Q.: Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812 (2020)

38. Zhao, C., Hua, T., Shen, Y., Lou, Q., Jin, H.: Automatic mixed-precision quantization search of bert. arXiv preprint arXiv:2112.14938 (2021)

39. Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., Zou, Y.: DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv:1606.06160 (2016)