Improving Efficiency in Large Language Models via Extendable Block Floating Point Representation
Authors: Dongyang Li, Zeyang Li, Bosheng Liu, Jigang Wu
Accepted at: acl25

Improving Efficiency in Large Language Models via Extendable Block Floating Point Representation

Dongyang Li1, Zeyang Li2, Bosheng Liu1*, Jigang Wu1*
1School of Computer Science and Technology, Guangdong University of Technology
2School of Automation, Guangdong University of Technology
liubosheng@gdut.edu.cn, asjgwucn@outlook.com

Abstract
Large language models (LLMs) have revolutionized natural language processing (NLP) tasks, yet their increasing size poses substantial challenges in terms of computational and memory resources. Block floating-point (BFP) arithmetic offers an effective solution by leveraging the strengths of both floating-point and fixed-point representations, leading to reductions in both storage and computational overhead. However, current low-bit BFP quantization approaches often struggle to handle extreme outliers, leading to significant accuracy degradation. To overcome this limitation, we introduce Extendable Exponent Sharing (EES), a novel BFP representation that extends the exponent bit width to capture a wider dynamic range. EES achieves this by embedding extendable exponent bits into the least significant mantissa bits, thereby increasing the shared exponent’s bit width without incurring additional storage costs. To optimize the trade-off between accuracy and energy efficiency, EES employs a design space exploration strategy to optimize the configuration of extendable exponent bit widths. Experimental results show that EES outperforms representative baselines in both accuracy and computational efficiency.

1 Introduction
Large language models (LLMs) have achieved remarkable success across a variety of natural language processing tasks, such as intelligent assistants, machine translation, and sentiment analysis (Nam et al., 2024; Kim et al., 2024; Zhao et al., 2024). However, as the size of models continues to scale, the computational and memory requirements grow substantially (Liu et al., 2024). Block floating point (BFP) representation provides an effective solution to address these challenges, balancing precision with efficiency, making it particularly well-suited to meet the resource-intensive demands of LLMs (Zeng et al., 2024; Qin et al., 2024). By grouping multiple values under a shared exponent, BFP enables efficient storage and calculation of large model parameters with minimal overhead (Darvish Rouhani et al., 2023; Zhang et al., 2024). In comparison to both fixed-point and floating-point formats, BFP offers distinct advantages, especially in large-scale computations. It minimizes memory usage by sharing an exponent across values, making it more storage-efficient than full-precision floating point, while providing greater numerical stability than fixed-point representation (Noh et al., 2023; Shen et al., 2024).

Recent advancements in low-bit BFP quantization have shown great promise in reducing memory and computational costs for LLM applications (Song et al., 2018; Lo et al., 2023). BFP effectively balances numerical precision and resource efficiency (Lo and Liu, 2023), but its limited exponent range in low-bit formats presents significant challenges. For example, in an 8-bit BFP representation using the E3M4 format, a block of numbers shares a 3-bit exponent and uses 4-bit mantissas. If the shared exponent is set to 4 (\(2^4 = 16\)), the largest representable value in the block is 240 (\(15 \times 16 = 240\)). Any value exceeding this, such as 300, results in overflow, as the shared exponent cannot accommodate larger values within the given bit width. This limitation becomes particularly problematic for extreme outliers, which are prevalent in the parameters and activations of LLMs, as highlighted in Fig. 1. The inability to accurately handle these outliers often causes significant accuracy degradation (Zhang et al., 2023; Nie et al., 2019; Chen et al., 2023). Solving this issue is essential for enabling the broader application of BFP in high-precision LLM workloads.

To mitigate the challenges posed by extreme outliers in low-bit BFP quantization, various techniques have been proposed. SuperWeight (Yu et al., 2024) tackles the issue by identifying and
Figure 1: Statistical exponent distribution of input activations and weights based on the layers.1.mlp.down_proj of the Llama-3.1-8B model.

preserving critical outlier weights during quantization, achieving a balance between compression efficiency and accuracy. Similarly, SmoothQuant (Xiao et al., 2023) adopts a per-channel scaling approach to redistribute value ranges, effectively shifting the quantization burden of outliers from activations to weights while preserving mathematical equivalence. Building on these strategies, BiE (Zou et al., 2024) introduces a dual-exponent mechanism to address outliers, ensuring improved accuracy. However, this method incurs additional resource overhead due to the inclusion of flag bits during computational implementation. Despite these advancements, existing methods only partially alleviate the challenges of low-bit BFP quantization (Jang et al., 2024). The inherent limitation of a narrow exponent width remains a critical bottleneck, restricting the efficient handling of extreme values and limiting computational performance.

In this work, we introduce Extendable Exponent Sharing (EES), a post-training quantization approach based on BFP designed to improve dynamic range and computational efficiency. EES extends the traditional shared exponent method by increasing the exponent bit width to cover a broader range of values. However, directly expanding the exponent width increases the storage requirements. To address this, we observe that changes in lower-order bits have a minimal effect on computations and model accuracy. For example, two 8-bit BFP numbers—1101.1010 and 1101.1001—differing only in the least significant bit show a minimal impact on precision, indicating that small changes in lower-order bits do not significantly affect accuracy. Using this insight, EES embeds the extended exponent bits into the least significant bits of the partial mantissas. These bits are then combined with the shared exponent to form an extended exponent, as shown in Fig. 2. Since this approach affects the precision of mantissas, we analyze the quantization error introduced by EES compared to standard BFP. To achieve an optimal balance between accuracy and computational efficiency, we use a design space exploration strategy to obtain the best configurations for both activations and weights. Evaluations in LLMs show that EES significantly improves both accuracy and energy efficiency. Key contributions of this work include:

• Introduction of EES: We introduce a new numerical representation, Extendable Exponent Sharing (EES), which uses extendable exponents to better handle outliers, resulting in improved accuracy and computational efficiency.

• Optimized Exponent Configuration: We analyze the distribution of exponents in activations and weights, and apply a design space exploration strategy to determine the optimal bit widths for the extended exponent.

• Enhanced Performance: Experimental results indicate that EES delivers up to a 1.38× improvement in energy efficiency, alongside a maximum accuracy boost of 0.36% when compared to cutting-edge BFP techniques.

2 Related Work

Quantization is a key technique used to compress LLMs by lower-precision data formats, significantly reducing memory usage and computational costs (Rokh et al., 2022). The two dominant data formats in this space are fixed-point (Lee et al., 2024) and BFP (Zeng et al., 2024) representations. Fixed-point representation simplifies operations like multiplication and accumulation compared to floating-point formats, but it has limited precision due to the fixed offset and scaling. (Nair et al., 2021; Benmaghnia et al., 2022) Differently, BFP combines the computational efficiency of fixed-point arithmetic with the dynamic range of floating-point formats, offering an effective balance between precision and performance (Basumallik et al., 2022).

Recent progress in BFP quantization, including techniques like DBPS (Lee et al., 2023), Static BFP (Fan et al., 2021), and MSFP12 (Darvish Rouhani et al., 2020), has shown its ability to achieve high accuracy, efficient computation, and reduced storage costs. Additionally, research into low-bit BFP representations aims to improve these advantages. For example, FAST (Zhang et al., 2022a) introduces low-bit BFP quantization for activations and
Figure 2: Profile of extended exponent sharing by compared with floating-point (FP) and BFP. The shared exponent of EES consists of \( l_S \) bits of original shared exponent and \( l_{EE} \) bits of extendable shared exponent.

weights, supporting efficient matrix multiplications with reduced precision. Similarly, Bucket (Lo and Liu, 2023) and FIGNA (Jang et al., 2024) adopt 11-bit and 8-bit BFP formats, respectively, to enhance computational performance and energy efficiency.

In the BFP representation, data can be classified into two categories: “normal values,” which are accurately represented, and “extreme outlier values,” which fall outside representable range and require approximation through shifting or scaling. These outliers are a major challenge in low-bit BFP quantization, as they can significantly degrade model accuracy. Techniques like AWQ (Lin et al., 2024) and SuperWeight (Yu et al., 2024) identify and preserve “super weights” — critical parameters with extreme values — to mitigate the negative effects of outliers on model quality. BIE (Zou et al., 2024) introduces dual shared exponents to manage both normal values and outliers in both activations and weights, ensuring high accuracy for LLM models.

While these methods address outlier challenges, the limited bit width of the shared exponent in BFP remains a key limitation for representing a wider range of values. We propose a scalable shared exponent strategy that extends the range of numerical representation and improves the performance of low-bit BFP quantization for LLMs.

3 Preliminary

As shown in Fig. 2(a), a set of \( N \) floating-point numbers, denoted as \( X \), can be given by:
\[
X = [(-1)^{s_0} m_0 2^{E_0}, \ldots, (-1)^{s_{N-1}} m_{N-1} 2^{E_{N-1}}],
\]
where \( s_i, m_i \), and \( E_i \) are the \( i \)-th sign, mantissa, and exponent (\( 0 \leq i \leq N - 1 \)), respectively.

When converting floating-point data into BFP representation, we obtain the new data set \( X' \), as illustrated in Fig. 2(b), which is given by:
\[
X' = [(-1)^{s_0} m'_0, \ldots, (-1)^{s_{N-1}} m'_{N-1}] \cdot 2^{E'_S},
\]
where \( E'_S \) is the shared exponent, calculated as:
\[
E'_S = \max E_i \mid i \in 0, \ldots, N - 1.
\]
The bit width of shared exponent \( E'_S \) equals to
\[
l_{E'_S} = l_S
\]
where \( l_S \) denotes the length of the shared exponent.

For each mantissa in this BFP representation, \( m'_i \) is computed by shifting the original mantissa \( m_i \) based on the difference between the maximum shared exponent (\( E'_S \)) and the individual exponent (\( E_i \)), using the right shift operation:
\[
m'_i = m_i \gg (E'_S - E_i)
\]
where \( \gg \) denotes the right shift operation.

Assuming the value of input activations and weights are known and the rounding-to-nearest method is used (Kalliojarvi and Astola, 1996; Song et al., 2018), the mean of \( \alpha \) is zero, and the error variance (\( \sigma^2_\alpha \)) can be formulated as
\[
\sigma^2 = \frac{2^{-2l_m}}{12} \cdot 2^{2E'_S}
\]
where \( l_m \) is the bit width of BFP mantissa.

4 Methodology

This section exhibits the EES framework, detailing the quantization workflow, error analysis, and hardware architecture.

Figure 3: Workflow. (a) Establishing exponent bit width.(b) Extended exponent sharing. (c) Bit width space exploration. Acc denotes the model accuracy, while EE denotes the energy efficiency.
4.1 Overview
Fig. 3 depicts the EES quantization workflow, which comprises three primary components: exponent bit width establishment, EES numerical representation, and bit width space exploration. The exponent bit width establishment component determines the bit widths for activations and weights by analyzing their exponent distributions. The EES numerical representation component defines the extendable exponent format during the transformation from FP to EES format. The bit width space exploration component optimizes the trade-off between model accuracy and energy efficiency to determine the optimal EES quantization configuration.

Exponent Bit Width Establishment. Fig. 3(a) illustrates the process of determining the exponent bit widths for activations and weights based on their statistical distribution ranges. The original shared exponent bit widths, \( l_{SA} \) for activations and \( l_{SW} \) for weights, restrict the representable ranges to \([ -2^{l_{SA}-1}, 2^{l_{SA}-1}-1 ]\) and \([ -2^{l_{SW}-1}, 2^{l_{SW}-1}-1 ]\), respectively. These ranges, however, are insufficient to fully capture data with large outliers, which poses challenges for accurate representation.

To address this, the shared exponent bit-width for activations and weights are augmented by introducing extendable shared exponent bits, thereby expanding the dynamic range. The total shared exponent bit width includes two components: the original shared exponent bit width (\( l_{SA} \) and \( l_{SW} \)) and the extendable shared exponent bit width (\( l_A \) for activations and \( l_W \) for weights). This unified representation is expressed as:

\[
l_{ES} = l_S + l_{EE}
\]

where \( l_{ES} \) represents the total shared exponent bit width, comprising the original bit width \( l_S \) and the extendable bit width \( l_{EE} \).

Following this expansion, the range of the shared exponent \( E_S \) is given by:

\[
E_S \in [ -2^{l_S+l_{EE}-1}, 2^{l_S+l_{EE}-1}-1 ]
\]

EES Numerical Representation. Fig. 3 (b) illustrates the EES numerical representation, which is derived from the conversion of original floating-point data within a tensor to the extendable exponent sharing format in a block. The EES representation, denoted as \( X'' \), is expressed as:

\[
X'' = [(-1)^{s_0}m''_0, ..., (-1)^{s_{N-1}}m''_{N-1}] \cdot 2^{E_S}
\]

where \( E_S \) represents the shared exponent.

The incorporation of the extendable exponent into the partial least significant bits of the private mantissas modifies the mantissa representation. The updated mantissa, \( m''_i \), is defined as:

\[
m''_i = \begin{cases}
m'_i[l_m-1:1], E_E[i]; & 0 \leq i \leq l_{EE} - 1 \\
m'_i[l_m-1:0]; & l_{EE} \leq i \leq N - 1
\end{cases}
\]

where \( m'_i[l_m-1:1] \) denotes the bits ranging from the first to the \( (l_m-1) \)-th position of the original mantissa \( m'_i \). Here, \( E_E \), the value of the extendable exponent, is derived from the least significant bits of the shared exponent \( E_S \) and is represented as:

\[
E_E = E_S[l_{EE} - 1 : 0]
\]

Vector Multiplication in Blocks. Given two blocks \( X_1 \) and \( X_2 \) with \( N \) elements in EES format, the dot product is computed as:

\[
X''_1 \cdot X''_2 = 2^{E_A+E_W} \sum_{i=0}^{N-1} (-1)^{S_{Ai} \oplus S_{Wi}} (M_{Ai} \cdot M_{Wi})
\]

where \( E_A \) and \( E_W \) represent the effective exponents for \( X''_1 \) and \( X''_2 \), respectively, defined as:

\[
\begin{align*}
E_A &= (E_{SA} << l_A) + E_{EA} \\
E_W &= (E_{SW} << l_W) + E_{EW}
\end{align*}
\]

where \( E_{SA} \) and \( E_{SW} \) denote the original shared exponent bit width for \( X''_1 \) and \( X''_2 \), respectively. \( E_{EA} \) and \( E_{EW} \) respectively correspond to the extendable exponent values for \( X''_1 \) and \( X''_2 \). \( l_A \) and \( l_W \) indicate the bit widths of extendable exponents.

Bit width Space Exploration. To optimize the extendable exponent bit width, the design space exploration process for activations and weights is illustrated in Fig. 3(c). The objective is to optimize the function \( O \), formulated as:

\[
\text{optimal}: O = \begin{cases}
AP(<l_A,l_W>,l_{m''}) + \alpha \cdot EE(<l_A,l_W>,l_{m''}) & AP = Acc \\
\frac{1}{AP(<l_A,l_W>,l_{m''})} + \alpha \cdot EE(<l_A,l_W>,l_{m''}) & AP = Per
\end{cases}
\]

where \( AP \) take values of \( Acc \) and \( Per \). \( Acc, Per \), and \( EE \) respectively denote the accuracy, perplexity, and energy efficiency. \( \alpha \) is a balancing factor. \( l_A \) and \( l_W \) correspond to the bit widths of the extendable exponents for activations and weights, respectively. \( l_{m''} \) specifies the length of the mantissa.
4.2 Error Analysis
The transition from FP to EES introduces errors originating from two primary sources. First, rounding errors arise during the quantization of FP values into the BFP format, governed by the specified mantissa and exponent lengths. Second, additional numerical errors occur due to the integration of the extendable exponent component (\( E_E \)) into the least significant bits of the BFP mantissas.

For a block containing \( N \) elements, the error \( \alpha \) for the conversion from FP to EES is given by:

\[
\alpha = \sum_{i=0}^{N-1} |x_i'' - x_i| \leq \sum_{i=0}^{N-1} ||x_i'' - x_i'| + |x_i' - x_i||
\]
\[
\leq \sum_{i=0}^{l_{E_E}-1} |m_i'' - m_i'| \cdot 2^{-l_{m_i''}} + \sum_{i=0}^{N-1} |m_i' - m_i| \cdot 2^{E_S}
\]
\[
\leq l_{E_E} \cdot 2^{-l_{m_i''}} + N \cdot (2^{-l_{m_i''+E_S}} - 2^{-l_{m_i''}+E_S})
\]

where \( x_i, x_i', x_i'' \) are the \( i \)-th number of the FP, quantified by the shared exponent and EES representations, respectively, while \( l_{m_i}, l_{m_i}' \), and \( l_{m_i}'' \) represent the mantissa width of \( x_i, x_i', \) and \( x_i'' \). \( l_{E_E} \) and \( E_S \) respectively denote the extendable exponent bit width and extended shared exponent.

Based on Equations (6)-(15), the error variance (\( \sigma_\alpha^2 \)) for the transition from FP to EES is given by:

\[
\frac{4^{-l_{m_i''}-1+E_S}}{3} \leq \sigma_\alpha^2 \leq \frac{4^{-l_{m_i''}+E_S}}{3}
\]

This range arises from replacing the least significant bit of the original mantissa in EES. Specifically, when the mantissa bit-width is \( l_{m_i}' \), the error variance \( \sigma_\alpha^2 \) is at least as large as the error variance introduced during the FP-to-BFP conversion. However, when accounting for the bit replacement, the error variance remains no greater than the variance observed in the FP-to-BFP conversion with a reduced mantissa bit-width of \( l_{m_i}' - 1 \).

4.3 Hardware Architecture
Fig. 4 depicts a hardware architecture designed to support EES-based computations, consisting of three primary components: the on-chip memory (AB, OB, and WB), the processing element (PE), the decoder (Dc) and encoder (Ec). The memory units (AB, OB, WB) temporarily store input activations, output activations and weights, respectively. The PE performs EES-based multiplication and accumulation. Dc converts EES-format data into BFP format for computation, while Ec encodes output data from FP format back into EES representation.

The PE is structured into three key components: SP, MP, and EP. SP handles the XOR operations for the sign bits, MP performs the mantissa multiplications, and EP processes the combined exponent, including both the original and the extended shared exponent. The intermediate results from these computations are stored in the Out registers, where they are accumulated to generate the final outputs.

Fig. 5 depicts the Dc and Ec components. The Dc component (Fig. 5(a)) decodes both activations and weights by converting EES-format data into BFP format for calculations. It uses a selector to extract the last bit of mantissas and employs two adders and shifters to concatenate the extended exponent. Ec (Fig. 5(b)) encodes the output activations back into the EES format. It first uses a selector and a shifter to convert the original data into BFP format. After that, two selectors separately choose the mantissa and shared exponent bits. Finally, an adder and a shifter concatenate the data into the final EES representation.

Figure 4: EES-based hardware architecture. ps denotes the partial sum propagated from the preceding PE.

5 Evaluation

5.1 Experimental setup
Baselines. We evaluate the EES framework across three key aspects. First, we compare EES against three representative post-training low-bit quantization baselines: Tender (Lee et al., 2024), FINGA (Jang et al., 2024), and BiE (Zou et al., 2024). Tender employs INT4 quantization with tensor decomposition and implicit re-quantization to achieve efficient low-precision acceleration of LLMs. FINGA adopts a BFP format with optimized multiplication operations to reduce storage overhead and en-
Figure 5: Dc (a) and Ec (b) components for decoding and encoding, respectively.

<table>
  <tr>
    <th>Extendable exponent bit width (\( l_{E_c} \))</th>
    <th>mantissa bit width (\( l''_{m_i} \))</th>
  </tr>
  <tr>
    <th>Activation (\( l_A \))</th>
    <th>Weight (\( l_W \))</th>
    <th>(2, 3, 4, 5, 6)</th>
  </tr>
  <tr>
    <td>\( \{1, 2, 3, 4, 5, 6\} \)</td>
    <td>\( \{1, 2, 3, 4, 5, 6\} \)</td>
    <td></td>
  </tr>
</table>

Table 1: Quantization space exploration under different extendable exponents (\( l_A, l_W \)) and mantissas (\( l''_{m_i} \)).

<h3>Enhance computational efficiency. BiE incorporates a double-exponential mechanism to handle outliers while preserving model accuracy. Second, we assess hardware performance, memory efficiency, and energy and area efficiency, benchmarking EES against these baselines. Finally, we perform a quantization space exploration for extendable shared exponents for activations and weights under various bit-width configurations (Table 1) for optimal accuracy performance and energy efficiency.</h3>

<b>Models and Datasets.</b> EES is evaluated on two major LLM families: the OPT suite (6.7B and 13B variants) (Zhang et al., 2022b) and Llama (Llama2-7B, Llama2-13B, and Llama3.1-8B) (Touvron et al., 2023). Benchmarks include Hellaswag (Zellers et al., 2019), ARC_challenge (Min, 2023), LAMBADA_OpenAI (Nguyen et al., 2024), and LAMBADA_standard (Paperno et al., 2016). All accuracy results are conducted in a high-performance computing environment with an NVIDIA RTX 4090 GPU, leveraging the PyTorch 2.1.0 framework.

<b>Implementations.</b> To simulate EES, activations and weights are quantized using PyTorch. Baseline configurations include Tender with 4-bit integer, FINGA with 4-bit private mantissa and 3-bit shared exponent, and BiE with a 4-bit private mantissa, two 5-bit shared exponents and a flag bit. EES uses a W4A4 configuration with 4-bit mantissas for both weights and activations. All BFP baselines and EES are evaluated in a block size of 16.

To evaluate hardware behavior, the EES architecture was implemented in Verilog RTL and verified through simulations. Power analysis was conducted using Synopsys Design Compiler with 45nm TSMC technology library. Table 2 lists parameter settings for comparisons. The power of baselines for each PE comes from their literature. Table 3 breaks down area and power consumption of EES, showing that encoding and decoding components contribute minimally (0.02%-0.03%). Performance and energy efficiency were assessed using the MAESTRO (Kwon et al., 2020) simulator, with LLM models as inputs. MAESTRO provides a framework for analyzing LLM deployments on hardware architectures, delivering detailed insights into performance and energy efficiency.

<table>
  <tr>
    <th>Architecture</th>
    <th>Config</th>
    <th>Frequency [HZ]</th>
    <th>Tech [nm]</th>
    <th>Power [mw]</th>
    <th>Area [\(\mu m^2\)]</th>
  </tr>
  <tr>
    <td>Tender</td>
    <td>W4A4</td>
    <td>1G</td>
    <td>28</td>
    <td>0.266</td>
    <td>48.28</td>
  </tr>
  <tr>
    <td>FINGA</td>
    <td>W4A4</td>
    <td>1G</td>
    <td>40</td>
    <td>0.317</td>
    <td>367.08</td>
  </tr>
  <tr>
    <td>BiE</td>
    <td>W4A4</td>
    <td>1G</td>
    <td>45</td>
    <td>0.426</td>
    <td>534.93</td>
  </tr>
  <tr>
    <td>EES</td>
    <td>W4A4</td>
    <td>1G</td>
    <td>45</td>
    <td>0.334</td>
    <td>384.37</td>
  </tr>
</table>

Table 2: Power and area consumption of one PE. The results of baselines come from their literature. The configuration labeled as “W4A4” indicates that both the weights and activations utilize 4-bit mantissas.

<table>
  <tr>
    <th>Component</th>
    <th>Number</th>
    <th>Power [w]</th>
    <th>Power Ratio(%)</th>
    <th>Area [\(\mu m^2\)]</th>
    <th>Area Ratio(%)</th>
  </tr>
  <tr>
    <td>Ec</td>
    <td>16</td>
    <td>0.00339</td>
    <td>0.0078</td>
    <td>0.00255</td>
    <td>0.0164</td>
  </tr>
  <tr>
    <td>Dc</td>
    <td>32</td>
    <td>0.00489</td>
    <td>0.0112</td>
    <td>0.00359</td>
    <td>0.0343</td>
  </tr>
  <tr>
    <td>EES-PE</td>
    <td>256</td>
    <td>0.428</td>
    <td>0.9810</td>
    <td>0.0984</td>
    <td>0.9413</td>
  </tr>
</table>

Table 3: Power and area breakdown of EES.

<b>5.2 Simulation Results</b>

<b>Accuracy and Perplexity Comparison.</b> Table 4 compares accuracy and perplexity across methods. While EES shows a slight accuracy decline compared to the original FP16 baseline, this is primarily due to the extendable shared exponent strategy introduced during quantization. Despite this small trade-off, EES significantly enhances energy efficiency via low-bit extendable shared exponent quantization. Compared to the INT4-based Tender method, EES achieves higher accuracy, thanks to the broader dynamic range of its data format.

EES demonstrates superior performance over BFP-based FINGA in both accuracy and perplexity. For example, in the OPT-6.7B model, using the Hellaswag dataset, EES improves accuracy by
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="1">FP</th>
    <th colspan="1">Tender</th>
    <th colspan="1">FINGA</th>
    <th colspan="1">BiE</th>
    <th colspan="1">EES</th>
  </tr>
  <tr>
    <td>Hellaswag (↑)</td>
    <td>48.06%</td>
    <td>45.58%</td>
    <td>46.50%</td>
    <td>47.33%</td>
    <td>47.88%</td>
  </tr>
  <tr>
    <td rowspan="3">OPT-6.7B</td>
    <td>ARC_challenge(↑)</td>
    <td>30.63%</td>
    <td>29.82%</td>
    <td>29.21%</td>
    <td>29.73%</td>
    <td>29.89%</td>
  </tr>
  <tr>
    <td>LAMBADA_OpenAI(↓)</td>
    <td>4.25</td>
    <td>5.30</td>
    <td>6.12</td>
    <td>4.93</td>
    <td>5.11</td>
  </tr>
  <tr>
    <td>LAMBADA_Standard(↓)</td>
    <td>5.38</td>
    <td>6.55</td>
    <td>6.72</td>
    <td>6.59</td>
    <td>5.68</td>
  </tr>
  <tr>
    <td rowspan="3">OPT-13B</td>
    <td>Hellaswag (↑)</td>
    <td>48.40%</td>
    <td>45.58%</td>
    <td>46.89%</td>
    <td>47.37%</td>
    <td>47.88%</td>
  </tr>
  <tr>
    <td>ARC_challenge(↑)</td>
    <td>33.31%</td>
    <td>31.99%</td>
    <td>31.90%</td>
    <td>33.06%</td>
    <td>32.78%</td>
  </tr>
  <tr>
    <td>LAMBADA_OpenAI(↓)</td>
    <td>4.08</td>
    <td>6.62</td>
    <td>5.67</td>
    <td>5.69</td>
    <td>5.01</td>
  </tr>
  <tr>
    <td rowspan="3">Llama2-7B</td>
    <td>LAMBADA_Standard(↓)</td>
    <td>5.67</td>
    <td>9.19</td>
    <td>7.13</td>
    <td>6.49</td>
    <td>6.48</td>
  </tr>
  <tr>
    <td>Hellaswag (↑)</td>
    <td>49.88%</td>
    <td>48.50%</td>
    <td>49.24%</td>
    <td>48.91%</td>
    <td>49.34%</td>
  </tr>
  <tr>
    <td>ARC_challenge(↑)</td>
    <td>44.99%</td>
    <td>40.69%</td>
    <td>44.42%</td>
    <td>43.01%</td>
    <td>44.45%</td>
  </tr>
  <tr>
    <td rowspan="3">Llama2-13B</td>
    <td>LAMBADA_OpenAI(↓)</td>
    <td>3.34</td>
    <td>5.42</td>
    <td>5.10</td>
    <td>3.97</td>
    <td>4.44</td>
  </tr>
  <tr>
    <td>LAMBADA_Standard(↓)</td>
    <td>4.27</td>
    <td>7.13</td>
    <td>6.10</td>
    <td>5.84</td>
    <td>5.60</td>
  </tr>
  <tr>
    <td>Hellaswag (↑)</td>
    <td>50.11%</td>
    <td>48.03%</td>
    <td>48.40%</td>
    <td>49.64%</td>
    <td>49.49%</td>
  </tr>
  <tr>
    <td rowspan="3">Llama2-13B</td>
    <td>ARC_challenge(↑)</td>
    <td>49.00%</td>
    <td>45.30%</td>
    <td>47.80%</td>
    <td>48.75%</td>
    <td>48.31%</td>
  </tr>
  <tr>
    <td>LAMBADA_OpenAI(↓)</td>
    <td>2.94</td>
    <td>4.36</td>
    <td>3.08</td>
    <td>3.66</td>
    <td>3.08</td>
  </tr>
  <tr>
    <td>LAMBADA_Standard(↓)</td>
    <td>3.77</td>
    <td>6.10</td>
    <td>3.94</td>
    <td>4.76</td>
    <td>3.83</td>
  </tr>
  <tr>
    <td rowspan="3">Llama3.1-8B</td>
    <td>Hellaswag (↑)</td>
    <td>52.20%</td>
    <td>50.11%</td>
    <td>50.80%</td>
    <td>51.18%</td>
    <td>51.73%</td>
  </tr>
  <tr>
    <td>ARC_challenge(↑)</td>
    <td>51.19%</td>
    <td>50.31%</td>
    <td>50.14%</td>
    <td>50.19%</td>
    <td>50.67%</td>
  </tr>
  <tr>
    <td>LAMBADA_OpenAI(↓)</td>
    <td>3.05</td>
    <td>4.95</td>
    <td>3.22</td>
    <td>3.62</td>
    <td>3.06</td>
  </tr>
  <tr>
    <td>LAMBADA_Standard(↓)</td>
    <td>4.01</td>
    <td>6.50</td>
    <td>4.34</td>
    <td>5.47</td>
    <td>4.16</td>
  </tr>
</table>

Table 4: Accuracy/perplexity comparison between EES and baselines. (↓) indicates that lower perplexity is better for the datasets, while (↑) indicates that higher accuracy is better.

<table>
  <tr>
    <th>Benchmark</th>
    <th colspan="2">OPT</th>
    <th colspan="2">Llama2</th>
    <th colspan="2">Llama3.1</th>
  </tr>
  <tr>
    <th></th>
    <th>-6.7B</th>
    <th>-13B</th>
    <th>-7B</th>
    <th>-13B</th>
    <th>-8B</th>
    <th>-8B</th>
  </tr>
  <tr>
    <th>Config</th>
    <td>W4A4</td>
    <td>W4A4</td>
    <td>W4A4</td>
    <td>W3A3</td>
    <td>W4A4</td>
    <td>W4A4</td>
  </tr>
  <tr>
    <th>\( l_{E_{SA}}, l_{E_{SW}} \)</th>
    <td>&lt;5,4&gt;</td>
    <td>&lt;5,4&gt;</td>
    <td>&lt;5,5&gt;</td>
    <td>&lt;5,4&gt;</td>
    <td>&lt;5,5&gt;</td>
    <td>&lt;5,5&gt;</td>
  </tr>
  <tr>
    <th>Accuracy (%)</th>
    <td>47.88</td>
    <td>47.67</td>
    <td>49.45</td>
    <td>48.76</td>
    <td>51.86</td>
    <td>51.86</td>
  </tr>
  <tr>
    <th>Energy Efficiency</th>
    <td>13.60×</td>
    <td>13.60×</td>
    <td>13.59×</td>
    <td>17.75×</td>
    <td>13.58×</td>
    <td>13.58×</td>
  </tr>
</table>

Table 5: The optimal accuracy and energy efficiency (normalized with FP16) are based on a bit-width space exploration conducted under different mantissa and extendable exponent bit-width setting on benchmarks and Hellaswag dataset. \( l_{E_{SA}}, l_{E_{SW}} \) denotes the combination of activation and weight of extended shared exponent bit-width. The exploration results on ARC_challenge, LAMBADA_OpenAI and LAMBADA_Standard datasets are shown in Appendix A.3.

1.38%, while on the LAMBADA_Standard dataset, it reduces perplexity by 1.11. This is attributed to the extendable shared exponent strategy, which ensures accurate representation of outliers, preventing errors from their misrepresentation. When compared to BiE, which also addresses outlier handling, EES achieves comparable accuracy while offering notable advantages in energy efficiency. Specifically, in the Llama3.1-8B model, using the Hellaswag dataset, EES boosts accuracy by up to 0.55% and reduces perplexity by 0.56% on the LAMBADA_Standard dataset. These results highlight the effectiveness of EES in maintaining high accuracy and energy efficiency, even with slight accuracy losses on certain benchmarks.

Bit-width Space Exploration. Table 5 summarizes results from exploring the bit-width space with different EES parameter configurations. The optimal configuration includes a mantissa bit-width of 4 and extendable shared exponent widths of 5 for activations and 4 for weights. During this exploration, accuracy loss between adjacent configurations was constrained to be no greater than 1.00%. On average, EES improves energy efficiency by 13.6× compared to FP16. The bit-width exploration is conducted offline, allowing the optimal configuration to be selected for online inference without adding hardware overhead. For the activation values generated online, we can obtain estimates through statistical data analysis and used them as the prior information for the guideline of the offline exploration. Details of the exploration process across benchmarks are provided in Appendix A.3.

The optimal configuration is derived using Equation (14), which balances precision and hardware overhead. The balance factor \( \alpha \) plays a key role in determining the trade-off between accuracy and energy efficiency. In this study, \( \alpha = 0.2 \) is used as
<table>
  <tr>
    <th>Benchmark</th>
    <th colspan="4">OPT-6/7B</th>
    <th colspan="4">OPT-13B</th>
    <th colspan="4">Llama2-7B</th>
  </tr>
  <tr>
    <th>Config</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
  </tr>
  <tr>
    <td>\( \langle E_{S^*}, E_{SW} \rangle \)</td>
    <td>&lt;5, 5&gt;</td>
    &lt;6, 4&gt;
    &lt;5, 4&gt;
    &lt;6, 5&gt;
    &lt;5, 5&gt;
    &lt;5, 4&gt;
    &lt;5, 4&gt;
    &lt;6, 5&gt;
    &lt;5, 5&gt;
    &lt;6, 4&gt;
    &lt;5, 4&gt;
    &lt;5, 5&gt;
  </tr>
  <tr>
    <td>Accuracy (%)</td>
    <td>43.84</td>
    <td>46.78</td>
    <td>47.92</td>
    <td>47.96</td>
    <td>44.89</td>
    <td>46.52</td>
    <td>48.01</td>
    <td>48.11</td>
    <td>43.27</td>
    <td>48.47</td>
    <td>49.74</td>
    <td>49.87</td>
  </tr>
  <tr>
    <td>Energy Efficiency</td>
    <td>25.33×</td>
    <td>17.74×</td>
    <td>11.03×</td>
    <td>8.02×</td>
    <td>25.33×</td>
    <td>17.75×</td>
    <td>11.03×</td>
    <td>8.02×</td>
    <td>25.33×</td>
    <td>17.74×</td>
    <td>11.02×</td>
    <td>8.03×</td>
  </tr>
  <tr>
    <th>Benchmark</th>
    <th colspan="4">Llama2-13B</th>
    <th colspan="4">Llama2-1.8B</th>
  </tr>
  <tr>
    <th>Config</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
  </tr>
  <tr>
    <td>\( \langle E_{S^*}, E_{SW} \rangle \)</td>
    <td>&lt;5, 5&gt;</td>
    &lt;5, 5&gt;
    &lt;5, 4&gt;
    &lt;6, 5&gt;
    &lt;5, 5&gt;
    &lt;5, 4&gt;
    &lt;5, 4&gt;
    &lt;6, 5&gt;
  </tr>
  <tr>
    <td>Accuracy (%)</td>
    <td>43.78</td>
    <td>48.76</td>
    <td>49.57</td>
    <td>49.76</td>
    <td>45.11</td>
    <td>50.55</td>
    <td>52.10</td>
    <td>52.22</td>
  </tr>
  <tr>
    <td>Energy Efficiency</td>
    <td>25.33×</td>
    <td>17.74×</td>
    <td>11.03×</td>
    <td>8.02×</td>
    <td>25.33×</td>
    <td>17.75×</td>
    <td>11.03×</td>
    <td>8.02×</td>
  </tr>
</table>

Table 6: Optimal accuracy and energy efficiency (normalized with FP16) results are based on a bit-width space exploration conducted under the optimal extended shared exponent bit width setting with different mantissa bit width on benchmarks and Hellaswag dataset. The optimal results of bit-width space exploration with different mantissa on ARC_challenge, LAMBADA_OpenAI and LAMBADA_Standard datasets are shown in Appendix A.3.

<table>
  <tr>
    <th>Method</th>
    <th>Config</th>
    <th>Throughput</th>
    <th>Memory Efficiency</th>
    <th>Energy Efficiency</th>
    <th>Area Efficiency</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>-</td>
    <td>1.00×</td>
    <td>1.00×</td>
    <td>1.00×</td>
    <td>1.00×</td>
  </tr>
  <tr>
    <td>Tender</td>
    <td>W4A4</td>
    <td>3.84×</td>
    <td>4.00×</td>
    <td>15.36×</td>
    <td>14.21×</td>
  </tr>
  <tr>
    <td>FINGA</td>
    <td>W4A4</td>
    <td>3.58×</td>
    <td>3.82×</td>
    <td>13.68×</td>
    <td>17.61×</td>
  </tr>
  <tr>
    <td>BiE</td>
    <td>W4A4</td>
    <td>3.51×</td>
    <td>2.80×</td>
    <td>9.83×</td>
    <td>11.85×</td>
  </tr>
  <tr>
    <td>EES</td>
    <td>W4A4</td>
    <td>3.56×</td>
    <td>3.82×</td>
    <td>13.60×</td>
    <td>16.73×</td>
  </tr>
</table>

Table 7: Hardware efficiency of the size PE array with different numerical representations compared with FP16. We highlight the hardware efficiency of EES.

<table>
  <tr>
    <th>Method</th>
    <th>Config</th>
    <th>Hellaswag</th>
    <th>arc challenge</th>
  </tr>
  <tr>
    <td>FP</td>
    <td>-</td>
    <td>52.16%</td>
    <td>51.19%</td>
  </tr>
  <tr>
    <td>BFP</td>
    <td>W4A4</td>
    <td>50.80%</td>
    <td>50.14%</td>
  </tr>
  <tr>
    <td>EES-EEE</td>
    <td>W4A4</td>
    <td>51.71%</td>
    <td>50.82%</td>
  </tr>
  <tr>
    <td>EES</td>
    <td>W4A4</td>
    <td>51.56%</td>
    <td>50.74%</td>
  </tr>
</table>

Table 8: Effect of shared exponent extending and embedding operations. “EES-EEE” denotes EES under the situation of removing the embedding.

the equilibrium value, with sensitivity analysis on \( \alpha \) provided in Appendix A.4.

Table 6 further presents the optimal extended shared exponent bit-widths and corresponding results across benchmarks under various mantissa bit-widths. For a fixed mantissa bit-width, changes in extendable shared exponent widths have minimal impact on energy efficiency. For instance, in the W3A3 configuration, shifting from an extended exponent combination of \( <6, 4> \) to \( <5, 4> \) reduces energy efficiency by only 0.01%. Thus, with a fixed mantissa bit-width, selecting the optimal extended exponent width is driven mainly by accuracy considerations. By systematically identifying the best configuration for both mantissa and extended shared exponents, EES ensures a balanced trade-off between precision and energy efficiency.

Hardware Efficiency. Table 7 highlights the normalized hardware performance of EES compared to baseline. EES achieves remarkable improvements across key metrics: throughput is increased by 3.6×, memory efficiency by 3.8×, energy efficiency by 13.6×, and area efficiency by 16.7×. These gains stem from the adoption of low-bit-width representations and innovative extendable exponent strategy. When compared to Tender, EES shows a slight decrease in energy efficiency (0.9×), primarily due to the added complexity of EES computations compared to simpler integer multiplication. Against FINGA, EES delivers comparable throughput and efficiency but outperforms in accuracy. Relative to BiE, EES improves both energy and area efficiency by 1.4×, as BiE’s use of an additional flag bit and dual-exponent mechanism incurs higher storage and computational costs.

Impact of Extendable Exponent Embedding. As shown in Table 8, removing the embedding operation for the extendable exponent leads to an average accuracy drop of only 0.12%. This indicates that embedding the extendable exponent into the last bit of partial mantissas minimally impacts accuracy while significantly reducing storage overhead. This result underscores the effectiveness of EES in balancing precision and hardware efficiency.

6 Conclusion

In this paper, we propose an Extendable Exponent Sharing (EES) method to overcome the limitations of existing low-bit quantization techniques. EES enhances dynamic range capture by embedding extendable exponent bits into lower-order bits of mantissa, without introducing additional overhead. To further optimize performance, we design a spatial exploration strategy to determine optimal bit-width configuration for both extendable exponent and mantissa. Comprehensive evaluation show that EES outperforms representative baseline methods in terms of both accuracy and energy efficiency.
7 Limitations

While EES offers an extended dynamic range, its accuracy can be affected by replacement errors when part of the mantissa is embedded within the extendable exponent bits. Furthermore, the optimal configuration of the extendable exponent and mantissa during the EES bit-width exploration can vary depending on the specific benchmarks or datasets considered. Currently, the exploration process involves searching the entire model, rather than tailoring it to individual layers, which both increases the exploration complexity and slows down the overall process. Future work will explore more efficient search strategies to accelerate this process.

On the hardware front, EES has shown improvements in both computational and energy efficiency. However, it may necessitate custom-designed computing units, encoders, and decoders, potentially adding extra overhead and complexity compared to conventional hardware designs. Future efforts will aim to refine the EES approach and address these challenges.

8 Acknowledgments

This work was supported in part by National Natural Science Foundation of China under Grants 62302102, in part by Guangdong Basic and Applied Basic Research Foundation (2023A1515012844, 2025A04J5208).

References

Ayon Basumallik, Darius Bunandar, Nicholas Dronen, Nicholas Harris, Ludmila Levkova, Calvin McCarter, Lakshmi Nair, David Walter, and David Peter Wiedemann. 2022. Adaptive block floating-point for analog deep learning hardware. ArXiv, abs/2205.06287.

Hanane Benmaghnia, Matthieu Martel, and Yassamine Seladji. 2022. Code generation for neural networks based on fixed-point arithmetic. ACM Transactions on Embedded Computing Systems (TECS).

Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, and Yusuke Miyao. 2023. Improving numeracy by input reframing and quantitative pre-finetuning task. In Findings of the Association for Computational Linguistics: EACL 2023, pages 69–77.

Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinosgradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. 2020. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Advances in neural information processing systems, 33:10271–10281.

Bita Darvish Rouhani, Ritchie Zhao, Venugul Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, et al. 2023. With shared microexponents, a little shifting goes a long way. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–13.

Hongxiang Fan, Shuanglong Liu, Zhiqiang Que, Xinyu Niu, and Wayne Luk. 2021. High-performance acceleration of 2-d and 3-d cnns on fpgas using static block floating point. IEEE Transactions on Neural Networks and Learning Systems, 34(8):4473–4487.

Jaeyong Jang, Yulhwa Kim, Juheun Lee, and Jae-Joon Kim. 2024. Figna: Integer unit-based accelerator design for fp-int gemm preserving numerical accuracy. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 760–773. IEEE.

Kari Kalliojarvi and Jaakko Astola. 1996. Roundoff errors in block-floating-point systems. IEEE transactions on signal processing, 44(4):783–790.

Callie Y Kim, Christine P Lee, and Bilge Mutlu. 2024. Understanding large-language model (llm)-powered human-robot interaction. In Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, pages 371–380.

Hyoukjun Kwon, Prasanth Chatarasi, Vivek Sarkar, Tushar Krishna, Michael Pellauer, and Angshuman Parashar. 2020. Maestro: A data-centric approach to understand reuse, performance, and hardware cost of dnn mappings. IEEE micro, 40(3):20–29.

Jungi Lee, Wonbeom Lee, and Jaewoong Sim. 2024. Tender: Accelerating large language models via tensor decomposition and runtime quantization. pages 1048–1062.

Seunghyun Lee, Jeik Choi, Seockhwan Noh, Jahyun Koo, and Jaeha Kung. 2023. Dbps: Dynamic block size and precision scaling for efficient dnn training supported by risc-v isa extensions. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Fangxin Liu, Ning Yang, Haomin Li, Zongwu Wang, Zhuoran Song, Songwen Pei, and Li Jiang. 2024. Spark: Scalable and precision-aware acceleration of neural networks via efficient encoding. In 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 1029–1042. IEEE.

Yun-Chen Lo, Tse-Kuang Lee, and Ren-Shuo Liu. 2023. Block and subword-scaling floating-point (bslfp): An
efficient non-uniform quantization for low precision inference. In The Eleventh International Conference on Learning Representations.

Yun-Chen Lo and Ren-Shuo Liu. 2023. Bucket getter: A bucket-based processing engine for low-bit block floating point (bfp) dnns. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pages 1002–1015.

Tan John Chong Min. 2023. An approach to solving the abstraction and reasoning corpus (arc) challenge.

Pravin Nair, Ruturaj G. Gavaskar, and Kunal Narayan Chaudhury. 2021. Fixed-point and objective convergence of plug-and-play algorithms. IEEE Transactions on Computational Imaging, 7:337–348.

Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13.

Trong-Hieu Nguyen, Anh-Cuong Le, and Viet-Cuong Nguyen. 2024. Villm-eval: A comprehensive evaluation suite for vietnamese large language models. ArXiv, abs/2404.11086.

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.

Seock-Hwan Noh, Jahyun Koo, Seunghyun Lee, Jongse Park, and Jaeha Kung. 2023. Flexblock: A flexible dn training accelerator with multi-mode block floating point support. IEEE Transactions on Computers, 72(9):2522–2535.

Denis Paperno, Germán Kruszewski, Angeliki Lazari-dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. ArXiv, abs/1606.06031.

Yubin Qin, Yang Wang, Zhiren Zhao, Xiaolong Yang, Yang Zhou, Shaojun Wei, Yang Hu, and Shouyi Yin. 2024. Mecla: Memory-compute-efficient llm accelerator with scaling sub-matrix partition. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1032–1047. IEEE.

Babak Rokh, Ali Azarpayvand, and Alireza Khantemoori. 2022. A comprehensive survey on model quantization for deep neural networks in image classification. ACM Transactions on Intelligent Systems and Technology, 14:1–50.

Xuan Shen, Peiyuan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, and Yanzhi Wang. 2024. Agile-quant: Activation-guided quantization for faster inference of llms on the edge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18944–18951.

Zhourui Song, Zhenyu Liu, and Dongsheng Wang. 2018. Computation error analysis of block floating point arithmetic oriented convolution neural network accelerator design. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.

Mengxia Yu, De Wang, Qi Shan, and Alvin Wan. 2024. The super weight in large language models. arXiv preprint arXiv:2411.07191.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyan Li, Zixiao Huang, et al. 2024. Flightllm: Efficient large language model inference with a complete mapping flow on fpgas. In Proceedings of the 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, pages 223–234.

Cheng Zhang, Jianyi Cheng, Ilia Shumailov, George A Constantinides, and Yiren Zhao. 2023. Revisiting block-based quantisation: What is important for sub-8-bit llm inference? arXiv preprint arXiv:2310.05079.

Hengrui Zhang, August Ning, Rohan Baskar Prabhakar, and David Wentzlaff. 2024. Llmcompass: Enabling efficient hardware design for large language model inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1080–1096. IEEE.

Sai Qian Zhang, Bradley McDaniel, and HT Kung. 2022a. Fast: Dnn training under variable precision block floating point with stochastic rounding. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 846–860. IEEE.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher Deenan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022b. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Youpeng Zhao, Di Wu, and Jun Wang. 2024. Alisa: Accelerating large language model inference via sparsity-aware kv caching. 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1005–1017.
A Appendix

A.1 Hardware Overhead
Table 9 presents a comparison of the encoding overhead for converting original FP format data to INT8, INT4, and BFP8, as well as the decoding overhead for restoring data to its original format. Taking the FP16-INT8 type as an example, it represents the overhead of encoding FP16 into INT8. From Table 9, it can be observed that, compared to conversion to other simpler data formats, the overhead of the encoder and decoder in the EES accelerator in terms of power consumption and area is relatively low, and can be considered negligible compared to the overall overhead of the accelerator.

A.2 Results of Accuracy and Perplexity
Table 10 demonstrates the experimental results of all baselines and our method on the more advanced LLM models DeepSeek_R1_Distill_Qwen_1.5B and DeepSeek_R1_Distill_Llama8B. Consistent with the results observed in other LLM models, EES has improved in both accuracy and perplexity compared to baseline, while maintaining performance close to the original accuracy.

Table 11 presents a comparison of the best performance data of basic BFP and our method EES across five models. It can be observed that, compared to the best-performing configuration of basic BFP, EES shows significant improvements in both accuracy and perplexity, while demonstrating similar performance in terms of energy efficiency.

A.3 Bit-width Space Exploration Results
Table 12 presents the optical accuracy and energy efficiency results obtained through the EES bit-width search across different benchmarks and datasets. Consistent with the findings on the Hellaswag dataset, The optimal configuration includes a mantissa bit-width of 4 and extendable shared exponent widths of 5 for activations and 4 for weights. Table 13 further describes the optimal accuracy and energy efficiency results for each benchmark and dataset under different mantissa bit-width settings obtained through the bit-width space search. As observed in the Hellaswag dataset, increasing the mantissa bit-width improves accuracy but also leads to a decrease in energy efficiency.

<table>
  <tr>
    <th>Type</th>
    <th>Component</th>
    <th>Power [w]</th>
    <th>area [mm<sup>2</sup>]</th>
  </tr>
  <tr>
    <td>FP16-INT8</td>
    <td>Encoder</td>
    <td>0.0025</td>
    <td>0.00362</td>
  </tr>
  <tr>
    <td>INT8-FP16</td>
    <td>Decoder</td>
    <td>0.0069</td>
    <td>0.00484</td>
  </tr>
  <tr>
    <td>FP16-INT4</td>
    <td>Encoder</td>
    <td>0.0015</td>
    <td>0.00289</td>
  </tr>
  <tr>
    <td>INT4-FP16</td>
    <td>Decoder</td>
    <td>0.0039</td>
    <td>0.00281</td>
  </tr>
  <tr>
    <td>FP16-BFP8</td>
    <td>Encoder</td>
    <td>0.0036</td>
    <td>0.00301</td>
  </tr>
  <tr>
    <td>BFP8-FP16</td>
    <td>Decoder</td>
    <td>0.0035</td>
    <td>0.00272</td>
  </tr>
  <tr>
    <td>FP16-EES</td>
    <td>Encoder</td>
    <td>0.0034</td>
    <td>0.00255</td>
  </tr>
  <tr>
    <td>EES-FP16</td>
    <td>Decoder</td>
    <td>0.0049</td>
    <td>0.00359</td>
  </tr>
</table>

Table 9: Comparison of Encoding and Decoding Costs with INT8, INT4, and BFP8.

Fig. 6 presents the accuracy and normalized energy efficiency results (compared to FP) for different mantissa bit width \( Lm \) using the Hellaswag dataset. It can be observed that for most benchmarks, when \( Lm \geq 4 \), EES can maintain relatively high accuracy, but the energy efficiency significantly decreases. For example, on the Llama3.1-8B benchmark, increasing \( Lm \) from 4 to 6 only improves accuracy by 0.32%, while energy efficiency drops by 1.7×. When \( Lm > 4 \), energy efficiency improves, but this comes with a significant loss in accuracy. Therefore, the performance across different benchmarks provides valuable guidance for selecting the optimal mantissa bit width that balances accuracy and energy efficiency effectively.

Fig. 7 shows the changes in perplexity and normalized energy efficiency for different values of \( Lm \) on the LAMBADA_OpenAI dataset. Consistent with the findings on the Hellaswag dataset, increasing \( Lm \) in EES leads to a decrease in perplexity, but energy efficiency also declines. This is primarily due to the use of different mantissa bit width in the EES implementation.

Fig. 8 illustrates the results of the extendable exponent bit-width search under a fixed mantissa bit-width using the Hellaswag dataset on the Llama3.1-8B model. As shown in Fig. 8 (a), when the mantissa bit-width is 6, the optimal extendable exponent configuration is <4, 3>. Fig. 8 (b) shows that when the mantissa bit-width is 4, the optimal extendable exponent configuration is <2, 1>. Since the impact of the extendable exponent on energy consumption is negligible when the mantissa bit-width is the same, we primarily determine the optimal extendable exponent bit-width configuration based on accuracy.

A.4 Determination of Balance Factor \( \alpha \)
Fig. 9 shows the impact of different balance factors \( \alpha \) on accuracy and energy efficiency across multi-
<table>
  <tr>
    <th rowspan="2">Tasks</th>
    <th colspan="6">FP</th>
    <th colspan="6">Tender</th>
    <th colspan="6">FIGNA</th>
    <th colspan="6">BiE</th>
    <th colspan="6">EES</th>
  </tr>
  <tr>
    <th>DeepSeek_R1_</th>
    <th>Hellaswag(↑)</th>
    <th>ARC_challenge(↑)</th>
    <th>lambda_openai(↓)</th>
    <th>lambda_standard(↓)</th>
    <th>DeepSeek_R1_</th>
    <th>Hellaswag(↑)</th>
    <th>ARC_challenge(↑)</th>
    <th>lambda_openai(↓)</th>
    <th>lambda_standard(↓)</th>
    <th>Distill_Qwen_1.5B</th>
    <th>Hellaswag(↑)</th>
    <th>ARC_challenge(↑)</th>
    <th>lambda_openai(↓)</th>
    <th>lambda_standard(↓)</th>
    <th>DeepSeek_R1_</th>
    <th>Hellaswag(↑)</th>
    <th>ARC_challenge(↑)</th>
    <th>lambda_openai(↓)</th>
    <th>lambda_standard(↓)</th>
    <th>Distill_Llama8B</th>
    <th>Hellaswag(↑)</th>
    <th>ARC_challenge(↑)</th>
    <th>lambda_openai(↓)</th>
    <th>lambda_standard(↓)</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>41.78%</td>
    <td>39.01%</td>
    <td>40.31%</td>
    <td>41.09%</td>
    <td>41.57%</td>
    <td>24.38%</td>
    <td>23.00%</td>
    <td>24.01%</td>
    <td>24.12%</td>
    <td>24.23%</td>
    <td>6.95</td>
    <td>20.92</td>
    <td>8.18</td>
    <td>8.04</td>
    <td>7.76</td>
    <td>10.65</td>
    <td>42.17</td>
    <td>12.54</td>
    <td>12.31</td>
    <td>11.98</td>
    <td>10.65</td>
    <td>42.17</td>
    <td>12.54</td>
    <td>12.31</td>
    <td>11.98</td>
  </tr>
</table>

Table 10: Accuracy/perplexity comparison between EES and baselines. (↓) indicates that lower perplexity is better for the datasets, while (↑) indicates that higher accuracy is better.

<table>
  <tr>
    <th>Model</th>
    <th>method</th>
    <th>Hellaswag(↑)</th>
    <th>ARC challenge(↑)</th>
    <th>lambda openai(↓)</th>
    <th>lambda standard(↓)</th>
    <th>energy efficiency</th>
  </tr>
  <tr>
    <td>OPT-6.7B</td>
    <td>BFP</td>
    <td>46.61%</td>
    <td>29.41%</td>
    <td>5.83</td>
    <td>6.90</td>
    <td>13.62x</td>
  </tr>
  <tr>
    <td></td>
    <td>EES</td>
    <td>47.88%</td>
    <td>29.89%</td>
    <td>5.11</td>
    <td>5.68</td>
    <td>13.60x</td>
  </tr>
  <tr>
    <td>OPT-13B</td>
    <td>BFP</td>
    <td>47.40%</td>
    <td>31.72%</td>
    <td>5.63</td>
    <td>7.08</td>
    <td>13.62x</td>
  </tr>
  <tr>
    <td></td>
    <td>EES</td>
    <td>47.88%</td>
    <td>32.58%</td>
    <td>5.01</td>
    <td>6.48</td>
    <td>13.60x</td>
  </tr>
  <tr>
    <td>LLama2-7B</td>
    <td>BFP</td>
    <td>48.80%</td>
    <td>43.71%</td>
    <td>5.61</td>
    <td>6.67</td>
    <td>13.62x</td>
  </tr>
  <tr>
    <td></td>
    <td>EES</td>
    <td>49.34%</td>
    <td>44.45%</td>
    <td>4.44</td>
    <td>5.60</td>
    <td>13.60x</td>
  </tr>
  <tr>
    <td>LLama2-13B</td>
    <td>BFP</td>
    <td>49.11%</td>
    <td>47.83%</td>
    <td>3.91</td>
    <td>4.42</td>
    <td>13.62x</td>
  </tr>
  <tr>
    <td></td>
    <td>EES</td>
    <td>49.49%</td>
    <td>48.31%</td>
    <td>3.08</td>
    <td>3.83</td>
    <td>13.60x</td>
  </tr>
  <tr>
    <td>LLama3-1.8B</td>
    <td>BFP</td>
    <td>50.87%</td>
    <td>49.50%</td>
    <td>3.98</td>
    <td>5.31</td>
    <td>13.62x</td>
  </tr>
  <tr>
    <td></td>
    <td>EES</td>
    <td>51.73%</td>
    <td>50.67%</td>
    <td>3.06</td>
    <td>4.16</td>
    <td>13.60x</td>
  </tr>
</table>

Table 11: The comparison between the best performance results of the basic BFP and EES, across five models is as follows, while the energy efficiency is expressed as a multiple relative to FP16.

<table>
  <tr>
    <th rowspan="2">Benchmark</th>
    <th colspan="2">OPT-6.7B</th>
    <th colspan="2">OPT-13B</th>
    <th colspan="2">Llama2-7B</th>
    <th colspan="2">Llama2-13B</th>
    <th colspan="2">Llama3-1.8B</th>
  </tr>
  <tr>
    <th>ARC_ch LAMBAD</th>
    <th>LAMBADA_OpenAI A_Standard</th>
    <th>ARC_ch LAMBAD</th>
    <th>LAMBADA_OpenAI A_Standard</th>
    <th>ARC_ch LAMBAD</th>
    <th>LAMBADA_OpenAI A_Standard</th>
    <th>ARC_ch LAMBAD</th>
    <th>LAMBADA_OpenAI A_Standard</th>
    <th>ARC_ch LAMBAD</th>
    <th>LAMBADA_OpenAI A_Standard</th>
  </tr>
  <tr>
    <td>Config</td>
    <td>W2A2</td>
    <td>W3A3</td>
    <td>W2A2</td>
    <td>W3A3</td>
    <td>W2A2</td>
    <td>W3A3</td>
    <td>W2A2</td>
    <td>W3A3</td>
    <td>W2A2</td>
    <td>W3A3</td>
  </tr>
  <tr>
    <td>\( <l_{ESA}, l_{ESW} > \)</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.45</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
  </tr>
  <tr>
    <td>Accuracy(%)</td>
    <td>29.89</td>
    <td>5.11</td>
    <td>5.68</td>
    <td>32.78</td>
    <td>5.01</td>
    <td>6.48</td>
    <td>44.53</td>
    <td>4.41</td>
    <td>5.56</td>
    <td>47.54</td>
    <td>3.35</td>
    <td>3.87</td>
    <td>50.72</td>
    <td>3.02</td>
    <td>4.07</td>
  </tr>
  <tr>
    <td>perplexity</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.59x</td>
    <td>13.59x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.58x</td>
    <td>13.58x</td>
    <td>13.58x</td>
    <td>13.58x</td>
    <td>13.58x</td>
  </tr>
  <tr>
    <td>Energy Efficiency</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.59x</td>
    <td>13.59x</td>
    <td>13.60x</td>
    <td>13.60x</td>
    <td>13.58x</td>
    <td>13.58x</td>
    <td>13.58x</td>
    <td>13.58x</td>
    <td>13.58x</td>
  </tr>
</table>

Table 12: The optimal accuracy, perplexity and energy efficiency (normalized with FP16) results are based on a quantization space exploration conducted under different mantissa and extendable exponent bit width setting on benchmarks evaluated using the ARC_challenge, LAMBADA_OpenAI and LAMBADA_Standard datasets. \( <l_{ESA}, l_{ESW} > \) denotes the combination of activation and weight of the extended shared exponent bit width.

<table>
  <tr>
    <th rowspan="2">Benchmark</th>
    <th colspan="2">OPT-6.7B</th>
    <th colspan="2">OPT-13B</th>
    <th colspan="2">Llama2-7B</th>
    <th colspan="2">Llama2-13B</th>
    <th colspan="2">Llama3-1.8B</th>
  </tr>
  <tr>
    <th>Config</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
    <th>W2A2</th>
    <th>W3A3</th>
    <th>W5A5</th>
    <th>W6A6</th>
  </tr>
  <tr>
    <td>\( <l_{ESA}, l_{ESW} > \)</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.45</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
    <td>5.50</td>
    <td>4.45</td>
  </tr>
  <tr>
    <td>LAMBADA_OpenAI</td>
    <td>25.11%</td>
    <td>29.23%</td>
    <td>30.12%</td>
    <td>30.43%</td>
    <td>30.19%</td>
    <td>32.47%</td>
    <td>32.83%</td>
    <td>32.99%</td>
    <td>33.68%</td>
    <td>42.96%</td>
    <td>44.52%</td>
    <td>44.99%</td>
    <td>42.23%</td>
    <td>47.54%</td>
    <td>48.65%</td>
    <td>48.79%</td>
    <td>38.11%</td>
    <td>48.77%</td>
    <td>52.88%</td>
    <td>52.01%</td>
  </tr>
  <tr>
    <td>LAMBADA_Standard(↓)</td>
    <td>5.76</td>
    <td>5.47</td>
    <td>4.66</td>
    <td>4.32</td>
    <td>5.71</td>
    <td>5.34</td>
    <td>4.86</td>
    <td>4.66</td>
    <td>5.83</td>
    <td>5.06</td>
    <td>3.69</td>
    <td>3.05</td>
    <td>3.62</td>
    <td>3.21</td>
    <td>3.03</td>
    <td>3.01</td>
    <td>6.73</td>
    <td>4.71</td>
    <td>3.11</td>
    <td>3.02</td>
  </tr>
  <tr>
    <td>Energy Efficiency</td>
    <td>25.33x</td>
    <td>17.74x</td>
    <td>11.03x</td>
    <td>8.02x</td>
    <td>25.33x</td>
    <td>17.75x</td>
    <td>11.03x</td>
    <td>8.02x</td>
    <td>25.33x</td>
    <td>17.74x</td>
    <td>11.02x</td>
    <td>8.03x</td>
    <td>25.33x</td>
    <td>17.74x</td>
    <td>11.03x</td>
    <td>8.02x</td>
    <td>25.33x</td>
    <td>17.75x</td>
    <td>11.03x</td>
    <td>8.02x</td>
  </tr>
</table>

Table 13: The optimal accuracy and energy efficiency (normalized with FP16) results are based on a bit-width space exploration conducted under the optimal extended shared exponent bit width setting with different mantissa bit width on benchmarks evaluated using the ARC_challenge, LAMBADA_OpenAI and LAMBADA_Standard datasets. \( <l_{ESA}, l_{ESW} > \) denotes the combination of activation and weight of the extended shared exponent bit width.

ple benchmarks. We selected six representative values (\( \alpha \in \{0.015, 0.05, 0.2, 0.5, 1, 2\} \)), ranging from 0 to 2) for sensitivity analysis at different levels of \( \alpha \). The results indicate that when \( \alpha \) is between 0.015 and 0.2, the objective function tends to favor accuracy, whereas when \( \alpha \) is between 0.5 and 2, it favors energy efficiency. Only when \( \alpha \) is between 0.2 and 0.5, the objective function can achieve a better balance between accuracy and energy efficiency. Based on the results across all benchmarks, we ultimately chose \( \alpha = 0.2 \) as the equilibrium value between accuracy and energy efficiency.
Figure 6: Bit-width space exploration results under different mantissa bit-width (\( Lm \)) on benchmarks evaluated using the Hellaswag dataset, with the optimal extended exponent bit-width settings. The red dashed box represents the optimal (\( Lm \)) configuration for benchmarks.

Figure 7: Bit-width space exploration results under different mantissa bit-width (\( Lm \)) on benchmarks evaluated using the LAMBADA_OpenAI dataset, with the optimal extended exponent bit-width settings. The red dashed box represents the optimal (\( Lm \)) configuration for benchmarks.

Figure 8: Bit-width space exploration results under different extendable exponent bit-width (\( LA, LW \)) on Llama3.1-8B evaluated using the Hellaswag dataset, with the same mantissa bit-width settings. (a) The original data format consists of a 6-bit mantissa and a 1-bit shared exponent. (b) The original data format consists of a 4-bit mantissa and a 3-bit shared exponent. \( LA \) and \( LW \) represent the extendable exponent bit width of activation and weight, respectively. The red dashed box represents the optimal extendable exponent bit-width configuration for benchmark.

Figure 9: Comprehensive sensitivity analysis of different \( \alpha \) values for accuracy and energy efficiency under different mantissa width on the benchmarks evaluated using the Hellaswag dataset. The red dashed box represents the optimal balance factor (\( \alpha = 0.2 \)) for benchmarks.

Reviewer paper summaries: This paper proposes an improved weight/activation representation method based on Block Floating-Point (BFP). Specifically, its core idea is to extend the exponent by utilizing the lower bits of the floating-point mantissa, thereby expanding the representational range of floating-point numbers. This allows for better representation of outliers, achieving a good balance between efficiency and accuracy.

This paper proposes Extendable Exponent Sharing (EES), a novel block floating point (BFP) representation method to address dynamic range limitations and precision degradation in low-bit BFP quantization caused by extreme outliers. By embedding extendable exponent bits into the least significant bits of the mantissa, EES expands the bit-width of shared exponents without increasing storage overhead, thereby enhancing dynamic range. The authors further optimize the bit-width configuration of extendable exponents and mantissa through a design space exploration strategy to balance precision and energy efficiency.

Experiments demonstrate that EES achieves up to 1.38× energy efficiency improvement and 0.36% accuracy gain over existing methods (e.g., Tender, BiE) across multiple large language models (LLMs, e.g., OPT, LLaMA) and benchmarks (e.g., Hellaswag, LAMBADA), with supporting hardware architecture enabling efficient computation.
Summary of strengths: 1. The paper proposes an innovative approach by utilizing the lower bits of the floating-point mantissa to extend the exponent representation while minimizing the impact on model capacity.
2. The paper also introduces a method for searching the optimal representation scheme within the floating-point bit-width space, providing a near-optimal solution for constructing the proposed EES, which enhances its practicality.
3. The paper comprehensively discusses algorithm design and hardware, closely bridging theory and practice.

1. **Methodological Innovation**: EES introduces a flexible exponent extension mechanism to resolve outlier challenges in low-bit BFP quantization, demonstrating originality in design.

2. **Comprehensive Validation**: Extensive experiments cover mainstream LLMs (OPT, LLaMA) and diverse benchmarks (Hellaswag, ARC), ensuring robust and credible results.

3. **Hardware Co-Design**: Proposes dedicated hardware components (e.g., decoders, encoders, processing units) and validates deployment feasibility through energy/area analysis.

4. **Quantization Strategy Optimization**: The design space exploration strategy effectively balances precision and energy efficiency, offering theoretical guidance for scenario-specific configurations.
Summary of weaknesses: 1. The presentation of the paper could be further improved. Although I was able to grasp the main ideas after a careful and thorough reading, the introduction section presents background and definitions in an overly technical manner, which limits readability. Incorporating simple illustrations might help clarify the research problem. Additionally, in the experimental tables (e.g., Table 4), highlighting the best results could make them more noticeable.
2. The search algorithm proposed in Bit Width Space Exploration primarily demonstrates a balance between accuracy and efficiency. However, the paper does not clearly illustrate how the EES algorithm operates on an existing LLM to convert a full FP16-format model into the proposed BPF-based extendable exponent format.
3. In Table 7, the bolded values do not appear to be the best results. In other words, EES does not achieve the top performance in any individual metric, making the impact of this table seem weak. As a reviewer, I understand that the authors aim to demonstrate EES’s strong efficiency (comparable) while maintaining good accuracy, but this message is not conveyed effectively by the table alone. I suggest using a different visualization method that jointly presents accuracy and efficiency metrics to make the comparison clearer.

1. **Insufficient Limitation Analysis**: While replacement errors and configuration complexity are noted, their specific impacts (e.g., error distribution, task-specific sensitivity) lack in-depth discussion.

2. **Neglect of Inter-Layer Heterogeneity**: Global optimization in design space exploration ignores layer-specific characteristics (e.g., attention vs. FFN layers), potentially leading to suboptimal configurations.

3. **Limited Hardware Verification**: Energy efficiency comparisons (e.g., Table 7) lack detailed evaluation of real-chip implementations or latency/throughput in practical scenarios.
Comments, suggestions and typos: All suggestions are shown in the weaknesses part.

1. **Dynamic Configuration Adaptation**: The offline nature of design space exploration may fail to adapt to dynamic input distributions. Is it possible to enable real-time configuration adjustments?

2. **Error Analysis**: How does replacement error accumulate across layers? Could it degrade performance in deep models?

3. **Deployment Challenges**: Compatibility with current frameworks and quantize/de-quantize overheads are undiscussed.
Overall assessment scores: 3

3
Metareview score: 3.5
Metareview: The paper proposes an Extendable Exponent Sharing (EES) idea to represent weight outliers, where the exponent bits are extended to mantissa bits. 

Overall, reviewers believe that this is a simple and smart idea and that the results are promising. 

On the other hand, certain suggestions are made by reviewers, such as using more advanced LLMs and not overcomplicating the ideas in writing
Summary of reasons to publish: Overall, reviewers believe that this is a simple and smart idea and that the results are promising.