BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
Authors: Wonsuk Jang
Thierry Tambe
Venue: ICML 2025 poster (2025)
Keywords: Quantization
Large Language Models
Hardware-supported scaling
Mixed format
Primary area: deep_learning->large_language_models
TL;DR: BlockDialect employs block-wise mixed format quantization with a tailored formatbook, optimizing both accuracy and energy efficiency for LLM inference.

Abstract: The rapidly increasing size of large language models (LLMs) presents significant challenges in memory usage and computational costs. Quantizing both weights and activations can address these issues, with hardware-supported fine-grained scaling emerging as a promising solution to mitigate outliers. However, existing methods struggle to capture nuanced block data distributions. We propose BlockDialect, a block-wise fine-grained mixed format technique that assigns a per-block optimal number format from a formatbook for better data representation. Additionally, we introduce DialectFP4, a formatbook of FP4 variants (akin to dialects) that adapt to diverse data distributions. To leverage this efficiently, we propose a two-stage approach for online DialectFP4 activation quantization. Importantly, DialectFP4 ensures energy efficiency by selecting representable values as scaled integers compatible with low-precision integer arithmetic. BlockDialect achieves 10.78% (7.48%) accuracy gain on the LLaMA3-8B (LLaMA2-7B) model compared to MXFP4 format with lower bit usage per data, while being only 5.45% (2.69%) below full precision even when quantizing full-path matrix multiplication. Focusing on how to represent over how to scale, our work presents a promising path for energy-efficient LLM inference.

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Effccient LLM Inference
Wonsuk Jang'
Thierry Tambe'
Abstract
Fine-grained block
FormatBook
Dialect 1
The rapidly increasing size of large language models (LLMMs) presents significant challenges in memory usage and computational costs. Quantiz- ing both weights and activations can address these issues, with hurdware-supporte fine- fine-grained scal- ing emerging
Dialect Dialect
Dialect 2
Dialect
Dialect
Dialect
Dialect 4
Dialect Dialect
Dialect
Dialect
3
3
Assign the best dialect (standard format variant)
promising solution to miti- gate outliers. However, existing methods strug- gle to capture nuanced block data distributions. We propose BlockDialect, block-wise fine- grained mixed format technique that assigns a per-block optimal number format from a format- book for better data representation. Additionally, we introduce DialectFP4,, formatbook of FP4 variants (akin dialects) that adapt to diverse data distributions. To leverage this efficiently, we propose two-stage approach for online Di- alectFP4 activation quantization, Importantly, Di- alectFP4 ensures energy efficiency by selecting representable values as scaled integers compatible with low-precision integer arithmetic. BlockDi- alect achieves 10.78 % (7.48 %) accuracy gain the LLaMA3-SB (LLaMA2-7B) model com- pared to MXFP4 format with lower bit usage per data, while being only 5.45 % (2.69% %) below full precision even when quantizing full-path matrix multiplication. Focusing on how to represent over how to scale, our work presents a promising path for energy-efficient L.LM inference.
Activation (Online quantization)
X
Weight
(Offline quantization)
Figure 1. Overview of BlockDialect technique workflow.
lami et al., 2024) and increased computational costs (Xiao et al., 2023). By reducing numerical precision, quantiza- tion effectively reduces memory usage and data movement overhead (Kim al., 2023). Additionally, leveraging low- precision operation results in improvements in inference speed, area, and energy efficiency (Xiao et al., 2023; Cao et al., 2024; Rouhani et al., 2023a). A key challenge in LLM quantization lies in handling outliers elements with much larger magnitudes compared to the rest (Dettmers et al., 2022). Outliers skew the scaling factor, diminishing the representation capacity for the majority of the elements (Liu et al., 2023). To counter this issue, block-wise quantization has been adopted a common solution (Frantar al., 2022; Dettmers et al., 2023; Lin al., 2024: Sheng et al., 2023). By partitioning the tensor into smaller blocks and quantizing each block separately, this method effectively mitigates the influence of outliers within localized areas.
While smaller blocks encapsulate outliers better, they in- troduce overhead in managing high-precision scaling fac- tors (Rouhani et al., 2023a). To balance this, recent advance- ments have focused hardware support for fine-grained block-wise quantization (e.g.. block size 32) (Rouhani et al., 2023a; Dai et al., 2021; Rouhani et al., 2023b). In line with this progress, Open Compute Project, backed by lead- ing tech companies, has established the Microscaling (MX) format specification !. This format enhances performance and hardware efficiency via fine-grained blocks and power- of-two scaling factors, and has been adopted by recent AI
1. Introduction
Quantization a crucial technique (Gholami et al., 2022) to address the challenges posed by the exponential growth in Large Language Models (LLMs), including memory bot- tlenecks (Frantar et al., 2022; Alizadch al., 2023; Gho-
Department of Electrical Engineering, Stanford Uni- versity, CA. USA. Correspondence to: Thierry Tambe <ttambe@ <taambe@stanford.ddi edd
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
formats-mx-v1-Oe --w-------------aaaa---- final-pdff

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Effiiient LLM Inference
accelerators like NVIDIA's Blackwell 2,
precision. Our contributions can be summarized follows:
In addition to advancements in block-wise quantization, re- search has progressed to sub-8-bit quantization, reaching even 2-bit precision (Egiazarian et al., 2024; Chee et al., 2024). However, most methods focus on weight-only quan- tization due to challenges in quantizing activations which are: (1) the need for real-time quantization, (2) a wider dy- namic range, and (3) channel-wise variances that misalign with matrix multiplication dimensions (Xiao et al., 2023). As a result, activations remain high-precision, requiring dequantization of weights and high-precision arithmetic op- erations (Dotzel et al., 2024), which reduces gains in energy efficiency and inference throughput. Moreover, increasing sequence lengths in modern I.LMs exacerbate these inef- ficiencies due to quadratic computational growth (Shyam et al., 2024). Thus, addressing activation quantization is critical for realizing energy-efficient L.M inference.
We introduce BlockDialect, novel block-wise fine- grained mixed format technique that assigns optimal number format to each block, enabling accurate repre- sentation of data distribution in L.Mss.
We propose DialectFP4, set of FP4 variants tailored for diverse block-level distributions, and achieve online DialectFP4 activation quantization through a practical two-stage approach, yielding accuracy comparable to an MSE-base approach.
We demonstrate that approach outperforms existing methods across multiple LLMs while leveraging low- precision, energy-efficient MAC units.
2. Related Work
Our work stems from the insight: "If a group of numbers deserves its own scaling factor, why not a number format?" Existing research primarily focuses on "how to scale" ac- tivations to make them quantization-frienn
2.1. Block-wise Quantization
often by mi- grating quantization difficulty weights (Xiao et al., 2023) or utilizing Hadamard matrices to reduce outliers (Ashk- boos et al., 2024). In contrast, we take a novel perspective by exploring "how to represent" each block. Leveraging harpowee-ssppported fine-grained block-wise quantization, we propose BlockDialect, which enables 4-bit weight and ac- tivation post-training quantization with each block assigned an optimal number format from the formatbook.
Block-wise (or group-wise) quantization is a widely adopted technique that assigns scaling factors sis, constraining the impact of outliers within each block. To determine these scaling factors, two methods can be employed: software-supported and hardware-supported. Software-supported methods (Frantar et al., 2022; Dettmers et al., 2023; Lin et al., 2024; Sheng et al., 2023) typically rely on high-pne precision scaling factors, enhancing accuracy but of- ten require larger blocks due the overhead of storing and applying scaling factors. In contrast, hardware-supporte techniques allow smaller blocks using werdware-friendle scaling factors, such as power-of-two shared exponents. VS- Quant (Dai et al., 2021) and Micro-exponents (Rouhani et al., 2023a) demonstrated the effectiveness of this ap- proach, further enhanced by multi-level scaling factors through dedicated hardware. Open Compute Project re- cently introduced the microscaling (MX) format (Rouhani et al., 2023b), which uses shared exponents across low- precision formats like FP4 and FP6. Its adoption in recent accelerators 2 highlights ongoing industry efforts to enhance hardware support for fine-grained scaling. Building on these advancements, our work introduces a novel approach that assigns number formats each fine-grained block.
per-block ba-
Additionally, we present DialectFP4: a formatbook of FP4 variants tailored to diverse block-wise data distributions. To leverage this efficiently, we propose a two-stage approach for online optimal format selection, achieving zero-shot performance comparable to the mean squared error (MSE)- based approach. Importantly, DialectFP4 ensures compat- ibility with low-precision integer arithmetic by selecting representable values as scaled integers (i.e., multiples of 0.5), enabling the proposed MAC units to achieve the area and energy efficiency of FP4 MAC units. To further en- hance energy efficiency, our approach extends to full-path matrix multiplication, including not only setvaattonweeghh multiplications in linear layers but also indivatioo-actiaatioon multiplications in attention blocks.
BlockDialect demonstrates significant improvements over the MXFP4 format, achieving 10.78% (7.48 %) higher zero- shot accuracy with lower bit usage per data, while showing only 5.45 % (2.69%) %) lower accuracy than full precision on the LLaMA3-8B (LLaMA2-7B) for full-path quantization. When quantizing only linear layers, BlockDialect achieves a marginal 1. 1.76% % (1.20%) accuracy drop compared full
2.2. Non-Uniform Quantization
Non-uniform quantization has been extensively explored as alternatives to integer formats, aiming to better capture data distributions in LLMs. tooaingg-poit formats have proven effective for handling wide value ranges encountered in deep learning models. FP8-Quantizatiin (Kuzmin et al., 2022) highlights how FP8 outperforms INT8 by effectively addressing outliers through its flexible exponent represen- tation. ZeroQuant-Fi (Wu et al., 2023) demonstrates that
2

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiient LLM Inference
floating-point formats strike better balance between dy-
Retio of deta points (%)
[_prroj (weight) _gate_proj (weight) attn_ingu _nnput (activatin) mip_input (activation)
40
namic range and precision compared to integer formats.
To improve the flexibility of representable values, lookup- based formats have been studied. NF4 (Dettmers et al., 2024) and SF4 (Dotzel et al., 2024) leverage statistical dis- tribution quantile functions (normal and Student's distribu- tion, respectively) to better align with LLM profiles. Vec- tor quantization extends this concept by performing vector- level matching with codebooks, with AQLM (Egiazarian
20
10
[0. (0.05))
[0.5 0.5.11
111.5
[1.1
[1.5.2]
(2.2.5)
[2 2253
(3.3.5]
(3.5.4)
(4,4.5)
(4.5.)
[5. (5.55)
(5.5.6)
16. (8.6.5)
[6 (.7)
17.7.50
(7.5.8)
Data ranges
Retio of deta points (%)
20
(a)
15
Maximum magnilude ls (4.8)
et al., 2024) introducing additive codebook quantization and QulP# (Tseng et al., 2024) proposing cache-efficient compressed codebook. Recently, NxFP (Lo et al., 2024)
normalization by 2
(0.0.5)
[0.5,1)
[1.1.5]
(1.5.2)
12.2.5)
12
[3.355
(3.5.4)
(4.4.5)
(4.5.)
[B.5.5]
[5.5.6]
[8.6.5)
(8.5.7)
[7.7.5)
(7.5.8)
introduced an enhanced MX format, appending mantissa bits to the shared exponent to address similar observations ours, such as inaccurate largest magnitude representation (detailed comparison in Appendix D). However, these meth- ods are typically limited to weight-only quantization and depend on high-precision operations, incurring significant compute and energy overhead. Our work uses FP4 vari- ants that capture block-level distributions while ensuring compatibility with low-precision integer arithmetic.
Dala ranges
Figure 2. LLaMA3-8B block-level profiling results: (a) matrix-
wise accumulated magnitude distribution, (b) block's 's maximum magnitude distribution. Each bar represents the average across layers O, 10, 20, and 30, with consistent trends across layers.
2.3. Activation Quantization
Additionally, unlike prior methods that rely on calibration or pre- me-training to reduce online activation processing overhead, BlockDialect supports efficient online processing via two- stage format selection and logical operation-basee quantiza- tion. It can also capture unstructured outliers through fine- grained blsek-wise localization, beyond easily calibrated structured outliers (e.g., channel-wise magnitude mean).
Addressing challenges of activation quantization such as real-time execution, large dynamic ranges, and channel- wise outliers, researchers have proposed several approaches: 1) LLM.int8() (Dettmers al., 2022) and Atom (Zhao al., 2024) employ mixed precision subgrouping, retaining outliers in high precision. However, this approach incurs non-negligible overhead from handling mixed precision. 2) SmoothQuant (Xiao et al., 2023) migrates the quantiza- tion difficulty to weights, enabling low-precision weight and activation quantization, thereby avoiding high- -ieeprrciiion operations. 3) Recent advancements using Hadamard ma- trices reduce outliers while maintaining computational in- variance. This enables effective 4-bit weight, activation, and KV cache quantization (Ashkboos et al., 2024; Liu et al., 2024b). However, it incurs overhead from online Hadamard transformation and retains some -igh-precision components (e.g.. queries). 4) Mixed format quantization selects optimal number formats for predefined granularities. For example, LLM-FP4 (Liu al., 2023) adjusts matrix-wise formats and exponent biases, while MoFQ (Zhang et al., 2024) applies layer-wise format selection between floating point and inte- ger. Yet, they lack adaptability to varying data distributions due to relatively coarse and limited customization strategies.
3. BlockDialect: Block-wise Fine-grained Mixed Format Quantization
achieve block-wise fine-grained mixed format quanti- zation, we address three key questions: Which dialects should be used? 2) How should the per-block dialect be selected? 3) How should online quantization and MAC operations be performed?
3.1. Which Dialects Should be Used?
Block-Level Profiling. To provide guideline for deter- mining dialects for the formatbook, we conduct profiling of Llama3-8B (Dubey et al., 2024), Llama2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023), and OPT- 6.7B (Zhang et al., 2022) models using WikiText2 (Merity et al., 2016) (Figure 2 shows results for LLaMA3, with results for other models in Appendix A). We split each ma- trix into blocks of size 32, scale each block by the shared exponent [log2 (block's maximum magnitude) - and ac- cumulate magnitude distribution histograms for each block. Normalization by the maximum exponent yields values in [0, 2). while FP4 E2MI spans [0, 6]. Subtracting 2 from the shared exponent shifts the range to [0, 8), enabling di-
Our work refines mixed format guantization by introduc- ing FP4 variants with minimal differences in representable values, assigning the optimal variant to fine ine-grained blocks. To emphasize the use of variants over entirely distinct for- mats, we use the term dialect throughout the paper instead of variants.
3

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.ficiiee LLM Inference
Data
@: FP4 E2M1
@: Better position
@
Different large magritude distributions
Block
Dialect 0
7.5
5.5
1.5
0.5
wasted
Dialect 1
7.5
4.5
2
1.5
0.5
(a)
Dialect 2
5.5
3
2
1.5
0.5
0
Block
Dialect
4.5
0.5
underestimated
Dialect
6.5
5
2
1.5
0.5
FP4 E2M1
Dialect 5
6.5
4
3
2
1.5
0.5
Dialect
5
2
1.5
0.5
(b)
Dialect
0.5
Block
Dialect 8
5.5
4.5
2
1.5
0.5
2 (power-of-two dynamic range)
Dialect 9
5.5
3.5
3
2
1.5
0.5
Dialect 10
4.5
2
1.5
0.5
Figure 3. Core observations shaping the formatbook design: (a) wasted underestinated ranges, (b) block-specift distributions deviating from the matrix-wisd pattern.
Dialect 11
5
3.5
1.5
0.5
Dialect 12
4.5
4
2
1.5
0.5
Dialect 13
4.5
3.5
2
1.5
0.5
Dialect 14
3.5
1.5
0.5
Dialect 15
3
2.5
2
1.5
0.5
Varicus dynamic ranges
@ Granularity of 0.5 & common values
rect comparison with FP4 E2M1. Note that we leverage hardware-supportee scaling with a power-of-two scaling factor, resulting in power-of-two dynamic range.
Figure 4. 16-dialect DialectFP4 example.
Our matrix-wise analysis revealed that FP4 E2MI closely aligns with the observed distribution. Specifically, the values are dense in the 0-2 range, sparser between 2-4, and highly sparse between 4-8, patterns that mirror the distribution of representable values in FP4 E2MI (Figure 2a). Based on this, we select FP4 E2MI (hereafter referred to as FP4) as the base format for our dialects.
Large magnitudes, especially outliers, are more likely to yield higher values after multiplication, indicating their greater importance, as similarly noted by other works (Lin et al., 2024; Dettmers et al., 2022). Likewise, we assume that larger magnitudes in each block are also of relatively greater importance. Given the constraints of 4-bit repre- sentation, with only distinct representable magnitudes, our approach prioritizes accurately expressing larger mag- nitudes over smaller ones. Our profiling reveals that each block's scaled data distribution does not always follow the matrix-wise trend, which becomes sparser at the outer bound (Figure 3b). Therefore, simply adjusting the exponent bias is insufficient as it fails to capture the nuanced distributions within specific blocks. Consequently, we establish our sec- ond criterion: generating dialects capable of representing a diverse range of magee-manniude distributions.
However, upon examination of individual block magnitude distributions, we identified two important trends. First, the maximum magnitude of each block is relatively evenly dis- tributed (Figure 2b). Second, some blocks deviate from the overall matrix-wise distribution, which shows more sparsity at the outer bound of the range. For instance, some blocks have multiple values around 7.5 but none in the [4, 7] range. These observations emphasize the importance of aligning with each block's specific distribution, leading three core principles for our formatbook: 1) minimizing wasted or underestimated ranges, 2) prioritizing the representation of larger magnitudes, and 3) ensuring hardware efficiency.
Ensuring Hardware Efficiency. To achieve hardware- efficient quantization, our dialects must support low- precision arithmetic. Hence, we maintain a minimum granu- larity 0.5. This approach limits the bit width per value and avoids floating -pintt operations, enabling more hardware- efficient implementation (detailed in Section 3.3). Also, using multiple dialects requires real-time activation quanti- zation for each selected dialect. This process cannot rely on conventional shift and round logic due to dialect variability, necessitating distinct quantization logic for each. However, implementing separate logic for each value across all di- alects would be inefficient. These considerations inform our third criterion: aligning to 0.5 granularity and preserving most FP4 values across dialects. This strategy balances representational flexibility with hardware efficiency.
Minimizing Wasted or Underestimated Ranges. Each block has its own dynamic range, but the use of power- of-two shared exponents results in power-of-two dynamic ranges. This mismatch leads wasted or underestimated range for certain blocks. As illustrated in Figure 3a, blocks with maximum magnitudes smaller than 6 (the maximum representable value of FP4) waste the range beyond their maximum magnitude, as no data points can be mapped to this unused range. For instance, if a block's maximum magnitude is 4.5, the maximum representable value of FP4 (6) could be more effectively utilized by scaling the range to lie between 0 and 4.5. Conversely, data points larger than 6 cannot be represented and are thus underestimated by FP4. This leads to our first criterion: including FP4 dialects with different maximum magnitudes.
16-Dialect DialectFP4 Example. Figure 4 illustrates 16- dialect formatbook, DialectFP4, that meets our three key requirements: ! The dialects cover all possible maximum magnitudes. 2 Each pair of dialects shares the maximum
Prioritizing the Representation of Larger Magnitudes.
4

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Effiiient LLM Inference
magnitude while differing in one large magnitude value, capturing various large magnitude distributions. 3 The unit of these dialects is 0.5, aligning with FP4, while most of the six smallest magnitude values remain consistent with FP4. Based on this DialectFP4, each data point is stored using 4 bits: 1-bit sign and 3-bit index indicating the value of the selected dialect. Additionally, a 4-bit dialect identifier (for 16 dialects) is assigned to each block.
characteristics, avoiding wasted or underestimated ranges.
In the second stage (Figure 5c), we determine the optimal di- alect from the chosen pair by evaluating which one has more block elements within its beneficial range. Since the two dialects differ by only one large magnitude value, the benefi- cial range is defined as the interval where incorporating this different value reduces quantization error. This range can be calculated as the midpoint between the differing value, its adjacent value, and the paired dialect's differing value. For example, the beneficial range for dialect 4 is [4.5, 5.75), where 4.5 is the midpoint between the differing values of dialect and 5 ((5.0 + 4.0)/2), and 5.75 is the midpoint between dialect 4 and its adjacent value ((5.0 + 6.5)
3.2. How Should the Per-Block Dialect be Selected?
While the optimal per-block dialect for preknown weights can be determined by calculating the exact mean square error (MSE) for each dialect, this approach is infeasible for activations due to their dynamic nature. In particular, Block- Dialect focuses on fine-grained activation blocks, where the selection of the dialect has a direct and significant impact on quantization outcomes. This heightened sensitivity de- mands a more precise and adaptive method. To address this, we adopt a sample dataset-agnostic strategy that performs on-the-fly dialect selection. Specifically, efficient two-stage selection process that operates after initial preprocessing step, as shown in Figure 5.
A naive approach to count elements within beneficial ranges requires four comparisons (two per each beneficial range) per element, which introduces compute and latency over- head. To optimize this, we convert range checks into effi- cient logical operations as illustrated in Figure Sc. If we represent each beneficial range as a 5-bit binary represen- tation and enumerate all possible cases, we can compress them into simple logical operations. In Figure Sc, we use 5' 3'110110 to represent the upper limit of dialect 4's beneficial range, excluding 5.75 15'11011).. It encompasses all values smaller than 5.75, as we truncate after the second fractional bit. Finally, if the four most significant bits are 4'b1001, the element falls within dialect 4's beneficial range.
propose
Preprocessing Stage. In the preprocessing stage, we com- pute a 5-bit shared exponent (FP16 exponent bit width) based on the block's maximum magnitude, adjusting it to ensure the expression range. [0,8), fully encompasses FP4's range, [0.6]. Each element's exponent is then adjusted by subtracting the shared exponent, enabling compact 2- exponent per element. For cases where the shared exponent exceeds an element's exponent, a compensatory mantissa shift is applied. The mantissa is then shifted by the adjusted exponent, and the lower bits are truncated to form a 5-bit representation: 3-bit integer part and 2-bit fractional part (Figure 5a). This representation covers DialectFPP's range of 0.0 to 7.5 with 0.5 granularity, requiring 3 integer bits and at least 1 fractional bit. However, to enable accurate rounding during quantization, an additional fractional bit is included, resulting in a 5-bit intermediate format. To clar- ify, this does not indicate 5-bit quantization; rather, it is an intermediate value used internally during dialect selection and quantization.
This logic can be pre-designed as we use a fixed DialectFP4. With these simple logical operations, we can efficiently count the number of elements falling within each dialect's s beneficial range in parallel. Given that we use fine-grained blocks with sizes up to 64 elements, the counting process can be further optimized using a reduction tree structure.
3.3. How Should Online Quantization and MAC Operations be Performed?
DialectFP4 uses 0.5 granularity for representable values, expressing values from 0 to 7.5 as 4-bit unsigned integers from 0 to 15 (0.5 * [0-15]). Consequently, multiplications can be efficiently performed with 4-bit unsigned integer arithmetic, followed by 2-bit right shift to account for the 0.5 factor of each number. Our quantization target is thus 4-bit (1-bit sign, 3-bit index), dequantized to 5-bit (previous 1-bit sign, 4-bit unsigned integer) before multiplication.
Two-Stage Dialect Selection Process. Comparing every dialect to find the best dialect for each block is computation- ally expensive and inefficient. Instead, we adopt two-stage approach. In the first stage, we narrow down the options by selecting a pair of dialects whose largest magnitudes match the block's s maximum (Figure 5b). Recall that each pair of dialects share the maximum magnitude. The block's maximum magnitude can be easily determined by round- ing from the second fractional bit (BlockmeaTrwwe)) This step not only streamlines the selection process but also en- sures that the chosen dynamic range aligns with the block's 's
For weights, the optimal dialect for each block is precom- puted, with pre-quantization performed prior to inference. During inference, the 3-bit index is converted to 4-bit inte- gers by indexing a pre-stored table of representable values (Figure 6). Most values are shared across dialects, minimiz- ing storage requirements. In contrast, activations require real-time quantization to the nearest representable value of the optimal dialect. For example, consider preprocessed data
5

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
Preprocessing stage
1st stage
2nd stage
Find Shared Exponent (SE) SE [log, Block 2
max
Dialect 4 beneficial range: [4.5, 5.75)
Dialect 0 17.5
Dialect beneficial range: [3.5, 4.5)
Exponent Subtract: expmexp-SE SE
Dialect 1
7.5
S-bit binery representation
2 bit
Dialect 2
7.0
Dialect 4: 100.10 z Mag $ 101.10
Dialect 5:
011.10 s Magm S 100.01
Mpy
Shift & Truncate: man << exp
Dialect 3
7.00...
Blocknaxtmin
3
May
2 bit
1st stage:
Select pair of dialects Dialect BlockneTuue
Dialect 5 6.5 4.0
Dialect 4 6.5 5.0 3.0
Dialect 4: (1001-, 1010:, 10110)
Dialect 5: (011-, 1000-)
+ Logical operation count
round
2nd stage: select the dialect with more elements its beneficial range
Dialect [4] [[3] ((1[2) (1D) [[2] ((8[1:0J)
Magrunc
meas
Block's maximum 6.5
Dialect 5: (1[4) & (8(3:1)) I ([4) (B:1))
(a)
(b)
(c)
Figure Per-block dialect selection process: a) overall process, b) 1st stage, and c) 2nd stage.
Optimal dialect index: 4'd4
and accumulated, right-shifted by 2 bits to apply the 0.5 scaling factor, and then requantized into 4-bit DialectFP4 format for the next matrix multiplication. Other specialized operations, such softmax, are performed in high precision. Importantly, the optimal dialect selection, quantization, and MAC operations can be pipelined to further accelerate the overall inference speed.
Quantized 4-bit data: 4'b1 110
7 T
Idx
Dialect 4 Dialect Dialect
sign
idx
7
6.5
6.5
6.0
6
5.0
4.0
5.0
0.5 -10 5.0
Common
5
3.0
Integer arithmetic operation
4
2.0
3
1.5
2
1.0
Apply to accumulated result
0.5
4. Experiments
Figure 6. Overview of dequantization process.
4.1. Experimental Setup
Models and dataset. We evaluate BlockDialect on three LLMs: LLaMA-2-7B (Touvron et al., 2023), LLaMA- 3-8B (Dubey et al., 2024), and Mistral-7B (Jiang et al., 2023). The evaluation includes seven zero-shot common-
(Magtrune in Figure 5) as 5' 5'b10001, representing 8.5 with 0.5 scaling factor. If the optimal dialect's representable val- ues are 0.5* * [13, 10,6 6, 4, 3, 2, 1, 0]. the value 8.5 would be quantized to 10. Note that this operation cannot be achieved using simple shift and rounding logic. After applying the 0.5 scaling, the final representation is 5.
sense reasoning tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021). ARC-easy, and ARC-challenge (Clark et al., 2018). We leverage the work, with 0-shol notation representing the average accuracy across seven tasks. Additionally, we report perplexity scores on WikiText2 (Merity et al., 2016) with a chunk of 2048.
We can also accelerate this efficiently with method similar to our dialect selection process. Each representable value is linked to simple checking logic derived from its binary form. For example, data is quantized to 10 it falls within [8.0, 11.5), expressed as [5 [5'610000, 5'610110). This can be verified by k(4) & 1A [3) & ()).. which holds true for the previous example, 5' 5'010001. The final 4-bit quantized value includes sign bit and 3'61 3'6110 (the index for 10). The overall quantization logic can be optimized by eliminating redundant logical operations, such as the repeated use of k([4] & [3[3] across different values. Additionally, since most representable values are shared among dialects, the logic can be further simplified.
(Gao et al., 2023) frame-
Baseline. We compare BlockDialect with the MXFP4 format (Rouhani et al., 2023b), which employs hardware- supported scaling. Additionally, we compare two recent methods employing coftware-supporte scaling: LLM- FP4 (Liu et al., 2023), and Quarot (Ashkboos et al., 2024). Quarot reduces outliers via Hadamard matrix, while LLM- FP4 adopts a matrix-wise mixed format. We run baselines using their open-source code3. To demonstrate BlockDi- alect's applicability for full-path matrix multiplications, we evaluate two scopes: linear (quantizing linear layers) and all (extends to attention block operations QK, Atin scoreV).
While multiplications are performed using 4-bit unsigned integers, all pairwise products between two blocks share the same exponent sum, enabling accumulation with integer operations and resulting in power- and area-efficient hard- ware (detailed in Section 4.3). Only when all elements in a block are processed, partial sums are converted to FP16
We set to avoid excessive calibration time, observing negligible LLaMA- 7B accuracy loss compared to the original paper.
reanch_round) (60, 2) in LLM-FP4
6

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
Table Perplexity on Wikitex12 and average zero-shot accuracy across seven common- reasoning tasks. dn: down.proj. Q: query. K: key, V: value. t: Quarot keeps query and attention scores in FP16 and performs the associated operations in FP16.
Block size (exception)
LLaMA3-8B
LLaMA2-7B
Mistral-7B 7B
Scope
Method
Eff. bit
Wiki,
AVG.
Eff. bit
Wiki
AVG.1
Eff. bit
Wikil,
AVG.1
I-P16
16
6.14
74.45
16
5.47
70.94
16
5.32
74.92
LLM-FPP4
Attensor, Wxch.
4
48.71
41.92
15.61
58.15
17.47
58.47
Quarot (W4A4)
Attoken, W:ch.
4
8.02
66.92
6.04
68.00
4
5.74
72.49
16
4.31
8.20
68.53
4.31
7.07
66.86
4.31
6.49
70.33
Linear
MXFP4
32
4.16
8.23
68.31
4.16
7.04
65.94
4.16
6.42
70.72
(A-W)
32
4.28
7.05
72.24
4.28
5.84
69.74
4.28
5.65
73.46
BlockDialect
64
W:4,25
W:4.23
W:4.25
7.12
72.69
5.88
69.51
5.68
73.30
(w/ DialectFP4)
(dv:16)
A:4.30
A:4.27
A:4.30
64
4.14
7.30
71,51
4.14
5.96
68.95
4.14
5.75
72.76
A:loken, Wich. (K, V:128)
Quarot (W4A4KV4)
W,K, V:4*
8.17
66.01
W.K.
6.10
67.50
W.K.
5.80
71.72
4.31
18.84
58.22
4.31
11.22
60.77
4.31
9.27
66.03
MXFP4
All (A-W.A-A A-A)
32
4.16
16.69
58.84
4.16
11.14
59.76
4.16
8.98
66.01
32
4.28
7.87
68.57
4.28
6.33
67.68
4.28
5.87
72.15
BlockDialect (w/ DinlectFP4)
64
W:4.25
W:4.23
W:4.25
(d1.
A:4.21
7.77
69.00
A:4.21
6.35
68.25
A:4.21
5.90
71.71
64
4.14
8.55
66.60
4.14
6.63
67.15
4.14
6.07
70.26
All operands are quantized along their respective multipli- cation dimensions, as detailed in Appendix C. We denote Effective bitwidth (Eff. bit) as the average bit width required per data, accounting for overhead from scaling factors or dialect identifiers as explained in Appendix I. In the linear scope, this metric covers linear layers; in the all scope, it includes all matrix multiplications. Block size is 32 unless otherwise specified. For simplicity, we denote each method by its block size, e.g., [method]-32 (block size of 32).
FP4 and Quarot, with 41.41/ 0.72-point lower perplexity, and 29.59 % 4.59 % higher accuracy, respectively. Com- pared to full-precision results, BlockDialect-32 exhibits marginal accuracy drops of 2.21 % / 1.20 % 1.46 % on the LLaMA3, LLaMA2, and Mistral models, respectively.
BlockDialect achieves full-path quantization (all) with min- imal performance loss. While MXFP4-16 suffers a signifi- cant accuracy drop (~~16.23 %), BlockDialect-32 shows re- silience, with 5.88 % / 3.26% / 2.77 % accuracy degradation compared full precision on the LLaMA3, LLaMA2, and Mistral models, respectively. BlockDialect-3) also outper- forms Quarot (W4A4KV4), which quantizes the linear layer, key, and value to 4-bit, demonstrating BlockDialect's supe- riority. Note that the comparison with W4A4K V4 is conser- vative, as it retains figh-precision components and performs high- gigh-precision activation -activatiooo multiplications.
Implementation. For performance evaluation, we imple- ment the BlockDialect emulation framework4 on top of HuggingFace Transformers using PyTorch. All experiments were conducted on a single NVIDIA H100 GPU. For hard- ware comparison, we model multiply-accumulate (MAC) units for various precision levels using System Verilog and synthesize them with Synopsys Design Compiler. The syn- thesis is performed at 0.5 GHz using the Nangate 45nm OpenCell Library to estimate area and power. Each MAC unit is sized to iteratively add 64 terms from dot product. For additional prototype hardware cost analysis, we syn- thesize the design using the Sky Water 130mm standard cell library, targeting a clock frequency of 100 MHz.
Finally, BlockDialect's performance improves further with smaller blocks in quansizationn-eensiiive sublayers, de- tailed in the following block size ablation study. This leads to accuracy drops of only 1.76 % in the LLaMA3 linear scope, with 5.45 % 2.69% in the all scope for the LLaMA3 and LLaMA2 models, respectively, compared to full preci- sion. Additionally, it outperforms MXFP4-16 16 by 10.78 7.48 % with lower effective bitwidth for these models.
4.2. Experiment Results
Main Results. As shown in Table 1, BlockDialect consis- tently outperforms MXFP4 across all models in the linear scope, even at a lower effective bitwidth. For instance, BlockDialect-64 achieves 0.93-point lower perplexity and 3.20% % higher accuracy than MXFP4-32 on the L.LMM33 model. Additionally, BlockDialect-64 surpasses both LLM-
Comparison with MSE-based Dialect Selection. We pro- posed an efficient two-stage approach to eliminate real-time MSE calculation for activation quantization. Table 2 com- pares our method with the MSE-basee approach, where both weight and activation dialects are selected based on MSE. Despite the absence of MSE computations, our approach in linear scope results in only minor perplexity increase (~044) and slight accuracy drop (~0.61 %) across mod-
'https://code stanforr..aoooddddooooo hww.:fordddd///ammt.tt. 1ab/ blockdialect
7

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
Table Comparison of dialect selection methods.
Table 4. Comparison across dialect numbers. For the 8-dialect case, two configurations are tested: prioritizing large magnitude
M.LMMA3-8B
T.LaMA2-7B
Mistral-7B Wikil,
distribution (dist.) and dynamic range (max. magnitude) (range).
Scope Method
Wikil
0-short
Wikil
0-short
0-shott
MSR
7.01
72.85
5,83
69.66
5.64
73.80
LLaMA3-8B
LLaMA2-7B
Mistral-77
Linear
Ours
7.05
72.24
5,84
69.74
5.65
73.46
Dialect
Wikil
0-shotf
Wikii
0-shot
Wikij
0-shott
All
MSE
7.72
69.19
6,25
68.31
5,85
72.12
(dist.)
8.29
67.96
6.51
66.75
6.01
71.88
Ours
7.87
68.57
6,33
67.68
5,87
72.15
(range)
8.20
68.06
6,45
67.51
5.94
71.42
16
7.87
68.57
6,33
67.68
5.87
72.15
24
8.84
67.57
6.97
67.30
6.05
71.69
Table 3. Impact of block size: dn: down.proj output.proo, q:
q.proj. k: k_proj, v. v.proj. Q: query, K: key.
accurately, leading to reduced performance. The results demonstrate that the 16-dialect formatbook strikes the best balance, effectively addressing both maximum magnitudes and large magnitude distribution. Interestingly, prioritizing maximum magnitude (range) results in better performance than prioritizing distribution (dist.) overall, aligning with our two-stage approach: select a pair of dialects based on dynamic range first, then choose the better one based on its distribution.
Block size (exception)
LLaMA3-8B
LLaMA2-7B
Mistral-7B Wikil,
Scope
Wikil,
0-shot"
Wikil,
0-shott
0-shott
16
6.82
72.98
5,76
69.91
5.55
73.39
32
7.05
72.24
5.84
69.74
5.65
73.46
64
7,30
71.51
5,96
68.95
5.75
72.76
Linear
64 (dn: 16)
7.12
72.69
5.88
69.51
5.68
73.30
64 (6:16)
7.24
71.68
5.94
69.60
5.73
72.64
64 v:
7.19
71.66
5.91
69.28
5.68
73.30
7.32
70.64
6.08
68.66
5.71
72.53
7.87
68.57
6,33
67.68
5.87
72.15
All
64
8.55
66.60
6.63
67.15
6.07
70.26
Additional Exploratory Studies. To explore various as- pects of BlockDialect, we further analyze the dialect selec- tion ratio for each model (Appendix B), confirming that all dialects are meaningful, with no extremely dominant or meaningless dialects. We also evaluate the performance of BlockDialect across various models and workloads (Ap- pendix E, F), demonstrating its generality for LLMS with diverse architectures, sizes, and tasks. In particular, we compare against another baseline format, NVFP4 a block scaling format with floating-poii scaling factors introduced by NVIDIA. Additionally, we investigate the impact of block shape (Appendix G) and examine the synergy with different activation quantization approaches (Appendix H).
64
7.77
69.00
6.35
68.25
5.90
71.71
(6:
els, even when evaluated on the all scope (~0.15, ~0.63 %) highlighting its effectiveness. This marginal gap also stems from the limitations of the MSE method, which overlooks the magnitude of data elements. This oversight can lead to suboptimal quantization, with inaccuracies for large magni- tude values while accurately quantizing smaller ones. By focusing on large magnitudes, our method achieves more efficient and balanced quantization.
Impact of Block Size. We explore the impact of block size in Table 3. Overall, smaller sizes improve performance by constraining outliers within smaller blocks and making it easier to represent all block data more effectively with dialects. However, this advantage comes with a higher effec- tive bitwidth, making it a trade-off between performance and memory footprint. We further investigate dynamic block size assignment by applying small blocks to specific projec- tion layers to assess block size sensitivity across sublayers. As in Table 3, down projection has higher sensitivity, which aligns with the observation from prior works (Li et al., 2023; Ashkboos et al., 2023). Based on these findings, we ob- tain comparable or superior results with block size of 64 by applying smaller blocks only to sublayers prone to out- liers (Hooper al., 2024; Liu et al., 2024a), compared a uniform block size of 32 in the all scope.
4.3. Hardware Cost Analysis
DialectFP4 is compatible with 5-bit integer arithmetic operations, enabling two implemen- tations: 1) leveraging the general INT4 MAC with simple logic (e.g.. shifter) to handle residual bits for 5-bit multipli- cation (Ours-INT44, and 2) designing optimized MACS with b-t unsigned integer multiplier and additional XOR gate for sign bit (Ours). Although the first option requires more power and area (still efficient than high nigh-precision MACs), it could be a practical option since many commercial accelera- tors already adopt INT4 MACS. The second option's 's MAC design achieves area and power efficiency comparable to FP4 (Table 5), providing significantly higher efficiency com- pared to higher precision MACs. Specifically, it is 1.58x (1.54x) more power (area) efficient than FP6 MACs, which are required to achieve better accuracy levels using the MX
MAC Unit Comparison.
Table 4 compares Impact of Number of Dialects. the performance across different numbers of dialects in all scope. Eight dialects are insufficient to cover both maximum magnitudes and large magnitude distribution, while the 24- dialect formatbook struggles to identify the optimal dialect
hhttpa: B1 olckkcaliig.
learniggg
8

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.ficient LLM Inference
Table Comparison of MAC units with different number formats (0,5GHz, 45am process). Ours-INT4 refers to the implementation that leverages the widely adopted INT4 MAC, with supportive logic. Area and power are in um" and ,W, respectively.
Table 7. Resource overhead comparison between format selec- tion methods: two-stage and based (mean square error).
Syn. frequency
Latency (elk cycle)
Power (mW))
Area (pm²²)
Method
(MH))
2-stage MSF
100
5
0.7
42833.6
Multiplier
Accumulator
Total
83.3
8
6.9
399409.3
Type
Area
Power
Area
Power
Area
Power
INT4
62.51
20.59
138.59
80.16
207
104.18
INTS
101.88
34.35
171.04
106.80
275.04
142.18
INT8
301.91
162.93
244.72
162.79
554.34
331.17
FP4
71,55
30.04
171.04
96.92
246.85
129.44
FP6
158.54
73.88
223. 17
139.80
381.71
213.68
method. Qualitatively, MSE-based method requires 16 rounds (per dialect) of quantization, each involving FP16 149.22
Ours
63.31
16.02
184.87
118.91
248.18
134.92
Ours-INT4
120.76
41.58
168.11
103.09
299.52
square mean error accumulations for every block element, whereas our 2-stage selection efficiently operates in single pass using 5-bit fixed-point values, logical operations, and simple counting.
Table 6. Overhead of on-the-fly quantization and dequantization (100MHz, 1 130mm process).
Quantitatively, we design in Verilog and synthesize quantization logic implementations with 130nm process. For fair comparison, we aim to match the latencies as closely as possible while evaluating area and power. Ad- ditionally, since FP16 operations in the exact MSF-based approach incur significant overhead, we convert into fixed- point representation and truncate the lower bits to reduce the complexity. Nevertheless, as shown in table 7, MSE- based logic is 9.32x larger and consumes 9.86x more power. Notably, MSE-based logic fails to meet timing at 100MHz (synthesized at 83.3MHz), whereas our logic meets timing constraints even at 250MHz, underscoring the efficiency of our approach and the impracticality of online MSE- based selection.
Latency (clk cycle)
Power (mW)
Area (um²)
Module
Quantization
5
0.7
42833.6
(incl. format selection)
Dequantization
1
0.2
6319.8
32 MACs (Ours)
1
2.2
41319.6
32 MACs (INT8)
6.1
85482.0
format (Rouhani et al., 2023b) and 2.45 x (2.23 x) more power (area) efficient than INT8 MACs.
Overhead of On-the-fly Quantization and Dequantiza- tion. Since the latency and energy benefits immproved computation efficiency through low-precision MACs and re- duced data movement via 4-bit quantization are clear (Yuan al., 2024; Argerich & Patiño-Martinne. 2024), assess- ing whether the overhead offsets these gains is crucial. To show this, synthesize and evaluate an implementation of BlockDialect modules for 32-element processing in Sys- tem tem Verilog. Since we share six values across dialects, we keep the number of cases managcable, enabling a compact combinational logic implementation. Even with a register file, the overhead remains minimal at 600B for 32-element parallel processing.
5. Conclusion
We introduce BlockDialect, a post-training quantization technique that assigns an optimal number format to fine- grained blocks. This approach allows the capture of nuanced data distributions often overlooked by existing methods. Complementing this, we develop DialectFP4, a set of FP4 variants, which ensure compatibility with an energy- and area-efficient integer MAC unit. To leverage this efficiently, we propose two-stage approach for online DialectFP4 activation quantization. Our 4-bit quantization results on the I. LLLMA3-8B (ILLaA2-7BB) model show only 5.45 % (2.69 %) accuracy gap compared to full precision for full- path quantization. By shifting the focus to how each block should be optimally represented in hardware-efficien man- ner, rather than solely scaling values, BlackDi sets foundation for energy-efficient LLM inference.
As shown in Table 6, quantization and dequantization logic takes only few clock eycles, which can be further over- lapped with pipelining. Their power and area are compara- ble to or lower than that of our 32 MAC units, indicating minimal overhead. The overhead of fny activation quantization can also be amortized the quantized activa- tion block is reused across large number of weight blocks. Compared to the resources required for INT8 MACs, the practicality of BlockDialect becomes more evident.
Acknowledgements
Resource Overhead of Real-time MSE Calculation. To evaluate the efficiency of our two-stage approach, we com- pare its resource overhead against that of an MSE-based
We thank Rangharajan Venkatesan (NVIDIA) for his valu- able and insightful feedback on our work.

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
Impact Statement
Dai, S., Venkatesan, R., Ren, M., Zimmer, B., Dally, W., and Khailany, B. VS-Quant: Per-Ved Vector Scaled Quantization for Accurate Low-Precision Neural Network Inference. Proceedings of Machine Learning and Systems, 3:873-
This paper introduces advancements quantization tech- niques for large language models, focusing on improving memory usage and computational efficiency which, as a so- cietal consequence, could help reduce the significant energy consumption associated with running LLMs, potentially making AI systems more environmentally sustainable and accessible.
884, 2021.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, GPT33int8(): 8-bit Matrix Multiplication for Transform- ers at Scale. Advances in Neural Information Processing Systems, 35:30318 042303183300 18-0033,, 2022.
References
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, TT, and Alistarh, D. SpQR: A Sparse-Quantizer Representa- tion for Losslessseess LLM Weight Compression. arXiv preprint arXiv:2306.03077. 2023.
Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Fara- jtabar, M. LLM in flash: Efficient Large Language Model Inference with Limited Memory, arXiv preprint arXiv:2312.11544 2023.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36, 2024.
Argerich, M. F. and Patiñoo Martínez, M. Measuring and Im- proving the Energy Efficiency of Large I Language Models Inference. IEEE Access, 2024.
Ashkboos, S., Markov, I,, Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D. Towards End-to-end 4-Bit Inference on Generative Large Language Models. arXiv preprint arXiv:2310.09255) 2023.
Dotzel, J., Chen, Y., Kotb, B., Prasad, S., Wu, G., Li, S., Abdelfattah, M. S., and Zhang, Z. Learning from Students: Applying t-Distributions to Explore Accu- rate and Efficient Formats for LLMs. arXiv preprint 24Xhv:2405.03/00 2024.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arXiv preprint 2Xiv:2404.00450 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama Herd of Models. arXiv preprint arXhv:2407.2178. 2024.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA: Reason- ing about Physical Commonsense in Natural Language. In Proceedings of the AAAI conference on artificial intel- ligence, volume 34, pp. 7432-7439, 2020.
Egiazarian, V., Panferov, A., Kuznedelex, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme Compression of Large Language Models via Additive Quantization. arXiv preprint arXiv:2401.061 18, 2024.
Cao, Y., Wen, M. Luo, Z., Ju, X., Huang. H., Shen, J., and Chen, H. ABS: Accumulation Bit-Width Scaling Method for Designing Low-Preciste Tensor Core. IEEE Trans- actions on Very large Scale Integration (VLSI) Systems, 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Tre-Trained Transformers. arXiv preprint arXiv:2210.173235 2022.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. QulP: 2-bit Quantization of Large Language Models with Guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, I.., Hsu, J., Le Noac"h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B.. Wang. K., and Zou, A. framework for few-shot language model evaluation,
Clark, C., Lec, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv preprint arXiv: 1905. 10044, 2019.
12 2023. URL https: / zenodo. org/ reggre/cdd///// 10256836.
Gholami, A., Kim, S., Dong, 7., Yao, Z., Mahoney, M. W., and Keutzer, K. A Survey of Quantization Methods for Efficient Neural Network Inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Chal- lenge. arXiv preprint arxiv: /00::880300000 1188000.5557 2018.
10

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.ficient LLM Inference
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., and Keutzer, K. Al and Memory Wall. IEEE Micro, 2024. Language Models for On-Device Use Cases. In Forty-first International Conference on Machine Learning, 2024c.
Liu, Z.., Zhao, C., landola, E,, Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., et al. MobileLLM: Optimizing Sub-billion Parameter
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring Massive Multitask Language Understanding. arXiv preprint arXiv:2009.03. 03:000 2020.
Lo. Y.-C,, Wei, G.-Y., and Brooks, D. Nanoscaling Floating-Powe PPint (NxFP): NanoMantissa, Adaptive Mi- croexponents, and Code Recycling for Direct-Cast Com- pression of Large Language Models. arXiv preprint arXiv:2412.19821, 2024.
Hooper, C.. Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. arXiv preprint arXiv:2401.18079 2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. arXiv preprint arXiv: net:::600078800 2016.
Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. Phi-2: The Surprising Power of Small Lan- guage Models. Microsoft Research Blog, 133:3, 2023.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.N. Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernánde7, R. The LAMBADA dataset: Word predic- tion requiring a broad discourse context. arXiv preprint arXiv: S.Gi:/600 606606031, 2016.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G.,
Lample, G., Saulnier, L., et al. Mistral 7B, arXiv preprint arXiv:2310.06822 2023.
Rouhani, B. D., Zhao, R., Flango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M.. More, A., Melnick, L.. Golub, M., Varatkar, G., et al. With Shared Microexpo- nents, A Little Shifting Goes Long Way. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1-13, 2023a.
Kim, S.. Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. SqueezeLLM: Sanse-and-Sparss Quantization. arXiv preprint ar ar.iv:2306.006222233 07629, 2023.
Kuzmin, A., Van Baalen, M., Ren, Y., Nagel, M., Peters, J., and Blankevoort, FP8 Quantization: The Power of the Exponent. Advances in Neural Information Processing Systems, 35:14651-14661 2022.
Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling Data Formats for Deep Learning. arXiv preprint arXiv:2310.10557 2023b.
Li, Q.. Zhang, Y., Li, I.., Yao, P., Zhang, B., Chu, X., Sun, Y., Du, L., and Xie, Y. FPTQ: Quantization for Large Language Models. arXiv preprint ar0iv:2308. 1 15987, 2023.
Post-Training
Sakaguchi, K., Bras, L., Bhagavatula, C., and Choi, Y. WinoGrande: An Adversarial Winograd Schema Chal- lenge Scale. Communications of the ACM, 64(9):99- 106, 2021.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation aataee Weight Quantization for On-Device LLM Compression and Acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Sheng, Y., Zheng, L.,, Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. FlexGen: High-Thhoughhy
Generative Inference of Large Lan- guage Models with a Single GPU. In International Con- ference on Machine Learning pp. 31094-31116 PMLR, 2023.
Liu, S.-y., Liu, Z., Huang, X., Dong, P., and Cheng, K.-T. LLM-FP4: 4-bit Ploating-Point Quantized Transformers. arXiv preprint 23X
2023.
Shyam, V., Pilault, J., Shepperd, E., Anthony, Q., and Mil- lidge, B. Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters. arXiv preprint 24000044493 2024.
Liu, Z. Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: A Tuning-Fre Frree Asym- metric 2bit Quantization for KV Cache. arXiv preprint Stti:220020025500 2024a.
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert- Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S.,, et al. Release Strategies and the Social Impacts of Language Models. arXiv preprint arXiv: 1908. 09203, 2019.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. SpinQuant: LLM Quantization with Learned Rotations. arXiv preprint arXiv:2405. /6406, 2024b.

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Effiiient LLM Inference
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P. Bhosale, S., et al. Llama 2: Open Foundation and Fine- Tuned Chat Models. arXiv preprint arXiv:2307.09288 2023.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. QulP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. arXiv preprint arXiv:2402.04396, 2024.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A Multi-tast Benchmark and Analysis Platform for Natural Language Understanding. arXiv preprint arXiv: (XX:::800.07466 2018.
Wu, X., Yao, Z., and He, Y. ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quanti- zation Using Floating-Powe Formats. arXiv preprint arXiv:2307.09782, 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, and Han, S. SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models. In In- ternational Conference on Machine Learning, pp. 38087- 38099. PMLR, 2023.
Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou, Z., Xue, C.,, Wu, B., Li, Z., Gu, Q. Lee, Y. J., al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv preprint arXiv:2402. /6363, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can Machine Really Finish Your Sentence? arXiv preprint arXiv: 1905.00905 2019.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01066 2022.
Zhang, Y., Zhao, L., Cao, S., Zhang, S., Wang, W., Cao,
T., Yang, F., Yang. M., Zhang, S., and Xu, N. Integer Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEFE, 2024.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L.,, Zheng, S., Ceze, I.., Krishnamurthy, A., Chen, T., and Kasikci, B. ATOM: Low-Bit Quantization for Efficient and Accurate LLM Serving. Proceedings of Machine Learning and Systems, 6:196-209, 2024.
12

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiient LLM Inference
A. Block-Level Profiling Results
proj (weight) gate_proj (weight) altn input (activation) = mip_inng input (activation)
40
Ralio of data points (%)
Ratio points
20
10
0
[0.0.5)
[0.5.1)
[1.15)
[15.2)
12.2.55
12.5.3]
(3,3.5]
(3.5.4)
(4,4.5)
(4.5.5)
[5,5 [5,5.5)
[5.5.6]
(6.65))
[6 (67.7)
[7.7.5)
[7.5.8)
[0.0.5]
[0.5,1]
[1.155
[1.5,2)
[2.25]
[2.5.3]
(3.3.5)
(3.5,4)
(4.4.5)
[455.5
[5.5.5
5.566
[6 68.6.5)
[8.5,7]
[7.7.5)
[7.58)
Data ranges
Data ranges
(B)
9 pro] (weight) gate__rooj (welght) m attn
aatn_input Input (activation) I mipp_inpp input (activation)
(%)
Ratio data points (%)
50
20
40
15
20
5
(0.0.5)
[0.5.1]
(1.1.5)
[1.5.2)
12.2.5)
[2.5.3)
(3.3.5)
(3.5.4)
(4.4.5)
(45.5)
(5.5.5)
[5.5.6]
[6,8.5)
(.777)
(7.7.5)
[7.5.8]
(0.0.5)
(0.5.1)
(1.1.5)
(1.5.2)
[2.2.5)
12.5.3)
(3.3.5)
(3.5.4)
(4.4.5)
(4.5.5)
15.5.5)
[5.5.6)
[8.6.5)
(6.5.7)
(7.7.5)
(7.5.8)
Data ranges
Data ranges
(c)
(d)
prajj (weight) (weight) matin_ings input (activation) fc1. fct_input (activation)
points (%)
Rallo points (%)
20
20
10
5
(0.0.5)
005:1)
(11.5))
(1.5.2)
12.2.5)
(2.5.3)
(33355
(3.5A)
(4.4.5)
(4.5.5)
(5.5.5)
[5.5,8]
(6.6.5)
(,,57)
(7.7.5)
(7.5,8)
(0.0.5)
(0.5.1)
(1.1.5)
(1.5,2)
(2.2.5)
(2.53)
13 (33.5)
83 4,444
(4.4.5)
(4.555)
[6 55
(5.5.8)
(8. 6.8.5)
(.577)
(7.7.5)
(7.5.8)
Data ranges
ranges
(e)
(f)
Figure 7. LLaMA2-7B (a,b), Mistral-7B (c.d), and OPT-6.7B (c.f) Block-level profiling results: (a), (c), (c) matrix-wise accumulated
magnitude distribution, (b), (d), (f) block's 's maximum magnitude distribution.
Figure presents the block-level profiling results for the LLaMA2-7B, Mistral-7B, and OPT-6.7B models. Each matrix is divided into blocks of size 32, with each block scaled by the shared exponent, llog, (block's 's maximum magnitude). Magnitude distribution histograms are then accumulated for each block. Figure 7 shows the average results for layers 0, 10. 20, and 30, showing similar trend to LLaMA3 as discussed in Section 3.1: the matrix-wise accumulated distribution aligns with FP4 E2M1's 's representable value distribution, while each block's maximum magnitude is relatively evenly distributed across the possible range.
B. Dialect Selection Ratio
Figure illustrates the selection ratio of each dialect across four models. While the Mistral model shows slightly higher concentration in selecting specific dialects, all dialects are effeetively utilized, with no dialect being overwhelmingly dominant insignificant. Interestingly, the weights of the Mistral model more likely select dialects with larger values eovennnumber dialects) compared to other models. Unlike weights, activations of all models tend to favor dialects skewed towards smaller values (odd-numbered dialects). It is worth mentioning that low selection ratio does not necessarily imply low importance for the corresponding dialect. For instance, even a dialect has low selection ratio, it can still be valuable if it effectively represents the large magnitudes certain blocks with high shared exponents, indicating high original magnitudes.
13

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiiint LLM Inference
Selection (%)
(weight) proj (weight) input (actvation) input (activation)
Selection rallo (%)
a__praj (waight] saato_rrjj_ (weight) input (activation) mip_innn input (activation)
15
10
10
0
2
3
4
6
7
8
10
11
12
13
14
15
0
1 2 3
4
5
7
8
9
10
11
12
13
14
15
Dialect index
Dialect Index
(b)
Selection (%)
@q_praj p.pra (weight) gate proj (weight) inpurt (activation) mip, Input (activabio)
Selection rallo (%)
#4_proj (weight) (teigght) input (activation)
imput (activation)
20
10
0
3
4
7
10 11
13
14
15
0
1
4
7
8
10
11
12
13
14
15
Dialect
Dialect index (d)
(c)
Figure Selection ratio each dialect (a) ILLMMA-88,, (b) LLaMA2-7B, (c) Mistral-7B, and (d) OPT-6.7B. Experiments were conducted Wikitext2 with block size of 32. Each bar represents the average across layers 0, 10, 20, and 30.
C. Quantization Dimension
KV cache structure
KV cache structure
New value
Sequence length
New value
Sequence length
Dim
4-bit
Need to requantize
Dim
4-bit
FP16
Block
Quantize append in block size chunk
Block size
(a)
(b)
Figure Proposed KV cache structure: (a) challenge of sub-channel-wisg value quantization, (b) proposed cache structure.
BlockDialect quantizes matrices and vectors along their respective multiplication dimensions. For example, in activation- weight multiplication, activations are quantized at the sub-token level, while weights are quantized at the sub-channel level. Existing KV cache quantization approaches, such as per-token quantization (Sheng et al., 2023), per-channel key quantization with non-uniform representation (Hooper et al., 2024) or group-wise quantization (Ashkboos et al., 2024), primarily focus on compressing and reducing I/O costs during the decode phase. These approaches often dequantize data to FP16 before performing multiplications, which limits computational efficiency. In contrast, BlockDialect addresses the full computational path, achieving both memory savings and hardware-efficien computation without FP16 multiplications. Note that BlockDialect's full-path low-precision matrix multiplication is significantly more efficient during the prefill phase.
To achieve this, BlockDialect employs sub-token-wet wise quantization for keys and subb-cannel-wiss quantization for values, aligning with the respective multiplication dimensions. However, this design introduces a challenge: repeated quantization overhead and numerical inconsistencies when updating the KV cache. Specifically, while sub-token--wie key quantization straightforward, as new key vectors can quantized before multiplication and appended to the KV cache, sub-channel-wiee value quantization more complex. When new value vector is added, the values of each sub-channel must be requantized (Figure 9a). However, BlockDialect discards the FP16 value and directly dequantizes to 5-bit integers with shared exponent to leverage integer arithmetic operations. As result, requantizing from the 5-bit integer quantized form risks quantization errors when integrating new value vectors.
To address this, we can leverage fine-grained block structure with a default size of 32. Values are stored in 4-bit chunks of blockssize token count, with only the most recent chunk (size of N mod block_size) maintained high precision. Instead of requantizing all blocks of the most recent chunk for every new value vector, once this chunk's token count reaches the
14

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiient LLM Inference
block size, then the chunk is stored in 4-bit form (Figure 9b). This strategy avoids excessive quantization and ensures accurate updates while keeping the small number of high precision tokens (low portion relative the total sequence length), resulting in minimal additional storage cost. Similar to Section 4.2, smaller blocks can be used for K.V cache quantization, further reducing the overhead and enhancing model performance.
D. Comparison with NxFP
Concurrent with our work, NxFP (Lo et al.. 2024) introduces the Nanoscaling format (NxFP), which improves the limitations of the Microscaling format (MX) (Rouhani et al., 2023b). While NxFP applies to weight-only quantization, requiring high-precision operations, BlockDialect utilizes energy mfery-efficcentt low-precision integer operations for both weight and activation quantization. However, NxFP and our work originate from the similar observations on the limitations of the MX format and both employ mixed formats, we clearly distinguish our solutions from NxFP's in this section.
NxFP identifies three limitations of the MX format: 1) inaccurate tracking of the largest value, 2) vacant quantization levels inherent to floating point representation, and 3) redundant O-representations. To address the first issue, NxFP appends a 2-bit mantissa the shared exponent to improve accuracy of the quantized largest value. The second issue is addressed by selecting the better option between BFP and MxFP. However, these approaches reduce the hardware efficiency of the MX format in scaling factor handling. Also, NxFP proposes techniques similar other mixed-formal quantization strategies, such as choosing between two distinct formats, like integer and floating-poit representations (akin to BFP vs. MX after dividing by the shared scaling factor) and adjusting exponent biases to indirectly better represent the distribution (akin to additional 2-bit mantissa for shared exponent). While these strategies demonstrate effectiveness, they do not fully account for the fine-grained block-level data distributions that we have observed in our profiling.
In contrast, BlockDialect takes a more tailored approach by selecting multiple (e.g., 16) dialects capable of expressing diverse block-level distributions, while still maintaining hardware-efficient power-of-two shared exponent. Moreover, BlockDialect dynamically selects dialects that align with block-specific data characteristies, offering a more direct and precise approach compared to adjusting exponent biases. Additionally, unlike NxFP, which only proposes MSE-based format selection, we present a practical two-stage approach to enable online activation quantization.
Finally, NxFP and prior works (Dotzel et al., 2024) have proposed remapping redundant O-representations to other quantization values to improve utilization. While this is also applicable to our work, BlockDialect emphasizes hardware simplicity and minimal dequantization overhead.
'Similar architecture to the residual cache in http://wggiigeeecccohhhohoraaaan which targets to reduce repeated quantization and preserve accuracy for recent keys and values.
15

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
E. Experimental Results on OPT-6.7B Model
Table 8, Results on OPT-6.7B model. Perplexity on Wikitex12 and zero-shot accuracy across seven common sense reasoning tasks:
LAMBADA (LA), WinoGrande (WG), BoolQ (BQ), PIQA (PQ), ARC-easy (A-e), ARC-challenge (A-c). and weg (HS).
Scope
Method
Block size
Feature
Eff. bit
Wiki
LA
WGt
BQ
PQT
A-ct
A-ct
HSt
AVG.
FP16
Full precision
16
10.86
67.63
65.43
66.12
76.55
60.14
34.81
67.19
62.55
16
4.31
19.17
49.78
52.41
50.06
69.31
47.39
28.92
56.64
50.64
MXFP4
HW-supported scaling
32
4.16
19.22
49.89
54.93
49.54
69.10
46.89
29.44
56.04
50.83
Linear
16
4.56
11.26
66.89
64.48
62.57
76.55
58.71
34.90
65.20
61.33
BlockDialect
32
ID block
4.28
11.31
65.73
64.40
62.11
75.68
57.58
32.17
64.69
60.34
(w/ DialectFP4)
64
4.14
11.73
63.21
61.33
63.43
74.81
58.12
33.02
63.16
59.58
16
4.31
22.94
44.93
53.67
48.65
68.34
45.45
28.33
53.99
49.05
MXFP4
HW-supported scaling
32
4.16
22.12
44.11
50.43
45.54
63.49
44.15
29.35
53.81
47.27
All
16
4.56
11.45
64.87
63.54
60.70
75.19
58.71
33.11
64.71
60.12
BlockDialect (w/ DialectFP4)
32
ID block
4.28
11.63
64.62
62.19
59.66
75,41
58.29
32.85
63.82
59.55
64
4.14
12.14
59.42
60.85
60.43
74.10
57.28
32.34
62.86
58.18
We further evaluate BlockDialect on an LLM with a different architecture, OPT-6.7B (Zhang et al., 2022) in Table 8.
Note that the effective bitwidth of BlockDialect-32 (64) is lower than that of MXFP4-16 (32). BlockDialect-32 (64)
achieves significant gains over MXFP4-16 (32), showing 7.86 (7.49) and 11.31 (9.98) lower perplexity points, along with 9.70% (8.75%) and 10.50% (10.91%) zero-shot accuracy improvements in linear and all scopes, respectively. Also, BlockDialect-16 is only 1.22% and 2.43% behind full precision in linear and all scopes, respectively.
F. Experimental Results across Architectures, Model Sizes, and Workloads
So far, our evaluation has focused on models with approximately 7-8 billion parameters. To evaluate the general applicability of BlockDialect, we compare performance across a range of architectures, model sizes, and workloads. Specifically, we test LLaMA3-IB, Phi-2.7B 7B (Javaheripi et al., 2023), MobileLLM- - (Liu al., 2024c), and GPT2-1.5B (Solaiman al., 2019). Perplexity (PI.) and accuracy of common reasoning (CR) tasks follow the same evaluation setup in previous experiments. For GLUE (Wang al., 2018) (GL), we report the average accuracy over six tasks: MRPC, SST-2, RTE, QQP, MNLI, and QNLI For MMLU (Hendrycks et al., 2020) (ML), we use representative accuracy scores from the Im-eval- harness framework. We evaluate both 16- and 32-block configurations for a comprehensive comparison. Additionally, we include NVFP4 as a baseline- a block scaling format introduced by NVIDIA, which uses FP8 E4M3 floating-point scaling factors and FP4 E2MI data elements.
Overall, BlockDialect outperforms both data types, demonstrating its versatility, with NVFP4 falling between MXFP4 and
Table 9, Performance comparison of BlockDialect (BDFP4), NVFP4, and MXFP4 across various model architectures, sizes, and workloads. Full indicates full- full-path quantization; if unspecified, only linear layers are quantized. MMLU results for GPT2 and MobileLLM are omitted as they are too low to be compared. Bold highlights the best result among comparable effective bitwidths (NVFP4-32, MXFP4-16. BDFP4-32).
Bik
Eff.
LLaMA3-IB
Phi-2.7B
MobileLLM-1251
GPT2-1.5B
LLaMA3-IB IB (Full)
GPT2-1.5B (Full)
bit
PLL
CRt
ML
GL
PLJ
CR$
ML1
GL
PL4
CR
GL1
PLI
CR1
GL
PLJ
CRI
ML1
GL1
PLJ
CR
GLf
FP16
9.75
60,38
37,58 52.62
9.71
72.43
54.50 64.32
12,53
46.31
51.20
17.41
53.15
48.66
9.75
60.38
37.58 52.62
17.41
53.15 48.66
16
4.5
12.40
55.18
31.96 52.72
11.28
68.76
52.09 64.88
14.91
44.18
49.63
18.60
50.88
47.65
17.46
49.16
27.15 50.81
18.81 50.42 47.76
NVEP4
32
4.25
12.82
54.12
29.06 53.14
11.62
68.28 50.96 64.93
15.39
43.57
50.53
18.54
50.42
47.22
19.99
48.53
26.11 49.89
18.77
49,80
47.20
16
4.31
15.71
51.40
27.08 50.66
12.59
69.41 50.04 61.39
18.33
42.22
49.75
19.11
51.28
48.25
53.75
41.84
24.16 49.30
20.32
49.21 47.74
MXFP4
32
4.16
15.91
50.60
26.26 50.55
12.83
69.09 50.01 61.09
18.16 42.18
49.24
19.00
51.28 48.03
60.04
40.04 24.05 48.91
20.07
50.00 47.53
16
4.56
11.47
56.32
31.35 52.74
11.12
70.58
52.13 62.87
14.33
44.51
50.38
17.85
51.82
48.08
14.83
52.24
27.89 51.35
18.02
51.37 47.59
BDFP4
32
4.28
12.09
55.54
30.17 53.43
11.79 69.91 52.11 64.70
15.06
43.59
50.38
18.07
$1.92
47.32
17.42 49.46
25.87 51.15
18.36 51.11 46.79
16

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiient LLM Inference
BiaekDi Also, in the full-path results, the accuracy gap widens, solidifying BiockDialect's superiority. Note that unlike BDFP4 and MXFP4, which a power-of-two shared exponent, NVFP4's floating-point scale factor requires costly floating-point operations (e.g., normalization, scale factor multiplication), with overhead increasing as block size decreases.
G. Impact of Block Shape
Table 10. Impact of block shape: 2D block shapes of sizes 16, 32, and 64 have dimensions of (4.4), A(4,8) or W(8,4), and (8.8). respectively.
Block size
LLaMA3-8B
LLaMA2-7B
Mistral-7B Wikil
Scope
(shape)
Wiki, 0-shotf
Wikil
0-shott
0-shotf
16 (ID)
6.82
72.98
5.76
69.91
5.55
73.39
16 (2D)
6.88
73.16
5.82
69.43
5.58
73.31
32 (ID)
7.05
72.24
5.84
69.74
5.65
73.46
Linear
32 (2I)
7.09
71.97
5.92
69.09
5.65
73.15
64 (ID)
7.30
71.51
5.96
68.95
5.75
72.76
64 (2I)
7.34
71.19
6.06
69.21
5.78
72.05
16 (1D)
7.32
70.64
6.08
68.66
5.71
72.53
16 (2D)
7.47
70.95
6.22
69.09
5.84
72.89
32 (1D)
7.87
68.57
6.33
67.68
5.87
72.15
All
32 (2D)
7.89
68.58
6.51
67.55
5.95
72.39
64 (ID)
8.55
66.60
6.63
67.15
6.07
70.26
64 (2D)
8.50
67.57
6.87
67.47
6.13
70.75
So far, we have experimented exclusively with 1D linear-shaped blocks. However, 2D square-shaped blocks may prove advantageous, they can better capture channel-wise activation variance compared to bub-token-wise linear-shaped blocks. We compare perplexity and zero-shot common-sense reasoning task accuracy between linear and square-shaped blocks in Table 10. While the 2D block shows slightly better accuracy for all scope, there is no clear superiority between 1D and 2D blocks in terms of accuracy. However, 2D block quantization generally results in higher perplexity. We infer that, due to the significant channel-win wise variance of the key (Liu al., 2024a), 2D block quantization for the key in all scope results in marginally better accuracy than sub-token-wist ID block key quantization, while 2D block quantization for the linear layer has minimal impact with small block size.
It is important to note the nn-eval-harneen framework processes multiple tokens in parallel, akin to the prefill phase. As a result, the reported numbers may not fully capture the impact of block shape during the decode phase. In the decode phase, operations typically involve GEMV or flat GEMM, which require zero padding for the square shape quantization. This results in an increased effective bitwidth for square-shaped blocks, as the scaling factor is calculated over fewer -adddin elements compared to the full block size. At the same time, could be beneficial for accuracy, as the effective block size for scaling becomes smaller for padded blocks.
H. Combination with Other Approach
To explore potential synergy with other approaches, we combine Block Dialect with SmoothQuant (Xiao et al., 2023), which shifts the challenge of activation quantization to the weights. We experiment with various migration strengths (a), controlling the aggressiveness
this shift with a granularity of 0.05, and select the most effective one with the lowest perplexity. For block size of 64, applying SmoothQuant results in 0.09, 0.07, 0.09, and 0.48 points of perplexity improvement, along with 0.03%, 0.13% 0.26%, and 1.69% accuracy gain in the LLaMA3-8B LLaMA2-7B, Mistral-7B, and OPT-6.7B models, respectively. This demonstrates overall improvement from the combination, though the gains are limited in some models.
We hypothesize that, despite the distinct perspectives of BlockDialect and SmoothQuant, they are not entirely orthogonal. Specifically, methods that flatten the distribution (like SmoothQuant or other techniques using a rotation matrix) may influence the performance of our approach, which focuses on selecting the best dialect for each distinct fluctuating distribution. We believe an optimal balance exists between both approaches. For example, extreme-magnituee outliers could
17

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.ficient LLM Inference
Table 11. Synergistic Effects of combining SmoothQuant with different L.LMs.
Method
Wikij
0-short
LLaMA3-8B w/o SmoothQuant
7.30
71.51
[LaMA3-8B w/ SmnothQuant
7.21
71.54
LLaMA2-7B w/o SmoothQuanl
5.96
68.95
LLaMA2-7B w/ SmnothQuant
5.89
69.08
Mistral-7B w/o SmoothQuant
5,75
72.76
Mistral-7B w/ SmoothQuamt
5.66
73.02
OPT-6.7B 7B w/o SmoothQuant
11.73
59.58
OPT-6.7B w/ SmoothQuant
11.25
61.27
be handled by flattening them using SmoothQuant (or other methods), while moderate outliers could be addressed with BlockDialect. We leave this an area for future investigation.
I. Effective Bitwidth Calculation
Effective bitwidth is defined as the average bitwidth required per data element, incorporating overhead from scaling factors and dialect identifiers. Based on FP16 for full precision, 5-bit shared exponent is used per block for the MX format, contributing an overhead of 5/ //bblook_ssii
BlockDialect, an additional 4-bit overhead per block is required to encode the optimal dialect index (with 16 dialects in the formatbook by default), resulting in total overhead of 9 b/block_size.
For mixed block sizes, we individually calculate the effective bitwidth for weights and activations to offer clearer and more precise understanding. The weight calculation is straightforward, but activation quantization considers two possible approaches due to shared activations across multiple projections: 1) weighted summation for shared activations, and 2) counting shared activations only once. The first approach captures computational overhead more accurately, while the second suited for memory-centric analyses. Since activations are more relevant to computational context, we adopt the first approach. Additionally, for the effective bitwidth. For example, the attention score involves two dimensions of sequence length, whereas other operands only We base calculations sequence length of 2048. Finally, for per-token per-channel quantization with software supported high precision scaling factor, we omit overhead calculations as they are negligible.
multiplications in the attention mechanism, the sequence length affects
J. Full Results
The following tables present the complete experimental results for LLaMA3-8B, LLaMA-7B: 7B, and Mistral-7B models.
18

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.ficient LLM Inference
Table 12. Full results on LLaMA3-BB model. Perplexity on Wikitext2 and zero-shot accuracy across seven common-sense reasoning
tasks: LAMBADA (LA), WinoGrande (WG), BoolQ (BQ), PIQA (PQ), ARC-easy (A-e), ARC-challenge (A-c), and HellaSwag (HS). dn: down_proj output_proj. q-proj, k: k.proj, v: v-proj, Q: query, K: key, and V: value, 2D block shapes of sizes 16, 32, and 64 have dimensions of (4,4), A(4,8) or W(8,4) and (8,8), respectively. Quarot keeps query and attention scores FP16 and performs the associated operations in FP16.
Scope
Method
Block
Feature
Eff.
Wiki
LAt
WGf
BQ1
PQT
A-ct
HS*
AVG.1
FP16
Full precision
16
6.14
76.05
72.77
81.38
80.79
77.74
53.24
79.17
74.45
LLM-FFP4
Attensor, Wich.
Mixed format
4
48.71
22.45
52.88
60.31
58.22
39.31
22.10
38.14
41.92
Quarot (W4A4)
Attoken, Wieh.
Rotation matrix
67.65
67.09
72.84
75.73
70.45
41.89
72.78
16
4.31
8.20
69.18
69.77
72.91
76.93
71.21
45.99
73.72
68.53
MXFP4
32
supporied scaling
4.16
8.23
68.14
67.01
72.66
77.15
72.73
46.93
73.57
68.31
4.08
8.34
67.03
67.09
73.06
77.09
71.63
45.56
73.27
67.82
ID block
4.56
6.82
75.10
71.74
80.76
80.41
74.75
50.77
77.35
72.98
16
2D block
4.56
6.88
74.40
71.43
81.13
79.22
76.26
52.22
77.46
73.16
bluck
4.28
73.96
72.14
78.62
78.40
74.92
$0.94
76.69
72.24
32
2D block
4.28
7.09
73.80
70.96
80.15
79.33
74.83
48.12
76.57
71.97
Lincar
Exact MSE
4.28
7.01
74.09
71.03
79.57
79.92
76.60
51.71
77.02
72.85
ID hlock
4.14
72.54
70.80
77.89
78.35
75.17
75.90
71.51
BlockDialect (w/ DialectFP4)
2D block
4.14
7.34
73.92
70.17
77.40
78.78
73.95
48.63
76.07
71.19
SmoothQuant
4.14
7.21
73.04
70.24
78.17
78.89
75.13
49.57
75.74
71.54
W:4.25
du bluck size: 16
7.12
73.22
71.59
78.81
79.33
77.53
$1.45
76.91
72.69
64
A:4.30
W:4.17
block size:16
7.24
72.97
68.98
78.35
78.29
76.05
50.68
76.47
71.68
A:4.19
W:4.19
q.K.v block size: 16
7.19
73.37
70.48
77.19
79.11 11
75.97
49.23
76.24
71.66
A:4.27
Quarot (W4A4KV4)
Attoken, Wioh.
K, block size:128
W.K.V:4
67.15
67.17
71.41
75.08
67.55
40.78
72.96
66.01
16
4.31
18.84
53.00
62.51
65.14
71.55
60.73
35.58
59.01
58.22
MXFP4
HW-supported scaling
32
4.16
16.69
58.49
61.25
64.74
71.22
58.04
36.18
61.9%
58. 84
bluck
4.56
73.24
69.69
77.71
77.69
73.11
47.01
76.06
70.66
16
2D block
4.56
7.47
73.36
70.17
77.19
77.53
74.92
47.44
76.06
70.95
1D block
4.28
7.87
71.76
66.34
74.89
76.33
71.25
44.62
74.62
68.57
hlock
4.28
72.04
75.68
69.23
44.03
74.96
68.58
All
Exact MSE
4.28
7.72
73.03
67.17
76.27
75.68
71.30
45.99
74.91
69.19
32
BlockDialect (w/ DialectFP4)
8-dialect (dist.)
4.25
8.29
70.97
66.85
74.86
75.73
69.87
44.37
73.04
67.96
8-dialect (range)
4.25
70.10
66.22
75.47
75.14
70.37
44.88
74.23
24-dialect
4.31
8.84
70.75
66.46
74.10
75.19
68.39
44.88
73.21
67.57
1D block
4.14
8.55
68.02
63.54
74.13
74.92
69.32
44.54
71.70
66.60
2D block
4.14
8.50
70.52
66.38
74.50
75.41
68.98
43.77
73.42
67.57
64
W:4.25
d.KQ.K K. block size:16
7.77
71.08
66.38
77.37
75.24
71.63
47.18
74.12
69.00
A:4.21
19

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiient LLM Inference
Table 13. Full results on LLaMA2-7B model. Perplexity on Wikitext2 and zero-shot accuracy across seven common-sense reasoning
tasks: LAMBADA (LA), WinoGrande (WG), BoolQ (BQ), PIQA (PQ), ARC-easy (A-e), ARC-challenge (A-c), and HellaSwag (HS). dn: down_proj output_proj. q-proj, k: k.proj, v: v-proj, Q: query, K: key, and V: value, 2D block shapes of sizes 16, 32, and 64 have dimensions of (4,4), A(4,8) or W(8,4), and (8,8), respectively. Quarot keeps query and attention scores FP16 and performs the associated operations in FP16.
Scope
Method
Block
Feature
Eff.
Wikil
LAt
WGf
BQ1
PQT
A-ct
HS*
AVG.
FP16
Full precision
16
5.47
73.88
69.06
77.77
79.05
74.58
46.25
76.00
70.94
LLM-FFP4
Attensor, Wich.
Mixed format
4
15.61
57.97
61.80
66.45
69.48
57.07
32.76
61.55
58.15
Quarot (W4A4)
Attoken, Wieh.
Rotation matrix
71.01
66.06
75.11
77.80
43.00
73.09
16
4.31
7.07
69.84
68.11
72.14
77.31
68.35
41.38
70.86
66.86
MXFP4
32
supporied scaling
4.16
7.04
69.73
65.51
70.89
76.61
67.59
40.36
70.91
65.94
4.08
70.58
71.25
76.50
68.35
40.70
70.81
ID block
4.56
5.76
73.92
68.67
76.54
77.64
73.74
44.03
74.82
69.91
16
2D block
4.56
5.82
72.93
66.69
76.94
78.02
72.77
44.28
74.37
69.43
bluck
4.28
73:70
69.46
76.02
78.13
72.81
43.60
74.47
69.74
32
2D block
4.28
5.92
72.27
66.30
76.85
77.80
72.73
43.43
74.24
69.09
Linear
Exact MSE
4.28
5.83
73.24
68.27
76.88
78.13
72.39
44.28
74.44
69.66
ID hlock
4.14
72.83
67.32
76.64
72.31
42.41
73.81
68.95
BlockDialect (w/ DialectFP4)
2D block
4.14
6.06
72.58
67.56
77.61
77.58
71.76
43.94
73.46
69.21
SmoothQuant
4.14
5.89
72.27
67.17
75.87
77.48
72.64
43.60
74.54
69.08
W:4.23
du bluck size: 16
5.88
72.40
68.90
76.88
78.62
72.26
43.69
73.82
69.51
64
A:4.27
W:4.18
block size:16
73.82
68.51
76.27
77.64
72.69
44.37
73.88
69.60
A:4.19
W:4.25
q.K.v block size: 16
5.91
73.26
67.88 88
76.82
77.58
72.43
42.66
74.35
69.28
A:4,29
Quarot (W4A4KV4)
Attoken, Wioh.
K, block size:128
W.K.V:4
6.10
70.79
64.33
74.40
77.20
70.12
42.92
72.72
67.50
16
4.31
11.22
60.95
61.01
66.30
74.59
61.74
35.84
64.94
60.77
MXFP4
HW-supported scaling
32
4.16
11.14
60.06
60.06
65.44
73.01
59,39
35.58
64.77
59.76
bluck
4.56
72.23
64.72
76.61
76.82
71.38
44.45
74.44
68.66 66.66
16
2D block
4.56
6.22
72.06
67.40
76.27
77.48
72.52
43.34
74.55
69.09
1D block
4.28
6,33
70.66
64.48
74.68
75.35
70.92
44.03
73.66
67.68
hlock
4.28
6.51
70.75
67.40
73.58
76.50
69.32
41.55
73.77
67.55
All
Exact MSE
4.28
6.25
71.16
66.38
75.35
77.64
71.17
42.83
73.65
68.31
32
BlockDialect (w/ DialectFP4)
8-dialect (dist.)
4.25
6,51
69,30
62,98
73.52
76.66
69.74
42.32
72.73
66.75
8-dialect (range)
4.25
70.75
63.22
74.37
77.26
70.20
43.34
73.41
24-dialect
4.31
6.97
69.84
66.38
72.84
75.73
71.00
42.58
72.74
67.30
1D block
4.14
6.63
70.39
65.19
73.52
75.35
69.99
43.09
72.55
67.15
2D block
4.14
6.87
70.60
65.67
73.91
76.71
69.57
42.92
72.94
67.47
64
W:4.23
d.KQ.K K. block size:16
6.35
70.81
67.17
75.44
75.84
70.96
44.28
73.28
68.25
A:4.21
20

---

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-E.fiiient LLM Inference
Table 14. Full results on Mistral-7B-v0.3 model. Perplexity on Wikitext2 and zero-shot accuracy across seven common-sense reasoning
tasks: LAMBADA (LA), WinoGrande (WG), BoolQ (BQ), PIQA (PQ), ARC-easy (A-e), ARC-challenge (A-c), and HellaSwag (HS). dn: down_proj output_proj. q-proj, k: k.proj, v: v-proj, Q: query, K: key, and V: value, 2D block shapes of sizes 16, 32, and 64 have dimensions of (4,4), A(4,8) or W(8,4) and (8,8), respectively. Quarot keeps query and attention scores FP16 and performs the associated operations in FP16.
Scope
Method
Block
Feature
Eff.
Wikil
LAt
WGf
BQ1
PQT
A-ct
HS*
AVG.1
FP16
16
5.32
75.32
73.88
82.11
82.26
78.24
52.22
80.42
74.92
LLM-FFP4
Attensor, Wich.
Mixed format
4
17.47
56.92
56.27
69.24
69.64
58.42
36.26
62.55
58.47
Quarot (W4A4)
Attoken, Wieh.
Rotation matrix
72.75
70.24
78.87
80.58
77.44
49.57
77.99
72.49
16
4.31
6.49
71.43
68.43
75.38
79.33
74.24
47.27
76.24
70.33
MXFP4
32
supporidd scaling
4.16
6.42
71.80
70.80
75.47
79.71
74.54
46.59
76.16
70.72
4.08
70.89
67.96
75.11
79.27
74.37
46.67
75.95
70.03
ID block
4.56
5.55
73.80
70.56
80.43
81.45
77.36
50.60
79.54
73.39
16
2D block
4.56
5,58
72.68
70.64
80.86
81.39
77.36
50.77
79.49
73.31
bluck
4.28
73.45
71.11
81.13
77.31
50.17
79.20
Linear
32
2D block
4.28
5.65
73.41
70.64
80.00
80.90
77.65
50.43
79.00
73.15
Exact MSE
4.28
5.64
74.40
71.35
81.38
80.85
77.99
51.28
79.38
73.80
ID hlock
4.14
5.75
73.08
69.85
79.63
80.69
77.19
49.83
79.04
72.76
BlockDialect (w/ DialectFP4)
2D block
4.14
5.78
71.18
66.69
80.64
80.69
76.52
50.68
77.95
72.05
SmoothQuant
4.14
5.66
73.06
71.59
80.98
80.09
77.06
49.49
78.84
73.02
W:4.25
du bluck size: 16
5.68
73.20
70.17
80.28
81.34
77.74
51.02
79.37
73.30
64
A:4.30
W:4.17
block size:16
72.79
69.69
78.38
80.52
77.40
50.43
79.27
72.64
A:4.19
W:4.19
q.K.v block size: 16
5.68
73.36
72.14
79.51
80.69
77.95
50.34
79.11
73.30
A:4.27
Quarot (W4A4KV4)
Attoken, Wioh.
K, block size:128
W.K. V:4l
73.10
68.35
79.30
79.16
76.94
47.35
77.81
71.72
16
4.31
9.27
64.64
64.40
70.89
77.15
69.99
42.92
72.23
66.03
HW-supported scaling
MXFP4
32
4.16
8.98
63.90
65.90
71.41
76.39
69.57
43.09
71.81
66.01
bluck
4.56
5.71
72.50
69.69
80.36
76.56
49.91
79.07
16
2D block
4.56
5.84
72.81
69.38
80.46
80.90
76.85
51.11
78.74
72.89
1D block
4.28
5.87
71.69
69.85
80.28
80.36
75.93
48.55
78.37
72.15
hlock
4.28
5.95
71.86
69.30
79.54
81.28
77.57
48.98
78.23
72.39
Exact MSE
4.28
5.85
71.63
69.06
80.40
80.41
75.72
49.06
78.57
72.12
All
32
BlockDialect (w/ DialeetFP4)
8-dialect (dist.)
4.25
6.01
72.06
68.82
81.22
80.03
75.25
48. 12
77.64
71.88
8-dialect (range)
4.25
71.51
68.19
79.69
79.87
75.67
46.93
78.10
24-dialect
4.31
6.05
71.01
69.38
79.14
80.58
75.04
48. 88
77.82
71.69
1D block
4.14
6.07
70.00
67.48
76.73
79.22
74.45
46.84
77.08
70.26
2D block
4.14
69.49
66.30
78.84
79.71
75.29
48.63
77.02
70.75
64
W:4.25
d.KQ.K K. block size:16
5.90
71.20
69.93
78.44
79.43
75.55
49.49
77.96
71.71
A:4.21
21


Reviews:
[{"review_id": "XY4OI7qCoV", "reviewer": "Reviewer_Vqff", "summary": "The paper introduces a block-wise finegrained mixed format technique (called BlockDialect) that assigns an optimal number format to each block and FP4 variants data format (called DialectFP4) that is built on shared exponent among a group of numbers. They also propose the method of efficient online quantization/dequantization and DialectFP4 computation method. They demonstrate that the proposed approach outperforms existing methods across multiple LLMs while leveraging lowprecision, energy-efficient MAC unit.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reason for Subtracting 2 from the Shared Exponent**\n* When normalizing by the maximum exponent, normalized values fall within [0, 2), whereas FP4 variants span [0, 7.5]. Subtracting 2 from the shared exponent extends the range to [0, 8), enabling FP4 variants to represent normalized values without additional scaling. This also enables direct comparisons and highlights FP4 E2M1's limitations, such as its inability to represent values in the (6, 8) range within a power-of-two dynamic range.\n\n**How to determine the representable values of DialectFP4?**\n* The representable values in DialectFP4 are designed to capture diverse block distributions that a single FP4 E2M1 format cannot express. \n  * To account for varying dynamic ranges of fine-grained blocks, each dialect's largest value is selected from a range of 7.5 to 4.0, while second-largest values vary to adapt to large-magnitude distributions.\n  * In the second stage of format selection, we choose the best dialect based on how many block values fall within each dialect's beneficial range. We adjust the second-largest value, which only varies between dialects with the same maximum, to ensure the widths of the beneficial ranges of the compared dialects are similar. This approach helps avoid bias toward any particular dialect.\n  * The representable values follow a fixed granularity of 0.5, ensuring all values are scaled integers (integer x 0.5). This facilitates efficient integer MAC operations. Additionally, six values remain consistent across dialects to simplify quantization/dequantization logic.\n* While DialectFP4 demonstrates strong performance, we acknowledge that it is not optimally constructed. There is potential to adaptively generate DialectFP4 tailored to specific models or layers.\n* Unexpected numbers may result from the shared exponent applied after quantization. For example, with a shared exponent of -3, a quantized value of 3.0 reverts to 0.375 (3/8). If this doesn't apply, please clarify where your values appear.\n\n**Justification for the 5-bit intermediate representation**\n* The representable magnitudes of DialectFP4 range from 0.0 to 7.5 with a 0.5 granularity, requiring three integer bits and one fractional bit. However, to handle rounding accurately, an additional fractional bit is needed to determine whether to round up or down to the nearest representable value, resulting in a total of 5 bits. \n\n**We appreciate the feedback and will clarify the points explained above in the next revision.**\n\n---\n**Limited network and workload diversity**\n* We conduct experiments across networks of varying sizes and additional workloads, such as MMLU. For further details, please refer to our response to Reviewer 4 (ID: 8c5N), “Performance comparison of BlockDialect, NVFP4, and MXFP4 across model sizes and architectures” section.\n\n**Comparison with table-based approaches and I-LLM**\n* While table-based approaches with high-precision entries may perform well, they typically rely on high-precision MACs, which conflicts with our goal of enabling low-precision MACs for better hardware efficiency. Additionally, codebook-based methods are generally restricted to weight-only quantization due to the challenges of adaptive codebook construction and online activation codebook quantization. This is why we exclude this method in our weight-activation quantization comparison.\n* We appreciate the mention of relevant papers. I-LLM functions as an adaptive online variant of SmoothQuant, migrating quantization difficulty to easier-to-quantize matrices, albeit with additional tracking overhead. As discussed in Appendix G, we believe such approaches can complement BlockDialect by managing outliers at different levels. In this context, I-LLM serves as a valuable reference for improving accelerators using BlockDialect.\n\n**Effective bitwidth calculation**\n* Due to character limitations, we have integrated this response with our reply to Reviewer 3 (ID: rDoa) under the \"Metadata Storage Requirements\" section.\n\n**Hardware evaluation parameters**\n* As stated in the implementation section and the caption of Table 5, we use a 0.5 GHz clock speed and a 45 nm process node for our hardware evaluation.\n\n**Comparison between proposed MAC and INT5 MAC**\n* The discrepancy mainly arises from the sign-magnitude (Ours) vs. 2's complement (INT5) implementation. As recent works$^{[1,2]}$ have noted, sign-magnitude representation offers advantages over 2's complement MAC due to its much lower toggle rate and simpler sign processing.\n  - ###### [1] Wang et al. \"14.3 A 28nm 17.83-to-62.84 TFLOPS/W Broadcast-Alignment Floating-Point CIM Macro with Non-Two's-Complement MAC for CNNs and Transformers.\" ISSCC, 2025. [2] Han & Chandrakasan. \"MEGA. mini: A Universal Generative AI Processor with a New Big/Little Core Architecture for NPU.\" ISSCC, 2025.\n\n**Use of log2 function**\n* That’s a great point. We just used log2 as it seems intuitive, but will consider changing using an efficient bit manipulation.", "claims_and_evidence": "Some claims lack sufficient support. For instance:\n1. The author states that \"We introduce BlockDialect, a novel block-wise fine\u0002grained mixed format technique that assigns an optimal number format to each block\" However, the proposed 4-bit BlockDialect may not be the optimal format; alternative methods, such as codebook-based quantization, could theoretically offer better performance. \n2. Additionally, the author mentions in Section 3.1 that subtracting 2 from the shared exponent allows for a direct comparison with FP4 E2M1, but there is no explanation or compelling evidence provided to support this claim.", "methods_and_evaluation_criteria": "The proposed methods make sense for energy-efficient LLM inference. \nThe evaluation criteria also makes sense.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "I carefully reviewed the author's experimental design.\n\n### Model and Dataset Selection\nThe author utilized the LLaMA-2-7B, LLaMA-3-8B, and Mistral-7B models, evaluating them on the WikiText2 dataset while employing zero-shot commonsense reasoning tasks .\nStrengths: The selection of widely used LLM models and standard datasets ensures the generalizability and comparability of the experiments.\nPotential Issues: All chosen models are around 7 billion parameters in size, without consideration for networks of varying sizes. Besides, the MMLU is not used.\n\n### Dataformat Compare\nThe study compares BlockDialect with baseline methods such as MXFP4, LLM-FP4, and Quarot. A variety of quantization methods were selected as baselines, encompassing both hardware-supported and software-supported quantization techniques, ensuring a comprehensive comparison.\nPotential Issues: The comparison does not include codebook-based methods, such as Vector Quantization. I wonder whether table-based approaches could potentially achieve similar or even higher performance through search methods, although storing the table entries might result in slightly increased storage requirements.\n\n### Effective Bit Width Calculation\nThe experimental design calculated the effective bit width (Eff. bit), taking into account the overhead of scaling factors or dialect identifiers. By calculating the effective bit width, the design more accurately reflects the memory footprint and computational efficiency of the quantization methods.\nPotential Issues: The specific calculation method for effective bit width is not detailed, which may lead to a lack of transparency in the results.\n\n### Evaluation of Hardware Implementation\nThe author modeled MAC units at different precision levels using SystemVerilog and synthesized them with Synopsys Design Compiler to evaluate area and power consumption. The assessment of hardware implementation demonstrates the efficiency of BlockDialect in practical hardware contexts.\nPotential Issues: The specific parameters for the hardware evaluation (e.g., clock frequency, process node) are not clearly stated, which could affect the comparability of the results. Additionally, the reported findings indicate that the method proposed by the author shows significantly lower overhead for the multiplier compared to INT5, raising questions about the validity of the evaluation settings.", "relation_to_broader_scientific_literature": "### Block-wise Quantization\nBlock-wise quantization is a widely adopted technique that assigns scaling factors on a per-block basis to constrain the impact of outliers. \nThe author use block-wise quantization is the same as previous methods.\n\n### Non-Uniform Quantization\n\nNon-uniform quantization serves as an alternative to integer formats, aiming to better capture data distributions in large language models. Floating-point formats excel in handling the wide value ranges encountered in deep learning models, while lookup-based formats better align with the distributions of large language models through statistical distribution quantile functions.\n\nThis paper introduces DialectFP4, a set of FP4 variants (akin to \"dialects\") tailored for diverse block-level data distributions, and achieves online DialectFP4 activation quantization through a practical two-stage approach. Compared to existing non-uniform quantization methods, this approach offers greater flexibility and hardware efficiency.\n\n### Activation Quantization\n\nActivation quantization faces challenges such as real-time execution, large dynamic ranges, and inter-channel outliers. Existing methods include mixed-precision subgrouping, migrating quantization difficulty to weights, and using Hadamard matrices to reduce outliers.\n\nThis paper achieves efficient activation quantization by introducing FP4 variants and adopting a two-stage approach for online optimal format selection. Compared to existing activation quantization methods, this approach reduces reliance on high-precision operations, improving energy efficiency and inference speed.", "essential_references_not_discussed": "1. Hu X, Cheng Y, Yang D, et al. \"I-LLM: Efficient Integer-Only Inference for Fully Quantized Low-Bit Large Language Models.\" arXiv preprint arXiv:2405.17849, 2024. (This paper mainly focuses on how to quantize all activations and weights in LLMs to enable fully integer-based computations in hardware, which is a good exploit of software-hardware co-design.)\n2. Yuan Z, Shang Y, Zhou Y, et al. Llm inference unveiled: Survey and roofline model insights[J]. arXiv preprint arXiv:2402.16363, 2024. (This paper provides an overview of how different methods impact inference efficiency from the perspective of software-hardware co-design.)", "other_strengths_and_weaknesses": "### Strengths\n1. Unlike existing methods that primarily focus on \"how to scale,\" this paper introduces a new perspective of \"how to represent\" each block, removing the reliance on a single scaling strategy and thereby better capturing the data distribution within the block.\n2. The hardware cost is considered in the design of data format, which is a successful explore of software-hardware co-design for efficient LLM inference.\n3. Experimental results show that BlockDialect significantly outperforms the existing MXFP4 format across multiple LLM models, especially in full-path matrix multiplication quantization, with only a 5.45% (LLaMA3-8B) and 2.69% (LLaMA2-7B) accuracy drop compared to full precision, while reducing the bit usage per data.\n\n\n### Weaknesses\n1. One of the main drawback of this paper lies in the insufficient clarity of the methodology section. Without referring to the code, the entire methodology section is difficult to fully understand.\n2. There are some questions when I execute the code (See Question for authors).", "other_comments_or_suggestions": "The comparison does not include codebook-based methods, such as Vector Quantization. I wonder whether table-based approaches could potentially achieve similar or even higher performance through search methods, although storing the table entries might result in slightly increased storage requirements.", "questions_for_authors": "1. Subtracting two facilitates direct comparison with FP4 E2M1 ? Why? Can you provide any proof?\n2. Hardcoded DialectFP4—where do these numbers come from? Why were these specific values chosen, such as 4.5, 7.5, 5.5, and 5.0, which cannot be represented by FP4-E2M1? What is the purpose of selecting these? I find the author's description very difficult to understand. Moreover, when I ran the W4A4Linear using the author's code, I found numbers like 0.25 and 0.275, which FP4 cannot represent at all. Why is that?\n3. The author mentions in Figure 5 the operation man << exp, followed by truncation, introducing an intermediate 5-bit data type. Why is this 5-bit data type used instead of other intermediate data types? How was the choice of 5 bits made? Is there proof that this is a result of a trade-off between precision and bit count?\n4. When sharing the exponent, why use log2 to determine the exponent of the absolute maximum value within a block, when bit manipulation could be employed directly (since the data format is based on FP16)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eEZq4he5ou", "reviewer": "Reviewer_6poj", "summary": "They proposed BlockDialect, a block-wise finegrained mixed format technique that assigns a per-block optimal number format from a formatbook for better data representation. DialectFP4 ensures energy efficiency by selecting representable values as scaled integers compatible with low-precision integer arithmetic.\n1. They introduce DialectFP4, a formatbook of FP4 variants (akin to dialects) that adapt to diverse data distributions. Three core principles of formatbook: 1) minimizing wasted or underestimated ranges, 2) prioritizing the representation of larger magnitudes, and 3) ensuring hardware efficiency.\n2. They propose a two-stage approach for online DialectFP4 activation quantization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Resource overhead of real-time MSE calculation**\n* Qualitatively, MSE-based method requires 16 rounds (per dialect) of quantization, each involving FP16 square mean error accumulations for every block element, whereas our 2-stage selection efficiently operates in a single pass using 5-bit fixed-point values, logical operations, and simple counting. \n* Quantitatively, we design in SystemVerilog and synthesize quantization logic implementations with a 130nm process. For a fair comparison, we aim to match the latencies as closely as possible while evaluating area and power. Additionally, since FP16 operations in the exact MSE-based approach incur significant overhead, we convert into fixed-point representation and truncate the lower bits to reduce the complexity. Nevertheless, as shown in the table, MSE-based logic is 9.3× larger and consumes 10× more power. Notably, MSE-based logic fails to meet timing at 100MHz (synthesized at 83.3MHz), whereas our logic meets timing constraints even at 250MHz, underscoring the efficiency of our approach and the impracticality of online MSE-based selection.\n\n|Format selection method|Synthesized freq. ($MHz$)|Latency ($clock\\ cycle$)|Power ($mW$)|Area ($\\mu m^2$)|\n|:-:|:-:|:-:|:-:|:-:|\n|2-stage based|100|5|0.7|42833.6|\n|MSE-based|83.3|8|6.9|399409.3|\n\n**Scalability of BlockDialect**\n* Model size upper bound is due to our GPU resource constraints, not BlockDialect limitations. A key advantage of block-wise quantization is its independent block processing, which ensures efficiency and accuracy remain consistent regardless of model size. While there is a potential concern regarding the linear increase in per-block metadata as the model scales, our results in the paper shows that we can keep this overhead below 0.3 bits per data. Additionally, as discussed in the following response, dynamic block sizing (e.g., larger blocks for quantization-insensitive layers) can help mitigate this overhead.\n\n**Dynamic block size strategy**\n* That's a great question. In Table 3 (page 8), we analyze the impact of the dynamic block size strategy and find that the sensitivity of each sub-layer varies. Additionally, using larger blocks for most layers, while reserving smaller blocks for quantization-sensitive sub-layers, improves accuracy in certain cases. Based on this observation, we expect that selectively applying small block to quantization-sensitive layers (e.g., those with more outliers or greater influence on the final output) could enhance overall performance.\n\n**Combining with other strategies**\n* In Appendix G, we present the experiment results and discussion on combining BlockDialect with SmoothQuant. In summary, the combination leads to overall improvement, but with limited gains, indicating that the two methods are not entirely orthogonal. A more refined approach, such as applying smoothing exclusively to 'extreme' outliers before applying BlockDialect, may be beneficial. Additionally, we further explored combining BlockDialect with the Hadamard transformation-based rotation method and observed similar trends (limited gains) as in the BlockDialect-SmoothQuant hybrid quantization.\n\n**Impact on the inference latency**\n* BlockDialect offers clear latency benefits: (1) higher throughput with full-path low-precision MACs, unlike methods relying on FP16 MACs after dequantization or some matrix multiplications, and (2) reduced data movement latency from 4-bit weight/activation quantization (including KV cache), both crucial for inference latency. To ensure these gains are not offset by additional logic overhead, we assess this in our response to Reviewer 4 (ID: 8c5N) (see the “overhead of quantization/dequantization” section). While BlockDialect is deployable on GPUs, further optimizations, such as specialized MACs for our formatbook and efficient quantization/dequantization logic, will reduce latency. To achieve this, we are also taping out a hardware accelerator to incorporate these optimizations.\n\n**Workload sensitivity of BlockDialect**\n* We conducted additional experiments with different networks and downstream tasks (GLUE benchmark). Please see our response to Reviewer 4 (ID: 8c5N), section \"Performance comparison of BlockDialect, NVFP4, and MXFP4 across various model sizes and architectures\". BlockDialect uses a power-of-two shared exponent and smaller block sizes compared to conventional scale factor methods, preserving accuracy through optimal dialect assignment. This allows BlockDialect to effectively localize outliers, making it particularly beneficial for tasks with high magnitude variation and unstructured outliers.\n\n**Sink in Figure 2**\n* Thanks for your valuable feedback. We will reorganize the layout and clarify any ambiguities in the next revision. If the term \"sink\" refers to the range between 0 and 4 in the Figure 2(b), this occurs because we normalize the values using $2^{MaxExponent-2}$ , making the maximum magnitude become $2^2\\cdot mantissa\\ ([1,2))\\rightarrow [4,8)$.", "claims_and_evidence": "The claims made in the submission supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods make sense.", "theoretical_claims": "Probably correct.", "experimental_designs_or_analyses": "Experimental soundness.", "relation_to_broader_scientific_literature": "Efficiency, model compression, and quantization.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "The experimental design of this paper considers the latest methods (Quarot) and model series (LLaMA3-8B), which are comprehensive. The authors deserve credit for providing the code.", "other_comments_or_suggestions": "I suggest moving Figure 2 to page 5 for easy viewing. In addition, Figure 2 lacks an explanation for some belonging, such as \"sink\".", "questions_for_authors": "1. The proposed method eliminates real-time MSE calculation for activation quantization and claims that MSE overlooks the magnitude of data elements. The authors should provide a comparison of resource overhead.\n2. The experiments are conducted on models with up to 8B parameters. How does BlockDialect scale with even larger models, such as those with 70B+ parameters? Are there any additional challenges or optimizations needed to maintain its efficiency and accuracy in such scenarios?\n3. The paper explores various block sizes and their impact on performance. However, does the optimal block size vary across different layers? Could a dynamic block size strategy be more effective, where different layers use different block sizes based on their specific characteristics?\n4. Given the growing interest in hybrid quantization techniques, how does BlockDialect interact with other methods? Please discuss the possibility of further combining with other strategies like SmoothQuant.\n5. While the paper focuses on accuracy and energy efficiency, real-time inference latency is also crucial for practical deployment. How does BlockDialect impact the inference latency compared to full-precision models and other quantization methods? Are there any specific hardware accelerators that could further optimize the latency of BlockDialect?\n6. The evaluation focuses on common-sense reasoning tasks. How does BlockDialect affect performance on other downstream tasks, such as machine translation and text summarization? Are there any specific tasks where BlockDialect might show more pronounced benefits or limitations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l8V6aNeBfN", "reviewer": "Reviewer_rDoa", "summary": "This work presents BlockDialect, a block-wise mixed format quantization method for energy-efficient LLM inference. It assigns each block an optimal number format from a predefined formatbook to better capture data distributions. The proposed DialectFP4, a set of FP4 variants, enhances flexibility while maintaining hardware efficiency. A two-stage online format selection method enables efficient activation quantization without costly MSE calculations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Evaluating the impact of BlockDialect on inference latency and energy consumption**\n* That’s a valid point. Given the clear energy and latency benefits from low-precision MACs and reduced data movement due to 4-bit weight/activation quantization (including KV cache), we assess the overhead from BlockDialect's additional logic across various aspects to determine if these benefits can be offset. We address this in our response to Reviewer 4 (ID: 8c5N) (see the \"Overhead of on-the-fly quantization/dequantization\" section).\n\n**Broader applicability of BlockDialect to architectures other than LLaMA and Mistral**\n* In appendix E, we provide experiment results for the OPT-6.7B model. Also, you can refer to our response to Reviewer 4 (ID: 8c5N), section \"Performance comparison of BlockDialect, NVFP4, and MXFP4 across various model sizes and architectures,\" for further experiments including OPT and Phi models.\n\n**Metadata storage requirements**\n* As mentioned on page 6, line 306, \"Effective bitwidth\" indicates the per-block metadata overhead and we provide the detailed calculation method in Appendix H (page 16). Our results show that BlockDialect achieves better accuracy than the MX format, even with a lower effective bitwidth. Additionally, metadata overhead is mitigated by sharing it across the block, reducing it to less than 0.3 bits per data. Note that even with larger models, the per-block data and metadata ratio remains constant.\n\n**Novelty of BlockDialect compared to existing block-wise mixed-precision quantization approaches**\n* Thanks for providing relevant references. We have carefully reviewed them to clarify the novelty of BlockDialect. While block-wise mixed-precision (MP) quantization methods and our mixed-format (same precision) approach share similarities, they differ in key aspects:\n  * MP requires either multiple types of MAC units or a complex modularized MAC with different precision modes, whereas our approach operates efficiently with a single low-precision MAC unit, improving hardware efficiency.\n  * MP introduces irregular memory access patterns or requires reordering mechanisms to manage distributed precision blocks, whereas our method maintains regular memory access without added complexity.\n  * MP relies on complex online tracking, pre-training or calibration with sample datasets to determine precision allocation before quantization, which may limit scalability or adaptability, whereas our approach avoids these requirements entirely.\n  * Due to these factors, MP demands extensive modifications to computation paths and kernel code, while our method only requires integrating quantization/dequantization functions before or after matrix multiplication, making it more portable and easier to deploy.\n* Additionally, BlockDialect introduces several novel aspects compared to existing quantization techniques:\n  * **Sample dataset agnostic**: Many activation quantization methods rely on sample dataset calibration or pre-training to avoid the high overhead of online processing, sacrificing adaptability. In contrast, BlockDialect eliminates the need for calibration or training by employing an efficient two-stage format selection (the efficiency of the two-stage approach is analyzed in the response to Reviewer 2 (ID: 6poj), section \"Resource overhead of real-time MSE calculation\") and logical operation-based quantization, enabling practical online processing.\n  * **Handling unstructured outliers**: Unlike many activation quantization methods that rely on easily tracked or calibrated structured outliers (e.g., channel-wise magnitude mean), BlockDialect can also handle unstructured outliers through fine-grained block-wise outlier localization.\n  * **Awareness of fine-grained block distribution**: To the best of our knowledge, this is the first mixed-format approach that constructs a diverse set of format candidates based on profiling fine-grained block distributions, rather than selecting from a few predefined standard formats or modifying exponent bias.\n  * **Addressing MX format limitations**: We identify and mitigate the limitations of the MX format, a block-wise quantization format already adopted in commercial products.\n  * **Efficient INT MAC utilization**: By carefully selecting a 0.5 granularity for representable values, BlockDialect enables floating-point representation using scaled integers ($0.5\\cdot integer$), allowing direct computation with efficient low-precision integer MAC units. Note that we do not require floating-point operations even for the scaling factor, as we use a power-of-two scaling factor, which can be handled by addition and shifting logic while preserving accuracy.", "claims_and_evidence": "Most of the claims in the paper are supported by strong empirical evidence and theoretical analysis, particularly regarding accuracy improvements, hardware efficiency, and training stability. However, while the method is evaluated on LLaMA3-8B, LLaMA2-7B, and Mistral-7B, it remains unclear how well BlockDialect generalizes to other architectures such as GPT, OPT, or hybrid transformer-based models.\nAdditional experiments on more diverse model families would strengthen this claim. Also, Not entirely. The method claims to be energy-efficient, but the evaluation of energy consumption provided in Table 5 seems not support this claim.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are well-structured and mostly relevant to the quantization challenges in LLM inference.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, I have reviewed the soundness and validity of the experimental designs and analyses presented in the submission. Overall, the experimental setup is reasonable.\n\nHowever, there are still some points that can be improve:\n- While the paper claims that BlockDialect is energy-efficient, it does not provide concrete measurements of energy savings in actual LLM inference workloads. The inclusion of MAC unit power analysis is useful but does not fully capture real-world energy consumption across the entire inference pipeline. A more rigorous validation of this claim would involve reporting end-to-end energy usage when running inference on real hardware accelerators, such as GPUs or TPUs.  \n\n- The experiments focus primarily on LLaMA and Mistral models, leaving uncertainty about whether BlockDialect generalizes to other architectures, such as GPT, OPT, or hybrid transformer-based models. Given that different model families may exhibit varying sensitivities to quantization, additional evaluations on a broader range of LLM architectures would provide a more comprehensive validation of the approach.  \n\n- Since BlockDialect assigns per-block format identifiers, there is an inherent trade-off between format flexibility and additional metadata storage requirements. However, the paper does not quantify how much extra memory is needed to store these identifiers or discuss scalability concerns for very large models. Providing a detailed breakdown of the memory overhead and its impact on inference efficiency would enhance the clarity of this trade-off.  \n\n- While the hardware synthesis results demonstrate energy and area efficiency at the MAC unit level, there is no empirical validation of how BlockDialect affects overall inference latency when deployed on real hardware. Without these benchmarks, it remains unclear whether the proposed method actually accelerates practical LLM inference. Including end-to-end latency measurements would significantly strengthen the practical relevance of BlockDialect.", "relation_to_broader_scientific_literature": "This work builds on prior research in block-wise and mixed-precision quantization, extending methods like MXFP4 and LLM-FP4 by introducing fine-grained format selection with DialectFP4. It aligns with recent advancements in adaptive numerical precision and hardware-efficient LLM training, particularly in optimizing low-power MAC operations.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "One of the key strengths of the paper is its practical approach to energy-efficient quantization by introducing a block-wise mixed-format strategy that balances flexibility and efficiency. \n\nHowever, I still have questions about the novelty of the paper, as it builds upon existing mixed-precision and block-wise quantization techniques rather than introducing fundamentally new principles. While the format book selection and DialectFP4 variants add flexibility, the core idea of assigning optimal numerical formats based on block-wise distributions has been explored in various forms before such as [1-3].\n\n[1]Liu R, Wei C, Yang Y, et al. Block-wise dynamic-precision neural network training acceleration via online quantization sensitivity analytics[C]//Proceedings of the 28th Asia and South Pacific Design Automation Conference. 2023: 372-377.\n[2]Wu X, Hanson E, Wang N, et al. Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Accelerators[J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2024.\n[3]Dettmers T, Lewis M, Shleifer S, et al. 8-bit optimizers via block-wise quantization, 2022[J]. URL https://arxiv. org/abs/2110.02861, 2022.", "other_comments_or_suggestions": "See Other Strengths And Weaknesses", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zFQBXg6yy3", "reviewer": "Reviewer_8c5N", "summary": "This paper introduces BlockDialect, a block-wise fine-grained mixed format quantization technique designed to enhance the energy efficiency of large language model (LLM) inference. Unlike traditional quantization methods that focus on scaling values, BlockDialect assigns a number format to each block using a predefined formatbook, DialectFP4, which consists of multiple FP4 variants (dialects) tailored to different data distributions. The proposed two-stage selection process aims to determine the best dialect for each activation block in real-time, ensuring compatibility with low-precision integer arithmetic. Experimental results demonstrate that BlockDialect outperforms MXFP4 format, achieving up to a 10.78% accuracy gain on LLaMA3-8B while maintaining lower bit usage and only a 5.45% drop compared to full precision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Performance comparison of BlockDialect (BDFP4), NVFP4, and MXFP4 across various model sizes and architectures**\n* We compare accuracy and perplexity for both `Linear` layer quantization and `Full`-path (including activation-activation multiplication) quantization. Perplexity (PL) and common reasoning tasks (CR) evaluations follow the BlockDialect paper; for GLUE (GL), we average six tasks (MRPC, SST2, RTE, QQP, MNLI, QNLI), and for MMLU (ML), we use the representative accuracy from lm-eval-harness framework. We test both 16- and 32-block sizes for a comprehensive comparison.\n* Overall, BlockDialect outperforms both data types, demonstrating its versatility, with NVFP4 falling between MXFP4 and BlockDialect. Also, in the full-path results, the accuracy gap widens, solidifying BlockDialect’s superiority.\n* Note that unlike BDFP4 and MXFP4, which use a power-of-two shared exponent, NVFP4’s floating-point scale factor requires costly floating-point operations (e.g., normalization, scale factor multiplication), with overhead increasing as block size decreases. \n\n###### N/A: Too low to compare  \n###### **Bold**: Best result among comparable effective bitwidths (NVFP4-32, MXFP4-16, BDFP4-32)\n||`Linear`||LLaMA3-1B|OPT-6.7B|Phi-2.7B|MobileLLM-125M|`Full`LLaMA3-1B|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**Format**|**BlkSize**|**Eff.bit.**|PL↓\\|CR↑\\|ML↑\\|GL↑|PL↓\\|CR↑\\|ML↑\\|GL↑|PL↓\\|CR↑\\|ML↑\\|GL↑|PL↓\\|CR↑\\|ML↑\\|GL↑|PL↓\\|CR↑\\|ML↑\\|GL↑\n|FP16|-||9.8 \\|60.4\\|37.6\\|52.6|10.9\\|62.6\\|N/A\\|52.8|9.7\\|72.4\\|54.5\\|64.3|12.5\\|46.3\\|N/A\\|51.2|9.8\\|60.4\\|37.6\\|52.6\n|NVFP4|16|4.5|12.4\\|55.2\\|32.0\\|52.7|12.4\\|59.3\\|N/A\\|47.9|11.3\\|68.8\\|52.1\\|64.9|14.9\\|44.2\\|N/A\\|49.6|17.5\\|49.2\\|27.2\\|50.8\n||32|4.25|12.8\\|54.1\\|29.1\\|53.1|12.5\\|59.2\\|N/A\\|**51.6**|**11.6**\\|68.3\\|51.0\\|**64.9**|15.4\\|43.6\\|N/A\\|**50.5**|20.0\\|48.5\\|**26.1**\\|49.9\n|MXFP4|16|4.31|15.7\\|51.4\\|27.1\\|50.7|19.2\\|50.6\\|N/A\\|49.9|12.6\\|69.4\\|50.0\\|61.4|18.3\\|42.2\\|N/A\\|49.8|53.8\\|41.8\\|24.2\\|49.3\n||32|4.16|15.9\\|50.6\\|26.3\\|50.6|19.2\\|50.8\\|N/A\\|50.3|12.8\\|69.1\\|50.0\\|61.1|18.2\\|42.2\\|N/A\\|49.2|60.0\\|40.0\\|24.1\\|48.9\n|BDFP4|16|4.56|11.5\\|56.3\\|31.4\\|52.7|11.3\\|61.3\\|N/A\\|51.3|11.1\\|70.6\\|52.1\\|62.9|14.3\\|44.5\\|N/A\\|50.4|14.8\\|52.2\\|27.9\\|51.4\n||32|4.28|**12.1**\\|**55.5**\\|**30.2**\\|**53.4**|**11.3**\\|**60.3**\\|N/A\\|51.3|11.8\\|**69.9**\\|**52.1**\\|64.7|**15.1**\\|**43.6**\\|N/A\\|50.4|**17.4**\\|**49.5**\\|25.9\\|**51.2**\n\n**Overhead for the 2-stage format selection**\n* We evaluate the overhead of the 2-stage format selection to highlight its superiority over the conventional Mean Square Error method. Please see our response to Reviewer 2 (ID: 6poj) in “Resource overhead of real-time MSE calculation” section.\n\n**Compatibility with prevalent GEMM accelerator architectures**\n* Given that BlockDialect uses 4-bit unsigned integer MAC and a logical operation-based quantization method, it can be integrated into existing GEMM accelerators. Moreover, as block-wise operations become more common and commercial accelerators start supporting block-wise quantization formats, integrating BlockDialect will become increasingly seamless.\n\n**Overhead of on-the-fly quantization/dequantization**\n* We synthesize and evaluate an implementation of BlockDialect modules for 32-element processing in SystemVerilog, as an alternative to implementing the GPU kernel within the time constraints. Since we share six values across dialects, we keep the number of cases manageable, enabling a compact combinational logic implementation. Even with a register file, the overhead remains minimal at 600B for 32-element parallel processing. The reported numbers use the 130nm process node at 100MHz.\n\n* Since the latency and energy benefits - improved computation efficiency through low-precision MACs and reduced data movement via 4-bit quantization - are clear$^{[1,2]}$, assessing whether the overhead offsets these gains is crucial. As shown in the results, quantization and dequantization logic takes only a few clock cycles, which can be further overlapped with pipelining. Their power and area are comparable to or lower than that of our 32 MAC units, indicating minimal overhead. The overhead of on-the-fly activation quantization can also be amortized as the quantized activation block is reused across a large number of weight blocks. Compared to the resources required for INT8 MACs, the practicality of BlockDialect becomes more evident. To fully realize BlockDialect’s potential, we are taping out an optimized accelerator that will provide end-to-end measurements.\n  - ###### [1] Yuan et al., \"LLM Inference Unveiled: Survey and Roofline Model Insights,\" arXiv, 2024. [2] Argerich & Patiño-Martínez, \"Measuring and improving the energy efficiency of large language models inference,\" IEEE Access, 2024.\n\n\n\n|Module|Latency ($clock\\ cycle$)|Power ($mW$)|Area ($\\mu m^2$)|\n|-|:-:|:-:|:-:|\n|Quantization (including format selection)|5|0.7|42833.6|\n|Dequantization|1|0.2|6319.8|\n|32 MACs (Ours)|1|2.2|41319.6|\n|32 MACs (INT8)|1|6.1|85482.0|", "claims_and_evidence": "Please refer to **“Methods And Evaluation Criteria”**.", "methods_and_evaluation_criteria": "The accuracy of the proposed method BlockDialect (DialectFP4) has been verified across different models and evaluation tasks, such as perplexity (PPL) and zero-shot downstream tasks. The evaluation results demonstrate BlockDialect can preserve the model’s accuracy better than baselines such as MXFP4. \n\nHowever, one of the important baselines, NVFP4, is missing in the evaluation. As [NVFP4](https://docs.nvidia.com/deeplearning/cudnn/frontend/latest/operations/BlockScaling.html) supports floating-point micro-scaling factors, it can preserve the model’s accuracy better than MXFP4.\n\nFor the efficiency benchmarks, the paper mainly presents the area and energy consumption of MAC units, while the costs (latency, memory, and energy) of on-the-fly quantization & dequantization units are missing.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Please refer to **“Methods And Evaluation Criteria”**.", "relation_to_broader_scientific_literature": "The paper contributes to accelerators for quantized LLM inference, with a particular focus on improving FP4 data format’s accuracy.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Please refer to **“Methods And Evaluation Criteria”**.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1) How is the accuracy of BlockDialect compared to NVFP4 quantization?\n2) What is the overhead for the 2-stage block-wise quantization in BlockDialect?\n3) Is the per-block dialect design compatible with prevalent GEMM accelerator architectures such as tensor core?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes BlockDialect, a block-wise, fine-grained mixed-format techniques that assigns an optimal number format to each block from a predefined format book, aiming to improve the FP4 format for low-precision inference of large language model (LLMs). The method is validated on popular LLMs, demonstrating improved performance in both accuracy and hardware efficiency.

The paper is well written, and the algorithm design is clearly presented. Unlike traditional quantization methods that rely on per-block scaling or mixed precisions, this paper assigns a specific number format to each block to better present block-wise data distributions, which is a novel approach. The proposed efficient two-stage format selection process, as well as the insights into block-level data distribution, are valuable contributions to the research in low-precision quantization of LLMs. 

During rebuttal, the authors provide additional analysis on quantization/dequantization overhead and a comparison with latest NvFP4 format, further strengthening the paper. Therefore, this paper is recommended for acceptance.
Decision: Accept (poster)