Addition is All You Need for Energy-efficient Language Models

## Abstract

Abstract Large neural networks spend most computation on floating point tensor multiplications. In this work, we find that a floating point multiplier can be approximated by one integer adder with high precision.
We propose the linear-complexity multiplication ( ℒ ℒ \mathcal{L} caligraphic_L -Mul) algorithm that approximates floating point number multiplication with integer addition operations.
The new algorithm costs significantly less computation resource than 8-bit floating point multiplication but achieves higher precision.
Compared to 8-bit floating point multiplications, the proposed method achieves higher precision but consumes significantly less bit-level computation.
Since multiplying floating point numbers requires substantially higher energy compared to integer addition operations, applying the ℒ ℒ \mathcal{L} caligraphic_L -Mul operation in tensor processing hardware can potentially reduce 95% energy cost by element-wise floating point tensor multiplications and 80% energy cost of dot products.
We calculated the theoretical error expectation of ℒ ℒ \mathcal{L} caligraphic_L -Mul, and evaluated the algorithm on a wide range of textual, visual, and symbolic tasks, including natural language understanding, structural reasoning, mathematics, and commonsense question answering.
Our numerical analysis experiments agree with the theoretical error estimation, which indicates that ℒ ℒ \mathcal{L} caligraphic_L -Mul with 4-bit mantissa achieves comparable precision as float8_e4m3 multiplications, and ℒ ℒ \mathcal{L} caligraphic_L -Mul with 3-bit mantissa outperforms float8_e5m2 .
Evaluation results on popular benchmarks show that directly applying ℒ ℒ \mathcal{L} caligraphic_L -Mul to the attention mechanism is almost lossless.
We further show that replacing all floating point multiplications with 3-bit mantissa ℒ ℒ \mathcal{L} caligraphic_L -Mul in a transformer model achieves equivalent precision as using float8_e4m3 as accumulation precision in both fine-tuning and inference.

## 1 Introduction

Modern artificial intelligence (AI) systems are significant energy consumers.
Because of the large scale computation needed for neural network inference, AI applications based on such models are consuming a considerable amount of electricity resource.
Reportly, the average electricity consumption of ChatGPT service in early 2023 was 564 MWh per day, equivalent to the total daily electricity usage of 18,000 families in the United States(^1^11[https://www.eia.gov/tools/faqs/faq.php?id=97](https://www.eia.gov/tools/faqs/faq.php?id=97)). It is estimated that Google’s AI service could consume as much electricity as Ireland (29.3 TWh per year) in the worst-case scenario (de Vries, ).

Reducing the amount of computation needed by neural networks is the key to reduce both energy consumption and inference speed for large-scale AI models.
Neural networks, especially large language models (LLMs) (Radford et al., ; Brown, ; Achiam et al., ; Touvron et al., ; Team et al., ), contain a large number of floating point parameters involved in element-wise and matrix multiplication computations.
In transformer (Vaswani, ) based LLMs, the attention mechanism is a major bottleneck that limits the computation efficiency. Given a input context of $N$ tokens, the complexity of standard attention mechanism computation is $O(N^{2})$, involving multiplying high dimensional tensors.
Besides attention, there are also a large amount of element-wise multiplication and linear transformation computations.
In this work, we propose the linear-complexity multiplication ($\mathcal{L}$-Mul) algorithm, which approximates floating point multiplication with integer addition operations. The algorithm can be integrated into existing models at various levels, such as replacing the multiplication in the attention mechanism or substituting all matrix and element-wise multiplications.

The proposed $\mathcal{L}$-Mul method will lead to a significantly reduced energy consumption for both model training and inference.
In modern computing hardware, multiplications between floating point numbers consumes significantly higher energy than addition operations (Horowitz, ). Specifically, multiplying two 32-bit floating point numbers (fp32) costs four times energy as adding two fp32 numbers, and 37 times higher cost than adding two 32-bit integers (int32). The rough energy costs for various operations are shown in Table 1.
In PyTorch (Paszke et al., ), the default precision for accumulating tensor multiplication results is set to fp32. While I/O and control operations are not considered, approximating fp32 multiplications with int32 additions consumes only $1/37\approx 2.7\%$ of the energy. When the accumulation precision is reduced to fp16, integer addition consumes approximately $4.7\%$ of the energy required for floating-point multiplication.

**Table 1: Energy cost of various arithmetic operations cited from Horowitz ().**
| Operation | Integer | Floating Point |  |  |
| --- | --- | --- | --- | --- |
|  | 8-bit | 32-bit | 16-bit | 32-bit |
| Addition | 0.03 pJ | 0.1 pJ | 0.4 pJ | 0.9 pJ |
| Multiplication | 0.2 pJ | 3.1 pJ | 1.1 pJ | 3.7 pJ |

We evaluate the numerical precision of $\mathcal{L}$-Mul algorithm on transformer-based language models with a wide range of language and vision tasks.
Experiments with full-precision model weights show that replacing standard multiplication operations with $\mathcal{L}$-Mul in the attention mechanism is almost lossless for transformer-based LLMs.
On natural language reasoning tasks, the average performance loss of $\mathcal{L}$-Mul-based attention is $0.07\%$ across commonsense, structured reasoning, language understanding. On vision tasks, $\mathcal{L}$-Mul-based attention gained $0.12\%$ accuracy improvement on visual question answering, object hallucination, and free-form visual instruction tasks.
The experiment results are obtained by directly adapting pretrained LLMs with the standard attention implementation to the new $\mathcal{L}$-Mul-based attention mechanism without any additional training.

The error estimation and ablation study show that under the training-free setting, $\mathcal{L}$-Mul with 4-bit mantissa can achieve comparable precision as multiplying float8_e4m3 numbers, and $\mathcal{L}$-Mul with 3-bit mantissa outperforms float8_e5m2 multiplication.
We also show that fine-tuning can fix the performance gap between $\mathcal{L}$-Mul and the standard multiplication.
Fine-tuning a model where all multiplication operations in attention mechanisms, linear transformations, and element-wise products are replaced by 3-bit-mantissa $\mathcal{L}$-Mul results in comparable performance to fine-tuning a standard model with an accumulation precision of float8_e4m3.

In the expansive landscape of AI efficiency research, our approach centers on enhancing the efficiency of tensor arithmetic algorithms—a direction that is orthogonal yet complementary to prevailing efforts in I/O and control optimization (Jouppi et al., ; Choquette et al., ; Abts et al., )(^2^22 Due to the absence of native implementation, GPUs cannot fully exploit the efficiency of the $\mathcal{L}$-Mul algorithm. We recommend training and hosting $\mathcal{L}$-Mul-based models on devices integrated with specialized architectural designs. Patent pending.). We believe that truly energy- and compute-efficient AI computation will emerge from a holistic integration of optimizations across I/O, control, and arithmetic operations.

## 2 Method

### 2.1 Background: Floating-point Numbers and Tensors

Most machine learning models, including neural networks, use floating point (FP) tensors to represent their inputs, outputs, and trainable parameters. Typical choices are 32-bit and 16-bit FP tensors (fp32 and fp16) defined by the IEEE 754 standard shown in Figure 1.

Figure: Figure 1: 16-bit, 8-bit floating point numbers defined in IEEE 754 and on various hardware for tensor computations, and the 16-bit integer. MSB stands for most significant bit and LSB stands for least significant bit.
Refer to caption: x1.png

Multiplication operations are generally more complicated than additions, and FP operation are more costly than integers (Horowitz, ). Table 1 shows that multiplying two fp32 numbers consumes 37 times higher energy than adding two 32-bit integers.
While the complexity of integer addition is $O(n)$ where $n$ is the number of bits used for representing the number, FP multiplication requires $O(e)$ exponent addition, $O(m^{2})$ mantissa multiplication, and rounding. Here $e$ and $m$ stand for the number of bits used for exponent and mantissa parts of the FP numbers.

Modern LLM training and inference involves a large number of FP calculations in tensor computation. Consider calculating the element-size and dot products of two 2-D tensors:

$$ $Y_{1}=A\circ X,\;Y_{2}=A\cdot X^{T};\;A,X\in R^{(N,k)}$ $$

Calculating $Y_{1}$ involves $N^{2}$ FP multiplications (Mul). If $A$ and $X$ are both fp32 tensors, $A\circ X$ consumes $37$ times higher energy than adding two int32 matrices of the save size. Similarly,
Calculating $Y_{2}$ involves $(m\times n\times k)$ FP Mul and the same number of FP additions (Add). When $A$ and $X$ are fp32 tensors, each Mul-Add operation for two numbers consumes $0.9+3.7=4.6$ (pJ) energy. If we replace the fp32 Mul with int32 Add, the energy cost becomes $0.1+0.9=1.0$ (pJ), only 21.7% of the original cost. Similarly, if the inference is conducted in fp16, replacing fp16 Mul with int16 Add result in a $1-(0.05+0.4)/(1.1+0.4)=70$% energy saving.

### 2.2 Linear-complexity Multiplication ( ℒ ℒ \mathcal{L} caligraphic_L -Mul)

We propose $\mathcal{L}$-Mul, a FP multiplication algorithm with $O(n)$ complexity, where $n$ is the bit size of its FP operands. Consider two FP numbers $x,y$, whose exponents and fractions are $x_{e},y_{e}$ and $x_{m},y_{m}$ respectively, the vanilla FP Mul result is

$$ $Mul(x,y)=(1+x_{m})\cdot 2^{x_{e}}\cdot(1+y_{m})\cdot 2^{y_{e}}=(1+x_{m}+y_{m}+ x_{m}\cdot y_{m})\cdot 2^{x_{e}+y_{e}}$ $$

plus an xor operation ($\oplus$) to decide the sign of the result. Assume $x_{m}$ and $y_{m}$ are mantissas of $m$ bits. The $O(m^{2})$ mantissa multiplication operation is the complexity bottleneck of this calculation. We remove this operation and introduce a new multiplication algorithm that processes mantissas with a computational complexity of $O(m)$:

$$ $\mathcal{L}\text{-Mul}(x,y)=(1+x_{m}+y_{m}+2^{-l(m)})\cdot 2^{x_{e}+y_{e}},\; \;l(m)=\begin{cases}m&\text{if }m\leq 3,\\ 3&\text{if }m=4,\\ 4&\text{if }m>4.\end{cases}$ (1) $$

The offset exponent $l(m)$ is defined according the observation shown in Figure 3. In the following sections, we show that (1) the $\mathcal{L}$-Mul operation can be implemented by integer Adders, and (2) the algorithm is more accurate and efficient than fp8 multiplications.

The implementation of the algorithm is shown in Figure 2, where we also added the Inline PTX Assembly code we used to simulate the process on Nvidia GPUs.

Figure: Figure 2: Coparing the process of regular floating-point multiplication and linear-complexity multiplication ($\mathcal{L}$-Mul) between two fp32 numbers. In the inline PTX Assembly code, $1 and $2 are fp32 registers storing inputs while $0 is the fp32 register for output. s1, s2, r0, r1, r2 are unsigned int32 registers storing intermediate results. Note that the assembly program is only for numerical simulation on Nvidia GPUs. The optimal implementation is at the hardware level.
Refer to caption: x2.png

While Equation (1) contains 4 addition operations, the bit format design of FP numbers helps us implement the $\mathcal{L}$-Mul algorithm with one adder. Since the FP format handles $1+x_{m}$ implicitly, we do not have to compute the value of $(1+\dots)$. The integer addition operation also automatically send the mantissa carry to the exponent. If the mantissa sum is greater than 2, a carry is automatically added to the exponent. This is different from the rounding process in traditional FP multiplier, where the fraction is manually rounded to $1.x$ and the carry is added to the exponent as an independent step. As a result, the $\mathcal{L}$-Mul algorithm is more efficient than traditional FP multiplication by skipping both mantissa multiplication and rounding operations.

The construction of $\mathcal{L}$-Mul results can be expressed using the following equation, where all bit-level calculations are performed as operations between unsigned integers.

$$ $\begin{split}\mathcal{L}\text{-mul}(x,y)[0]&=x[0]\oplus y[0]\\ \mathcal{L}\text{-mul}(x,y)[1\text{:}]&=x[1\text{:}]+y[1\text{:}]-\text{offset }\end{split}$ (2) $$

We further implement the attention mechanism with $\mathcal{L}$-Mul. In transformer models, the attention mechanism has a high computation cost because of its $O(|C|^{2})$ complexity to process the input context $C$. We found that $\mathcal{L}$-Mul can replace the complicated tensor multiplications with minimal performance loss needing no additional training. In this work we implement a more efficient attention mechanism as follows,

$$ $\begin{split}K=H\cdot W_{k},\;Q=H\cdot W_{q},\;V&=H\cdot W_{V}\\ A=softmax\left[\frac{\mathcal{L}\text{-matmul}(Q,K^{T})}{\sqrt{d}}\right],\;H^ {\prime}&=\mathcal{L}\text{-matmul}(A,H)\end{split}$ (3) $$

where $\mathcal{L}\text{-matmul}(Q,K^{T})$ stands for a matrix multiplication operation where all regular FP multiplications are implemented in $\mathcal{L}$-Mul. By doing this, all FP multiplications are replaced with integer additions, which consumes significantly lower computation resource.

### 2.3 Precision and Cost Analysis

In this section, we show that $\mathcal{L}$-Mul is more precise than fp8_e4m3 multiplications but uses less computation resource than fp_e5m2. To be concise, we do not consider the rounding to nearest even mode in both error analysis and complexity estimation for both Mul and $\mathcal{L}$-Mul.

#### 2.3.1 Precision Estimation

The goal of the precision analysis is to find the precision of the $\mathcal{L}$-Mul algorithm is equivalent to rounding the fraction of a FP number to how many bits, e.g., fp8 with 2- or 3-bit mantissas (e5m2 or e4m3). Consider positive FP numbers $x=(1+x_{m})\cdot 2^{x_{e}}$ and $y=(1+y_{m})\cdot 2^{y_{e}}$, they can be written in the following format if we explicitly highlight the $k$ bits to be kept after rounding:

$$ $x=(1+x_{k}+x_{r})\cdot 2^{x_{e}},\;\;x^{\prime}=(1+x_{k})\cdot 2^{x_{e}}$ $$

$$ $y=(1+y_{k}+y_{r})\cdot 2^{y_{e}},\;\;y^{\prime}=(1+y_{k})\cdot 2^{y_{e}}$ $$

where $x_{k},y_{k}$ are the first k bits of $x_{m},y_{m}$, and $x_{r},y_{r}$ are the value of remaining bits that will be ignored after the k-bit rounding. $x^{\prime},y^{\prime}$ are the rounded value of $x,y$ by keeping the first k bits of the mantissa. Consider $x$ and $y$ has $m$-bit mantissa in their full precision. For example, Float16 numbers have 10-bit mantissa and BFloat16 contain 7 bits.
The error of $Mul(x,y)=x\cdot y$ and its expectation are

$$ $\begin{split}e_{mul}^{k}=Mul(x,y)-Mul(x^{\prime},y^{\prime})&=(x_{k}y_{r}+y_{k }x_{r}+x_{r}+y_{r}+x_{r}y_{r})\cdot 2^{x_{e}+y_{e}}\\ E[e_{mul}^{k}]&=f_{1}(m,k)\cdot E[2^{x_{e}+y_{e}}]\end{split}$ (4) $$

Comparing with a $k$-bit mantissa FP multiplication, the error of $k$-bit mantissa $\mathcal{L}$-Mul is

$$ $\begin{split}e_{lmul}^{k}&=e_{mul}^{k}+(x_{k}\*y_{k}-2^{-l(k)})\cdot 2^{x_{e}+ y_{e}}\\ E[e_{lmul}^{k}]&=E[e_{mul}^{k}]+E[x_{k}\,y_{k}-2^{-l(k)}]\cdot E[2^{x_{e}+y_{e }}]\end{split}$ (5) $$

With the equations above, we can compute the expectation of the precision gap between $k$-bit $\mathcal{L}$-Mul and FP multiplication:

$$ $E[e_{lmul}^{k}]-E[e_{mul}^{k}]=f_{2}(k)\cdot E[2^{e_{x}+e_{y}}],\;\;E[e_{lmul} ^{k}]=[f_{1}(m,k)+f_{2}(k)]\cdot E[2^{e_{x}+e_{y}}]$ $$

When $x_{m},y_{m}$ are evenly distributed, we can calculate the following expectations,

$$ $E[x_{k}]=\frac{1}{2}(1-2^{-k}),\;E[x_{r}]=\frac{1}{2}(2^{-k}-2^{-m})$ $$

By estimating $f_{1}(m,k)$ and $f_{2}(k)$ and further inferring $E[e_{lmul}^{k}]$ and $E[e_{mul}^{k}]$,
we find that $\mathcal{L}$-Mul is more accurate than fp8_e5m2 with evenly distributed operands.
However, the weight distribution is often biased in pretrained LLMs. Based on the combined weight distribution of five popular LLMs, we find that $\mathcal{L}$-Mul can achieve higher precision beyond fp8_e4m3 with 5-bit mantissa operands in practice. We support both claims with estimated errors detailed in Appendix A.

#### 2.3.2 Gate Complexity Estimation

In this section, we make a rough estimation for the amount of gate-level computations needed by $\mathcal{L}$-Mul and fp8 multiplications. Multiplying two fpn_eimj number require the following computation: sign prediction, exponent addition with offset, a $j+1$-bit mantissa multiplication, and exponent rounding. The mantissa multiplication includes $(j+1)^{2}$ AND operations, $3$ half adders and $2j-2$ full adders. The exponent rounding needs $i$ half adders. In a regular circuit design, a full adder involves 2 AND, 2 XOR, and 1 OR. Each XOR has 4 NAND gates. As a result, a full adder consumes 11 gate-level computation, while a half adder (no incoming carry) consumes 5 gate-level computations (1 AND and 1 XOR).

In conclusion, the total amount of gate-level computation needed by fp8 Mul can be estimated as

$$ $N_{\text{fp16}}^{\times}\approx 584,\;N_{\text{fp8-e4m3}}^{\times}\approx 325, \;N_{\text{fp8-e5m2}}^{\times}\approx 296$ (6) $$

$\mathcal{L}$-Mul consumes $1$ XOR for sign prediction, $1$ half adder, and $k-2$ full adders. The total gate count needed by 16-bit and 8-bit $\mathcal{L}$-Mul can be estimated as follows,

$$ $\begin{split}N_{eimj}^{\mathcal{L}\text{-mul}}=N_{1}^{\oplus}+N_{int(i+j)}^{+} &+N_{int8}^{+}\\ N_{\text{fp16}}^{\mathcal{L}\text{-mul}}\approx 256,\;N_{\text{fp8}}^{\mathcal {L}\text{-mul}}&\approx 157\end{split}$ (7) $$

$\mathcal{L}$-Mul with fp8_e4m3 and fp8_e5m2 operands have similar complexity since exponent offsets are typically implemented by 8-bit unsigned integer adders. As estimated, fp16 $\mathcal{L}$-Mul requires less gates than fp8 multiplications, and fp8 $\mathcal{L}$-Mul is significantly more efficient.

To summarize the error and complexity analysis, $\mathcal{L}$-Mul is both more efficient and more accurate than fp8 multiplication.

## 3 Experiments

To prove the theoretical precision estimation and find out how $\mathcal{L}$-Mul-based LLMs perform on real tasks, we conducted experiments on various benchmarks with different transformer-based large language models. We evaluated Llama-3.1-8b-Instruct (Dubey et al., ), mistral-7b-v0.3-Instruct (Jiang et al., ), Gemma2-2b-It (Team et al., ), and Llava-v1.5-7b (Liu et al., ) models, and found that the proposed method can replace different modules in transformer layers under fine-tuning or training-free settings. In this section, we first introduce the benchmarks and tasks used for evaluation, then compare the numerical error of the $\mathcal{L}$-Mul algorithm against models with fp8 parameters. We also report the benchmarking results of LLMs under different precision settings.

### 3.1 Tasks

Massive Multitask Language Understanding (MMLU) (Hendrycks et al., ) contains 57 multi-choice natural language understanding tasks covering various high-school and college subjects. With 5 few-shot examples, the LLMs for evaluation are required to find the most appropriate answer option to each question. The benchmark focuses on evaluating the language understanding and knowledge abilities related to given subjects.

BigBench-Hard (BBH) (Srivastava et al., ) contains a set of complex symbolic tasks to evaluate the structural and logic reasoning abilities of language models. In this work, we select a subset of 17 multi-choice tasks to evaluate Llama and Mistral LLMs. We evaluate language models under the few-shot prompting setting for all BBH tasks.

Common Sense. We put together a set of 5 question answering tasks to evaluate the commonsense knowledge reasoning ability of LLMs. The set of task includes ARC-Challenge (Clark et al., ), CSQA (Saha et al., ), OBQA (Mihaylov et al., ), PIQA (Bisk et al., ), and SIQA (Sap et al., ), covering different aspects of factual and social knowledge.

Visual Question Answering. We select a set of multi-choice question answering tasks based on images for evaluating both vision and language understanding abilities of visual language models. The tasks include VQAv2 (Goyal et al., ), VizWiz (Gurari et al., ), and TextVQA (Singh et al., ), containing both unanswerable and answerable questions with different types of answers.

Visual Instruction following. We test the instruction following ability of Llava-1.5-7b model with the Llava-bench task (Liu et al., ) by generating free-form responses given images and corresponding instructions. Following the official evaluation guide, we evaluate the instruction following quality with GPT4o and compare the relative performance.

Object Hallucination. We explore if conducting inference with lower precision infects the truthfulness of the Llava model using the POPE benchmark (Li et al., ), which prompt visual language models with a sequence of yes/no questions about positive and negative objects.

GSM8k (Cobbe et al., ) consists of 8.5k human-crafted grade school math problems, with a test split of 1,000 problems designed to evaluate the arithmetic capabilities of language models. We conduct experiments on GSM8k in two different settings. In the training-free setting, we assess LLMs with few-shot, chain-of-thought prompting (Wei et al., ). Additionally, we fine-tune the Gemma2-2b-It model on the training split and evaluate its performance in a zero-shot setting.

### 3.2 Precision Analysis

Selection of $l(k)$. We first visualize the mean square errors obtained by different $l(k)$ selections with different models on the GSM8k dataset in Figure 3. In the plot, we highlight the $l(k)$ configurations that leads to lower average error than float8_e4m3 multiplications in model inference in red, and the $k,l(k)$ combinations leading to an error between e4m3 and e5m2 are underlined. In both models, $\mathcal{L}$-Mul with 3-bit mantissas is more accurate than fp8_e5m2 and $\mathcal{L}$-Mul with 4-bit mantissas achieves comparable or lower error than fp8_e4m3.

Figure: Figure 3: Mean square errors obtained by different $l(k)$ selections on Llama and Gemma models. The combinations achieving higher precision than fp8_e4m3 are highlighted in read, and those outperforming fp8_e5m2 are underlined. When $k=4$ and $l(k)=3$, the average error of the llama model is slighly lower but very close to fp8_e4m3.
Refer to caption: x3.png

Mantissa size. In section 2.3.1, we argued that the error expectation of $\mathcal{L}$-Mul can be lower than multiplying fp8_e4m3 multiplication while using less computation resource than multiplying fp8_e5m2 numbers. We hereby confirm the correctness of our theoretical precision estimates for the $\mathcal{L}$-Mul algorithm with experimental analysis. The average errors of Llama and Gemma models are illustrated in Figure 4.

Figure: Figure 4: Comparing the error levels of linear-complexity multiplication ($\mathcal{L}$-Mul) against the number of mantissa bits comparing with 8-bit FP multiplication operation in different formats.
Refer to caption: x4.png

The experiments demonstrated that across various sizes of LLMs, the $\mathcal{L}$-Mul algorithm using 6-bit mantissa FP operands approximates the lowest average error, significantly outperforming both fp8 formats. Additionally, the 3- and 4-bit mantissa $\mathcal{L}$-Mul achieved accuracy on par with or exceeding that of fp8_e5m2 and fp8_e4m3 multiplication operations, respectively.

In the IEEE 754 format (with a 1-bit sign and a 5-bit exponent), using a 6-bit mantissa is equivalent to rounding fp16 numbers down to fp12. By applying the complexity estimation method outlined in Equation (7), we can compute the gate count for 12-bit $\mathcal{L}$-Mul operations as follows:

$$ $N_{12}^{\mathcal{L}\text{-mul}}\approx 201<N_{fp8}^{\times}\approx 300$ (8) $$

The experimental results further confirm that $\mathcal{L}$-Mul is more efficient and accurate than fp8 multiplication. Although we estimated gate counts as an indicator of computational complexity, the actual difference in energy cost is greater than the complexity gap suggests. Based on the energy consumption reported in Horowitz (), an fp8 multiplication consumes approximately 0.25 pJ to 0.4 pJ, while a 16-bit $\mathcal{L}$-Mul uses around 0.06 pJ of energy.

### 3.3 Benchmarking

In this section, we demonstrate that $\mathcal{L}$-Mul can replace tensor multiplications in the attention mechanism without any loss of performance, whereas using fp8 multiplications for the same purpose degrades inference accuracy. This indicates that we can achieve the same model inference performance while reducing the energy cost of attention computations by 80%. Additionally, we present the full-model fine-tuning performance when all tensor multiplication operations are replaced with $\mathcal{L}$-Mul on the GSM8k benchmark.

Textual tasks. Table 2 presents the evaluation results of the Llama and Mistral models on various natural language benchmarks, including MMLU, BBH, ARC-C, CSQA, PIQA, OBQA, and SIQA. In these experiments, the matrix multiplications in the attention layers, both before and after the softmax operation, were replaced with 8-bit tensor computations in different formats or $\mathcal{L}$-Matmul following the implementation we discussed in Equation (3).

**Table 2: Comparing the attention mechanism implemented with 16- and 8-bit tensor multiplication operations and $\mathcal{L}$-Mul approximations. Note that the $\mathcal{L}$-Mul computations cost significantly less resource than fp8 tensors.**
| Precision | BBH | MMLU | ARC-R | CSQA | OBQA | PIQA | SIQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral-7b-Instruction-v0.3 |  |  |  |  |  |  |  |  |
| BFloat16 | 55.85 | 62.20 | 75.94 | 71.42 | 76.20 | 80.74 | 44.83 | 69.83 |
| Float8_e4m3 | 55.16 | 62.18 | 75.39 | 71.25 | 76.00 | 80.47 | 44.63 | 69.55 |
| Float8_e5m2 | 53.20 | 61.75 | 74.91 | 71.25 | 74.40 | 79.76 | 44.52 | 68.97 |
| $\mathcal{L}$-Mul | 55.87 | 62.19 | 76.11 | 71.09 | 76.60 | 80.52 | 45.34 | 69.93 |
| Llama-3.1-8B-Instruct |  |  |  |  |  |  |  |  |
| BFloat16 | 70.79 | 68.86 | 82.51 | 74.53 | 84.20 | 84.00 | 45.96 | 74.24 |
| Float8_e4m3 | 69.91 | 68.16 | 81.66 | 74.28 | 82.20 | 83.51 | 45.34 | 73.40 |
| Float8_e5m2 | 62.94 | 66.61 | 80.12 | 73.30 | 79.40 | 81.07 | 45.39 | 71.86 |
| $\mathcal{L}$-Mul | 70.78 | 68.54 | 82.17 | 74.28 | 84.20 | 83.30 | 46.06 | 74.00 |

The results indicate that $\mathcal{L}$-Mul not only requires significantly fewer computational resources but also delivers higher precision than float8-e4m3 tensors in 12 out of 14 experiments using Mistral and Llama models. This leads to a minimal performance gap when compared to bf16 inference. On average, across the two models, the performance difference between bf16 and $\mathcal{L}$-Mul is just 0.07%. These findings suggest that matrix multiplication operations in the attention mechanism can be seamlessly replaced with the $\mathcal{L}$-Mul algorithm without any loss of accuracy or the need for additional training.

GSM8k.
We evaluated the performance of three language models—Mistral-7b-Instruct-v0.3, Llama3.1-7b-Instruct, and Gemma2-2b-It—on the GSM8k dataset using few-shot prompting and $\mathcal{L}$-Mul-based attention. The models were tested under different numerical precision formats: bf16, fp8_e4m3, fp8_e5m2, and the $\mathcal{L}$-Mul method. The results are summarized in Table 3.

Notably, the $\mathcal{L}$-Mul-based attention mechanism slightly improved the average performance compared to the bf16 baseline.
Mistral-7b-Instruct-v0.3 and Gemma2-2b-It both exhibited improved accuracies with $\mathcal{L}$-Mul, achieving 52.92% and 47.01% respectively. Llama3.1-7b-Instruct’s accuracy with $\mathcal{L}$-Mul was slightly lower than its bf16 performance but still higher than with fp8_e4m3 and fp8_e5m2. On contrary, rounding the tensors in the attention computation to fp8_e5m2 leads to a significant performance drop although it’s more complicated than $\mathcal{L}$-Mul.

**Table 3: GSM8k accuracy with Mistral, Llama, and Gemma models with few-shot prompting and attention mechanism implemented in different precision levels.**
| Model | Bfloat16 | Float8_e4m3 | Float8_e5m2 | $\mathcal{L}$-Mul |
| --- | --- | --- | --- | --- |
| Mistral-7b-Instruct-v0.3 | 52.54 | 52.39 | 50.19 | 52.92 |
| Llama3.1-7b-Instruct | 76.12 | 75.44 | 71.80 | 75.63 |
| Gemma2-2b-It | 45.87 | 45.94 | 44.43 | 47.01 |
| Average | 58.17 | 57.92 | 55.47 | 58.52 |

Vision-language tasks. The performance of the Llava-v1.5-7b model on VQA, object hallucination, and instruction following tasks are shown in Table 4. Similar to the experiments on language tasks, the attention computation is conducted with different precision/methods while other linear transformation layers are unchanged. Except for TextVQA where the accuracy gap is $0.5\%$, the performance of $\mathcal{L}$-Mul and BFloat16 attentions are comparable. The VQA tasks are evaluated with the official evaluation scripts and the Llava-Bench results are generated by GPT-4o.

**Table 4: Evaluating the performance of different attention implementation on the Llava-v1.5-7b model. VQAv2, VizWiz, and TextVQA are visual question answering tasks, POPE evaluates object hallucination, and Llava-Bench assesses the instruction following ability scored by GPT-4o.**
| Task | POPE | Llava-Bench | TextVQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Split | rand. | adv. | pop. | all | comp. | conv. | detail. | all | all |
| BFloat16 | 86.20 | 83.17 | 85.13 | 84.83 | 66.80 | 57.60 | 41.40 | 57.50 | 57.90 |
| $\mathcal{L}$-Mul | 86.57 | 83.19 | 85.34 | 85.03 | 64.90 | 58.70 | 43.30 | 57.50 | 57.41 |
| Task | VQAv2 | VizWiz |  |  |  |  |  |  |  |
| Split | yes/no | num. | other | all | yes/no | num. | unans. | other | all |
| BFloat16 | 91.88 | 59.04 | 70.56 | 78.03 | 77.19 | 45.24 | 71.75 | 38.19 | 49.31 |
| $\mathcal{L}$-Mul | 91.78 | 58.93 | 70.73 | 78.06 | 78.54 | 50.48 | 73.78 | 38.41 | 50.16 |

$\mathcal{L}$-Mul with fewer bits. In this section, we explore how $\mathcal{L}$-Mul-based attention precision influences the overall model performance using the MMLU benchmark with Mistral and Llama models. We implement the attention mechanism with $\mathcal{L}$-Mul and only keep the first $k$ bits of the operand tensors. The results of $\mathcal{L}$-Mul attention with different precision are listed in Table 6.
As expected, using $\mathcal{L}$-Mul with a 4-bit mantissa achieves performance comparable to or slightly better than that of bf16 and fp8_e4m3. However, performance drops proportionally to the estimated error depicted in Figure 4. When $k=3$, both models significantly outperform their fp8_e5m2 counterparts, with the Llama model’s performance remaining close to that of fp8_e4m3. When $k=2$, the Llama model’s performance is comparable to that of fp8_e5m2 rounding. This suggests that with the Llama model, we can perform $\mathcal{L}$-Mul directly on fp8 models without compromising performance.

**Table 5: The performance of Mistral models with attention mechanism implemented with $k$-bit tensor $\mathcal{L}$-Mul.**
| Model | e4m3 | e5m2 | $k=4$ | $k=3$ | $k=2$ |
| --- | --- | --- | --- | --- | --- |
| Mitral | 62.18 | 61.75 | 62.16 | 62.06 | 61.08 |
| Llama | 68.16 | 66.61 | 68.43 | 68.12 | 66.67 |

Full-model fine-tuning.
To further explore the impact of the $\mathcal{L}$-Mul algorithm, we go beyond implementing attention layers with $\mathcal{L}$-Mul by replacing all multiplication operations—including matrix multiplications in linear transformations, element-wise multiplications, and those within attention layers—with fp8_e4m3 $\mathcal{L}$-Mul for the Gemma2-2b-It model. We then fine-tune the updated model on the training set of the GSM8k corpus and evaluate both the fine-tuned fp8 and $\mathcal{L}$-Mul models under a zero-shot setting on the GSM8k test set. Note that the $\mathcal{L}$-Mul operations in this experiment takes operands with 3-bit mantissas ($k=3$) and the accumulation precision is fp8_e4m3 to explore an extremely efficient setting.

The experimental results demonstrate that a fine-tuned fp8_e4m3 $\mathcal{L}$-Mul model achieves performance comparable to a standard fine-tuned fp8_e4m3 model under fp8 accumulation precision. This suggests that $\mathcal{L}$-Mul can enhance training efficiency without compromising the fine-tuned model’s performance. Moreover, it reveals the potential of training $\mathcal{L}$-Mul native LLMs for accurate and energy-efficient model hosting.

## 4 Related Work

Reducing the computation needed by neural networks while maintain the performance is an important problem which entailed multiple research directions. Typical methods include neural network pruning, quantization, and improved tensor I/O implementations.

Pruning. Neural network pruning focuses on improving the inference efficiency by reducing the number of connections among layers (Han et al., ; ; Wang et al., ). Neural network pruning methods usually involves training. After important weights are identified, the neural networks are re-trained to further update the selected weights for specific tasks. Different from model pruning, the method we proposed is designed for general tasks, requiring no task-specific re-training.

Optimizing tensor I/O. On regular GPUs, moving tensors between GPU SRAM and high-bandwidth memory (HBM) is the main bottleneck of time and energy consumption. Reducing the I/O operations in transformer models and making the best use of the HBM can significantly improve the efficiency of AI training and inference (Dao et al., ; ; Kwon et al., ). Our method, which focuses on optimizing arithmetic operations, is orthogonal to this direction.

Rounding and quantization. Standard neural network weights are stored as 32-bit or 16-bit FP tensors. However, the full-sized weights takes a considerable amount of GPU memory. To improve the storage efficiency, both weights storage and computation can be conducted in a lower precision, for example, using 16-bit, 8-bit, or 4-bit FP and Int (fp16, bf16 (Kalamkar et al., ), fp8-e4m3, fp8-e5m2 (Micikevicius et al., ), int8 (Dettmers et al., ), fp4, and int4 (Dettmers et al., )) tensors to represent model weights. Inference with lower-bit parameters usually hurts the computation accuracy and impacts the performance of pretrained models, and Integer-based quantization methods spend significant time to handle outlier weights. comparing to the quantization methods, our method requires less computation but achieves higher accuracy.

## 5 Future Work

To unlock the full potential of our proposed method, we will implement the $\mathcal{L}$-Mul and $\mathcal{L}$-Matmul kernel algorithms on hardware level and develop programming APIs for high-level model design. Furthermore, we will train textual, symbolic, and multi-modal generative AI models optimized for deployment on $\mathcal{L}$-Mul native hardware. This will deliver high-speed and energy-efficient AI hosting solutions, reducing the energy cost for data centers, robotics, and a wide spectrum of edge-computing devices.

## 6 Conclusion

In this paper, we introduced $\mathcal{L}$-Mul, an efficient algorithm that approximates floating-point multiplication using integer addition. We first demonstrated that the algorithm exhibits linear complexity relative to the bit size of its floating-point operands. We then showed that the expected accuracy of $\mathcal{L}$-Mul surpasses that of fp8 multiplications while requiring significantly less computational power. To assess the practical impact of $\mathcal{L}$-Mul, we evaluated it on natural language, vision, and mathematics benchmarks using popular language models. Our experiments indicate that $\mathcal{L}$-Mul outperforms 8-bit transformers with lower computational consumption and achieves lossless performance when applied to computation-intensive attention layers without additional training. Based on this evidence, we argue that tensor multiplications in language models can be effectively implemented using $\mathcal{L}$-Mul to preserve performance while enabling energy-efficient model deployment.

## Appendix A Error Estimation

We calculate the error expectations with different $(n,k)$ combinations as follows in Table 7.
The values are calculated with the actual parameters of Mistral, Llama, and Gemma models. For even distribution, we use the expectations introduced in Section 2.3.1. For real distribution, we estimate the average value of possible operands using the parameters of five popular pretrained LLMs.

**Table 7: Average error expectation with five different language models on floating point multiplication and $\mathcal{L}$-Mul with different rounding representations when the full precision is BFloat16. $K$ stands for the bit number of the operand mantissa.**
| K values | 1 | 2 | 3 | 4 | 5 | 6 |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | $abs[f_{1}(n=7,k)]$ | 0.68 | 0.35 | 0.17 | 0.081 | 0.035 | 0.012 |
| Even<br>Distribution | $abs[f_{1}(n=7,k)+f_{2}(k)]$ | 0.68 | 0.43 | 0.30 | 0.24 | 0.20 | 0.19 |
|  | $abs[f_{1}(n=7,k)]$ | 0.61 | 0.33 | 0.16 | 0.077 | 0.033 | 0.011 |
| Real<br>Distribution | $abs[f_{1}(n=7,k)+f_{2}(k)]$ | 0.16 | 0.18 | 0.18 | 0.12 | 0.15 | 0.14 |

We find that when the operands are distributed evenlly, $\mathcal{L}$-Mul is more accurate than float8_e5m2 multiplications. However with real models, $\mathcal{L}$-Mul can achieve higher precision than float8_e4m3 calculations.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - why the focus on replacement of the attention matmul, instead of all matmul across the model?
- can the authors provide estimates of energy savings at model level?
- extensive proofreading is needed to improve readability and consistency
soundness: 3
strengths: - improving energy efficiency with limited or no accuracy degradation is a topic of great interest for both academia and industry
- to the best of my knowledge, the proposed approach is novel
- 12-bit L-Mul achieves better or comparable accuracies to FP8 formats across multiple tasks. Model and benchmark selection is relevant and appropriate, and provides convincing evidence on the accuracy claims insofar they relate to replacement of the attention matmuls
- energy estimates are more handwaved and reported in a confusing way, but I agree with the claim that there is an expectation of potentially significant energy savings, at least at the level the individual computations that are being replaced by L-Mul
summary: This paper presents L-Mul, an algorithm that approximates floating point multiplications with integer additions, performed at higher precision. The trade-off is favorable towards L-Mul against other data formats, as significant energy savings may be achieved thanks to the use of fewer gate-level computations, with each Add consuming less energy than Mul. The authors primary focus is replacing the attention matmuls with L-Mul. 12-bit L-Mul performance are compared against fp16, fp8_e4m3, fp8_e5m2. Results are reported on various LLM (llama-3.1-8b, mistral-7b-v0.3, gemma2-2b, llava-v1.5-7b) across several common NLP tasks.
weaknesses: - "is all you need" title format is generally overused and in this case does not even fairly represents the content of the paper: most experiments leave regular matmul in place everywhere across the network, with the exception of the attention matmuls. I strongly recommend the authors to use a more informative title for this paper
- a main limitation is that the matmul replacement (and related accuracy results) is primarily focused on attention modules. There is only one very limited example (Table 6) where L-Mul is applied across the full model. What reasons drove the authors to this apply L-Mul only to the attention, in the first place?
- discussion on energy savings is inconsistent and confusing:
	- 95% saving for element-wise computation mentioned in the abstract is not discussed further in the paper
	- 80% saving for dot product is mentioned at the beginning of Section 3.3 but not supported by evidence (it's unclear how this is derived, and for what precision of L-Mul)
	- it is not clear at all how these two numbers are derived, may the authors clarify this?
	- crucially, a discussion of energy saving *at model level* is not made. I would encourage the authors to provide such estimate
- energy savings are estimates (not measurements) and the behavior of the L-Mul algorithm is simulated, with the authors leaving as future work an actual implementation of this concept at hardware level
- the meaning of Fig 3 should be better explained: is this the combined mse of every matmul and every example of GSM8k, for the two given models? Or a subset of these combinations?
- the abstract mentions that L-Mul with 4 and 3 bits mantissa (= fp10 and fp9, respectively) achieve comparable performance to fp8_e4m3 and fp8_e5m2, respectively. This is reported in Fig. 3 and 4. However, most of the results (Table 2,3,4, possibly 6) relate to L-Mul with 6 bits mantissa (fp12). At least this is my interpretation of the text, the precision used across different experiments is not stated as clearly as it should have been, and I would encourage the authors to amend the text throughout to clarify this point
- the paper shows an embarrassing lack of proofreading: it is full of typos, minor errors, and does not use notation consistently. Some examples include:
	- a sentence is repeated in the *abstract* with slightly different wording
	- Section 2.1: element-size -> element-wise
	- Section 2.1: A and X matrices are defined to be (N,k) but these dimensions do not match the subsequent calculations:
		- Y1 element-wise product of (N,k) matrices involves N*k multiplications, not N^2
		- Y2 dot product uses N * N * k multiplications, not m * n * k (m was defined as mantissa bits); the number of additions is similar but not exactly the same
	- Section 2.1: a comparison is made between fp16 and int16 energy numbers but the origin of the int16 numbers is not reported (they are not provided in Table 1 along with the other formats)
	- Fig 2 caption: coparing --> comparing
	- Section 2.2 (and elsewhere): the distinction between L-Mul and L-MatMul is unclear and appears to be inconsistent. They are the same op applied to different part of the network. Reported results usually refer to L-Mul but in most cases the new op is applied to the attention matmuls (which eq. 3 calls L-MatMul). Future work talks about L-Mul and L-Matmul. Would recommend to use a single name, as the op is the same
	- Section 2.2: when l(m) is defined in Eq. 1, the text states that it "is defined according to observation in Figure 3" but at this point it is totally unclear what observations are being referred to
	- Section 2.3.1: the meaning of the first sentence is unclear, the grammar is incorrect
	- Section 2.3.2: the texts mentions "the total amount of gate-level computations needed by fp8 Mul" but reports fp16 as well
	- Eq. 4 uses k,r in the explicit form of e^k_mul but f1 is then expressed as function of m,k instead (m being k+r)
	- fp8_m4e3 and fp8_m5e2 baseline errors are used in Fig. 3 as reference but their value is only reported later in Fig. 4!
	- Fig 3 caption: read --> red
	- Section 3.2: "the experimental results further confirm that L-Mul is more efficient and accurate than FP8 multiplication" --> would probably remove this sentence. There aren't additional experiments on efficiency. Accuracy is reported in the following section
	- Appendix A uses n for number of mantissa bits that are called m everywhere else

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: Interesting idea that approximates floating-point multiplications with integer additions to reduce energy consumption.
summary: This paper proposes the linear-complexity multiplication algorithm that approximates floating point multiplications with integer additions to save compute energy. Evaluations on various tasks show that the accuracy is comparable to FP8 multiplications.
weaknesses: While the proposed approximation method is interesting, the practical value of simply replacing floating-point multiplications with high-precision additions is not convincing, especially in the context of LLMs. 

+ First, in an LLM serving system (software + hardware), it is known that the on-chip and off-chip data movement and storage (induced by weights and kv caches), rather than computations, dominate energy consumption. Although computations are streamlined, the proposed method introduces non-negligible overheads on memory footprint and data traffic (due to high-precision additions), which might offset the energy saving of computations.
 
+ Second, I think a system-level comparison with INT4 quantization on a real system (e.g., FPGA-based system) is helpful to demonstrate the efficiency of the proposed method, since the energy consumption of an INT4 MAC is also far less than that of FP, and the storage and bandwidth requirements of INT4 data can also be significantly reduced.

Moreover, the authors claim that the proposed method is orthogonal and complementary to methods that optimize I/O and control. However, evaluations of this (such as incorporation with weight/kv cache pruning) are missing.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The reviewer has some doubts regarding the gate count calculations. According to Section 2.3.2, the gate count for FP8-e5m2 should include four parts: sign prediction, exponent addition with offset, a (j + 1)-bit mantissa multiplication, and exponent rounding.
   For sign prediction, an XOR is required, using four gates; exponent addition with offset, even with all additions implemented as full adders, would require only 5 * 11 * 2 = 110 gates. Mantissa multiplication, as described in the first paragraph, requires (j + 1)^2 AND gates, three half adders, and 2j - 2 full adders, totaling 9 + 3 * 5 + 2 * 11 = 46 gates; exponent rounding requires five half adders, which requires 5 * 5 = 25 gates. 
   Summing these, we get 4 + 110 + 46 + 25 = 185 gates, which is still far from the 296 gates stated in Equation 6. 
   Can the method proposed in this paper indeed significantly reduce gate count? It's suggested that the authors provide a more accurate calculation method and the calculation details.
2. Additionally, the proposed L-Mul involves converting FP32 to int32, performing integer addition, and then converting the result back to FP32. The reviewer is curious about the communication and resource overhead and hopes for more experiments and profiling results. It seems the proposed method helps reduce the computation costs while doing no favor to the crucial communication bottlenecks of LLMs.
soundness: 2
strengths: 1. The method proposed in this paper, using integer adders in place of floating-point multipliers, is innovative and may support the reduction of energy consumption in large language models.
2. The experiments examine the method's effectiveness across a wide range of task types.
3. The paper is clearly presented, with visually appealing and easy-to-understand figures and tables that also exhibit a novel presentation style.
summary: Large neural networks dedicate the majority of their computation to floating-point tensor multiplications. This work approximates a floating-point multiplier using a high-precision integer adder.
Through theoretical derivation, it is shown that, compared to 8-bit floating-point multiplication, this method achieves higher precision while reducing computational demand. The application of the L-Mul operation can potentially reduce energy consumption by 95% for element-wise floating-point tensor multiplications and by 80% for dot product calculations.
Theoretical error expectations for L-Mul have been calculated, and the algorithm has been evaluated across textual, visual, and symbolic tasks. Results demonstrate that L-Mul with a 4-bit mantissa achieves comparable precision to float8 e4m3 multiplication, while L-Mul with a 3-bit mantissa outperforms float8 e5m2.
weaknesses: 1. The paper contains several spelling errors, such as 'Reportly' which should be 'Reportedly,' 'of the save size' which should be 'same size,' 'highlighted in read' which should be 'in red,' as well as 'slighly' for 'slightly,' 'evenlly' for 'evenly,' and 'coparing' for 'comparing,' among others.
2. The selection of L(k) is based on Figure 3, which only includes experimental analysis for the Llama3.1-8b and Gemma2-2b models, making it difficult to establish the general applicability of the current L(k) values across different models.
3. The experimental section lacks model diversity, as it only includes Mistral-7b, Llama3.1-7b, and Gemma2-2b. Additional experiments could be conducted on models such as OPT and Mamba, as well as across different model scales, such as Llama3-13B and Llama3-70B.
4. There is a lack of hardware experiments, with metrics like gate complexity and power consumption only theoretically estimated. Power consumption calculations are based on results published in 2014, which provides limited guidance to the readers. It's suggested that the authors provide RTL implementations, synthesis, and simulation results to demonstrate the ideas. 
5. From the perspective of hardware designers, similar approximation techniques have been widely discussed. The authors are required to provide more discussions and comparisons. 
6. Some transitions lack clarity. For instance, moving from Equation (1) to Figure 2 requires readers to infer the offset transformation, which could be more explicitly explained.
7. The current experiments cover a variety of tasks, but the scope of applicability for the L-Mul algorithm is not clearly defined.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: It would be beneficial to present experimental results and comparisons on real hardware, showcasing the outcomes in terms of energy and power consumption. This would provide concrete evidence of the efficiency gains from the proposed approach.
soundness: 3
strengths: This paper explores an effective approach to reducing the energy consumption of large language models (LLMs) by replacing complex multi-operations with more energy-efficient summation operations.
summary: This paper proposes replacing the multi-operation process in large language models (LLMs) with summation operations, significantly reducing energy consumption.
weaknesses: The primary concern is hardware support, as current GPUs are highly optimized for multiplication operations. If model inference relies solely on summation, a significant portion of the GPU's resources dedicated to multiplication would be underutilized, leading to inefficiency.